Nov 02 12:24:05.597192 ip-10-0-128-132 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Nov 02 12:24:05.597204 ip-10-0-128-132 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Nov 02 12:24:05.597211 ip-10-0-128-132 systemd[1]: kubelet.service: Failed with result 'resources'.
Nov 02 12:24:05.597520 ip-10-0-128-132 systemd[1]: Failed to start Kubernetes Kubelet.
Nov 02 12:24:15.315380 ip-10-0-128-132 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Nov 02 12:24:15.315412 ip-10-0-128-132 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 24d2a71e3aa44267952201a9aacb5268 --
Nov 02 12:26:00.710163 ip-10-0-128-132 systemd[1]: Starting Kubernetes Kubelet...
Nov 02 12:26:01.122063 ip-10-0-128-132 kubenswrapper[2574]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 02 12:26:01.122063 ip-10-0-128-132 kubenswrapper[2574]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 02 12:26:01.122063 ip-10-0-128-132 kubenswrapper[2574]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 02 12:26:01.122063 ip-10-0-128-132 kubenswrapper[2574]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 02 12:26:01.125648 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.125564    2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 12:26:01.125648 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.125648    2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 12:26:01.125718 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.125652    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 12:26:01.125718 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.125656    2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 12:26:01.125718 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.125660    2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 12:26:01.125718 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.125666    2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 12:26:01.125718 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.125671    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 12:26:01.125718 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.125675    2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 12:26:01.125718 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.125678    2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 12:26:01.125718 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.125681    2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 12:26:01.125718 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.125683    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 12:26:01.125718 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.125686    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 12:26:01.125718 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.125689    2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 12:26:01.125718 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.125691    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 12:26:01.125718 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.125694    2574 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 12:26:01.125718 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.125696    2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 12:26:01.125718 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.125699    2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 12:26:01.125718 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.125702    2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 12:26:01.125718 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.125704    2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 12:26:01.125718 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.125707    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 12:26:01.125718 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.125709    2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 12:26:01.126180 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.125717    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 12:26:01.126180 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.125720    2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 12:26:01.126180 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.125723    2574 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 12:26:01.126180 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.125726    2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 12:26:01.126180 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.125729    2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 12:26:01.126180 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.125732    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 12:26:01.126180 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.125734    2574 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 12:26:01.126180 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.125737    2574 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 12:26:01.126180 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.125740    2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 12:26:01.126180 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.125742    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 12:26:01.126180 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.125745    2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 12:26:01.126180 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.125747    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 12:26:01.126180 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.125750    2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 12:26:01.126180 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.125752    2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 12:26:01.126180 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.125755    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 12:26:01.126180 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.125757    2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 12:26:01.126180 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.125760    2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 12:26:01.126180 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.125762    2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 12:26:01.126180 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.125765    2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 12:26:01.126632 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.125768    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 12:26:01.126632 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.125770    2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 12:26:01.126632 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.125773    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 12:26:01.126632 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.125775    2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 12:26:01.126632 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.125778    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 12:26:01.126632 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.125780    2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 12:26:01.126632 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.125783    2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 12:26:01.126632 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.125785    2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 12:26:01.126632 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.125788    2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 12:26:01.126632 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.125790    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 12:26:01.126632 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.125792    2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 12:26:01.126632 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.125795    2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 12:26:01.126632 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.125797    2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 12:26:01.126632 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.125799    2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 12:26:01.126632 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.125803    2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 12:26:01.126632 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.125806    2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 12:26:01.126632 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.125808    2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 12:26:01.126632 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.125812    2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 12:26:01.126632 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.125814    2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 12:26:01.127094 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.125817    2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 12:26:01.127094 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.125819    2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 12:26:01.127094 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.125822    2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 12:26:01.127094 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.125824    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 12:26:01.127094 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.125827    2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 12:26:01.127094 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.125829    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 12:26:01.127094 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.125832    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 12:26:01.127094 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.125834    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 12:26:01.127094 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.125837    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 12:26:01.127094 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.125839    2574 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 12:26:01.127094 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.125842    2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 12:26:01.127094 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.125845    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 12:26:01.127094 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.125848    2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 12:26:01.127094 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.125851    2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 12:26:01.127094 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.125854    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 12:26:01.127094 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.125856    2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 12:26:01.127094 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.125859    2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 12:26:01.127094 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.125863    2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 12:26:01.127094 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.125866    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 12:26:01.127546 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.125868    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 12:26:01.127546 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.125883    2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 12:26:01.127546 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.125886    2574 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 12:26:01.127546 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.125889    2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 12:26:01.127546 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.125891    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 12:26:01.127546 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.125894    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 12:26:01.127546 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.125896    2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 12:26:01.127546 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.125899    2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 12:26:01.127546 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.125903    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 12:26:01.127546 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.125906    2574 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 12:26:01.127546 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.125909    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 12:26:01.127546 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.125912    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 12:26:01.127546 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.125914    2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 12:26:01.127546 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.125917    2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 12:26:01.127546 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.125920    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 12:26:01.127546 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.125922    2574 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 12:26:01.127546 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.125925    2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 12:26:01.127546 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.125928    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 12:26:01.127546 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.125930    2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 12:26:01.127546 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.125933    2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 12:26:01.128044 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.125935    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 12:26:01.128044 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.125938    2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 12:26:01.128044 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.125940    2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 12:26:01.128044 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.125943    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 12:26:01.128044 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.125945    2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 12:26:01.128044 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.125948    2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 12:26:01.128044 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.125950    2574 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 12:26:01.128044 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.125955    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 12:26:01.128044 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.126727    2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 12:26:01.128044 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.126734    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 12:26:01.128044 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.126738    2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 12:26:01.128044 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.126741    2574 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 12:26:01.128044 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.126744    2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 12:26:01.128044 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.126747    2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 12:26:01.128044 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.126750    2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 12:26:01.128044 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.126753    2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 12:26:01.128044 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.126756    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 12:26:01.128044 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.126758    2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 12:26:01.128044 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.126761    2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 12:26:01.128044 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.126764    2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 12:26:01.128563 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.126767    2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 12:26:01.128563 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.126769    2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 12:26:01.128563 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.126772    2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 12:26:01.128563 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.126775    2574 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 12:26:01.128563 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.126779    2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 12:26:01.128563 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.126783    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 12:26:01.128563 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.126786    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 12:26:01.128563 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.126788    2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 12:26:01.128563 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.126791    2574 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 12:26:01.128563 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.126793    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 12:26:01.128563 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.126796    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 12:26:01.128563 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.126799    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 12:26:01.128563 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.126801    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 12:26:01.128563 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.126804    2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 12:26:01.128563 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.126806    2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 12:26:01.128563 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.126809    2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 12:26:01.128563 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.126811    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 12:26:01.128563 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.126814    2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 12:26:01.128563 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.126817    2574 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 12:26:01.129029 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.126820    2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 12:26:01.129029 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.126823    2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 12:26:01.129029 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.126825    2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 12:26:01.129029 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.126828    2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 12:26:01.129029 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.126830    2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 12:26:01.129029 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.126834    2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 12:26:01.129029 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.126837    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 12:26:01.129029 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.126839    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 12:26:01.129029 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.126842    2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 12:26:01.129029 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.126844    2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 12:26:01.129029 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.126847    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 12:26:01.129029 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.126850    2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 12:26:01.129029 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.126852    2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 12:26:01.129029 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.126855    2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 12:26:01.129029 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.126858    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 12:26:01.129029 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.126860    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 12:26:01.129029 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.126863    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 12:26:01.129029 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.126865    2574 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 12:26:01.129029 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.126868    2574 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 12:26:01.129029 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.126884    2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 12:26:01.129586 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.126887    2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 12:26:01.129586 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.126890    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 12:26:01.129586 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.126893    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 12:26:01.129586 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.126896    2574 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 12:26:01.129586 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.126898    2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 12:26:01.129586 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.126901    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 12:26:01.129586 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.126904    2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 12:26:01.129586 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.126907    2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 12:26:01.129586 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.126909    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 12:26:01.129586 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.126912    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 12:26:01.129586 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.126914    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 12:26:01.129586 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.126917    2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 12:26:01.129586 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.126919    2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 12:26:01.129586 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.126922    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 12:26:01.129586 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.126925    2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 12:26:01.129586 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.126927    2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 12:26:01.129586 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.126930    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 12:26:01.129586 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.126933    2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 12:26:01.129586 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.126936    2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 12:26:01.130053 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.126939    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 12:26:01.130053 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.126941    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 12:26:01.130053 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.126944    2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 12:26:01.130053 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.126946    2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 12:26:01.130053 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.126949    2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 12:26:01.130053 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.126952    2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 12:26:01.130053 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.126955    2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 12:26:01.130053 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.126959    2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 12:26:01.130053 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.126963    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 12:26:01.130053 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.126966    2574 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 12:26:01.130053 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.126969    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 12:26:01.130053 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.126971    2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 12:26:01.130053 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.126974    2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 12:26:01.130053 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.126976    2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 12:26:01.130053 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.126979    2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 12:26:01.130053 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.126981    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 12:26:01.130053 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.126984    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 12:26:01.130053 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.126987    2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 12:26:01.130053 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.126990    2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 12:26:01.130505 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.126992    2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 12:26:01.130505 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.126995    2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 12:26:01.130505 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.126998    2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 12:26:01.130505 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.127000    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 12:26:01.130505 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.127002    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 12:26:01.130505 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.127005    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 12:26:01.130505 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.127008    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 12:26:01.130505 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.127010    2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 12:26:01.130505 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.127013    2574 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 12:26:01.130505 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.127016    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 12:26:01.130505 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.127019    2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 12:26:01.130505 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.127021    2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 12:26:01.130505 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.127024    2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 12:26:01.130505 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.127026    2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 12:26:01.130505 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.127029    2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 12:26:01.130505 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.127031    2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 12:26:01.130505 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.127034    2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 12:26:01.130505 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.127528    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 12:26:01.130505 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.127533    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 12:26:01.130505 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.127536    2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 12:26:01.130996 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.127539    2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 12:26:01.130996 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.127541    2574 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 12:26:01.130996 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.127544    2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 12:26:01.130996 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.127547    2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 12:26:01.130996 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.127550    2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 12:26:01.130996 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.127554    2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 12:26:01.130996 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.127558    2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 12:26:01.130996 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.127561    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 12:26:01.130996 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.127565    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 12:26:01.130996 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.127567    2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 12:26:01.130996 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.127570    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 12:26:01.130996 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.127573    2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 12:26:01.130996 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.127575    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 12:26:01.130996 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.127578    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 12:26:01.130996 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.127580    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 12:26:01.130996 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.127583    2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 12:26:01.130996 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.127586    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 12:26:01.130996 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.127588    2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 12:26:01.130996 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.127590    2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 12:26:01.131446 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.127593    2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 12:26:01.131446 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.127596    2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 12:26:01.131446 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.127599    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 12:26:01.131446 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.127603    2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 12:26:01.131446 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.127623    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 12:26:01.131446 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.127627    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 12:26:01.131446 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.127630    2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 12:26:01.131446 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.127633    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 12:26:01.131446 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.127636    2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 12:26:01.131446 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.127638    2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 12:26:01.131446 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.127641    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 12:26:01.131446 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.127644    2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 12:26:01.131446 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.127647    2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 12:26:01.131446 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.127650    2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 12:26:01.131446 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.127653    2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 12:26:01.131446 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.127655    2574 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 12:26:01.131446 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.127658    2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 12:26:01.131446 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.127661    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 12:26:01.131446 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.127663    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 12:26:01.131924 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.127666    2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 12:26:01.131924 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.127668    2574 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 12:26:01.131924 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.127671    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 12:26:01.131924 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.127674    2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 12:26:01.131924 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.127677    2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 12:26:01.131924 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.127679    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 12:26:01.131924 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.127682    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 12:26:01.131924 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.127685    2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 12:26:01.131924 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.127688    2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 12:26:01.131924 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.127690    2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 12:26:01.131924 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.127693    2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 12:26:01.131924 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.127696    2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 12:26:01.131924 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.127698    2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 12:26:01.131924 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.127701    2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 12:26:01.131924 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.127703    2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 12:26:01.131924 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.127706    2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 12:26:01.131924 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.127710    2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 12:26:01.131924 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.127712    2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 12:26:01.131924 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.127715    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 12:26:01.131924 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.127718    2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 12:26:01.132403 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.127721    2574 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 12:26:01.132403 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.127723    2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 12:26:01.132403 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.127725    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 12:26:01.132403 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.127728    2574 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 12:26:01.132403 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.127733    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 12:26:01.132403 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.127736    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 12:26:01.132403 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.127738    2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 12:26:01.132403 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.127741    2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 12:26:01.132403 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.127743    2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 12:26:01.132403 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.127746    2574 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 12:26:01.132403 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.127748    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 12:26:01.132403 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.127751    2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 12:26:01.132403 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.127754    2574 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 12:26:01.132403 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.127756    2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 12:26:01.132403 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.127758    2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 12:26:01.132403 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.127761    2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 12:26:01.132403 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.127763    2574 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 12:26:01.132403 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.127766    2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 12:26:01.132403 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.127768    2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 12:26:01.132403 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.127771    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 12:26:01.132937 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.127773    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 12:26:01.132937 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.127776    2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 12:26:01.132937 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.127779    2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 12:26:01.132937 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.127781    2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 12:26:01.132937 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.127783    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 12:26:01.132937 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.127786    2574 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 12:26:01.132937 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.127788    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 12:26:01.132937 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.127791    2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 12:26:01.132937 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.127793    2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 12:26:01.132937 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.127796    2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 12:26:01.132937 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.127799    2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 12:26:01.132937 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.127802    2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 12:26:01.132937 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.127804    2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 12:26:01.132937 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.127807    2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 12:26:01.132937 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.127817    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 12:26:01.132937 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.127823    2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 12:26:01.132937 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.127829    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 12:26:01.132937 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.127831    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 12:26:01.132937 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.127834    2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 12:26:01.133394 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.127837    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 12:26:01.133394 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.127839    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 12:26:01.133394 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.127842    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 12:26:01.133394 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.127844    2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 12:26:01.133394 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.127847    2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 12:26:01.133394 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.127850    2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 12:26:01.133394 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.128943    2574 flags.go:64] FLAG: --address="0.0.0.0"
Nov 02 12:26:01.133394 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.128959    2574 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 02 12:26:01.133394 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.128965    2574 flags.go:64] FLAG: --anonymous-auth="true"
Nov 02 12:26:01.133394 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.128970    2574 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 02 12:26:01.133394 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.128974    2574 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 02 12:26:01.133394 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.128978    2574 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 02 12:26:01.133394 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.128982    2574 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 02 12:26:01.133394 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.128991    2574 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 02 12:26:01.133394 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.128994    2574 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 02 12:26:01.133394 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.128997    2574 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 02 12:26:01.133394 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.129001    2574 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 02 12:26:01.133394 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.129004    2574 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 02 12:26:01.133394 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.129007    2574 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 02 12:26:01.133394 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.129010    2574 flags.go:64] FLAG: --cgroup-root=""
Nov 02 12:26:01.133394 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.129013    2574 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 02 12:26:01.133394 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.129016    2574 flags.go:64] FLAG: --client-ca-file=""
Nov 02 12:26:01.133933 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.129019    2574 flags.go:64] FLAG: --cloud-provider="external"
Nov 02 12:26:01.133933 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.129024    2574 flags.go:64] FLAG: --cluster-dns="[]"
Nov 02 12:26:01.133933 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.129030    2574 flags.go:64] FLAG: --cluster-domain=""
Nov 02 12:26:01.133933 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.129034    2574 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 02 12:26:01.133933 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.129037    2574 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 02 12:26:01.133933 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.129040    2574 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 02 12:26:01.133933 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.129044    2574 flags.go:64] FLAG: --container-log-max-files="5"
Nov 02 12:26:01.133933 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.129049    2574 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 02 12:26:01.133933 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.129052    2574 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 02 12:26:01.133933 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.129055    2574 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 02 12:26:01.133933 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.129059    2574 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 02 12:26:01.133933 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.129062    2574 flags.go:64] FLAG: --contention-profiling="false"
Nov 02 12:26:01.133933 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.129066    2574 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 02 12:26:01.133933 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.129069    2574 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 02 12:26:01.133933 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.129072    2574 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 02 12:26:01.133933 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.129075    2574 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 02 12:26:01.133933 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.129080    2574 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 02 12:26:01.133933 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.129083    2574 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 02 12:26:01.133933 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.129086    2574 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 02 12:26:01.133933 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.129089    2574 flags.go:64] FLAG: --enable-load-reader="false"
Nov 02 12:26:01.133933 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.129092    2574 flags.go:64] FLAG: --enable-server="true"
Nov 02 12:26:01.133933 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.129095    2574 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 02 12:26:01.133933 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.129100    2574 flags.go:64] FLAG: --event-burst="100"
Nov 02 12:26:01.133933 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.129103    2574 flags.go:64] FLAG: --event-qps="50"
Nov 02 12:26:01.133933 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.129107    2574 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 02 12:26:01.134522 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.129110    2574 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 02 12:26:01.134522 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.129113    2574 flags.go:64] FLAG: --eviction-hard=""
Nov 02 12:26:01.134522 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.129117    2574 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 02 12:26:01.134522 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.129120    2574 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 02 12:26:01.134522 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.129124    2574 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 02 12:26:01.134522 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.129127    2574 flags.go:64] FLAG: --eviction-soft=""
Nov 02 12:26:01.134522 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.129130    2574 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 02 12:26:01.134522 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.129133    2574 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 02 12:26:01.134522 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.129137    2574 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 02 12:26:01.134522 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.129140    2574 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 02 12:26:01.134522 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.129144    2574 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 02 12:26:01.134522 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.129146    2574 flags.go:64] FLAG: --fail-swap-on="true"
Nov 02 12:26:01.134522 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.129149    2574 flags.go:64] FLAG: --feature-gates=""
Nov 02 12:26:01.134522 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.129154    2574 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 02 12:26:01.134522 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.129157    2574 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 02 12:26:01.134522 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.129160    2574 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 02 12:26:01.134522 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.129163    2574 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 02 12:26:01.134522 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.129166    2574 flags.go:64] FLAG: --healthz-port="10248"
Nov 02 12:26:01.134522 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.129170    2574 flags.go:64] FLAG: --help="false"
Nov 02 12:26:01.134522 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.129173    2574 flags.go:64] FLAG: --hostname-override="ip-10-0-128-132.ec2.internal"
Nov 02 12:26:01.134522 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.129176    2574 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 02 12:26:01.134522 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.129179    2574 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 02 12:26:01.134522 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.129182    2574 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 02 12:26:01.135074 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.129185    2574 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 02 12:26:01.135074 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.129188    2574 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 02 12:26:01.135074 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.129191    2574 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 02 12:26:01.135074 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.129194    2574 flags.go:64] FLAG: --image-service-endpoint=""
Nov 02 12:26:01.135074 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.129197    2574 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 02 12:26:01.135074 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.129200    2574 flags.go:64] FLAG: --kube-api-burst="100"
Nov 02 12:26:01.135074 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.129203    2574 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 02 12:26:01.135074 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.129206    2574 flags.go:64] FLAG: --kube-api-qps="50"
Nov 02 12:26:01.135074 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.129208    2574 flags.go:64] FLAG: --kube-reserved=""
Nov 02 12:26:01.135074 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.129211    2574 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 02 12:26:01.135074 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.129214    2574 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 02 12:26:01.135074 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.129217    2574 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 02 12:26:01.135074 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.129220    2574 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 02 12:26:01.135074 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.129223    2574 flags.go:64] FLAG: --lock-file=""
Nov 02 12:26:01.135074 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.129226    2574 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 02 12:26:01.135074 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.129229    2574 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 02 12:26:01.135074 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.129233    2574 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 02 12:26:01.135074 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.129242    2574 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 02 12:26:01.135074 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.129245    2574 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 02 12:26:01.135074 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.129248    2574 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 02 12:26:01.135074 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.129251    2574 flags.go:64] FLAG: --logging-format="text"
Nov 02 12:26:01.135074 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.129254    2574 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 02 12:26:01.135074 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.129258    2574 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 02 12:26:01.135074 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.129261    2574 flags.go:64] FLAG: --manifest-url=""
Nov 02 12:26:01.135638 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.129264    2574 flags.go:64] FLAG: --manifest-url-header=""
Nov 02 12:26:01.135638 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.129268    2574 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 02 12:26:01.135638 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.129271    2574 flags.go:64] FLAG: --max-open-files="1000000"
Nov 02 12:26:01.135638 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.129275    2574 flags.go:64] FLAG: --max-pods="110"
Nov 02 12:26:01.135638 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.129279    2574 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 02 12:26:01.135638 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.129282    2574 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 02 12:26:01.135638 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.129285    2574 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 02 12:26:01.135638 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.129288    2574 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 02 12:26:01.135638 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.129302    2574 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 02 12:26:01.135638 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.129306    2574 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 02 12:26:01.135638 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.129309    2574 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 02 12:26:01.135638 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.129316    2574 flags.go:64] FLAG: --node-status-max-images="50"
Nov 02 12:26:01.135638 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.129320    2574 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 02 12:26:01.135638 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.129323    2574 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 02 12:26:01.135638 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.129326    2574 flags.go:64] FLAG: --pod-cidr=""
Nov 02 12:26:01.135638 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.129329    2574 flags.go:64] FLAG: --pod-manifest-path=""
Nov 02 12:26:01.135638 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.129332    2574 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 02 12:26:01.135638 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.129335    2574 flags.go:64] FLAG: --pods-per-core="0"
Nov 02 12:26:01.135638 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.129338    2574 flags.go:64] FLAG: --port="10250"
Nov 02 12:26:01.135638 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.129341    2574 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 02 12:26:01.135638 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.129344    2574 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-082bd4ecc28470df4"
Nov 02 12:26:01.135638 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.129347    2574 flags.go:64] FLAG: --qos-reserved=""
Nov 02 12:26:01.135638 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.129350    2574 flags.go:64] FLAG: --read-only-port="10255"
Nov 02 12:26:01.135638 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.129353    2574 flags.go:64] FLAG: --register-node="true"
Nov 02 12:26:01.135638 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.129356    2574 flags.go:64] FLAG: --register-with-taints=""
Nov 02 12:26:01.135638 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.129362    2574 flags.go:64] FLAG: --registry-burst="10"
Nov 02 12:26:01.136309 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.129365    2574 flags.go:64] FLAG: --registry-qps="5"
Nov 02 12:26:01.136309 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.129369    2574 flags.go:64] FLAG: --reserved-cpus=""
Nov 02 12:26:01.136309 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.129372    2574 flags.go:64] FLAG: --reserved-memory=""
Nov 02 12:26:01.136309 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.129377    2574 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 02 12:26:01.136309 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.129380    2574 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 02 12:26:01.136309 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.129383    2574 flags.go:64] FLAG: --rotate-certificates="false"
Nov 02 12:26:01.136309 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.129386    2574 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 02 12:26:01.136309 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.129389    2574 flags.go:64] FLAG: --runonce="false"
Nov 02 12:26:01.136309 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.129392    2574 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 02 12:26:01.136309 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.129395    2574 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 02 12:26:01.136309 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.129398    2574 flags.go:64] FLAG: --seccomp-default="false"
Nov 02 12:26:01.136309 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.129401    2574 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 02 12:26:01.136309 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.129404    2574 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 02 12:26:01.136309 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.129407    2574 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 02 12:26:01.136309 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.129411    2574 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 02 12:26:01.136309 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.129414    2574 flags.go:64] FLAG: --storage-driver-password="root"
Nov 02 12:26:01.136309 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.129417    2574 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 02 12:26:01.136309 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.129420    2574 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 02 12:26:01.136309 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.129423    2574 flags.go:64] FLAG: --storage-driver-user="root"
Nov 02 12:26:01.136309 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.129426    2574 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 02 12:26:01.136309 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.129429    2574 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 02 12:26:01.136309 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.129432    2574 flags.go:64] FLAG: --system-cgroups=""
Nov 02 12:26:01.136309 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.129434    2574 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 02 12:26:01.136309 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.129440    2574 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 02 12:26:01.136309 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.129443    2574 flags.go:64] FLAG: --tls-cert-file=""
Nov 02 12:26:01.136931 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.129446    2574 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 02 12:26:01.136931 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.129450    2574 flags.go:64] FLAG: --tls-min-version=""
Nov 02 12:26:01.136931 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.129453    2574 flags.go:64] FLAG: --tls-private-key-file=""
Nov 02 12:26:01.136931 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.129456    2574 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 02 12:26:01.136931 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.129460    2574 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 02 12:26:01.136931 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.129463    2574 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 02 12:26:01.136931 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.129468    2574 flags.go:64] FLAG: --v="2"
Nov 02 12:26:01.136931 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.129472    2574 flags.go:64] FLAG: --version="false"
Nov 02 12:26:01.136931 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.129477    2574 flags.go:64] FLAG: --vmodule=""
Nov 02 12:26:01.136931 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.129481    2574 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 02 12:26:01.136931 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.129485    2574 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 02 12:26:01.136931 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.129624    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 12:26:01.136931 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.129628    2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 12:26:01.136931 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.129631    2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 12:26:01.136931 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.129634    2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 12:26:01.136931 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.129637    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 12:26:01.136931 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.129639    2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 12:26:01.136931 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.129642    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 12:26:01.136931 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.129644    2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 12:26:01.136931 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.129647    2574 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 12:26:01.136931 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.129650    2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 12:26:01.136931 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.129653    2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 12:26:01.136931 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.129655    2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 12:26:01.137717 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.129658    2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 12:26:01.137717 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.129660    2574 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 12:26:01.137717 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.129663    2574 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 12:26:01.137717 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.129666    2574 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 12:26:01.137717 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.129668    2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 12:26:01.137717 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.129671    2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 12:26:01.137717 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.129674    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 12:26:01.137717 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.129676    2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 12:26:01.137717 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.129679    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 12:26:01.137717 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.129681    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 12:26:01.137717 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.129684    2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 12:26:01.137717 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.129686    2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 12:26:01.137717 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.129689    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 12:26:01.137717 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.129693    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 12:26:01.137717 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.129695    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 12:26:01.137717 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.129699    2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 12:26:01.137717 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.129702    2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 12:26:01.137717 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.129704    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 12:26:01.137717 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.129707    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 12:26:01.138541 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.129710    2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 12:26:01.138541 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.129713    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 12:26:01.138541 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.129716    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 12:26:01.138541 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.129718    2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 12:26:01.138541 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.129721    2574 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 12:26:01.138541 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.129725    2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 12:26:01.138541 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.129728    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 12:26:01.138541 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.129731    2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 12:26:01.138541 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.129734    2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 12:26:01.138541 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.129736    2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 12:26:01.138541 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.129739    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 12:26:01.138541 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.129741    2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 12:26:01.138541 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.129744    2574 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 12:26:01.138541 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.129747    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 12:26:01.138541 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.129750    2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 12:26:01.138541 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.129752    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 12:26:01.138541 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.129755    2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 12:26:01.138541 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.129758    2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 12:26:01.138541 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.129760    2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 12:26:01.139042 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.129763    2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 12:26:01.139042 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.129766    2574 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 12:26:01.139042 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.129768    2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 12:26:01.139042 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.129770    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 12:26:01.139042 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.129773    2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 12:26:01.139042 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.129776    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 12:26:01.139042 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.129778    2574 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 12:26:01.139042 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.129782    2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 12:26:01.139042 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.129785    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 12:26:01.139042 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.129789    2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 12:26:01.139042 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.129792    2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 12:26:01.139042 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.129794    2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 12:26:01.139042 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.129797    2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 12:26:01.139042 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.129800    2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 12:26:01.139042 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.129802    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 12:26:01.139042 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.129805    2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 12:26:01.139042 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.129808    2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 12:26:01.139042 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.129810    2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 12:26:01.139042 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.129813    2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 12:26:01.139042 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.129815    2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 12:26:01.139516 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.129819    2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 12:26:01.139516 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.129823    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 12:26:01.139516 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.129827    2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 12:26:01.139516 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.129831    2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 12:26:01.139516 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.129833    2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 12:26:01.139516 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.129836    2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 12:26:01.139516 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.129839    2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 12:26:01.139516 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.129842    2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 12:26:01.139516 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.129844    2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 12:26:01.139516 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.129847    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 12:26:01.139516 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.129850    2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 12:26:01.139516 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.129852    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 12:26:01.139516 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.129855    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 12:26:01.139516 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.129858    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 12:26:01.139516 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.129861    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 12:26:01.139516 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.129863    2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 12:26:01.139516 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.129866    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 12:26:01.139516 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.129868    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 12:26:01.139516 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.129882    2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 12:26:01.140000 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.129887    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 12:26:01.140000 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.129889    2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 12:26:01.140000 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.129893    2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 12:26:01.140000 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.129896    2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 12:26:01.140000 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.129899    2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 12:26:01.140000 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.129901    2574 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 12:26:01.140000 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.129904    2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 12:26:01.140000 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.129907    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 12:26:01.140000 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.129909    2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 12:26:01.140000 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.129912    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 12:26:01.140000 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.129915    2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 12:26:01.140000 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.129918    2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 12:26:01.140000 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.129920    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 12:26:01.140000 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.129923    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 12:26:01.140000 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.129926    2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 12:26:01.140000 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.129928    2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 12:26:01.140000 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.129931    2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 12:26:01.140399 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.129958    2574 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 02 12:26:01.140399 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.129992    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 12:26:01.140399 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.129997    2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 12:26:01.140399 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.130000    2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 12:26:01.140399 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.130002    2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 12:26:01.140399 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.130005    2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 12:26:01.140399 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.130008    2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 12:26:01.140399 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.130010    2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 12:26:01.140399 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.130013    2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 12:26:01.140399 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.130016    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 12:26:01.140399 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.130019    2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 12:26:01.140399 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.130021    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 12:26:01.140399 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.130024    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 12:26:01.140399 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.130026    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 12:26:01.140399 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.130029    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 12:26:01.140399 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.130033    2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 12:26:01.140783 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.130036    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 12:26:01.140783 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.130038    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 12:26:01.140783 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.130042    2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 12:26:01.140783 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.130045    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 12:26:01.140783 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.130047    2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 12:26:01.140783 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.130050    2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 12:26:01.140783 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.130053    2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 12:26:01.140783 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.130056    2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 12:26:01.140783 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.130059    2574 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 12:26:01.140783 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.130062    2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 12:26:01.140783 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.130065    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 12:26:01.140783 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.130067    2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 12:26:01.140783 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.130070    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 12:26:01.140783 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.130073    2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 12:26:01.140783 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.130075    2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 12:26:01.140783 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.130078    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 12:26:01.140783 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.130080    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 12:26:01.140783 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.130083    2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 12:26:01.140783 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.130086    2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 12:26:01.141373 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.130088    2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 12:26:01.141373 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.130091    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 12:26:01.141373 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.130094    2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 12:26:01.141373 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.130097    2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 12:26:01.141373 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.130099    2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 12:26:01.141373 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.130102    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 12:26:01.141373 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.130104    2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 12:26:01.141373 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.130107    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 12:26:01.141373 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.130109    2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 12:26:01.141373 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.130112    2574 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 12:26:01.141373 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.130114    2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 12:26:01.141373 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.130117    2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 12:26:01.141373 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.130121    2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 12:26:01.141373 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.130123    2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 12:26:01.141373 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.130126    2574 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 12:26:01.141373 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.130130    2574 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 12:26:01.141373 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.130133    2574 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 12:26:01.141373 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.130136    2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 12:26:01.141373 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.130138    2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 12:26:01.141373 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.130141    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 12:26:01.141373 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.130143    2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 12:26:01.141885 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.130146    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 12:26:01.141885 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.130149    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 12:26:01.141885 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.130152    2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 12:26:01.141885 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.130155    2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 12:26:01.141885 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.130157    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 12:26:01.141885 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.130160    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 12:26:01.141885 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.130162    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 12:26:01.141885 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.130165    2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 12:26:01.141885 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.130168    2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 12:26:01.141885 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.130170    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 12:26:01.141885 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.130173    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 12:26:01.141885 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.130175    2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 12:26:01.141885 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.130178    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 12:26:01.141885 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.130180    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 12:26:01.141885 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.130183    2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 12:26:01.141885 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.130185    2574 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 12:26:01.141885 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.130189    2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 12:26:01.141885 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.130193    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 12:26:01.141885 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.130196    2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 12:26:01.142344 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.130199    2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 12:26:01.142344 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.130202    2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 12:26:01.142344 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.130204    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 12:26:01.142344 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.130207    2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 12:26:01.142344 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.130212    2574 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 12:26:01.142344 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.130216    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 12:26:01.142344 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.130219    2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 12:26:01.142344 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.130224    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 12:26:01.142344 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.130226    2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 12:26:01.142344 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.130229    2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 12:26:01.142344 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.130231    2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 12:26:01.142344 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.130234    2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 12:26:01.142344 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.130237    2574 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 12:26:01.142344 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.130239    2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 12:26:01.142344 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.130242    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 12:26:01.142344 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.130244    2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 12:26:01.142344 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.130247    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 12:26:01.142344 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.130250    2574 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 12:26:01.142344 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.130252    2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 12:26:01.142344 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.130255    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 12:26:01.142856 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.130258    2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 12:26:01.142856 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.130261    2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 12:26:01.142856 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.130263    2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 12:26:01.142856 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.130266    2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 12:26:01.142856 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.130268    2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 12:26:01.142856 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.130271    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 12:26:01.142856 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.130273    2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 12:26:01.142856 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.130276    2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 12:26:01.142856 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.130278    2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 12:26:01.142856 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.130280    2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 12:26:01.142856 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.130283    2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 12:26:01.142856 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.130287    2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 12:26:01.142856 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.139287    2574 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 02 12:26:01.142856 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.139305    2574 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 02 12:26:01.142856 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.139322    2574 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 02 12:26:01.142856 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.139327    2574 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 02 12:26:01.142856 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.139386    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 12:26:01.142856 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.139392    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 12:26:01.142856 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.139396    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 12:26:01.143338 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.139399    2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 12:26:01.143338 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.139401    2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 12:26:01.143338 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.139404    2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 12:26:01.143338 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.139407    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 12:26:01.143338 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.139410    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 12:26:01.143338 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.139412    2574 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 12:26:01.143338 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.139415    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 12:26:01.143338 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.139418    2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 12:26:01.143338 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.139420    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 12:26:01.143338 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.139423    2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 12:26:01.143338 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.139426    2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 12:26:01.143338 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.139428    2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 12:26:01.143338 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.139432    2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 12:26:01.143338 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.139434    2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 12:26:01.143338 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.139437    2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 12:26:01.143338 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.139440    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 12:26:01.143338 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.139442    2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 12:26:01.143338 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.139445    2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 12:26:01.143338 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.139448    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 12:26:01.143783 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.139450    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 12:26:01.143783 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.139453    2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 12:26:01.143783 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.139455    2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 12:26:01.143783 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.139458    2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 12:26:01.143783 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.139461    2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 12:26:01.143783 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.139464    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 12:26:01.143783 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.139471    2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 12:26:01.143783 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.139475    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 12:26:01.143783 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.139478    2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 12:26:01.143783 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.139480    2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 12:26:01.143783 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.139483    2574 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 12:26:01.143783 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.139486    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 12:26:01.143783 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.139489    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 12:26:01.143783 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.139492    2574 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 12:26:01.143783 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.139494    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 12:26:01.143783 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.139497    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 12:26:01.143783 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.139500    2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 12:26:01.143783 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.139502    2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 12:26:01.143783 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.139506    2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 12:26:01.144284 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.139509    2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 12:26:01.144284 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.139513    2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 12:26:01.144284 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.139516    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 12:26:01.144284 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.139518    2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 12:26:01.144284 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.139521    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 12:26:01.144284 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.139524    2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 12:26:01.144284 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.139527    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 12:26:01.144284 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.139530    2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 12:26:01.144284 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.139533    2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 12:26:01.144284 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.139536    2574 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 12:26:01.144284 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.139539    2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 12:26:01.144284 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.139541    2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 12:26:01.144284 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.139544    2574 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 12:26:01.144284 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.139547    2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 12:26:01.144284 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.139549    2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 12:26:01.144284 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.139552    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 12:26:01.144284 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.139555    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 12:26:01.144284 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.139558    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 12:26:01.144284 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.139561    2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 12:26:01.144284 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.139564    2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 12:26:01.144757 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.139568    2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 12:26:01.144757 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.139572    2574 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 12:26:01.144757 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.139575    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 12:26:01.144757 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.139579    2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 12:26:01.144757 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.139581    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 12:26:01.144757 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.139584    2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 12:26:01.144757 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.139587    2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 12:26:01.144757 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.139589    2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 12:26:01.144757 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.139592    2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 12:26:01.144757 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.139594    2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 12:26:01.144757 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.139597    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 12:26:01.144757 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.139600    2574 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 12:26:01.144757 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.139602    2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 12:26:01.144757 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.139605    2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 12:26:01.144757 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.139608    2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 12:26:01.144757 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.139610    2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 12:26:01.144757 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.139613    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 12:26:01.144757 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.139615    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 12:26:01.144757 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.139619    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 12:26:01.145298 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.139622    2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 12:26:01.145298 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.139624    2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 12:26:01.145298 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.139627    2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 12:26:01.145298 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.139629    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 12:26:01.145298 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.139632    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 12:26:01.145298 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.139635    2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 12:26:01.145298 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.139638    2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 12:26:01.145298 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.139640    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 12:26:01.145298 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.139643    2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 12:26:01.145298 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.139646    2574 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 12:26:01.145298 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.139648    2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 12:26:01.145298 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.139651    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 12:26:01.145298 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.139654    2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 12:26:01.145298 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.139656    2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 12:26:01.145298 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.139659    2574 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 12:26:01.145298 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.139661    2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 12:26:01.145298 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.139664    2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 12:26:01.145298 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.139666    2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 12:26:01.145298 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.139669    2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 12:26:01.145745 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.139672    2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 12:26:01.145745 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.139674    2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 12:26:01.145745 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.139677    2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 12:26:01.145745 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.139679    2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 12:26:01.145745 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.139682    2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 12:26:01.145745 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.139684    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 12:26:01.145745 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.139687    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 12:26:01.145745 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.139713    2574 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 02 12:26:01.145745 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.139888    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 12:26:01.145745 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.139895    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 12:26:01.145745 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.139898    2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 12:26:01.145745 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.139901    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 12:26:01.145745 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.139903    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 12:26:01.145745 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.139907    2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 12:26:01.145745 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.139910    2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 12:26:01.145745 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.139912    2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 12:26:01.146136 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.139915    2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 12:26:01.146136 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.139919    2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 12:26:01.146136 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.139923    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 12:26:01.146136 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.139926    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 12:26:01.146136 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.139929    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 12:26:01.146136 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.139932    2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 12:26:01.146136 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.139934    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 12:26:01.146136 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.139937    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 12:26:01.146136 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.139940    2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 12:26:01.146136 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.139943    2574 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 12:26:01.146136 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.139946    2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 12:26:01.146136 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.139949    2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 12:26:01.146136 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.139951    2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 12:26:01.146136 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.139954    2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 12:26:01.146136 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.139957    2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 12:26:01.146136 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.139960    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 12:26:01.146136 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.139963    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 12:26:01.146136 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.139966    2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 12:26:01.146136 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.139968    2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 12:26:01.146614 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.139971    2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 12:26:01.146614 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.139974    2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 12:26:01.146614 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.139976    2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 12:26:01.146614 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.139979    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 12:26:01.146614 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.139981    2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 12:26:01.146614 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.139984    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 12:26:01.146614 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.139987    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 12:26:01.146614 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.139989    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 12:26:01.146614 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.139992    2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 12:26:01.146614 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.139994    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 12:26:01.146614 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.139997    2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 12:26:01.146614 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.140001    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 12:26:01.146614 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.140004    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 12:26:01.146614 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.140006    2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 12:26:01.146614 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.140009    2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 12:26:01.146614 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.140012    2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 12:26:01.146614 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.140015    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 12:26:01.146614 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.140018    2574 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 12:26:01.146614 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.140021    2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 12:26:01.146614 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.140023    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 12:26:01.147114 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.140025    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 12:26:01.147114 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.140028    2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 12:26:01.147114 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.140031    2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 12:26:01.147114 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.140034    2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 12:26:01.147114 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.140036    2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 12:26:01.147114 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.140039    2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 12:26:01.147114 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.140041    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 12:26:01.147114 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.140044    2574 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 12:26:01.147114 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.140046    2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 12:26:01.147114 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.140049    2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 12:26:01.147114 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.140051    2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 12:26:01.147114 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.140054    2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 12:26:01.147114 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.140056    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 12:26:01.147114 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.140059    2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 12:26:01.147114 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.140061    2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 12:26:01.147114 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.140064    2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 12:26:01.147114 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.140066    2574 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 12:26:01.147114 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.140069    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 12:26:01.147114 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.140072    2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 12:26:01.147562 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.140074    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 12:26:01.147562 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.140076    2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 12:26:01.147562 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.140079    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 12:26:01.147562 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.140081    2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 12:26:01.147562 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.140084    2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 12:26:01.147562 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.140087    2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 12:26:01.147562 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.140090    2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 12:26:01.147562 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.140094    2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 12:26:01.147562 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.140098    2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 12:26:01.147562 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.140101    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 12:26:01.147562 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.140104    2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 12:26:01.147562 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.140107    2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 12:26:01.147562 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.140110    2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 12:26:01.147562 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.140113    2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 12:26:01.147562 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.140116    2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 12:26:01.147562 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.140118    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 12:26:01.147562 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.140121    2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 12:26:01.147562 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.140124    2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 12:26:01.147562 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.140126    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 12:26:01.148041 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.140129    2574 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 12:26:01.148041 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.140131    2574 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 12:26:01.148041 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.140134    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 12:26:01.148041 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.140137    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 12:26:01.148041 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.140139    2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 12:26:01.148041 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.140142    2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 12:26:01.148041 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.140144    2574 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 12:26:01.148041 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.140147    2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 12:26:01.148041 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.140149    2574 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 12:26:01.148041 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.140152    2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 12:26:01.148041 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.140154    2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 12:26:01.148041 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.140157    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 12:26:01.148041 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.140160    2574 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 12:26:01.148041 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.140162    2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 12:26:01.148041 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.140165    2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 12:26:01.148041 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.140167    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 12:26:01.148041 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.140170    2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 12:26:01.148041 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.140172    2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 12:26:01.148041 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.140175    2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 12:26:01.148041 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.140178    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 12:26:01.148510 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.140180    2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 12:26:01.148510 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.140205    2574 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 02 12:26:01.148510 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.140238    2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 12:26:01.148510 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.140242    2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 12:26:01.148510 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.140245    2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 12:26:01.148510 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.140249    2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 12:26:01.148510 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.140254    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 12:26:01.148510 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.140257    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 12:26:01.148510 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.140260    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 12:26:01.148510 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.140263    2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 12:26:01.148510 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.140266    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 12:26:01.148510 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.140268    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 12:26:01.148510 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.140271    2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 12:26:01.148510 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.140273    2574 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 12:26:01.148510 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.140276    2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 12:26:01.148862 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.140279    2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 12:26:01.148862 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.140281    2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 12:26:01.148862 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.140285    2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 12:26:01.148862 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.140287    2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 12:26:01.148862 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.140290    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 12:26:01.148862 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.140292    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 12:26:01.148862 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.140295    2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 12:26:01.148862 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.140297    2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 12:26:01.148862 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.140300    2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 12:26:01.148862 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.140303    2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 12:26:01.148862 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.140305    2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 12:26:01.148862 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.140307    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 12:26:01.148862 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.140310    2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 12:26:01.148862 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.140312    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 12:26:01.148862 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.140315    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 12:26:01.148862 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.140317    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 12:26:01.148862 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.140320    2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 12:26:01.148862 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.140323    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 12:26:01.148862 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.140326    2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 12:26:01.149411 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.140329    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 12:26:01.149411 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.140331    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 12:26:01.149411 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.140334    2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 12:26:01.149411 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.140337    2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 12:26:01.149411 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.140339    2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 12:26:01.149411 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.140342    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 12:26:01.149411 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.140344    2574 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 12:26:01.149411 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.140347    2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 12:26:01.149411 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.140349    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 12:26:01.149411 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.140352    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 12:26:01.149411 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.140354    2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 12:26:01.149411 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.140357    2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 12:26:01.149411 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.140359    2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 12:26:01.149411 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.140362    2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 12:26:01.149411 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.140364    2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 12:26:01.149411 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.140367    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 12:26:01.149411 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.140369    2574 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 12:26:01.149411 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.140372    2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 12:26:01.149411 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.140374    2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 12:26:01.149411 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.140377    2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 12:26:01.149864 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.140380    2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 12:26:01.149864 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.140382    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 12:26:01.149864 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.140385    2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 12:26:01.149864 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.140387    2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 12:26:01.149864 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.140390    2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 12:26:01.149864 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.140393    2574 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 12:26:01.149864 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.140395    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 12:26:01.149864 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.140398    2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 12:26:01.149864 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.140400    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 12:26:01.149864 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.140403    2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 12:26:01.149864 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.140405    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 12:26:01.149864 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.140409    2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 12:26:01.149864 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.140413    2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 12:26:01.149864 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.140415    2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 12:26:01.149864 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.140418    2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 12:26:01.149864 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.140422    2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 12:26:01.149864 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.140426    2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 12:26:01.149864 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.140429    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 12:26:01.149864 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.140432    2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 12:26:01.150319 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.140434    2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 12:26:01.150319 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.140437    2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 12:26:01.150319 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.140440    2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 12:26:01.150319 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.140442    2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 12:26:01.150319 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.140445    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 12:26:01.150319 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.140447    2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 12:26:01.150319 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.140450    2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 12:26:01.150319 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.140453    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 12:26:01.150319 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.140455    2574 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 12:26:01.150319 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.140458    2574 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 12:26:01.150319 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.140460    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 12:26:01.150319 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.140463    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 12:26:01.150319 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.140465    2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 12:26:01.150319 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.140468    2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 12:26:01.150319 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.140470    2574 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 12:26:01.150319 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.140473    2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 12:26:01.150319 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.140475    2574 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 12:26:01.150319 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.140478    2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 12:26:01.150319 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.140480    2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 12:26:01.150319 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.140483    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 12:26:01.150823 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.140485    2574 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 12:26:01.150823 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.140487    2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 12:26:01.150823 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.140490    2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 12:26:01.150823 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.140493    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 12:26:01.150823 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.140495    2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 12:26:01.150823 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.140502    2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 12:26:01.150823 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.140505    2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 12:26:01.150823 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.140507    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 12:26:01.150823 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.140510    2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 12:26:01.150823 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.140513    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 12:26:01.150823 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.140516    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 12:26:01.150823 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.140519    2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 12:26:01.150823 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.140521    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 12:26:01.150823 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.140524    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 12:26:01.150823 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:01.140526    2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 12:26:01.150823 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.140584    2574 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 02 12:26:01.150823 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.143043    2574 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Nov 02 12:26:01.150823 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.144129    2574 server.go:1014] "Starting client certificate rotation"
Nov 02 12:26:01.150823 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.144221    2574 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 02 12:26:01.151293 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.145100    2574 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 02 12:26:01.151293 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.147031    2574 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 02 12:26:01.151293 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.148161    2574 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 02 12:26:01.161504 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.161487    2574 log.go:25] "Validated CRI v1 runtime API"
Nov 02 12:26:01.166204 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.166184    2574 log.go:25] "Validated CRI v1 image API"
Nov 02 12:26:01.167075 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.167045    2574 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 02 12:26:01.169997 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.169977    2574 fs.go:135] Filesystem UUIDs: map[6c917a51-8570-43b9-84b7-2d38dd82c25b:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 d917a991-98d5-4d8a-9534-b799f518d4a4:/dev/nvme0n1p4]
Nov 02 12:26:01.170066 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.169996    2574 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Nov 02 12:26:01.175703 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.175594    2574 manager.go:217] Machine: {Timestamp:2026-11-02 12:26:01.174141153 +0000 UTC m=+0.368849086 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3102950 MemoryCapacity:33164345344 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec25cc89266f0ccc37e35a5d281e33db SystemUUID:ec25cc89-266f-0ccc-37e3-5a5d281e33db BootID:24d2a71e-3aa4-4267-9522-01a9aacb5268 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582172672 Type:vfs Inodes:4048382 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:f7:a6:90:79 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:f7:a6:90:79 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:b2:8d:24:bb:6d:49 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164345344 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 02 12:26:01.175703 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.175702    2574 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 02 12:26:01.175832 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.175820    2574 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 02 12:26:01.177079 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.177058    2574 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 02 12:26:01.177223 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.177082    2574 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-132.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 02 12:26:01.177306 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.177224    2574 topology_manager.go:143] "Creating topology manager with none policy"
Nov 02 12:26:01.177306 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.177233    2574 container_manager_linux.go:311] "Creating device plugin manager"
Nov 02 12:26:01.177306 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.177269    2574 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 02 12:26:01.177306 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.177286    2574 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 02 12:26:01.178551 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.178539    2574 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 02 12:26:01.183247 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.183234    2574 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 02 12:26:01.183368 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.183358    2574 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 02 12:26:01.184241 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.184229    2574 kubelet.go:493] "Attempting to sync node with API server"
Nov 02 12:26:01.184298 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.184246    2574 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 02 12:26:01.184298 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.184257    2574 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 02 12:26:01.184298 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.184266    2574 kubelet.go:405] "Adding apiserver pod source"
Nov 02 12:26:01.184298 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.184275    2574 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 02 12:26:01.186308 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.186278    2574 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 02 12:26:01.189839 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.189822    2574 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 02 12:26:01.197321 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.197305    2574 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 02 12:26:01.198255 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.198241    2574 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 02 12:26:01.198333 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.198280    2574 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 02 12:26:01.198475 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.198462    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 02 12:26:01.198533 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.198484    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 02 12:26:01.198533 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.198494    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 02 12:26:01.198533 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.198504    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 02 12:26:01.198533 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.198513    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 02 12:26:01.198533 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.198522    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 02 12:26:01.199340 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.199330    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 02 12:26:01.199393 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.199343    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 02 12:26:01.199393 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.199357    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 02 12:26:01.199393 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.199367    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 02 12:26:01.199393 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.199381    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 02 12:26:01.199564 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.199395    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 02 12:26:01.200598 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.200583    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 02 12:26:01.200678 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.200601    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 02 12:26:01.204596 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.204576    2574 server.go:1263] "Started kubelet"
Nov 02 12:26:01.204704 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.204649    2574 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 02 12:26:01.204775 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.204721    2574 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 02 12:26:01.204825 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.204806    2574 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 02 12:26:01.205393 ip-10-0-128-132 systemd[1]: Started Kubernetes Kubelet.
Nov 02 12:26:01.206290 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.206263    2574 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 02 12:26:01.206904 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.206887    2574 server.go:317] "Adding debug handlers to kubelet server"
Nov 02 12:26:01.211217 ip-10-0-128-132 kubenswrapper[2574]: E1102 12:26:01.211197    2574 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Nov 02 12:26:01.214350 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.214334    2574 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 02 12:26:01.214449 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.214352    2574 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 02 12:26:01.214449 ip-10-0-128-132 kubenswrapper[2574]: E1102 12:26:01.214416    2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-132.ec2.internal\" not found"
Nov 02 12:26:01.215141 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.215122    2574 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 02 12:26:01.215281 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.215267    2574 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 02 12:26:01.215351 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.215286    2574 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 02 12:26:01.215384 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.215363    2574 reconstruct.go:97] "Volume reconstruction finished"
Nov 02 12:26:01.215384 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.215368    2574 reconciler.go:29] "Reconciler: start to sync state"
Nov 02 12:26:01.216985 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.216969    2574 factory.go:153] Registering CRI-O factory
Nov 02 12:26:01.216985 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.216985    2574 factory.go:223] Registration of the crio container factory successfully
Nov 02 12:26:01.217125 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.217029    2574 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 02 12:26:01.217125 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.217035    2574 factory.go:55] Registering systemd factory
Nov 02 12:26:01.217125 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.217041    2574 factory.go:223] Registration of the systemd container factory successfully
Nov 02 12:26:01.217125 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.217052    2574 factory.go:103] Registering Raw factory
Nov 02 12:26:01.217125 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.217062    2574 manager.go:1199] Started watching for new ooms in manager
Nov 02 12:26:01.218153 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.218131    2574 manager.go:319] Starting recovery of all containers
Nov 02 12:26:01.231224 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.231201    2574 manager.go:324] Recovery completed
Nov 02 12:26:01.233453 ip-10-0-128-132 kubenswrapper[2574]: E1102 12:26:01.233408    2574 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-128-132.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Nov 02 12:26:01.233453 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.233371    2574 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-128-132.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Nov 02 12:26:01.236768 ip-10-0-128-132 kubenswrapper[2574]: E1102 12:26:01.233351    2574 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-132.ec2.internal.18e43a08d4cb6bfb  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-132.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-128-132.ec2.internal,},FirstTimestamp:2026-11-02 12:26:01.204550651 +0000 UTC m=+0.399258583,LastTimestamp:2026-11-02 12:26:01.204550651 +0000 UTC m=+0.399258583,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-132.ec2.internal,}"
Nov 02 12:26:01.237574 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.237559    2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 12:26:01.241608 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.241591    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-132.ec2.internal" event="NodeHasSufficientMemory"
Nov 02 12:26:01.241684 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.241628    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-132.ec2.internal" event="NodeHasNoDiskPressure"
Nov 02 12:26:01.241684 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.241642    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-132.ec2.internal" event="NodeHasSufficientPID"
Nov 02 12:26:01.242184 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.242168    2574 cpu_manager.go:226] "Starting" policy="none"
Nov 02 12:26:01.242184 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.242180    2574 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 02 12:26:01.242302 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.242197    2574 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 02 12:26:01.242302 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.242181    2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 12:26:01.243174 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.243156    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-132.ec2.internal" event="NodeHasSufficientMemory"
Nov 02 12:26:01.243275 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.243185    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-132.ec2.internal" event="NodeHasNoDiskPressure"
Nov 02 12:26:01.243275 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.243198    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-132.ec2.internal" event="NodeHasSufficientPID"
Nov 02 12:26:01.244551 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.244538    2574 policy_none.go:50] "Start"
Nov 02 12:26:01.244602 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.244557    2574 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 02 12:26:01.244602 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.244568    2574 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 02 12:26:01.246578 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.246564    2574 policy_none.go:44] "Start"
Nov 02 12:26:01.249960 ip-10-0-128-132 kubenswrapper[2574]: E1102 12:26:01.249937    2574 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service"
Nov 02 12:26:01.249960 ip-10-0-128-132 kubenswrapper[2574]: E1102 12:26:01.249944    2574 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-128-132.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node"
Nov 02 12:26:01.250724 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.250710    2574 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 02 12:26:01.250767 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.250759    2574 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-vrmh5"
Nov 02 12:26:01.268335 ip-10-0-128-132 kubenswrapper[2574]: E1102 12:26:01.268239    2574 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-132.ec2.internal.18e43a08d700dec0  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-132.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-128-132.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-128-132.ec2.internal,},FirstTimestamp:2026-11-02 12:26:01.241607872 +0000 UTC m=+0.436315808,LastTimestamp:2026-11-02 12:26:01.241607872 +0000 UTC m=+0.436315808,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-132.ec2.internal,}"
Nov 02 12:26:01.268452 ip-10-0-128-132 kubenswrapper[2574]: E1102 12:26:01.268378    2574 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver"
Nov 02 12:26:01.294445 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.293574    2574 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 02 12:26:01.296498 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.294905    2574 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 02 12:26:01.296498 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.294927    2574 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 02 12:26:01.296498 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.294955    2574 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 02 12:26:01.296498 ip-10-0-128-132 kubenswrapper[2574]: E1102 12:26:01.294994    2574 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Nov 02 12:26:01.311291 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.311274    2574 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 12:26:01.315239 ip-10-0-128-132 kubenswrapper[2574]: E1102 12:26:01.315215    2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-132.ec2.internal\" not found"
Nov 02 12:26:01.365499 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.365469    2574 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 02 12:26:01.367823 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.367803    2574 manager.go:341] "Starting Device Plugin manager"
Nov 02 12:26:01.367950 ip-10-0-128-132 kubenswrapper[2574]: E1102 12:26:01.367935    2574 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 02 12:26:01.368016 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.367952    2574 server.go:87] "Starting device plugin registration server"
Nov 02 12:26:01.368225 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.368213    2574 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 02 12:26:01.368289 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.368227    2574 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 02 12:26:01.368339 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.368318    2574 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 02 12:26:01.368415 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.368401    2574 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 02 12:26:01.368415 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.368413    2574 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 02 12:26:01.368840 ip-10-0-128-132 kubenswrapper[2574]: E1102 12:26:01.368812    2574 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 02 12:26:01.368936 ip-10-0-128-132 kubenswrapper[2574]: E1102 12:26:01.368866    2574 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-128-132.ec2.internal\" not found"
Nov 02 12:26:01.395595 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.395513    2574 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-132.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-128-132.ec2.internal"]
Nov 02 12:26:01.395595 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.395580    2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 12:26:01.396588 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.396571    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-132.ec2.internal" event="NodeHasSufficientMemory"
Nov 02 12:26:01.396655 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.396603    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-132.ec2.internal" event="NodeHasNoDiskPressure"
Nov 02 12:26:01.396655 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.396619    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-132.ec2.internal" event="NodeHasSufficientPID"
Nov 02 12:26:01.398954 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.398941    2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 12:26:01.399101 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.399084    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-132.ec2.internal"
Nov 02 12:26:01.399135 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.399124    2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 12:26:01.399676 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.399663    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-132.ec2.internal" event="NodeHasSufficientMemory"
Nov 02 12:26:01.399749 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.399688    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-132.ec2.internal" event="NodeHasNoDiskPressure"
Nov 02 12:26:01.399749 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.399697    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-132.ec2.internal" event="NodeHasSufficientPID"
Nov 02 12:26:01.399749 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.399702    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-132.ec2.internal" event="NodeHasSufficientMemory"
Nov 02 12:26:01.399749 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.399721    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-132.ec2.internal" event="NodeHasNoDiskPressure"
Nov 02 12:26:01.399749 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.399731    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-132.ec2.internal" event="NodeHasSufficientPID"
Nov 02 12:26:01.401864 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.401851    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-132.ec2.internal"
Nov 02 12:26:01.401915 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.401887    2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 12:26:01.402547 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.402533    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-132.ec2.internal" event="NodeHasSufficientMemory"
Nov 02 12:26:01.402613 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.402564    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-132.ec2.internal" event="NodeHasNoDiskPressure"
Nov 02 12:26:01.402613 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.402579    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-132.ec2.internal" event="NodeHasSufficientPID"
Nov 02 12:26:01.416961 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.416930    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6e3a47dd3a64bdc7ae97c7ee36b68697-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-132.ec2.internal\" (UID: \"6e3a47dd3a64bdc7ae97c7ee36b68697\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-132.ec2.internal"
Nov 02 12:26:01.417055 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.416969    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6e3a47dd3a64bdc7ae97c7ee36b68697-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-132.ec2.internal\" (UID: \"6e3a47dd3a64bdc7ae97c7ee36b68697\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-132.ec2.internal"
Nov 02 12:26:01.417055 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.416991    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9f29c1002c42911f93b255b0f3495fd0-config\") pod \"kube-apiserver-proxy-ip-10-0-128-132.ec2.internal\" (UID: \"9f29c1002c42911f93b255b0f3495fd0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-132.ec2.internal"
Nov 02 12:26:01.428493 ip-10-0-128-132 kubenswrapper[2574]: E1102 12:26:01.428472    2574 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-132.ec2.internal\" not found" node="ip-10-0-128-132.ec2.internal"
Nov 02 12:26:01.431176 ip-10-0-128-132 kubenswrapper[2574]: E1102 12:26:01.431159    2574 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-132.ec2.internal\" not found" node="ip-10-0-128-132.ec2.internal"
Nov 02 12:26:01.465440 ip-10-0-128-132 kubenswrapper[2574]: E1102 12:26:01.465416    2574 nodelease.go:50] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-128-132.ec2.internal\" not found" node="ip-10-0-128-132.ec2.internal"
Nov 02 12:26:01.468526 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.468512    2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 12:26:01.469450 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.469428    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-132.ec2.internal" event="NodeHasSufficientMemory"
Nov 02 12:26:01.469542 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.469459    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-132.ec2.internal" event="NodeHasNoDiskPressure"
Nov 02 12:26:01.469542 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.469473    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-132.ec2.internal" event="NodeHasSufficientPID"
Nov 02 12:26:01.469542 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.469498    2574 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-128-132.ec2.internal"
Nov 02 12:26:01.491448 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.491430    2574 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-128-132.ec2.internal"
Nov 02 12:26:01.491448 ip-10-0-128-132 kubenswrapper[2574]: E1102 12:26:01.491451    2574 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-128-132.ec2.internal\": node \"ip-10-0-128-132.ec2.internal\" not found"
Nov 02 12:26:01.517618 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.517584    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6e3a47dd3a64bdc7ae97c7ee36b68697-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-132.ec2.internal\" (UID: \"6e3a47dd3a64bdc7ae97c7ee36b68697\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-132.ec2.internal"
Nov 02 12:26:01.517748 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.517640    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6e3a47dd3a64bdc7ae97c7ee36b68697-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-132.ec2.internal\" (UID: \"6e3a47dd3a64bdc7ae97c7ee36b68697\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-132.ec2.internal"
Nov 02 12:26:01.517748 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.517665    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9f29c1002c42911f93b255b0f3495fd0-config\") pod \"kube-apiserver-proxy-ip-10-0-128-132.ec2.internal\" (UID: \"9f29c1002c42911f93b255b0f3495fd0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-132.ec2.internal"
Nov 02 12:26:01.517748 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.517678    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6e3a47dd3a64bdc7ae97c7ee36b68697-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-132.ec2.internal\" (UID: \"6e3a47dd3a64bdc7ae97c7ee36b68697\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-132.ec2.internal"
Nov 02 12:26:01.517748 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.517724    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9f29c1002c42911f93b255b0f3495fd0-config\") pod \"kube-apiserver-proxy-ip-10-0-128-132.ec2.internal\" (UID: \"9f29c1002c42911f93b255b0f3495fd0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-132.ec2.internal"
Nov 02 12:26:01.517889 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.517754    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6e3a47dd3a64bdc7ae97c7ee36b68697-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-132.ec2.internal\" (UID: \"6e3a47dd3a64bdc7ae97c7ee36b68697\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-132.ec2.internal"
Nov 02 12:26:01.531892 ip-10-0-128-132 kubenswrapper[2574]: E1102 12:26:01.531857    2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-132.ec2.internal\" not found"
Nov 02 12:26:01.632168 ip-10-0-128-132 kubenswrapper[2574]: E1102 12:26:01.632133    2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-132.ec2.internal\" not found"
Nov 02 12:26:01.733180 ip-10-0-128-132 kubenswrapper[2574]: E1102 12:26:01.733122    2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-132.ec2.internal\" not found"
Nov 02 12:26:01.733431 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.733413    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-132.ec2.internal"
Nov 02 12:26:01.736561 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:01.736544    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-132.ec2.internal"
Nov 02 12:26:03.311411 ip-10-0-128-132 kubenswrapper[2574]: E1102 12:26:03.311226    2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-132.ec2.internal\" not found"
Nov 02 12:26:03.411720 ip-10-0-128-132 kubenswrapper[2574]: E1102 12:26:03.411677    2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-132.ec2.internal\" not found"
Nov 02 12:26:03.512293 ip-10-0-128-132 kubenswrapper[2574]: E1102 12:26:03.512212    2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-132.ec2.internal\" not found"
Nov 02 12:26:03.612727 ip-10-0-128-132 kubenswrapper[2574]: E1102 12:26:03.612689    2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-132.ec2.internal\" not found"
Nov 02 12:26:03.621794 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:03.621771    2574 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Nov 02 12:26:03.621943 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:03.621920    2574 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Nov 02 12:26:03.713675 ip-10-0-128-132 kubenswrapper[2574]: E1102 12:26:03.713613    2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-132.ec2.internal\" not found"
Nov 02 12:26:03.734736 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:03.734699    2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-01 12:21:02 +0000 UTC" deadline="2028-08-01 01:52:52.96462666 +0000 UTC"
Nov 02 12:26:03.734736 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:03.734730    2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15301h26m49.22989967s"
Nov 02 12:26:03.813612 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:03.813558    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f29c1002c42911f93b255b0f3495fd0.slice/crio-947a04ccbe41c2f96116327d8affb1d55f29d958c984d2c07e166f1457ed1df4 WatchSource:0}: Error finding container 947a04ccbe41c2f96116327d8affb1d55f29d958c984d2c07e166f1457ed1df4: Status 404 returned error can't find the container with id 947a04ccbe41c2f96116327d8affb1d55f29d958c984d2c07e166f1457ed1df4
Nov 02 12:26:03.813797 ip-10-0-128-132 kubenswrapper[2574]: E1102 12:26:03.813778    2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-132.ec2.internal\" not found"
Nov 02 12:26:03.814133 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:03.814109    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e3a47dd3a64bdc7ae97c7ee36b68697.slice/crio-6428f691ff96f07129a9481aec168662f7b0c994b973c88e00f854d59b8c5700 WatchSource:0}: Error finding container 6428f691ff96f07129a9481aec168662f7b0c994b973c88e00f854d59b8c5700: Status 404 returned error can't find the container with id 6428f691ff96f07129a9481aec168662f7b0c994b973c88e00f854d59b8c5700
Nov 02 12:26:03.819373 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:03.819355    2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 12:26:03.822078 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:03.822061    2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 12:26:03.822984 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:03.822970    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-132.ec2.internal" event="NodeHasSufficientMemory"
Nov 02 12:26:03.823033 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:03.822996    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-132.ec2.internal" event="NodeHasNoDiskPressure"
Nov 02 12:26:03.823033 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:03.823012    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-132.ec2.internal" event="NodeHasSufficientPID"
Nov 02 12:26:03.914685 ip-10-0-128-132 kubenswrapper[2574]: E1102 12:26:03.914627    2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-132.ec2.internal\" not found"
Nov 02 12:26:03.992006 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:03.991976    2574 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 12:26:04.015278 ip-10-0-128-132 kubenswrapper[2574]: E1102 12:26:04.015252    2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-132.ec2.internal\" not found"
Nov 02 12:26:04.116088 ip-10-0-128-132 kubenswrapper[2574]: E1102 12:26:04.116012    2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-132.ec2.internal\" not found"
Nov 02 12:26:04.127474 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.127450    2574 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 12:26:04.127757 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.127740    2574 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 12:26:04.193025 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.192987    2574 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-132.ec2.internal"
Nov 02 12:26:04.217984 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.217960    2574 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 02 12:26:04.219184 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.219161    2574 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-132.ec2.internal"
Nov 02 12:26:04.240085 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.240057    2574 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 02 12:26:04.663757 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.663718    2574 apiserver.go:52] "Watching apiserver"
Nov 02 12:26:04.692516 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.692472    2574 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Nov 02 12:26:04.692830 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.692807    2574 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 02 12:26:04.693211 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.693179    2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/node-ca-9t4f5","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-132.ec2.internal","openshift-multus/multus-z5nxc","openshift-network-diagnostics/network-check-target-wx6zw","openshift-network-operator/iptables-alerter-7gc6b","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kbhhk","openshift-dns/node-resolver-7trhz","openshift-multus/network-metrics-daemon-dp7h7","openshift-ovn-kubernetes/ovnkube-node-qgv46","kube-system/global-pull-secret-syncer-6grht","kube-system/konnectivity-agent-dnm8d","kube-system/kube-apiserver-proxy-ip-10-0-128-132.ec2.internal","openshift-multus/multus-additional-cni-plugins-v7bcl","openshift-cluster-node-tuning-operator/tuned-vnv9x"]
Nov 02 12:26:04.695882 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.695859    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kbhhk"
Nov 02 12:26:04.698023 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.698002    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-vnv9x"
Nov 02 12:26:04.700881 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.700799    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-z5nxc"
Nov 02 12:26:04.703228 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.703209    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dp7h7"
Nov 02 12:26:04.703327 ip-10-0-128-132 kubenswrapper[2574]: E1102 12:26:04.703289    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dp7h7" podUID="09f70dd9-4dbe-46df-ac95-1c351a541a18"
Nov 02 12:26:04.705407 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.705389    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-7trhz"
Nov 02 12:26:04.708061 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.707649    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-7gc6b"
Nov 02 12:26:04.709974 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.709953    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-9t4f5"
Nov 02 12:26:04.711244 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.711217    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 02 12:26:04.711244 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.711228    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 02 12:26:04.711244 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.711241    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-7btt2\""
Nov 02 12:26:04.711409 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.711319    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 02 12:26:04.712291 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.712180    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wx6zw"
Nov 02 12:26:04.712291 ip-10-0-128-132 kubenswrapper[2574]: E1102 12:26:04.712254    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wx6zw" podUID="089d7bd4-d019-471c-960f-51346831fe45"
Nov 02 12:26:04.714544 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.714526    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-qgv46"
Nov 02 12:26:04.717272 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.717251    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6grht"
Nov 02 12:26:04.717400 ip-10-0-128-132 kubenswrapper[2574]: E1102 12:26:04.717324    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6grht" podUID="83964713-5989-4b35-91e8-3d3993688ccd"
Nov 02 12:26:04.719514 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.719492    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-dnm8d"
Nov 02 12:26:04.720088 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.720067    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 02 12:26:04.720088 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.720077    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-wm9fw\""
Nov 02 12:26:04.720320 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.720304    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-vg52x\""
Nov 02 12:26:04.720384 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.720340    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 02 12:26:04.720384 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.720373    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-bbkt9\""
Nov 02 12:26:04.720496 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.720414    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 02 12:26:04.720496 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.720429    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 02 12:26:04.720596 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.720516    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 02 12:26:04.721427 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.721409    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 02 12:26:04.721516 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.721502    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 02 12:26:04.721653 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.721598    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 02 12:26:04.722001 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.721981    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-v7bcl"
Nov 02 12:26:04.724883 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.724857    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 02 12:26:04.761969 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.761721    2574 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 02 12:26:04.761969 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.761824    2574 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-6pdcl"
Nov 02 12:26:04.771270 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.771240    2574 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-6pdcl"
Nov 02 12:26:04.777509 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.777462    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-132.ec2.internal" event={"ID":"6e3a47dd3a64bdc7ae97c7ee36b68697","Type":"ContainerStarted","Data":"6428f691ff96f07129a9481aec168662f7b0c994b973c88e00f854d59b8c5700"}
Nov 02 12:26:04.778797 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.778772    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-132.ec2.internal" event={"ID":"9f29c1002c42911f93b255b0f3495fd0","Type":"ContainerStarted","Data":"947a04ccbe41c2f96116327d8affb1d55f29d958c984d2c07e166f1457ed1df4"}
Nov 02 12:26:04.793835 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.793815    2574 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 02 12:26:04.804889 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.804855    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/993b825c-e7c4-445e-b8e8-5e92eb205f3e-host-var-lib-cni-bin\") pod \"multus-z5nxc\" (UID: \"993b825c-e7c4-445e-b8e8-5e92eb205f3e\") " pod="openshift-multus/multus-z5nxc"
Nov 02 12:26:04.805015 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.804903    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkl5z\" (UniqueName: \"kubernetes.io/projected/5507fd15-57b9-422a-af1d-5daf266bd9ca-kube-api-access-qkl5z\") pod \"node-resolver-7trhz\" (UID: \"5507fd15-57b9-422a-af1d-5daf266bd9ca\") " pod="openshift-dns/node-resolver-7trhz"
Nov 02 12:26:04.805015 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.804958    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42zsx\" (UniqueName: \"kubernetes.io/projected/09f70dd9-4dbe-46df-ac95-1c351a541a18-kube-api-access-42zsx\") pod \"network-metrics-daemon-dp7h7\" (UID: \"09f70dd9-4dbe-46df-ac95-1c351a541a18\") " pod="openshift-multus/network-metrics-daemon-dp7h7"
Nov 02 12:26:04.805015 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.805000    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a81d60e5-2049-4600-9b3f-d3826aee59fb-host-slash\") pod \"ovnkube-node-qgv46\" (UID: \"a81d60e5-2049-4600-9b3f-d3826aee59fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgv46"
Nov 02 12:26:04.805163 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.805032    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a81d60e5-2049-4600-9b3f-d3826aee59fb-etc-openvswitch\") pod \"ovnkube-node-qgv46\" (UID: \"a81d60e5-2049-4600-9b3f-d3826aee59fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgv46"
Nov 02 12:26:04.805163 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.805066    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a81d60e5-2049-4600-9b3f-d3826aee59fb-run-ovn\") pod \"ovnkube-node-qgv46\" (UID: \"a81d60e5-2049-4600-9b3f-d3826aee59fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgv46"
Nov 02 12:26:04.805163 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.805111    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a81d60e5-2049-4600-9b3f-d3826aee59fb-log-socket\") pod \"ovnkube-node-qgv46\" (UID: \"a81d60e5-2049-4600-9b3f-d3826aee59fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgv46"
Nov 02 12:26:04.805163 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.805156    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8a2e896a-489b-43fa-ba40-b2e3c0d56176-etc-modprobe-d\") pod \"tuned-vnv9x\" (UID: \"8a2e896a-489b-43fa-ba40-b2e3c0d56176\") " pod="openshift-cluster-node-tuning-operator/tuned-vnv9x"
Nov 02 12:26:04.805299 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.805184    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8a2e896a-489b-43fa-ba40-b2e3c0d56176-etc-sysctl-d\") pod \"tuned-vnv9x\" (UID: \"8a2e896a-489b-43fa-ba40-b2e3c0d56176\") " pod="openshift-cluster-node-tuning-operator/tuned-vnv9x"
Nov 02 12:26:04.805299 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.805207    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/993b825c-e7c4-445e-b8e8-5e92eb205f3e-host-var-lib-cni-multus\") pod \"multus-z5nxc\" (UID: \"993b825c-e7c4-445e-b8e8-5e92eb205f3e\") " pod="openshift-multus/multus-z5nxc"
Nov 02 12:26:04.805299 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.805234    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/993b825c-e7c4-445e-b8e8-5e92eb205f3e-multus-conf-dir\") pod \"multus-z5nxc\" (UID: \"993b825c-e7c4-445e-b8e8-5e92eb205f3e\") " pod="openshift-multus/multus-z5nxc"
Nov 02 12:26:04.805299 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.805259    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a81d60e5-2049-4600-9b3f-d3826aee59fb-env-overrides\") pod \"ovnkube-node-qgv46\" (UID: \"a81d60e5-2049-4600-9b3f-d3826aee59fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgv46"
Nov 02 12:26:04.805299 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.805284    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/83964713-5989-4b35-91e8-3d3993688ccd-kubelet-config\") pod \"global-pull-secret-syncer-6grht\" (UID: \"83964713-5989-4b35-91e8-3d3993688ccd\") " pod="kube-system/global-pull-secret-syncer-6grht"
Nov 02 12:26:04.805465 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.805330    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/83964713-5989-4b35-91e8-3d3993688ccd-original-pull-secret\") pod \"global-pull-secret-syncer-6grht\" (UID: \"83964713-5989-4b35-91e8-3d3993688ccd\") " pod="kube-system/global-pull-secret-syncer-6grht"
Nov 02 12:26:04.805465 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.805355    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/993b825c-e7c4-445e-b8e8-5e92eb205f3e-hostroot\") pod \"multus-z5nxc\" (UID: \"993b825c-e7c4-445e-b8e8-5e92eb205f3e\") " pod="openshift-multus/multus-z5nxc"
Nov 02 12:26:04.805465 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.805372    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/20cc95cd-9718-49ee-8082-e9ac00fcc147-host\") pod \"node-ca-9t4f5\" (UID: \"20cc95cd-9718-49ee-8082-e9ac00fcc147\") " pod="openshift-image-registry/node-ca-9t4f5"
Nov 02 12:26:04.805465 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.805401    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/af96531f-d9a0-4acc-ab22-62949e7400aa-socket-dir\") pod \"aws-ebs-csi-driver-node-kbhhk\" (UID: \"af96531f-d9a0-4acc-ab22-62949e7400aa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kbhhk"
Nov 02 12:26:04.805465 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.805415    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a81d60e5-2049-4600-9b3f-d3826aee59fb-host-kubelet\") pod \"ovnkube-node-qgv46\" (UID: \"a81d60e5-2049-4600-9b3f-d3826aee59fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgv46"
Nov 02 12:26:04.805465 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.805433    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f7c6ba1b-8808-4b6b-8965-2e4e7a067030-konnectivity-ca\") pod \"konnectivity-agent-dnm8d\" (UID: \"f7c6ba1b-8808-4b6b-8965-2e4e7a067030\") " pod="kube-system/konnectivity-agent-dnm8d"
Nov 02 12:26:04.805465 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.805447    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8a2e896a-489b-43fa-ba40-b2e3c0d56176-host\") pod \"tuned-vnv9x\" (UID: \"8a2e896a-489b-43fa-ba40-b2e3c0d56176\") " pod="openshift-cluster-node-tuning-operator/tuned-vnv9x"
Nov 02 12:26:04.805465 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.805464    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/af2e7d8e-b1df-499c-9a92-9ebe9d7bc7b2-iptables-alerter-script\") pod \"iptables-alerter-7gc6b\" (UID: \"af2e7d8e-b1df-499c-9a92-9ebe9d7bc7b2\") " pod="openshift-network-operator/iptables-alerter-7gc6b"
Nov 02 12:26:04.805762 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.805487    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/af96531f-d9a0-4acc-ab22-62949e7400aa-sys-fs\") pod \"aws-ebs-csi-driver-node-kbhhk\" (UID: \"af96531f-d9a0-4acc-ab22-62949e7400aa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kbhhk"
Nov 02 12:26:04.805762 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.805529    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9zns\" (UniqueName: \"kubernetes.io/projected/8a2e896a-489b-43fa-ba40-b2e3c0d56176-kube-api-access-q9zns\") pod \"tuned-vnv9x\" (UID: \"8a2e896a-489b-43fa-ba40-b2e3c0d56176\") " pod="openshift-cluster-node-tuning-operator/tuned-vnv9x"
Nov 02 12:26:04.805762 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.805549    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5507fd15-57b9-422a-af1d-5daf266bd9ca-tmp-dir\") pod \"node-resolver-7trhz\" (UID: \"5507fd15-57b9-422a-af1d-5daf266bd9ca\") " pod="openshift-dns/node-resolver-7trhz"
Nov 02 12:26:04.805762 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.805572    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/af2e7d8e-b1df-499c-9a92-9ebe9d7bc7b2-host-slash\") pod \"iptables-alerter-7gc6b\" (UID: \"af2e7d8e-b1df-499c-9a92-9ebe9d7bc7b2\") " pod="openshift-network-operator/iptables-alerter-7gc6b"
Nov 02 12:26:04.805762 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.805598    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a81d60e5-2049-4600-9b3f-d3826aee59fb-host-cni-bin\") pod \"ovnkube-node-qgv46\" (UID: \"a81d60e5-2049-4600-9b3f-d3826aee59fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgv46"
Nov 02 12:26:04.805762 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.805622    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8a2e896a-489b-43fa-ba40-b2e3c0d56176-var-lib-kubelet\") pod \"tuned-vnv9x\" (UID: \"8a2e896a-489b-43fa-ba40-b2e3c0d56176\") " pod="openshift-cluster-node-tuning-operator/tuned-vnv9x"
Nov 02 12:26:04.805997 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.805859    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/993b825c-e7c4-445e-b8e8-5e92eb205f3e-cni-binary-copy\") pod \"multus-z5nxc\" (UID: \"993b825c-e7c4-445e-b8e8-5e92eb205f3e\") " pod="openshift-multus/multus-z5nxc"
Nov 02 12:26:04.805997 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.805899    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/993b825c-e7c4-445e-b8e8-5e92eb205f3e-multus-socket-dir-parent\") pod \"multus-z5nxc\" (UID: \"993b825c-e7c4-445e-b8e8-5e92eb205f3e\") " pod="openshift-multus/multus-z5nxc"
Nov 02 12:26:04.805997 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.805928    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/af96531f-d9a0-4acc-ab22-62949e7400aa-kubelet-dir\") pod \"aws-ebs-csi-driver-node-kbhhk\" (UID: \"af96531f-d9a0-4acc-ab22-62949e7400aa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kbhhk"
Nov 02 12:26:04.806125 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.806011    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/feba123c-eeb3-412f-911e-a66da1981358-tuning-conf-dir\") pod \"multus-additional-cni-plugins-v7bcl\" (UID: \"feba123c-eeb3-412f-911e-a66da1981358\") " pod="openshift-multus/multus-additional-cni-plugins-v7bcl"
Nov 02 12:26:04.806125 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.806046    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8a2e896a-489b-43fa-ba40-b2e3c0d56176-etc-systemd\") pod \"tuned-vnv9x\" (UID: \"8a2e896a-489b-43fa-ba40-b2e3c0d56176\") " pod="openshift-cluster-node-tuning-operator/tuned-vnv9x"
Nov 02 12:26:04.806125 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.806063    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/993b825c-e7c4-445e-b8e8-5e92eb205f3e-cnibin\") pod \"multus-z5nxc\" (UID: \"993b825c-e7c4-445e-b8e8-5e92eb205f3e\") " pod="openshift-multus/multus-z5nxc"
Nov 02 12:26:04.806125 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.806114    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/993b825c-e7c4-445e-b8e8-5e92eb205f3e-host-run-multus-certs\") pod \"multus-z5nxc\" (UID: \"993b825c-e7c4-445e-b8e8-5e92eb205f3e\") " pod="openshift-multus/multus-z5nxc"
Nov 02 12:26:04.806316 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.806148    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a81d60e5-2049-4600-9b3f-d3826aee59fb-run-systemd\") pod \"ovnkube-node-qgv46\" (UID: \"a81d60e5-2049-4600-9b3f-d3826aee59fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgv46"
Nov 02 12:26:04.806316 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.806172    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/993b825c-e7c4-445e-b8e8-5e92eb205f3e-multus-daemon-config\") pod \"multus-z5nxc\" (UID: \"993b825c-e7c4-445e-b8e8-5e92eb205f3e\") " pod="openshift-multus/multus-z5nxc"
Nov 02 12:26:04.806316 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.806197    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/feba123c-eeb3-412f-911e-a66da1981358-system-cni-dir\") pod \"multus-additional-cni-plugins-v7bcl\" (UID: \"feba123c-eeb3-412f-911e-a66da1981358\") " pod="openshift-multus/multus-additional-cni-plugins-v7bcl"
Nov 02 12:26:04.806316 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.806229    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/feba123c-eeb3-412f-911e-a66da1981358-cni-binary-copy\") pod \"multus-additional-cni-plugins-v7bcl\" (UID: \"feba123c-eeb3-412f-911e-a66da1981358\") " pod="openshift-multus/multus-additional-cni-plugins-v7bcl"
Nov 02 12:26:04.806316 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.806265    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5507fd15-57b9-422a-af1d-5daf266bd9ca-hosts-file\") pod \"node-resolver-7trhz\" (UID: \"5507fd15-57b9-422a-af1d-5daf266bd9ca\") " pod="openshift-dns/node-resolver-7trhz"
Nov 02 12:26:04.806545 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.806314    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/20cc95cd-9718-49ee-8082-e9ac00fcc147-serviceca\") pod \"node-ca-9t4f5\" (UID: \"20cc95cd-9718-49ee-8082-e9ac00fcc147\") " pod="openshift-image-registry/node-ca-9t4f5"
Nov 02 12:26:04.806545 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.806356    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/af96531f-d9a0-4acc-ab22-62949e7400aa-device-dir\") pod \"aws-ebs-csi-driver-node-kbhhk\" (UID: \"af96531f-d9a0-4acc-ab22-62949e7400aa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kbhhk"
Nov 02 12:26:04.806545 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.806381    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a81d60e5-2049-4600-9b3f-d3826aee59fb-host-run-netns\") pod \"ovnkube-node-qgv46\" (UID: \"a81d60e5-2049-4600-9b3f-d3826aee59fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgv46"
Nov 02 12:26:04.806545 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.806415    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/af96531f-d9a0-4acc-ab22-62949e7400aa-registration-dir\") pod \"aws-ebs-csi-driver-node-kbhhk\" (UID: \"af96531f-d9a0-4acc-ab22-62949e7400aa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kbhhk"
Nov 02 12:26:04.806545 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.806464    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a81d60e5-2049-4600-9b3f-d3826aee59fb-node-log\") pod \"ovnkube-node-qgv46\" (UID: \"a81d60e5-2049-4600-9b3f-d3826aee59fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgv46"
Nov 02 12:26:04.806545 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.806491    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8a2e896a-489b-43fa-ba40-b2e3c0d56176-sys\") pod \"tuned-vnv9x\" (UID: \"8a2e896a-489b-43fa-ba40-b2e3c0d56176\") " pod="openshift-cluster-node-tuning-operator/tuned-vnv9x"
Nov 02 12:26:04.806545 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.806520    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/993b825c-e7c4-445e-b8e8-5e92eb205f3e-etc-kubernetes\") pod \"multus-z5nxc\" (UID: \"993b825c-e7c4-445e-b8e8-5e92eb205f3e\") " pod="openshift-multus/multus-z5nxc"
Nov 02 12:26:04.806545 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.806536    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9k4x\" (UniqueName: \"kubernetes.io/projected/20cc95cd-9718-49ee-8082-e9ac00fcc147-kube-api-access-c9k4x\") pod \"node-ca-9t4f5\" (UID: \"20cc95cd-9718-49ee-8082-e9ac00fcc147\") " pod="openshift-image-registry/node-ca-9t4f5"
Nov 02 12:26:04.806845 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.806563    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztjcq\" (UniqueName: \"kubernetes.io/projected/af96531f-d9a0-4acc-ab22-62949e7400aa-kube-api-access-ztjcq\") pod \"aws-ebs-csi-driver-node-kbhhk\" (UID: \"af96531f-d9a0-4acc-ab22-62949e7400aa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kbhhk"
Nov 02 12:26:04.806845 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.806655    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a81d60e5-2049-4600-9b3f-d3826aee59fb-host-cni-netd\") pod \"ovnkube-node-qgv46\" (UID: \"a81d60e5-2049-4600-9b3f-d3826aee59fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgv46"
Nov 02 12:26:04.806845 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.806717    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a81d60e5-2049-4600-9b3f-d3826aee59fb-ovn-node-metrics-cert\") pod \"ovnkube-node-qgv46\" (UID: \"a81d60e5-2049-4600-9b3f-d3826aee59fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgv46"
Nov 02 12:26:04.806845 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.806738    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/feba123c-eeb3-412f-911e-a66da1981358-cnibin\") pod \"multus-additional-cni-plugins-v7bcl\" (UID: \"feba123c-eeb3-412f-911e-a66da1981358\") " pod="openshift-multus/multus-additional-cni-plugins-v7bcl"
Nov 02 12:26:04.806845 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.806753    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/feba123c-eeb3-412f-911e-a66da1981358-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-v7bcl\" (UID: \"feba123c-eeb3-412f-911e-a66da1981358\") " pod="openshift-multus/multus-additional-cni-plugins-v7bcl"
Nov 02 12:26:04.806845 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.806774    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8a2e896a-489b-43fa-ba40-b2e3c0d56176-lib-modules\") pod \"tuned-vnv9x\" (UID: \"8a2e896a-489b-43fa-ba40-b2e3c0d56176\") " pod="openshift-cluster-node-tuning-operator/tuned-vnv9x"
Nov 02 12:26:04.806845 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.806792    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/993b825c-e7c4-445e-b8e8-5e92eb205f3e-multus-cni-dir\") pod \"multus-z5nxc\" (UID: \"993b825c-e7c4-445e-b8e8-5e92eb205f3e\") " pod="openshift-multus/multus-z5nxc"
Nov 02 12:26:04.806845 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.806814    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/993b825c-e7c4-445e-b8e8-5e92eb205f3e-host-run-k8s-cni-cncf-io\") pod \"multus-z5nxc\" (UID: \"993b825c-e7c4-445e-b8e8-5e92eb205f3e\") " pod="openshift-multus/multus-z5nxc"
Nov 02 12:26:04.806845 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.806836    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4629q\" (UniqueName: \"kubernetes.io/projected/af2e7d8e-b1df-499c-9a92-9ebe9d7bc7b2-kube-api-access-4629q\") pod \"iptables-alerter-7gc6b\" (UID: \"af2e7d8e-b1df-499c-9a92-9ebe9d7bc7b2\") " pod="openshift-network-operator/iptables-alerter-7gc6b"
Nov 02 12:26:04.807148 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.806855    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a81d60e5-2049-4600-9b3f-d3826aee59fb-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qgv46\" (UID: \"a81d60e5-2049-4600-9b3f-d3826aee59fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgv46"
Nov 02 12:26:04.807148 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.806871    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2j4q8\" (UniqueName: \"kubernetes.io/projected/a81d60e5-2049-4600-9b3f-d3826aee59fb-kube-api-access-2j4q8\") pod \"ovnkube-node-qgv46\" (UID: \"a81d60e5-2049-4600-9b3f-d3826aee59fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgv46"
Nov 02 12:26:04.807148 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.806886    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wphgd\" (UniqueName: \"kubernetes.io/projected/feba123c-eeb3-412f-911e-a66da1981358-kube-api-access-wphgd\") pod \"multus-additional-cni-plugins-v7bcl\" (UID: \"feba123c-eeb3-412f-911e-a66da1981358\") " pod="openshift-multus/multus-additional-cni-plugins-v7bcl"
Nov 02 12:26:04.807148 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.806901    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8a2e896a-489b-43fa-ba40-b2e3c0d56176-run\") pod \"tuned-vnv9x\" (UID: \"8a2e896a-489b-43fa-ba40-b2e3c0d56176\") " pod="openshift-cluster-node-tuning-operator/tuned-vnv9x"
Nov 02 12:26:04.807148 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.806915    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8a2e896a-489b-43fa-ba40-b2e3c0d56176-etc-tuned\") pod \"tuned-vnv9x\" (UID: \"8a2e896a-489b-43fa-ba40-b2e3c0d56176\") " pod="openshift-cluster-node-tuning-operator/tuned-vnv9x"
Nov 02 12:26:04.807148 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.806929    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nn9gd\" (UniqueName: \"kubernetes.io/projected/993b825c-e7c4-445e-b8e8-5e92eb205f3e-kube-api-access-nn9gd\") pod \"multus-z5nxc\" (UID: \"993b825c-e7c4-445e-b8e8-5e92eb205f3e\") " pod="openshift-multus/multus-z5nxc"
Nov 02 12:26:04.807148 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.806949    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/af96531f-d9a0-4acc-ab22-62949e7400aa-etc-selinux\") pod \"aws-ebs-csi-driver-node-kbhhk\" (UID: \"af96531f-d9a0-4acc-ab22-62949e7400aa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kbhhk"
Nov 02 12:26:04.807148 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.806969    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a81d60e5-2049-4600-9b3f-d3826aee59fb-run-openvswitch\") pod \"ovnkube-node-qgv46\" (UID: \"a81d60e5-2049-4600-9b3f-d3826aee59fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgv46"
Nov 02 12:26:04.807148 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.806993    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a81d60e5-2049-4600-9b3f-d3826aee59fb-ovnkube-config\") pod \"ovnkube-node-qgv46\" (UID: \"a81d60e5-2049-4600-9b3f-d3826aee59fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgv46"
Nov 02 12:26:04.807148 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.807017    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a81d60e5-2049-4600-9b3f-d3826aee59fb-ovnkube-script-lib\") pod \"ovnkube-node-qgv46\" (UID: \"a81d60e5-2049-4600-9b3f-d3826aee59fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgv46"
Nov 02 12:26:04.807148 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.807034    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/993b825c-e7c4-445e-b8e8-5e92eb205f3e-host-var-lib-kubelet\") pod \"multus-z5nxc\" (UID: \"993b825c-e7c4-445e-b8e8-5e92eb205f3e\") " pod="openshift-multus/multus-z5nxc"
Nov 02 12:26:04.807148 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.807048    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/09f70dd9-4dbe-46df-ac95-1c351a541a18-metrics-certs\") pod \"network-metrics-daemon-dp7h7\" (UID: \"09f70dd9-4dbe-46df-ac95-1c351a541a18\") " pod="openshift-multus/network-metrics-daemon-dp7h7"
Nov 02 12:26:04.807148 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.807062    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8a2e896a-489b-43fa-ba40-b2e3c0d56176-tmp\") pod \"tuned-vnv9x\" (UID: \"8a2e896a-489b-43fa-ba40-b2e3c0d56176\") " pod="openshift-cluster-node-tuning-operator/tuned-vnv9x"
Nov 02 12:26:04.807148 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.807077    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/993b825c-e7c4-445e-b8e8-5e92eb205f3e-host-run-netns\") pod \"multus-z5nxc\" (UID: \"993b825c-e7c4-445e-b8e8-5e92eb205f3e\") " pod="openshift-multus/multus-z5nxc"
Nov 02 12:26:04.807148 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.807094    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cws77\" (UniqueName: \"kubernetes.io/projected/089d7bd4-d019-471c-960f-51346831fe45-kube-api-access-cws77\") pod \"network-check-target-wx6zw\" (UID: \"089d7bd4-d019-471c-960f-51346831fe45\") " pod="openshift-network-diagnostics/network-check-target-wx6zw"
Nov 02 12:26:04.807148 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.807118    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a81d60e5-2049-4600-9b3f-d3826aee59fb-systemd-units\") pod \"ovnkube-node-qgv46\" (UID: \"a81d60e5-2049-4600-9b3f-d3826aee59fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgv46"
Nov 02 12:26:04.807817 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.807135    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a81d60e5-2049-4600-9b3f-d3826aee59fb-host-run-ovn-kubernetes\") pod \"ovnkube-node-qgv46\" (UID: \"a81d60e5-2049-4600-9b3f-d3826aee59fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgv46"
Nov 02 12:26:04.807817 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.807159    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/feba123c-eeb3-412f-911e-a66da1981358-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-v7bcl\" (UID: \"feba123c-eeb3-412f-911e-a66da1981358\") " pod="openshift-multus/multus-additional-cni-plugins-v7bcl"
Nov 02 12:26:04.807817 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.807180    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8a2e896a-489b-43fa-ba40-b2e3c0d56176-etc-kubernetes\") pod \"tuned-vnv9x\" (UID: \"8a2e896a-489b-43fa-ba40-b2e3c0d56176\") " pod="openshift-cluster-node-tuning-operator/tuned-vnv9x"
Nov 02 12:26:04.807817 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.807201    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8a2e896a-489b-43fa-ba40-b2e3c0d56176-etc-sysctl-conf\") pod \"tuned-vnv9x\" (UID: \"8a2e896a-489b-43fa-ba40-b2e3c0d56176\") " pod="openshift-cluster-node-tuning-operator/tuned-vnv9x"
Nov 02 12:26:04.807817 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.807221    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/993b825c-e7c4-445e-b8e8-5e92eb205f3e-system-cni-dir\") pod \"multus-z5nxc\" (UID: \"993b825c-e7c4-445e-b8e8-5e92eb205f3e\") " pod="openshift-multus/multus-z5nxc"
Nov 02 12:26:04.807817 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.807235    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a81d60e5-2049-4600-9b3f-d3826aee59fb-var-lib-openvswitch\") pod \"ovnkube-node-qgv46\" (UID: \"a81d60e5-2049-4600-9b3f-d3826aee59fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgv46"
Nov 02 12:26:04.807817 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.807251    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/83964713-5989-4b35-91e8-3d3993688ccd-dbus\") pod \"global-pull-secret-syncer-6grht\" (UID: \"83964713-5989-4b35-91e8-3d3993688ccd\") " pod="kube-system/global-pull-secret-syncer-6grht"
Nov 02 12:26:04.807817 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.807266    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f7c6ba1b-8808-4b6b-8965-2e4e7a067030-agent-certs\") pod \"konnectivity-agent-dnm8d\" (UID: \"f7c6ba1b-8808-4b6b-8965-2e4e7a067030\") " pod="kube-system/konnectivity-agent-dnm8d"
Nov 02 12:26:04.807817 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.807280    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8a2e896a-489b-43fa-ba40-b2e3c0d56176-etc-sysconfig\") pod \"tuned-vnv9x\" (UID: \"8a2e896a-489b-43fa-ba40-b2e3c0d56176\") " pod="openshift-cluster-node-tuning-operator/tuned-vnv9x"
Nov 02 12:26:04.811780 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.811653    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 02 12:26:04.811780 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.811729    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 02 12:26:04.812017 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.811997    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 02 12:26:04.812093 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.812034    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 02 12:26:04.812093 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.812069    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 02 12:26:04.812270 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.812250    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-fhvpr\""
Nov 02 12:26:04.812373 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.812313    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 02 12:26:04.812433 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.812383    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 02 12:26:04.812486 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.812444    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 02 12:26:04.812486 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.812456    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-l5rh2\""
Nov 02 12:26:04.812575 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.812491    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 02 12:26:04.812575 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.812522    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 02 12:26:04.812690 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.812575    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-hrc62\""
Nov 02 12:26:04.812690 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.812600    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 02 12:26:04.812690 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.812665    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 02 12:26:04.812690 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.812688    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-scnn5\""
Nov 02 12:26:04.812873 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.812606    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 02 12:26:04.812873 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.812707    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 02 12:26:04.812873 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.812741    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-2bc7v\""
Nov 02 12:26:04.812873 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.812789    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 02 12:26:04.885361 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.885333    2574 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 12:26:04.907868 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.907836    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2j4q8\" (UniqueName: \"kubernetes.io/projected/a81d60e5-2049-4600-9b3f-d3826aee59fb-kube-api-access-2j4q8\") pod \"ovnkube-node-qgv46\" (UID: \"a81d60e5-2049-4600-9b3f-d3826aee59fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgv46"
Nov 02 12:26:04.908052 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.907879    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wphgd\" (UniqueName: \"kubernetes.io/projected/feba123c-eeb3-412f-911e-a66da1981358-kube-api-access-wphgd\") pod \"multus-additional-cni-plugins-v7bcl\" (UID: \"feba123c-eeb3-412f-911e-a66da1981358\") " pod="openshift-multus/multus-additional-cni-plugins-v7bcl"
Nov 02 12:26:04.908052 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.907906    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8a2e896a-489b-43fa-ba40-b2e3c0d56176-run\") pod \"tuned-vnv9x\" (UID: \"8a2e896a-489b-43fa-ba40-b2e3c0d56176\") " pod="openshift-cluster-node-tuning-operator/tuned-vnv9x"
Nov 02 12:26:04.908052 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.907933    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8a2e896a-489b-43fa-ba40-b2e3c0d56176-etc-tuned\") pod \"tuned-vnv9x\" (UID: \"8a2e896a-489b-43fa-ba40-b2e3c0d56176\") " pod="openshift-cluster-node-tuning-operator/tuned-vnv9x"
Nov 02 12:26:04.908052 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.907956    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nn9gd\" (UniqueName: \"kubernetes.io/projected/993b825c-e7c4-445e-b8e8-5e92eb205f3e-kube-api-access-nn9gd\") pod \"multus-z5nxc\" (UID: \"993b825c-e7c4-445e-b8e8-5e92eb205f3e\") " pod="openshift-multus/multus-z5nxc"
Nov 02 12:26:04.908052 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.907980    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/af96531f-d9a0-4acc-ab22-62949e7400aa-etc-selinux\") pod \"aws-ebs-csi-driver-node-kbhhk\" (UID: \"af96531f-d9a0-4acc-ab22-62949e7400aa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kbhhk"
Nov 02 12:26:04.908052 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.907999    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a81d60e5-2049-4600-9b3f-d3826aee59fb-run-openvswitch\") pod \"ovnkube-node-qgv46\" (UID: \"a81d60e5-2049-4600-9b3f-d3826aee59fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgv46"
Nov 02 12:26:04.908052 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.908021    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a81d60e5-2049-4600-9b3f-d3826aee59fb-ovnkube-config\") pod \"ovnkube-node-qgv46\" (UID: \"a81d60e5-2049-4600-9b3f-d3826aee59fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgv46"
Nov 02 12:26:04.908052 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.908046    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a81d60e5-2049-4600-9b3f-d3826aee59fb-ovnkube-script-lib\") pod \"ovnkube-node-qgv46\" (UID: \"a81d60e5-2049-4600-9b3f-d3826aee59fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgv46"
Nov 02 12:26:04.908362 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.908069    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/993b825c-e7c4-445e-b8e8-5e92eb205f3e-host-var-lib-kubelet\") pod \"multus-z5nxc\" (UID: \"993b825c-e7c4-445e-b8e8-5e92eb205f3e\") " pod="openshift-multus/multus-z5nxc"
Nov 02 12:26:04.908362 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.908101    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/09f70dd9-4dbe-46df-ac95-1c351a541a18-metrics-certs\") pod \"network-metrics-daemon-dp7h7\" (UID: \"09f70dd9-4dbe-46df-ac95-1c351a541a18\") " pod="openshift-multus/network-metrics-daemon-dp7h7"
Nov 02 12:26:04.908362 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.908123    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8a2e896a-489b-43fa-ba40-b2e3c0d56176-tmp\") pod \"tuned-vnv9x\" (UID: \"8a2e896a-489b-43fa-ba40-b2e3c0d56176\") " pod="openshift-cluster-node-tuning-operator/tuned-vnv9x"
Nov 02 12:26:04.908362 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.908139    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/993b825c-e7c4-445e-b8e8-5e92eb205f3e-host-run-netns\") pod \"multus-z5nxc\" (UID: \"993b825c-e7c4-445e-b8e8-5e92eb205f3e\") " pod="openshift-multus/multus-z5nxc"
Nov 02 12:26:04.908362 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.908153    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cws77\" (UniqueName: \"kubernetes.io/projected/089d7bd4-d019-471c-960f-51346831fe45-kube-api-access-cws77\") pod \"network-check-target-wx6zw\" (UID: \"089d7bd4-d019-471c-960f-51346831fe45\") " pod="openshift-network-diagnostics/network-check-target-wx6zw"
Nov 02 12:26:04.908362 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.908169    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a81d60e5-2049-4600-9b3f-d3826aee59fb-systemd-units\") pod \"ovnkube-node-qgv46\" (UID: \"a81d60e5-2049-4600-9b3f-d3826aee59fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgv46"
Nov 02 12:26:04.908362 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.908190    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a81d60e5-2049-4600-9b3f-d3826aee59fb-host-run-ovn-kubernetes\") pod \"ovnkube-node-qgv46\" (UID: \"a81d60e5-2049-4600-9b3f-d3826aee59fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgv46"
Nov 02 12:26:04.908362 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.908215    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/feba123c-eeb3-412f-911e-a66da1981358-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-v7bcl\" (UID: \"feba123c-eeb3-412f-911e-a66da1981358\") " pod="openshift-multus/multus-additional-cni-plugins-v7bcl"
Nov 02 12:26:04.908362 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.908237    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8a2e896a-489b-43fa-ba40-b2e3c0d56176-etc-kubernetes\") pod \"tuned-vnv9x\" (UID: \"8a2e896a-489b-43fa-ba40-b2e3c0d56176\") " pod="openshift-cluster-node-tuning-operator/tuned-vnv9x"
Nov 02 12:26:04.908362 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.908258    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8a2e896a-489b-43fa-ba40-b2e3c0d56176-etc-sysctl-conf\") pod \"tuned-vnv9x\" (UID: \"8a2e896a-489b-43fa-ba40-b2e3c0d56176\") " pod="openshift-cluster-node-tuning-operator/tuned-vnv9x"
Nov 02 12:26:04.908362 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.908285    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/993b825c-e7c4-445e-b8e8-5e92eb205f3e-system-cni-dir\") pod \"multus-z5nxc\" (UID: \"993b825c-e7c4-445e-b8e8-5e92eb205f3e\") " pod="openshift-multus/multus-z5nxc"
Nov 02 12:26:04.908362 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.908306    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a81d60e5-2049-4600-9b3f-d3826aee59fb-var-lib-openvswitch\") pod \"ovnkube-node-qgv46\" (UID: \"a81d60e5-2049-4600-9b3f-d3826aee59fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgv46"
Nov 02 12:26:04.908362 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.908320    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/83964713-5989-4b35-91e8-3d3993688ccd-dbus\") pod \"global-pull-secret-syncer-6grht\" (UID: \"83964713-5989-4b35-91e8-3d3993688ccd\") " pod="kube-system/global-pull-secret-syncer-6grht"
Nov 02 12:26:04.908362 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.908334    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f7c6ba1b-8808-4b6b-8965-2e4e7a067030-agent-certs\") pod \"konnectivity-agent-dnm8d\" (UID: \"f7c6ba1b-8808-4b6b-8965-2e4e7a067030\") " pod="kube-system/konnectivity-agent-dnm8d"
Nov 02 12:26:04.908362 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.908350    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8a2e896a-489b-43fa-ba40-b2e3c0d56176-etc-sysconfig\") pod \"tuned-vnv9x\" (UID: \"8a2e896a-489b-43fa-ba40-b2e3c0d56176\") " pod="openshift-cluster-node-tuning-operator/tuned-vnv9x"
Nov 02 12:26:04.908944 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.908380    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/993b825c-e7c4-445e-b8e8-5e92eb205f3e-host-var-lib-cni-bin\") pod \"multus-z5nxc\" (UID: \"993b825c-e7c4-445e-b8e8-5e92eb205f3e\") " pod="openshift-multus/multus-z5nxc"
Nov 02 12:26:04.908944 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.908404    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkl5z\" (UniqueName: \"kubernetes.io/projected/5507fd15-57b9-422a-af1d-5daf266bd9ca-kube-api-access-qkl5z\") pod \"node-resolver-7trhz\" (UID: \"5507fd15-57b9-422a-af1d-5daf266bd9ca\") " pod="openshift-dns/node-resolver-7trhz"
Nov 02 12:26:04.908944 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.908428    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42zsx\" (UniqueName: \"kubernetes.io/projected/09f70dd9-4dbe-46df-ac95-1c351a541a18-kube-api-access-42zsx\") pod \"network-metrics-daemon-dp7h7\" (UID: \"09f70dd9-4dbe-46df-ac95-1c351a541a18\") " pod="openshift-multus/network-metrics-daemon-dp7h7"
Nov 02 12:26:04.908944 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.908452    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a81d60e5-2049-4600-9b3f-d3826aee59fb-host-slash\") pod \"ovnkube-node-qgv46\" (UID: \"a81d60e5-2049-4600-9b3f-d3826aee59fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgv46"
Nov 02 12:26:04.908944 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.908476    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a81d60e5-2049-4600-9b3f-d3826aee59fb-etc-openvswitch\") pod \"ovnkube-node-qgv46\" (UID: \"a81d60e5-2049-4600-9b3f-d3826aee59fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgv46"
Nov 02 12:26:04.908944 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.908501    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a81d60e5-2049-4600-9b3f-d3826aee59fb-run-ovn\") pod \"ovnkube-node-qgv46\" (UID: \"a81d60e5-2049-4600-9b3f-d3826aee59fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgv46"
Nov 02 12:26:04.908944 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.908524    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a81d60e5-2049-4600-9b3f-d3826aee59fb-log-socket\") pod \"ovnkube-node-qgv46\" (UID: \"a81d60e5-2049-4600-9b3f-d3826aee59fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgv46"
Nov 02 12:26:04.908944 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.908549    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8a2e896a-489b-43fa-ba40-b2e3c0d56176-etc-modprobe-d\") pod \"tuned-vnv9x\" (UID: \"8a2e896a-489b-43fa-ba40-b2e3c0d56176\") " pod="openshift-cluster-node-tuning-operator/tuned-vnv9x"
Nov 02 12:26:04.908944 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.908572    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8a2e896a-489b-43fa-ba40-b2e3c0d56176-etc-sysctl-d\") pod \"tuned-vnv9x\" (UID: \"8a2e896a-489b-43fa-ba40-b2e3c0d56176\") " pod="openshift-cluster-node-tuning-operator/tuned-vnv9x"
Nov 02 12:26:04.908944 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.908593    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/993b825c-e7c4-445e-b8e8-5e92eb205f3e-host-var-lib-cni-multus\") pod \"multus-z5nxc\" (UID: \"993b825c-e7c4-445e-b8e8-5e92eb205f3e\") " pod="openshift-multus/multus-z5nxc"
Nov 02 12:26:04.908944 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.908618    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/993b825c-e7c4-445e-b8e8-5e92eb205f3e-multus-conf-dir\") pod \"multus-z5nxc\" (UID: \"993b825c-e7c4-445e-b8e8-5e92eb205f3e\") " pod="openshift-multus/multus-z5nxc"
Nov 02 12:26:04.908944 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.908662    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a81d60e5-2049-4600-9b3f-d3826aee59fb-env-overrides\") pod \"ovnkube-node-qgv46\" (UID: \"a81d60e5-2049-4600-9b3f-d3826aee59fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgv46"
Nov 02 12:26:04.908944 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.908687    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/83964713-5989-4b35-91e8-3d3993688ccd-kubelet-config\") pod \"global-pull-secret-syncer-6grht\" (UID: \"83964713-5989-4b35-91e8-3d3993688ccd\") " pod="kube-system/global-pull-secret-syncer-6grht"
Nov 02 12:26:04.908944 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.908713    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/83964713-5989-4b35-91e8-3d3993688ccd-original-pull-secret\") pod \"global-pull-secret-syncer-6grht\" (UID: \"83964713-5989-4b35-91e8-3d3993688ccd\") " pod="kube-system/global-pull-secret-syncer-6grht"
Nov 02 12:26:04.909540 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.908996    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/993b825c-e7c4-445e-b8e8-5e92eb205f3e-hostroot\") pod \"multus-z5nxc\" (UID: \"993b825c-e7c4-445e-b8e8-5e92eb205f3e\") " pod="openshift-multus/multus-z5nxc"
Nov 02 12:26:04.909540 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.909024    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/20cc95cd-9718-49ee-8082-e9ac00fcc147-host\") pod \"node-ca-9t4f5\" (UID: \"20cc95cd-9718-49ee-8082-e9ac00fcc147\") " pod="openshift-image-registry/node-ca-9t4f5"
Nov 02 12:26:04.909540 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.909047    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/af96531f-d9a0-4acc-ab22-62949e7400aa-socket-dir\") pod \"aws-ebs-csi-driver-node-kbhhk\" (UID: \"af96531f-d9a0-4acc-ab22-62949e7400aa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kbhhk"
Nov 02 12:26:04.909540 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.909071    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a81d60e5-2049-4600-9b3f-d3826aee59fb-host-kubelet\") pod \"ovnkube-node-qgv46\" (UID: \"a81d60e5-2049-4600-9b3f-d3826aee59fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgv46"
Nov 02 12:26:04.909540 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.909100    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f7c6ba1b-8808-4b6b-8965-2e4e7a067030-konnectivity-ca\") pod \"konnectivity-agent-dnm8d\" (UID: \"f7c6ba1b-8808-4b6b-8965-2e4e7a067030\") " pod="kube-system/konnectivity-agent-dnm8d"
Nov 02 12:26:04.909540 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.909128    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8a2e896a-489b-43fa-ba40-b2e3c0d56176-host\") pod \"tuned-vnv9x\" (UID: \"8a2e896a-489b-43fa-ba40-b2e3c0d56176\") " pod="openshift-cluster-node-tuning-operator/tuned-vnv9x"
Nov 02 12:26:04.909540 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.909160    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/af2e7d8e-b1df-499c-9a92-9ebe9d7bc7b2-iptables-alerter-script\") pod \"iptables-alerter-7gc6b\" (UID: \"af2e7d8e-b1df-499c-9a92-9ebe9d7bc7b2\") " pod="openshift-network-operator/iptables-alerter-7gc6b"
Nov 02 12:26:04.909540 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.909182    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/af96531f-d9a0-4acc-ab22-62949e7400aa-sys-fs\") pod \"aws-ebs-csi-driver-node-kbhhk\" (UID: \"af96531f-d9a0-4acc-ab22-62949e7400aa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kbhhk"
Nov 02 12:26:04.909540 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.909207    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9zns\" (UniqueName: \"kubernetes.io/projected/8a2e896a-489b-43fa-ba40-b2e3c0d56176-kube-api-access-q9zns\") pod \"tuned-vnv9x\" (UID: \"8a2e896a-489b-43fa-ba40-b2e3c0d56176\") " pod="openshift-cluster-node-tuning-operator/tuned-vnv9x"
Nov 02 12:26:04.909540 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.909224    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5507fd15-57b9-422a-af1d-5daf266bd9ca-tmp-dir\") pod \"node-resolver-7trhz\" (UID: \"5507fd15-57b9-422a-af1d-5daf266bd9ca\") " pod="openshift-dns/node-resolver-7trhz"
Nov 02 12:26:04.909540 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.909243    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/af2e7d8e-b1df-499c-9a92-9ebe9d7bc7b2-host-slash\") pod \"iptables-alerter-7gc6b\" (UID: \"af2e7d8e-b1df-499c-9a92-9ebe9d7bc7b2\") " pod="openshift-network-operator/iptables-alerter-7gc6b"
Nov 02 12:26:04.909540 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.909262    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a81d60e5-2049-4600-9b3f-d3826aee59fb-host-cni-bin\") pod \"ovnkube-node-qgv46\" (UID: \"a81d60e5-2049-4600-9b3f-d3826aee59fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgv46"
Nov 02 12:26:04.909540 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.909280    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8a2e896a-489b-43fa-ba40-b2e3c0d56176-var-lib-kubelet\") pod \"tuned-vnv9x\" (UID: \"8a2e896a-489b-43fa-ba40-b2e3c0d56176\") " pod="openshift-cluster-node-tuning-operator/tuned-vnv9x"
Nov 02 12:26:04.909540 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.909304    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/993b825c-e7c4-445e-b8e8-5e92eb205f3e-cni-binary-copy\") pod \"multus-z5nxc\" (UID: \"993b825c-e7c4-445e-b8e8-5e92eb205f3e\") " pod="openshift-multus/multus-z5nxc"
Nov 02 12:26:04.909540 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.909323    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/993b825c-e7c4-445e-b8e8-5e92eb205f3e-multus-socket-dir-parent\") pod \"multus-z5nxc\" (UID: \"993b825c-e7c4-445e-b8e8-5e92eb205f3e\") " pod="openshift-multus/multus-z5nxc"
Nov 02 12:26:04.909540 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.909341    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/af96531f-d9a0-4acc-ab22-62949e7400aa-kubelet-dir\") pod \"aws-ebs-csi-driver-node-kbhhk\" (UID: \"af96531f-d9a0-4acc-ab22-62949e7400aa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kbhhk"
Nov 02 12:26:04.909540 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.909361    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/feba123c-eeb3-412f-911e-a66da1981358-tuning-conf-dir\") pod \"multus-additional-cni-plugins-v7bcl\" (UID: \"feba123c-eeb3-412f-911e-a66da1981358\") " pod="openshift-multus/multus-additional-cni-plugins-v7bcl"
Nov 02 12:26:04.910274 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.909381    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8a2e896a-489b-43fa-ba40-b2e3c0d56176-etc-systemd\") pod \"tuned-vnv9x\" (UID: \"8a2e896a-489b-43fa-ba40-b2e3c0d56176\") " pod="openshift-cluster-node-tuning-operator/tuned-vnv9x"
Nov 02 12:26:04.910274 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.909396    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/993b825c-e7c4-445e-b8e8-5e92eb205f3e-cnibin\") pod \"multus-z5nxc\" (UID: \"993b825c-e7c4-445e-b8e8-5e92eb205f3e\") " pod="openshift-multus/multus-z5nxc"
Nov 02 12:26:04.910274 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.909416    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/993b825c-e7c4-445e-b8e8-5e92eb205f3e-host-run-multus-certs\") pod \"multus-z5nxc\" (UID: \"993b825c-e7c4-445e-b8e8-5e92eb205f3e\") " pod="openshift-multus/multus-z5nxc"
Nov 02 12:26:04.910274 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.909437    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a81d60e5-2049-4600-9b3f-d3826aee59fb-run-systemd\") pod \"ovnkube-node-qgv46\" (UID: \"a81d60e5-2049-4600-9b3f-d3826aee59fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgv46"
Nov 02 12:26:04.910274 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.909458    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/993b825c-e7c4-445e-b8e8-5e92eb205f3e-multus-daemon-config\") pod \"multus-z5nxc\" (UID: \"993b825c-e7c4-445e-b8e8-5e92eb205f3e\") " pod="openshift-multus/multus-z5nxc"
Nov 02 12:26:04.910274 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.909478    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/feba123c-eeb3-412f-911e-a66da1981358-system-cni-dir\") pod \"multus-additional-cni-plugins-v7bcl\" (UID: \"feba123c-eeb3-412f-911e-a66da1981358\") " pod="openshift-multus/multus-additional-cni-plugins-v7bcl"
Nov 02 12:26:04.910274 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.909495    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/feba123c-eeb3-412f-911e-a66da1981358-cni-binary-copy\") pod \"multus-additional-cni-plugins-v7bcl\" (UID: \"feba123c-eeb3-412f-911e-a66da1981358\") " pod="openshift-multus/multus-additional-cni-plugins-v7bcl"
Nov 02 12:26:04.910274 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.909545    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5507fd15-57b9-422a-af1d-5daf266bd9ca-hosts-file\") pod \"node-resolver-7trhz\" (UID: \"5507fd15-57b9-422a-af1d-5daf266bd9ca\") " pod="openshift-dns/node-resolver-7trhz"
Nov 02 12:26:04.910274 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.909570    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/20cc95cd-9718-49ee-8082-e9ac00fcc147-serviceca\") pod \"node-ca-9t4f5\" (UID: \"20cc95cd-9718-49ee-8082-e9ac00fcc147\") " pod="openshift-image-registry/node-ca-9t4f5"
Nov 02 12:26:04.910274 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.909598    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/af96531f-d9a0-4acc-ab22-62949e7400aa-device-dir\") pod \"aws-ebs-csi-driver-node-kbhhk\" (UID: \"af96531f-d9a0-4acc-ab22-62949e7400aa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kbhhk"
Nov 02 12:26:04.910274 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.909622    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a81d60e5-2049-4600-9b3f-d3826aee59fb-host-run-netns\") pod \"ovnkube-node-qgv46\" (UID: \"a81d60e5-2049-4600-9b3f-d3826aee59fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgv46"
Nov 02 12:26:04.910274 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.909663    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/af96531f-d9a0-4acc-ab22-62949e7400aa-registration-dir\") pod \"aws-ebs-csi-driver-node-kbhhk\" (UID: \"af96531f-d9a0-4acc-ab22-62949e7400aa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kbhhk"
Nov 02 12:26:04.910274 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.909695    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a81d60e5-2049-4600-9b3f-d3826aee59fb-node-log\") pod \"ovnkube-node-qgv46\" (UID: \"a81d60e5-2049-4600-9b3f-d3826aee59fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgv46"
Nov 02 12:26:04.910274 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.909730    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8a2e896a-489b-43fa-ba40-b2e3c0d56176-sys\") pod \"tuned-vnv9x\" (UID: \"8a2e896a-489b-43fa-ba40-b2e3c0d56176\") " pod="openshift-cluster-node-tuning-operator/tuned-vnv9x"
Nov 02 12:26:04.910274 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.909745    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/993b825c-e7c4-445e-b8e8-5e92eb205f3e-host-var-lib-cni-multus\") pod \"multus-z5nxc\" (UID: \"993b825c-e7c4-445e-b8e8-5e92eb205f3e\") " pod="openshift-multus/multus-z5nxc"
Nov 02 12:26:04.910274 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.909757    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/993b825c-e7c4-445e-b8e8-5e92eb205f3e-etc-kubernetes\") pod \"multus-z5nxc\" (UID: \"993b825c-e7c4-445e-b8e8-5e92eb205f3e\") " pod="openshift-multus/multus-z5nxc"
Nov 02 12:26:04.910274 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.909797    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/993b825c-e7c4-445e-b8e8-5e92eb205f3e-etc-kubernetes\") pod \"multus-z5nxc\" (UID: \"993b825c-e7c4-445e-b8e8-5e92eb205f3e\") " pod="openshift-multus/multus-z5nxc"
Nov 02 12:26:04.910274 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.909834    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9k4x\" (UniqueName: \"kubernetes.io/projected/20cc95cd-9718-49ee-8082-e9ac00fcc147-kube-api-access-c9k4x\") pod \"node-ca-9t4f5\" (UID: \"20cc95cd-9718-49ee-8082-e9ac00fcc147\") " pod="openshift-image-registry/node-ca-9t4f5"
Nov 02 12:26:04.911008 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.909840    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/993b825c-e7c4-445e-b8e8-5e92eb205f3e-multus-conf-dir\") pod \"multus-z5nxc\" (UID: \"993b825c-e7c4-445e-b8e8-5e92eb205f3e\") " pod="openshift-multus/multus-z5nxc"
Nov 02 12:26:04.911008 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.909857    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztjcq\" (UniqueName: \"kubernetes.io/projected/af96531f-d9a0-4acc-ab22-62949e7400aa-kube-api-access-ztjcq\") pod \"aws-ebs-csi-driver-node-kbhhk\" (UID: \"af96531f-d9a0-4acc-ab22-62949e7400aa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kbhhk"
Nov 02 12:26:04.911008 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.909881    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a81d60e5-2049-4600-9b3f-d3826aee59fb-host-cni-netd\") pod \"ovnkube-node-qgv46\" (UID: \"a81d60e5-2049-4600-9b3f-d3826aee59fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgv46"
Nov 02 12:26:04.911008 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.909902    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a81d60e5-2049-4600-9b3f-d3826aee59fb-ovn-node-metrics-cert\") pod \"ovnkube-node-qgv46\" (UID: \"a81d60e5-2049-4600-9b3f-d3826aee59fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgv46"
Nov 02 12:26:04.911008 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.909923    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/feba123c-eeb3-412f-911e-a66da1981358-cnibin\") pod \"multus-additional-cni-plugins-v7bcl\" (UID: \"feba123c-eeb3-412f-911e-a66da1981358\") " pod="openshift-multus/multus-additional-cni-plugins-v7bcl"
Nov 02 12:26:04.911008 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.909941    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/feba123c-eeb3-412f-911e-a66da1981358-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-v7bcl\" (UID: \"feba123c-eeb3-412f-911e-a66da1981358\") " pod="openshift-multus/multus-additional-cni-plugins-v7bcl"
Nov 02 12:26:04.911008 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.909964    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8a2e896a-489b-43fa-ba40-b2e3c0d56176-lib-modules\") pod \"tuned-vnv9x\" (UID: \"8a2e896a-489b-43fa-ba40-b2e3c0d56176\") " pod="openshift-cluster-node-tuning-operator/tuned-vnv9x"
Nov 02 12:26:04.911008 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.909985    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/993b825c-e7c4-445e-b8e8-5e92eb205f3e-multus-cni-dir\") pod \"multus-z5nxc\" (UID: \"993b825c-e7c4-445e-b8e8-5e92eb205f3e\") " pod="openshift-multus/multus-z5nxc"
Nov 02 12:26:04.911008 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.910005    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/993b825c-e7c4-445e-b8e8-5e92eb205f3e-host-run-k8s-cni-cncf-io\") pod \"multus-z5nxc\" (UID: \"993b825c-e7c4-445e-b8e8-5e92eb205f3e\") " pod="openshift-multus/multus-z5nxc"
Nov 02 12:26:04.911008 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.910028    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4629q\" (UniqueName: \"kubernetes.io/projected/af2e7d8e-b1df-499c-9a92-9ebe9d7bc7b2-kube-api-access-4629q\") pod \"iptables-alerter-7gc6b\" (UID: \"af2e7d8e-b1df-499c-9a92-9ebe9d7bc7b2\") " pod="openshift-network-operator/iptables-alerter-7gc6b"
Nov 02 12:26:04.911008 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.910046    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a81d60e5-2049-4600-9b3f-d3826aee59fb-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qgv46\" (UID: \"a81d60e5-2049-4600-9b3f-d3826aee59fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgv46"
Nov 02 12:26:04.911008 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.910137    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a81d60e5-2049-4600-9b3f-d3826aee59fb-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qgv46\" (UID: \"a81d60e5-2049-4600-9b3f-d3826aee59fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgv46"
Nov 02 12:26:04.911008 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.910247    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a81d60e5-2049-4600-9b3f-d3826aee59fb-env-overrides\") pod \"ovnkube-node-qgv46\" (UID: \"a81d60e5-2049-4600-9b3f-d3826aee59fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgv46"
Nov 02 12:26:04.911008 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.910806    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8a2e896a-489b-43fa-ba40-b2e3c0d56176-run\") pod \"tuned-vnv9x\" (UID: \"8a2e896a-489b-43fa-ba40-b2e3c0d56176\") " pod="openshift-cluster-node-tuning-operator/tuned-vnv9x"
Nov 02 12:26:04.911586 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.911555    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8a2e896a-489b-43fa-ba40-b2e3c0d56176-lib-modules\") pod \"tuned-vnv9x\" (UID: \"8a2e896a-489b-43fa-ba40-b2e3c0d56176\") " pod="openshift-cluster-node-tuning-operator/tuned-vnv9x"
Nov 02 12:26:04.911635 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.911599    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/feba123c-eeb3-412f-911e-a66da1981358-cnibin\") pod \"multus-additional-cni-plugins-v7bcl\" (UID: \"feba123c-eeb3-412f-911e-a66da1981358\") " pod="openshift-multus/multus-additional-cni-plugins-v7bcl"
Nov 02 12:26:04.911707 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.911673    2574 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 02 12:26:04.911866 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.911844    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/af96531f-d9a0-4acc-ab22-62949e7400aa-registration-dir\") pod \"aws-ebs-csi-driver-node-kbhhk\" (UID: \"af96531f-d9a0-4acc-ab22-62949e7400aa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kbhhk"
Nov 02 12:26:04.911921 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.911857    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8a2e896a-489b-43fa-ba40-b2e3c0d56176-etc-kubernetes\") pod \"tuned-vnv9x\" (UID: \"8a2e896a-489b-43fa-ba40-b2e3c0d56176\") " pod="openshift-cluster-node-tuning-operator/tuned-vnv9x"
Nov 02 12:26:04.911921 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.911890    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/993b825c-e7c4-445e-b8e8-5e92eb205f3e-multus-cni-dir\") pod \"multus-z5nxc\" (UID: \"993b825c-e7c4-445e-b8e8-5e92eb205f3e\") " pod="openshift-multus/multus-z5nxc"
Nov 02 12:26:04.912008 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.911923    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a81d60e5-2049-4600-9b3f-d3826aee59fb-node-log\") pod \"ovnkube-node-qgv46\" (UID: \"a81d60e5-2049-4600-9b3f-d3826aee59fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgv46"
Nov 02 12:26:04.912008 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.911986    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8a2e896a-489b-43fa-ba40-b2e3c0d56176-sys\") pod \"tuned-vnv9x\" (UID: \"8a2e896a-489b-43fa-ba40-b2e3c0d56176\") " pod="openshift-cluster-node-tuning-operator/tuned-vnv9x"
Nov 02 12:26:04.912008 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.911990    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8a2e896a-489b-43fa-ba40-b2e3c0d56176-etc-sysctl-conf\") pod \"tuned-vnv9x\" (UID: \"8a2e896a-489b-43fa-ba40-b2e3c0d56176\") " pod="openshift-cluster-node-tuning-operator/tuned-vnv9x"
Nov 02 12:26:04.912138 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.912030    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a81d60e5-2049-4600-9b3f-d3826aee59fb-host-run-netns\") pod \"ovnkube-node-qgv46\" (UID: \"a81d60e5-2049-4600-9b3f-d3826aee59fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgv46"
Nov 02 12:26:04.912138 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.912034    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/993b825c-e7c4-445e-b8e8-5e92eb205f3e-hostroot\") pod \"multus-z5nxc\" (UID: \"993b825c-e7c4-445e-b8e8-5e92eb205f3e\") " pod="openshift-multus/multus-z5nxc"
Nov 02 12:26:04.912138 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.912080    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/993b825c-e7c4-445e-b8e8-5e92eb205f3e-host-run-k8s-cni-cncf-io\") pod \"multus-z5nxc\" (UID: \"993b825c-e7c4-445e-b8e8-5e92eb205f3e\") " pod="openshift-multus/multus-z5nxc"
Nov 02 12:26:04.912138 ip-10-0-128-132 kubenswrapper[2574]: E1102 12:26:04.912084    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 02 12:26:04.912138 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.912104    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/feba123c-eeb3-412f-911e-a66da1981358-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-v7bcl\" (UID: \"feba123c-eeb3-412f-911e-a66da1981358\") " pod="openshift-multus/multus-additional-cni-plugins-v7bcl"
Nov 02 12:26:04.912362 ip-10-0-128-132 kubenswrapper[2574]: E1102 12:26:04.912161    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/83964713-5989-4b35-91e8-3d3993688ccd-original-pull-secret podName:83964713-5989-4b35-91e8-3d3993688ccd nodeName:}" failed. No retries permitted until 2026-11-02 12:26:05.412133613 +0000 UTC m=+3.129076384 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/83964713-5989-4b35-91e8-3d3993688ccd-original-pull-secret") pod "global-pull-secret-syncer-6grht" (UID: "83964713-5989-4b35-91e8-3d3993688ccd") : object "kube-system"/"original-pull-secret" not registered
Nov 02 12:26:04.912362 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.912198    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/83964713-5989-4b35-91e8-3d3993688ccd-kubelet-config\") pod \"global-pull-secret-syncer-6grht\" (UID: \"83964713-5989-4b35-91e8-3d3993688ccd\") " pod="kube-system/global-pull-secret-syncer-6grht"
Nov 02 12:26:04.912362 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.912216    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/993b825c-e7c4-445e-b8e8-5e92eb205f3e-system-cni-dir\") pod \"multus-z5nxc\" (UID: \"993b825c-e7c4-445e-b8e8-5e92eb205f3e\") " pod="openshift-multus/multus-z5nxc"
Nov 02 12:26:04.912362 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.912261    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a81d60e5-2049-4600-9b3f-d3826aee59fb-host-kubelet\") pod \"ovnkube-node-qgv46\" (UID: \"a81d60e5-2049-4600-9b3f-d3826aee59fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgv46"
Nov 02 12:26:04.912362 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.912263    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a81d60e5-2049-4600-9b3f-d3826aee59fb-var-lib-openvswitch\") pod \"ovnkube-node-qgv46\" (UID: \"a81d60e5-2049-4600-9b3f-d3826aee59fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgv46"
Nov 02 12:26:04.912362 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.911989    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/af96531f-d9a0-4acc-ab22-62949e7400aa-etc-selinux\") pod \"aws-ebs-csi-driver-node-kbhhk\" (UID: \"af96531f-d9a0-4acc-ab22-62949e7400aa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kbhhk"
Nov 02 12:26:04.912621 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.912362    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/20cc95cd-9718-49ee-8082-e9ac00fcc147-host\") pod \"node-ca-9t4f5\" (UID: \"20cc95cd-9718-49ee-8082-e9ac00fcc147\") " pod="openshift-image-registry/node-ca-9t4f5"
Nov 02 12:26:04.912775 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.912758    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f7c6ba1b-8808-4b6b-8965-2e4e7a067030-konnectivity-ca\") pod \"konnectivity-agent-dnm8d\" (UID: \"f7c6ba1b-8808-4b6b-8965-2e4e7a067030\") " pod="kube-system/konnectivity-agent-dnm8d"
Nov 02 12:26:04.912845 ip-10-0-128-132 kubenswrapper[2574]: E1102 12:26:04.912833    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 12:26:04.912908 ip-10-0-128-132 kubenswrapper[2574]: E1102 12:26:04.912892    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/09f70dd9-4dbe-46df-ac95-1c351a541a18-metrics-certs podName:09f70dd9-4dbe-46df-ac95-1c351a541a18 nodeName:}" failed. No retries permitted until 2026-11-02 12:26:05.412878637 +0000 UTC m=+3.129821389 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/09f70dd9-4dbe-46df-ac95-1c351a541a18-metrics-certs") pod "network-metrics-daemon-dp7h7" (UID: "09f70dd9-4dbe-46df-ac95-1c351a541a18") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 12:26:04.912963 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.912935    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a81d60e5-2049-4600-9b3f-d3826aee59fb-run-systemd\") pod \"ovnkube-node-qgv46\" (UID: \"a81d60e5-2049-4600-9b3f-d3826aee59fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgv46"
Nov 02 12:26:04.913021 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.913002    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/af2e7d8e-b1df-499c-9a92-9ebe9d7bc7b2-host-slash\") pod \"iptables-alerter-7gc6b\" (UID: \"af2e7d8e-b1df-499c-9a92-9ebe9d7bc7b2\") " pod="openshift-network-operator/iptables-alerter-7gc6b"
Nov 02 12:26:04.913071 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.913020    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a81d60e5-2049-4600-9b3f-d3826aee59fb-run-openvswitch\") pod \"ovnkube-node-qgv46\" (UID: \"a81d60e5-2049-4600-9b3f-d3826aee59fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgv46"
Nov 02 12:26:04.913177 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.913155    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/993b825c-e7c4-445e-b8e8-5e92eb205f3e-host-run-netns\") pod \"multus-z5nxc\" (UID: \"993b825c-e7c4-445e-b8e8-5e92eb205f3e\") " pod="openshift-multus/multus-z5nxc"
Nov 02 12:26:04.913299 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.913207    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a81d60e5-2049-4600-9b3f-d3826aee59fb-etc-openvswitch\") pod \"ovnkube-node-qgv46\" (UID: \"a81d60e5-2049-4600-9b3f-d3826aee59fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgv46"
Nov 02 12:26:04.913299 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.913245    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a81d60e5-2049-4600-9b3f-d3826aee59fb-host-slash\") pod \"ovnkube-node-qgv46\" (UID: \"a81d60e5-2049-4600-9b3f-d3826aee59fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgv46"
Nov 02 12:26:04.913299 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.913253    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a81d60e5-2049-4600-9b3f-d3826aee59fb-run-ovn\") pod \"ovnkube-node-qgv46\" (UID: \"a81d60e5-2049-4600-9b3f-d3826aee59fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgv46"
Nov 02 12:26:04.913431 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.913370    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8a2e896a-489b-43fa-ba40-b2e3c0d56176-var-lib-kubelet\") pod \"tuned-vnv9x\" (UID: \"8a2e896a-489b-43fa-ba40-b2e3c0d56176\") " pod="openshift-cluster-node-tuning-operator/tuned-vnv9x"
Nov 02 12:26:04.913431 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.913413    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a81d60e5-2049-4600-9b3f-d3826aee59fb-systemd-units\") pod \"ovnkube-node-qgv46\" (UID: \"a81d60e5-2049-4600-9b3f-d3826aee59fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgv46"
Nov 02 12:26:04.913521 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.913477    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/83964713-5989-4b35-91e8-3d3993688ccd-dbus\") pod \"global-pull-secret-syncer-6grht\" (UID: \"83964713-5989-4b35-91e8-3d3993688ccd\") " pod="kube-system/global-pull-secret-syncer-6grht"
Nov 02 12:26:04.914012 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.913613    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a81d60e5-2049-4600-9b3f-d3826aee59fb-host-cni-bin\") pod \"ovnkube-node-qgv46\" (UID: \"a81d60e5-2049-4600-9b3f-d3826aee59fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgv46"
Nov 02 12:26:04.914012 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.913935    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8a2e896a-489b-43fa-ba40-b2e3c0d56176-etc-systemd\") pod \"tuned-vnv9x\" (UID: \"8a2e896a-489b-43fa-ba40-b2e3c0d56176\") " pod="openshift-cluster-node-tuning-operator/tuned-vnv9x"
Nov 02 12:26:04.914171 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.913963    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/af96531f-d9a0-4acc-ab22-62949e7400aa-sys-fs\") pod \"aws-ebs-csi-driver-node-kbhhk\" (UID: \"af96531f-d9a0-4acc-ab22-62949e7400aa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kbhhk"
Nov 02 12:26:04.914171 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.914097    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/af2e7d8e-b1df-499c-9a92-9ebe9d7bc7b2-iptables-alerter-script\") pod \"iptables-alerter-7gc6b\" (UID: \"af2e7d8e-b1df-499c-9a92-9ebe9d7bc7b2\") " pod="openshift-network-operator/iptables-alerter-7gc6b"
Nov 02 12:26:04.914418 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.914396    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/993b825c-e7c4-445e-b8e8-5e92eb205f3e-cnibin\") pod \"multus-z5nxc\" (UID: \"993b825c-e7c4-445e-b8e8-5e92eb205f3e\") " pod="openshift-multus/multus-z5nxc"
Nov 02 12:26:04.914559 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.914430    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5507fd15-57b9-422a-af1d-5daf266bd9ca-tmp-dir\") pod \"node-resolver-7trhz\" (UID: \"5507fd15-57b9-422a-af1d-5daf266bd9ca\") " pod="openshift-dns/node-resolver-7trhz"
Nov 02 12:26:04.914953 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.914477    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/993b825c-e7c4-445e-b8e8-5e92eb205f3e-host-var-lib-kubelet\") pod \"multus-z5nxc\" (UID: \"993b825c-e7c4-445e-b8e8-5e92eb205f3e\") " pod="openshift-multus/multus-z5nxc"
Nov 02 12:26:04.914953 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.914493    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/993b825c-e7c4-445e-b8e8-5e92eb205f3e-multus-socket-dir-parent\") pod \"multus-z5nxc\" (UID: \"993b825c-e7c4-445e-b8e8-5e92eb205f3e\") " pod="openshift-multus/multus-z5nxc"
Nov 02 12:26:04.914953 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.914510    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/993b825c-e7c4-445e-b8e8-5e92eb205f3e-cni-binary-copy\") pod \"multus-z5nxc\" (UID: \"993b825c-e7c4-445e-b8e8-5e92eb205f3e\") " pod="openshift-multus/multus-z5nxc"
Nov 02 12:26:04.915136 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.914592    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8a2e896a-489b-43fa-ba40-b2e3c0d56176-host\") pod \"tuned-vnv9x\" (UID: \"8a2e896a-489b-43fa-ba40-b2e3c0d56176\") " pod="openshift-cluster-node-tuning-operator/tuned-vnv9x"
Nov 02 12:26:04.915136 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.914589    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/af96531f-d9a0-4acc-ab22-62949e7400aa-kubelet-dir\") pod \"aws-ebs-csi-driver-node-kbhhk\" (UID: \"af96531f-d9a0-4acc-ab22-62949e7400aa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kbhhk"
Nov 02 12:26:04.915136 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.914617    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a81d60e5-2049-4600-9b3f-d3826aee59fb-ovnkube-script-lib\") pod \"ovnkube-node-qgv46\" (UID: \"a81d60e5-2049-4600-9b3f-d3826aee59fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgv46"
Nov 02 12:26:04.915136 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.914681    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/feba123c-eeb3-412f-911e-a66da1981358-system-cni-dir\") pod \"multus-additional-cni-plugins-v7bcl\" (UID: \"feba123c-eeb3-412f-911e-a66da1981358\") " pod="openshift-multus/multus-additional-cni-plugins-v7bcl"
Nov 02 12:26:04.915136 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.914703    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/af96531f-d9a0-4acc-ab22-62949e7400aa-socket-dir\") pod \"aws-ebs-csi-driver-node-kbhhk\" (UID: \"af96531f-d9a0-4acc-ab22-62949e7400aa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kbhhk"
Nov 02 12:26:04.915136 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.914716    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/af96531f-d9a0-4acc-ab22-62949e7400aa-device-dir\") pod \"aws-ebs-csi-driver-node-kbhhk\" (UID: \"af96531f-d9a0-4acc-ab22-62949e7400aa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kbhhk"
Nov 02 12:26:04.915136 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.914848    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/feba123c-eeb3-412f-911e-a66da1981358-tuning-conf-dir\") pod \"multus-additional-cni-plugins-v7bcl\" (UID: \"feba123c-eeb3-412f-911e-a66da1981358\") " pod="openshift-multus/multus-additional-cni-plugins-v7bcl"
Nov 02 12:26:04.915136 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.915048    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a81d60e5-2049-4600-9b3f-d3826aee59fb-host-cni-netd\") pod \"ovnkube-node-qgv46\" (UID: \"a81d60e5-2049-4600-9b3f-d3826aee59fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgv46"
Nov 02 12:26:04.915136 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.915093    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a81d60e5-2049-4600-9b3f-d3826aee59fb-log-socket\") pod \"ovnkube-node-qgv46\" (UID: \"a81d60e5-2049-4600-9b3f-d3826aee59fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgv46"
Nov 02 12:26:04.915136 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.915097    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a81d60e5-2049-4600-9b3f-d3826aee59fb-host-run-ovn-kubernetes\") pod \"ovnkube-node-qgv46\" (UID: \"a81d60e5-2049-4600-9b3f-d3826aee59fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgv46"
Nov 02 12:26:04.915617 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.915161    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/993b825c-e7c4-445e-b8e8-5e92eb205f3e-multus-daemon-config\") pod \"multus-z5nxc\" (UID: \"993b825c-e7c4-445e-b8e8-5e92eb205f3e\") " pod="openshift-multus/multus-z5nxc"
Nov 02 12:26:04.915617 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.915181    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5507fd15-57b9-422a-af1d-5daf266bd9ca-hosts-file\") pod \"node-resolver-7trhz\" (UID: \"5507fd15-57b9-422a-af1d-5daf266bd9ca\") " pod="openshift-dns/node-resolver-7trhz"
Nov 02 12:26:04.915617 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.915238    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/993b825c-e7c4-445e-b8e8-5e92eb205f3e-host-run-multus-certs\") pod \"multus-z5nxc\" (UID: \"993b825c-e7c4-445e-b8e8-5e92eb205f3e\") " pod="openshift-multus/multus-z5nxc"
Nov 02 12:26:04.915617 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.915279    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8a2e896a-489b-43fa-ba40-b2e3c0d56176-etc-sysctl-d\") pod \"tuned-vnv9x\" (UID: \"8a2e896a-489b-43fa-ba40-b2e3c0d56176\") " pod="openshift-cluster-node-tuning-operator/tuned-vnv9x"
Nov 02 12:26:04.915617 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.915303    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/993b825c-e7c4-445e-b8e8-5e92eb205f3e-host-var-lib-cni-bin\") pod \"multus-z5nxc\" (UID: \"993b825c-e7c4-445e-b8e8-5e92eb205f3e\") " pod="openshift-multus/multus-z5nxc"
Nov 02 12:26:04.915617 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.915333    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8a2e896a-489b-43fa-ba40-b2e3c0d56176-etc-modprobe-d\") pod \"tuned-vnv9x\" (UID: \"8a2e896a-489b-43fa-ba40-b2e3c0d56176\") " pod="openshift-cluster-node-tuning-operator/tuned-vnv9x"
Nov 02 12:26:04.915929 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.915656    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/feba123c-eeb3-412f-911e-a66da1981358-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-v7bcl\" (UID: \"feba123c-eeb3-412f-911e-a66da1981358\") " pod="openshift-multus/multus-additional-cni-plugins-v7bcl"
Nov 02 12:26:04.915929 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.915657    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/20cc95cd-9718-49ee-8082-e9ac00fcc147-serviceca\") pod \"node-ca-9t4f5\" (UID: \"20cc95cd-9718-49ee-8082-e9ac00fcc147\") " pod="openshift-image-registry/node-ca-9t4f5"
Nov 02 12:26:04.915929 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.915704    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8a2e896a-489b-43fa-ba40-b2e3c0d56176-etc-sysconfig\") pod \"tuned-vnv9x\" (UID: \"8a2e896a-489b-43fa-ba40-b2e3c0d56176\") " pod="openshift-cluster-node-tuning-operator/tuned-vnv9x"
Nov 02 12:26:04.915929 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.915796    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/feba123c-eeb3-412f-911e-a66da1981358-cni-binary-copy\") pod \"multus-additional-cni-plugins-v7bcl\" (UID: \"feba123c-eeb3-412f-911e-a66da1981358\") " pod="openshift-multus/multus-additional-cni-plugins-v7bcl"
Nov 02 12:26:04.916120 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.916024    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a81d60e5-2049-4600-9b3f-d3826aee59fb-ovnkube-config\") pod \"ovnkube-node-qgv46\" (UID: \"a81d60e5-2049-4600-9b3f-d3826aee59fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgv46"
Nov 02 12:26:04.916173 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.916114    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8a2e896a-489b-43fa-ba40-b2e3c0d56176-tmp\") pod \"tuned-vnv9x\" (UID: \"8a2e896a-489b-43fa-ba40-b2e3c0d56176\") " pod="openshift-cluster-node-tuning-operator/tuned-vnv9x"
Nov 02 12:26:04.916305 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.916286    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8a2e896a-489b-43fa-ba40-b2e3c0d56176-etc-tuned\") pod \"tuned-vnv9x\" (UID: \"8a2e896a-489b-43fa-ba40-b2e3c0d56176\") " pod="openshift-cluster-node-tuning-operator/tuned-vnv9x"
Nov 02 12:26:04.918280 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.917381    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a81d60e5-2049-4600-9b3f-d3826aee59fb-ovn-node-metrics-cert\") pod \"ovnkube-node-qgv46\" (UID: \"a81d60e5-2049-4600-9b3f-d3826aee59fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgv46"
Nov 02 12:26:04.918280 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.918165    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f7c6ba1b-8808-4b6b-8965-2e4e7a067030-agent-certs\") pod \"konnectivity-agent-dnm8d\" (UID: \"f7c6ba1b-8808-4b6b-8965-2e4e7a067030\") " pod="kube-system/konnectivity-agent-dnm8d"
Nov 02 12:26:04.934392 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.934360    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2j4q8\" (UniqueName: \"kubernetes.io/projected/a81d60e5-2049-4600-9b3f-d3826aee59fb-kube-api-access-2j4q8\") pod \"ovnkube-node-qgv46\" (UID: \"a81d60e5-2049-4600-9b3f-d3826aee59fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgv46"
Nov 02 12:26:04.934909 ip-10-0-128-132 kubenswrapper[2574]: E1102 12:26:04.934888    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 02 12:26:04.935030 ip-10-0-128-132 kubenswrapper[2574]: E1102 12:26:04.934918    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 02 12:26:04.935030 ip-10-0-128-132 kubenswrapper[2574]: E1102 12:26:04.934933    2574 projected.go:196] Error preparing data for projected volume kube-api-access-cws77 for pod openshift-network-diagnostics/network-check-target-wx6zw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 12:26:04.935030 ip-10-0-128-132 kubenswrapper[2574]: E1102 12:26:04.934990    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/089d7bd4-d019-471c-960f-51346831fe45-kube-api-access-cws77 podName:089d7bd4-d019-471c-960f-51346831fe45 nodeName:}" failed. No retries permitted until 2026-11-02 12:26:05.434974093 +0000 UTC m=+3.151916854 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-cws77" (UniqueName: "kubernetes.io/projected/089d7bd4-d019-471c-960f-51346831fe45-kube-api-access-cws77") pod "network-check-target-wx6zw" (UID: "089d7bd4-d019-471c-960f-51346831fe45") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 12:26:04.935847 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.935823    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9zns\" (UniqueName: \"kubernetes.io/projected/8a2e896a-489b-43fa-ba40-b2e3c0d56176-kube-api-access-q9zns\") pod \"tuned-vnv9x\" (UID: \"8a2e896a-489b-43fa-ba40-b2e3c0d56176\") " pod="openshift-cluster-node-tuning-operator/tuned-vnv9x"
Nov 02 12:26:04.936178 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.936094    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4629q\" (UniqueName: \"kubernetes.io/projected/af2e7d8e-b1df-499c-9a92-9ebe9d7bc7b2-kube-api-access-4629q\") pod \"iptables-alerter-7gc6b\" (UID: \"af2e7d8e-b1df-499c-9a92-9ebe9d7bc7b2\") " pod="openshift-network-operator/iptables-alerter-7gc6b"
Nov 02 12:26:04.938149 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.938119    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wphgd\" (UniqueName: \"kubernetes.io/projected/feba123c-eeb3-412f-911e-a66da1981358-kube-api-access-wphgd\") pod \"multus-additional-cni-plugins-v7bcl\" (UID: \"feba123c-eeb3-412f-911e-a66da1981358\") " pod="openshift-multus/multus-additional-cni-plugins-v7bcl"
Nov 02 12:26:04.940239 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.940217    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nn9gd\" (UniqueName: \"kubernetes.io/projected/993b825c-e7c4-445e-b8e8-5e92eb205f3e-kube-api-access-nn9gd\") pod \"multus-z5nxc\" (UID: \"993b825c-e7c4-445e-b8e8-5e92eb205f3e\") " pod="openshift-multus/multus-z5nxc"
Nov 02 12:26:04.940335 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.940310    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkl5z\" (UniqueName: \"kubernetes.io/projected/5507fd15-57b9-422a-af1d-5daf266bd9ca-kube-api-access-qkl5z\") pod \"node-resolver-7trhz\" (UID: \"5507fd15-57b9-422a-af1d-5daf266bd9ca\") " pod="openshift-dns/node-resolver-7trhz"
Nov 02 12:26:04.940384 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.940338    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztjcq\" (UniqueName: \"kubernetes.io/projected/af96531f-d9a0-4acc-ab22-62949e7400aa-kube-api-access-ztjcq\") pod \"aws-ebs-csi-driver-node-kbhhk\" (UID: \"af96531f-d9a0-4acc-ab22-62949e7400aa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kbhhk"
Nov 02 12:26:04.940384 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.940355    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-42zsx\" (UniqueName: \"kubernetes.io/projected/09f70dd9-4dbe-46df-ac95-1c351a541a18-kube-api-access-42zsx\") pod \"network-metrics-daemon-dp7h7\" (UID: \"09f70dd9-4dbe-46df-ac95-1c351a541a18\") " pod="openshift-multus/network-metrics-daemon-dp7h7"
Nov 02 12:26:04.941752 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:04.941729    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9k4x\" (UniqueName: \"kubernetes.io/projected/20cc95cd-9718-49ee-8082-e9ac00fcc147-kube-api-access-c9k4x\") pod \"node-ca-9t4f5\" (UID: \"20cc95cd-9718-49ee-8082-e9ac00fcc147\") " pod="openshift-image-registry/node-ca-9t4f5"
Nov 02 12:26:05.016578 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:05.016367    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kbhhk"
Nov 02 12:26:05.019535 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:05.019502    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-vnv9x"
Nov 02 12:26:05.028675 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:05.028634    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-z5nxc"
Nov 02 12:26:05.029119 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:05.029092    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf96531f_d9a0_4acc_ab22_62949e7400aa.slice/crio-edb20a3e6fb60b89fc7006db3c3f05d04cd7418759a6fda5a329897763afc65c WatchSource:0}: Error finding container edb20a3e6fb60b89fc7006db3c3f05d04cd7418759a6fda5a329897763afc65c: Status 404 returned error can't find the container with id edb20a3e6fb60b89fc7006db3c3f05d04cd7418759a6fda5a329897763afc65c
Nov 02 12:26:05.030054 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:05.030014    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a2e896a_489b_43fa_ba40_b2e3c0d56176.slice/crio-284a70308f24b9f2203b19b2d22b7a5e1865ce2ebf93ca54509cb83e2c2a636e WatchSource:0}: Error finding container 284a70308f24b9f2203b19b2d22b7a5e1865ce2ebf93ca54509cb83e2c2a636e: Status 404 returned error can't find the container with id 284a70308f24b9f2203b19b2d22b7a5e1865ce2ebf93ca54509cb83e2c2a636e
Nov 02 12:26:05.035890 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:05.035870    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-7trhz"
Nov 02 12:26:05.039267 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:05.039242    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod993b825c_e7c4_445e_b8e8_5e92eb205f3e.slice/crio-62ef2cf548134a07d3fd774b87b4967be8dc5cedd69e89d5a6d90fa52d758b32 WatchSource:0}: Error finding container 62ef2cf548134a07d3fd774b87b4967be8dc5cedd69e89d5a6d90fa52d758b32: Status 404 returned error can't find the container with id 62ef2cf548134a07d3fd774b87b4967be8dc5cedd69e89d5a6d90fa52d758b32
Nov 02 12:26:05.047815 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:05.047655    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-7gc6b"
Nov 02 12:26:05.056462 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:05.056442    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-9t4f5"
Nov 02 12:26:05.058017 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:05.057993    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf2e7d8e_b1df_499c_9a92_9ebe9d7bc7b2.slice/crio-d367350c201505849466be8254a4de268d7df96d9b246672c6d288afe8b8bedb WatchSource:0}: Error finding container d367350c201505849466be8254a4de268d7df96d9b246672c6d288afe8b8bedb: Status 404 returned error can't find the container with id d367350c201505849466be8254a4de268d7df96d9b246672c6d288afe8b8bedb
Nov 02 12:26:05.067814 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:05.067787    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20cc95cd_9718_49ee_8082_e9ac00fcc147.slice/crio-56c29b248f147512b03791da0b50b810e2c43849264148c426a9a6d4abd4e9cb WatchSource:0}: Error finding container 56c29b248f147512b03791da0b50b810e2c43849264148c426a9a6d4abd4e9cb: Status 404 returned error can't find the container with id 56c29b248f147512b03791da0b50b810e2c43849264148c426a9a6d4abd4e9cb
Nov 02 12:26:05.075765 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:05.075743    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-dnm8d"
Nov 02 12:26:05.081880 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:05.081854    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-qgv46"
Nov 02 12:26:05.087439 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:05.087415    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7c6ba1b_8808_4b6b_8965_2e4e7a067030.slice/crio-21395fe98db6e90d01f7b1c1b735c298221b387616a9f95e4f33a972a04ea8d5 WatchSource:0}: Error finding container 21395fe98db6e90d01f7b1c1b735c298221b387616a9f95e4f33a972a04ea8d5: Status 404 returned error can't find the container with id 21395fe98db6e90d01f7b1c1b735c298221b387616a9f95e4f33a972a04ea8d5
Nov 02 12:26:05.093175 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:05.093153    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda81d60e5_2049_4600_9b3f_d3826aee59fb.slice/crio-d3f9609da159aded0018387be5cfe1f3a8fce539c124f377e1cd39b8d0144242 WatchSource:0}: Error finding container d3f9609da159aded0018387be5cfe1f3a8fce539c124f377e1cd39b8d0144242: Status 404 returned error can't find the container with id d3f9609da159aded0018387be5cfe1f3a8fce539c124f377e1cd39b8d0144242
Nov 02 12:26:05.096851 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:05.096827    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-v7bcl"
Nov 02 12:26:05.107488 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:05.107458    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfeba123c_eeb3_412f_911e_a66da1981358.slice/crio-9519b1bdf6e9646240a4c501ff13fa301b69a8b9a474cc5f445adff3e6ddd126 WatchSource:0}: Error finding container 9519b1bdf6e9646240a4c501ff13fa301b69a8b9a474cc5f445adff3e6ddd126: Status 404 returned error can't find the container with id 9519b1bdf6e9646240a4c501ff13fa301b69a8b9a474cc5f445adff3e6ddd126
Nov 02 12:26:05.412662 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:05.412610    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/83964713-5989-4b35-91e8-3d3993688ccd-original-pull-secret\") pod \"global-pull-secret-syncer-6grht\" (UID: \"83964713-5989-4b35-91e8-3d3993688ccd\") " pod="kube-system/global-pull-secret-syncer-6grht"
Nov 02 12:26:05.412815 ip-10-0-128-132 kubenswrapper[2574]: E1102 12:26:05.412769    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 02 12:26:05.412873 ip-10-0-128-132 kubenswrapper[2574]: E1102 12:26:05.412851    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/83964713-5989-4b35-91e8-3d3993688ccd-original-pull-secret podName:83964713-5989-4b35-91e8-3d3993688ccd nodeName:}" failed. No retries permitted until 2026-11-02 12:26:06.412830272 +0000 UTC m=+4.129773031 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/83964713-5989-4b35-91e8-3d3993688ccd-original-pull-secret") pod "global-pull-secret-syncer-6grht" (UID: "83964713-5989-4b35-91e8-3d3993688ccd") : object "kube-system"/"original-pull-secret" not registered
Nov 02 12:26:05.513544 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:05.513463    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/09f70dd9-4dbe-46df-ac95-1c351a541a18-metrics-certs\") pod \"network-metrics-daemon-dp7h7\" (UID: \"09f70dd9-4dbe-46df-ac95-1c351a541a18\") " pod="openshift-multus/network-metrics-daemon-dp7h7"
Nov 02 12:26:05.513544 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:05.513509    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cws77\" (UniqueName: \"kubernetes.io/projected/089d7bd4-d019-471c-960f-51346831fe45-kube-api-access-cws77\") pod \"network-check-target-wx6zw\" (UID: \"089d7bd4-d019-471c-960f-51346831fe45\") " pod="openshift-network-diagnostics/network-check-target-wx6zw"
Nov 02 12:26:05.513788 ip-10-0-128-132 kubenswrapper[2574]: E1102 12:26:05.513707    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 02 12:26:05.513788 ip-10-0-128-132 kubenswrapper[2574]: E1102 12:26:05.513730    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 02 12:26:05.513788 ip-10-0-128-132 kubenswrapper[2574]: E1102 12:26:05.513743    2574 projected.go:196] Error preparing data for projected volume kube-api-access-cws77 for pod openshift-network-diagnostics/network-check-target-wx6zw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 12:26:05.513933 ip-10-0-128-132 kubenswrapper[2574]: E1102 12:26:05.513801    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/089d7bd4-d019-471c-960f-51346831fe45-kube-api-access-cws77 podName:089d7bd4-d019-471c-960f-51346831fe45 nodeName:}" failed. No retries permitted until 2026-11-02 12:26:06.513784017 +0000 UTC m=+4.230726767 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-cws77" (UniqueName: "kubernetes.io/projected/089d7bd4-d019-471c-960f-51346831fe45-kube-api-access-cws77") pod "network-check-target-wx6zw" (UID: "089d7bd4-d019-471c-960f-51346831fe45") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 12:26:05.515348 ip-10-0-128-132 kubenswrapper[2574]: E1102 12:26:05.514129    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 12:26:05.515348 ip-10-0-128-132 kubenswrapper[2574]: E1102 12:26:05.514196    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/09f70dd9-4dbe-46df-ac95-1c351a541a18-metrics-certs podName:09f70dd9-4dbe-46df-ac95-1c351a541a18 nodeName:}" failed. No retries permitted until 2026-11-02 12:26:06.514177812 +0000 UTC m=+4.231120573 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/09f70dd9-4dbe-46df-ac95-1c351a541a18-metrics-certs") pod "network-metrics-daemon-dp7h7" (UID: "09f70dd9-4dbe-46df-ac95-1c351a541a18") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 12:26:05.773700 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:05.773387    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wx6zw"
Nov 02 12:26:05.774400 ip-10-0-128-132 kubenswrapper[2574]: E1102 12:26:05.773774    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wx6zw" podUID="089d7bd4-d019-471c-960f-51346831fe45"
Nov 02 12:26:05.774400 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:05.774164    2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 12:21:04 +0000 UTC" deadline="2028-03-29 01:14:01.206352741 +0000 UTC"
Nov 02 12:26:05.774400 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:05.774205    2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12300h47m55.432151553s"
Nov 02 12:26:05.792519 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:05.792482    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-7gc6b" event={"ID":"af2e7d8e-b1df-499c-9a92-9ebe9d7bc7b2","Type":"ContainerStarted","Data":"d367350c201505849466be8254a4de268d7df96d9b246672c6d288afe8b8bedb"}
Nov 02 12:26:05.797801 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:05.797728    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-7trhz" event={"ID":"5507fd15-57b9-422a-af1d-5daf266bd9ca","Type":"ContainerStarted","Data":"650a2fafdbcfc0deaeea5cf5d1f9421dda4993a5b5fb3cab9d39f65189d15577"}
Nov 02 12:26:05.800226 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:05.800157    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-z5nxc" event={"ID":"993b825c-e7c4-445e-b8e8-5e92eb205f3e","Type":"ContainerStarted","Data":"62ef2cf548134a07d3fd774b87b4967be8dc5cedd69e89d5a6d90fa52d758b32"}
Nov 02 12:26:05.807461 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:05.806545    2574 generic.go:356] "Generic (PLEG): container finished" podID="6e3a47dd3a64bdc7ae97c7ee36b68697" containerID="02cdf71f74b8f9e80b67e03511cf60785517f161c05b664e0190b920d570cd06" exitCode=0
Nov 02 12:26:05.807461 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:05.806623    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-132.ec2.internal" event={"ID":"6e3a47dd3a64bdc7ae97c7ee36b68697","Type":"ContainerDied","Data":"02cdf71f74b8f9e80b67e03511cf60785517f161c05b664e0190b920d570cd06"}
Nov 02 12:26:05.821656 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:05.817517    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v7bcl" event={"ID":"feba123c-eeb3-412f-911e-a66da1981358","Type":"ContainerStarted","Data":"9519b1bdf6e9646240a4c501ff13fa301b69a8b9a474cc5f445adff3e6ddd126"}
Nov 02 12:26:05.830179 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:05.830151    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-dnm8d" event={"ID":"f7c6ba1b-8808-4b6b-8965-2e4e7a067030","Type":"ContainerStarted","Data":"21395fe98db6e90d01f7b1c1b735c298221b387616a9f95e4f33a972a04ea8d5"}
Nov 02 12:26:05.856965 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:05.856878    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-9t4f5" event={"ID":"20cc95cd-9718-49ee-8082-e9ac00fcc147","Type":"ContainerStarted","Data":"56c29b248f147512b03791da0b50b810e2c43849264148c426a9a6d4abd4e9cb"}
Nov 02 12:26:05.865533 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:05.865501    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-vnv9x" event={"ID":"8a2e896a-489b-43fa-ba40-b2e3c0d56176","Type":"ContainerStarted","Data":"284a70308f24b9f2203b19b2d22b7a5e1865ce2ebf93ca54509cb83e2c2a636e"}
Nov 02 12:26:05.883618 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:05.883578    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kbhhk" event={"ID":"af96531f-d9a0-4acc-ab22-62949e7400aa","Type":"ContainerStarted","Data":"edb20a3e6fb60b89fc7006db3c3f05d04cd7418759a6fda5a329897763afc65c"}
Nov 02 12:26:05.888366 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:05.888337    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-132.ec2.internal" event={"ID":"9f29c1002c42911f93b255b0f3495fd0","Type":"ContainerStarted","Data":"7d401c8eda83f0557bccbe07a60d0d9ec9a79aa79285b1ab14d186ee9da4e4e6"}
Nov 02 12:26:05.899214 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:05.899184    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qgv46" event={"ID":"a81d60e5-2049-4600-9b3f-d3826aee59fb","Type":"ContainerStarted","Data":"d3f9609da159aded0018387be5cfe1f3a8fce539c124f377e1cd39b8d0144242"}
Nov 02 12:26:06.421292 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:06.421251    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/83964713-5989-4b35-91e8-3d3993688ccd-original-pull-secret\") pod \"global-pull-secret-syncer-6grht\" (UID: \"83964713-5989-4b35-91e8-3d3993688ccd\") " pod="kube-system/global-pull-secret-syncer-6grht"
Nov 02 12:26:06.421455 ip-10-0-128-132 kubenswrapper[2574]: E1102 12:26:06.421398    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 02 12:26:06.421530 ip-10-0-128-132 kubenswrapper[2574]: E1102 12:26:06.421457    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/83964713-5989-4b35-91e8-3d3993688ccd-original-pull-secret podName:83964713-5989-4b35-91e8-3d3993688ccd nodeName:}" failed. No retries permitted until 2026-11-02 12:26:08.421439765 +0000 UTC m=+6.138382528 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/83964713-5989-4b35-91e8-3d3993688ccd-original-pull-secret") pod "global-pull-secret-syncer-6grht" (UID: "83964713-5989-4b35-91e8-3d3993688ccd") : object "kube-system"/"original-pull-secret" not registered
Nov 02 12:26:06.522555 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:06.521626    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/09f70dd9-4dbe-46df-ac95-1c351a541a18-metrics-certs\") pod \"network-metrics-daemon-dp7h7\" (UID: \"09f70dd9-4dbe-46df-ac95-1c351a541a18\") " pod="openshift-multus/network-metrics-daemon-dp7h7"
Nov 02 12:26:06.522555 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:06.521697    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cws77\" (UniqueName: \"kubernetes.io/projected/089d7bd4-d019-471c-960f-51346831fe45-kube-api-access-cws77\") pod \"network-check-target-wx6zw\" (UID: \"089d7bd4-d019-471c-960f-51346831fe45\") " pod="openshift-network-diagnostics/network-check-target-wx6zw"
Nov 02 12:26:06.522555 ip-10-0-128-132 kubenswrapper[2574]: E1102 12:26:06.521926    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 02 12:26:06.522555 ip-10-0-128-132 kubenswrapper[2574]: E1102 12:26:06.521948    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 02 12:26:06.522555 ip-10-0-128-132 kubenswrapper[2574]: E1102 12:26:06.521959    2574 projected.go:196] Error preparing data for projected volume kube-api-access-cws77 for pod openshift-network-diagnostics/network-check-target-wx6zw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 12:26:06.522555 ip-10-0-128-132 kubenswrapper[2574]: E1102 12:26:06.522021    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/089d7bd4-d019-471c-960f-51346831fe45-kube-api-access-cws77 podName:089d7bd4-d019-471c-960f-51346831fe45 nodeName:}" failed. No retries permitted until 2026-11-02 12:26:08.522002126 +0000 UTC m=+6.238944892 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-cws77" (UniqueName: "kubernetes.io/projected/089d7bd4-d019-471c-960f-51346831fe45-kube-api-access-cws77") pod "network-check-target-wx6zw" (UID: "089d7bd4-d019-471c-960f-51346831fe45") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 12:26:06.522555 ip-10-0-128-132 kubenswrapper[2574]: E1102 12:26:06.522462    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 12:26:06.522555 ip-10-0-128-132 kubenswrapper[2574]: E1102 12:26:06.522522    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/09f70dd9-4dbe-46df-ac95-1c351a541a18-metrics-certs podName:09f70dd9-4dbe-46df-ac95-1c351a541a18 nodeName:}" failed. No retries permitted until 2026-11-02 12:26:08.522506645 +0000 UTC m=+6.239449393 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/09f70dd9-4dbe-46df-ac95-1c351a541a18-metrics-certs") pod "network-metrics-daemon-dp7h7" (UID: "09f70dd9-4dbe-46df-ac95-1c351a541a18") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 12:26:06.773870 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:06.773787    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dp7h7"
Nov 02 12:26:06.773870 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:06.773837    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6grht"
Nov 02 12:26:06.774393 ip-10-0-128-132 kubenswrapper[2574]: E1102 12:26:06.773944    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dp7h7" podUID="09f70dd9-4dbe-46df-ac95-1c351a541a18"
Nov 02 12:26:06.774393 ip-10-0-128-132 kubenswrapper[2574]: E1102 12:26:06.774086    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6grht" podUID="83964713-5989-4b35-91e8-3d3993688ccd"
Nov 02 12:26:06.774393 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:06.774334    2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 12:21:04 +0000 UTC" deadline="2028-08-13 19:44:41.60753477 +0000 UTC"
Nov 02 12:26:06.774393 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:06.774370    2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15607h18m34.833169095s"
Nov 02 12:26:06.917499 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:06.917434    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-132.ec2.internal" event={"ID":"6e3a47dd3a64bdc7ae97c7ee36b68697","Type":"ContainerStarted","Data":"448d328d2ca223f172b36ed842c2880b47dc762e3748c61f4b9dfa3544509dac"}
Nov 02 12:26:06.960127 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:06.959711    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-132.ec2.internal" podStartSLOduration=2.959691609 podStartE2EDuration="2.959691609s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 12:26:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:26:05.9286304 +0000 UTC m=+3.645573169" watchObservedRunningTime="2026-11-02 12:26:06.959691609 +0000 UTC m=+4.676634380"
Nov 02 12:26:07.773268 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:07.773055    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wx6zw"
Nov 02 12:26:07.773268 ip-10-0-128-132 kubenswrapper[2574]: E1102 12:26:07.773188    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wx6zw" podUID="089d7bd4-d019-471c-960f-51346831fe45"
Nov 02 12:26:07.928686 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:07.928650    2574 generic.go:356] "Generic (PLEG): container finished" podID="feba123c-eeb3-412f-911e-a66da1981358" containerID="3f3d71f0353934ab8912213a609d87e77f16aec3544799bdaa34c10d0c88ef60" exitCode=0
Nov 02 12:26:07.929786 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:07.929270    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v7bcl" event={"ID":"feba123c-eeb3-412f-911e-a66da1981358","Type":"ContainerDied","Data":"3f3d71f0353934ab8912213a609d87e77f16aec3544799bdaa34c10d0c88ef60"}
Nov 02 12:26:08.002559 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:08.002449    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-132.ec2.internal" podStartSLOduration=4.002430567 podStartE2EDuration="4.002430567s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 12:26:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:26:06.959889868 +0000 UTC m=+4.676832640" watchObservedRunningTime="2026-11-02 12:26:08.002430567 +0000 UTC m=+5.719373337"
Nov 02 12:26:08.440669 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:08.440014    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/83964713-5989-4b35-91e8-3d3993688ccd-original-pull-secret\") pod \"global-pull-secret-syncer-6grht\" (UID: \"83964713-5989-4b35-91e8-3d3993688ccd\") " pod="kube-system/global-pull-secret-syncer-6grht"
Nov 02 12:26:08.440669 ip-10-0-128-132 kubenswrapper[2574]: E1102 12:26:08.440158    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 02 12:26:08.440669 ip-10-0-128-132 kubenswrapper[2574]: E1102 12:26:08.440223    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/83964713-5989-4b35-91e8-3d3993688ccd-original-pull-secret podName:83964713-5989-4b35-91e8-3d3993688ccd nodeName:}" failed. No retries permitted until 2026-11-02 12:26:12.440206446 +0000 UTC m=+10.157149198 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/83964713-5989-4b35-91e8-3d3993688ccd-original-pull-secret") pod "global-pull-secret-syncer-6grht" (UID: "83964713-5989-4b35-91e8-3d3993688ccd") : object "kube-system"/"original-pull-secret" not registered
Nov 02 12:26:08.542078 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:08.541217    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/09f70dd9-4dbe-46df-ac95-1c351a541a18-metrics-certs\") pod \"network-metrics-daemon-dp7h7\" (UID: \"09f70dd9-4dbe-46df-ac95-1c351a541a18\") " pod="openshift-multus/network-metrics-daemon-dp7h7"
Nov 02 12:26:08.542078 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:08.541269    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cws77\" (UniqueName: \"kubernetes.io/projected/089d7bd4-d019-471c-960f-51346831fe45-kube-api-access-cws77\") pod \"network-check-target-wx6zw\" (UID: \"089d7bd4-d019-471c-960f-51346831fe45\") " pod="openshift-network-diagnostics/network-check-target-wx6zw"
Nov 02 12:26:08.542078 ip-10-0-128-132 kubenswrapper[2574]: E1102 12:26:08.541434    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 02 12:26:08.542078 ip-10-0-128-132 kubenswrapper[2574]: E1102 12:26:08.541456    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 02 12:26:08.542078 ip-10-0-128-132 kubenswrapper[2574]: E1102 12:26:08.541471    2574 projected.go:196] Error preparing data for projected volume kube-api-access-cws77 for pod openshift-network-diagnostics/network-check-target-wx6zw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 12:26:08.542078 ip-10-0-128-132 kubenswrapper[2574]: E1102 12:26:08.541532    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/089d7bd4-d019-471c-960f-51346831fe45-kube-api-access-cws77 podName:089d7bd4-d019-471c-960f-51346831fe45 nodeName:}" failed. No retries permitted until 2026-11-02 12:26:12.54151181 +0000 UTC m=+10.258454568 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-cws77" (UniqueName: "kubernetes.io/projected/089d7bd4-d019-471c-960f-51346831fe45-kube-api-access-cws77") pod "network-check-target-wx6zw" (UID: "089d7bd4-d019-471c-960f-51346831fe45") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 12:26:08.542078 ip-10-0-128-132 kubenswrapper[2574]: E1102 12:26:08.541968    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 12:26:08.542078 ip-10-0-128-132 kubenswrapper[2574]: E1102 12:26:08.542018    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/09f70dd9-4dbe-46df-ac95-1c351a541a18-metrics-certs podName:09f70dd9-4dbe-46df-ac95-1c351a541a18 nodeName:}" failed. No retries permitted until 2026-11-02 12:26:12.542000244 +0000 UTC m=+10.258942996 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/09f70dd9-4dbe-46df-ac95-1c351a541a18-metrics-certs") pod "network-metrics-daemon-dp7h7" (UID: "09f70dd9-4dbe-46df-ac95-1c351a541a18") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 12:26:08.774336 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:08.774258    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dp7h7"
Nov 02 12:26:08.774497 ip-10-0-128-132 kubenswrapper[2574]: E1102 12:26:08.774408    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dp7h7" podUID="09f70dd9-4dbe-46df-ac95-1c351a541a18"
Nov 02 12:26:08.774901 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:08.774739    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6grht"
Nov 02 12:26:08.774901 ip-10-0-128-132 kubenswrapper[2574]: E1102 12:26:08.774860    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6grht" podUID="83964713-5989-4b35-91e8-3d3993688ccd"
Nov 02 12:26:09.773452 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:09.772931    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wx6zw"
Nov 02 12:26:09.773452 ip-10-0-128-132 kubenswrapper[2574]: E1102 12:26:09.773078    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wx6zw" podUID="089d7bd4-d019-471c-960f-51346831fe45"
Nov 02 12:26:09.949475 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:09.949436    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-dnm8d" event={"ID":"f7c6ba1b-8808-4b6b-8965-2e4e7a067030","Type":"ContainerStarted","Data":"a6c150170ada7f9b271f63e07e8d3c6ad7f8a0db4d2b03d831956c6c4b67862b"}
Nov 02 12:26:10.773983 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:10.773948    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dp7h7"
Nov 02 12:26:10.773983 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:10.773974    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6grht"
Nov 02 12:26:10.774440 ip-10-0-128-132 kubenswrapper[2574]: E1102 12:26:10.774092    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6grht" podUID="83964713-5989-4b35-91e8-3d3993688ccd"
Nov 02 12:26:10.774440 ip-10-0-128-132 kubenswrapper[2574]: E1102 12:26:10.774188    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dp7h7" podUID="09f70dd9-4dbe-46df-ac95-1c351a541a18"
Nov 02 12:26:10.956061 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:10.956023    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-9t4f5" event={"ID":"20cc95cd-9718-49ee-8082-e9ac00fcc147","Type":"ContainerStarted","Data":"aa07bf1bd30a0cfc2c41603bf687917c87bf8876d166f9387fb5c78c19530594"}
Nov 02 12:26:10.959745 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:10.959678    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kbhhk" event={"ID":"af96531f-d9a0-4acc-ab22-62949e7400aa","Type":"ContainerStarted","Data":"98cba43d445a713f54c2f2efc2195d4d9ba57b05ddd7d006975a58fb46d8501d"}
Nov 02 12:26:10.998275 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:10.997580    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-dnm8d" podStartSLOduration=4.1543568539999995 podStartE2EDuration="7.997552096s" totalImagesPullingTime="3.843195242s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:26:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:26:09.991570317 +0000 UTC m=+7.708513086" watchObservedRunningTime="2026-11-02 12:26:10.997552096 +0000 UTC m=+8.714494868"
Nov 02 12:26:11.773161 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:11.773124    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wx6zw"
Nov 02 12:26:11.773316 ip-10-0-128-132 kubenswrapper[2574]: E1102 12:26:11.773269    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wx6zw" podUID="089d7bd4-d019-471c-960f-51346831fe45"
Nov 02 12:26:12.240100 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:12.240073    2574 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 02 12:26:12.254232 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:12.253694    2574 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-dnm8d"
Nov 02 12:26:12.254979 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:12.254823    2574 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-dnm8d"
Nov 02 12:26:12.298192 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:12.298118    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-9t4f5" podStartSLOduration=4.457914663 podStartE2EDuration="9.298096114s" totalImagesPullingTime="4.840181451s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 12:26:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:26:10.998241566 +0000 UTC m=+8.715184339" watchObservedRunningTime="2026-11-02 12:26:12.298096114 +0000 UTC m=+10.015038885"
Nov 02 12:26:12.475042 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:12.475003    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/83964713-5989-4b35-91e8-3d3993688ccd-original-pull-secret\") pod \"global-pull-secret-syncer-6grht\" (UID: \"83964713-5989-4b35-91e8-3d3993688ccd\") " pod="kube-system/global-pull-secret-syncer-6grht"
Nov 02 12:26:12.475192 ip-10-0-128-132 kubenswrapper[2574]: E1102 12:26:12.475170    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 02 12:26:12.475236 ip-10-0-128-132 kubenswrapper[2574]: E1102 12:26:12.475232    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/83964713-5989-4b35-91e8-3d3993688ccd-original-pull-secret podName:83964713-5989-4b35-91e8-3d3993688ccd nodeName:}" failed. No retries permitted until 2026-11-02 12:26:20.475218056 +0000 UTC m=+18.192160807 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/83964713-5989-4b35-91e8-3d3993688ccd-original-pull-secret") pod "global-pull-secret-syncer-6grht" (UID: "83964713-5989-4b35-91e8-3d3993688ccd") : object "kube-system"/"original-pull-secret" not registered
Nov 02 12:26:12.576004 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:12.575969    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/09f70dd9-4dbe-46df-ac95-1c351a541a18-metrics-certs\") pod \"network-metrics-daemon-dp7h7\" (UID: \"09f70dd9-4dbe-46df-ac95-1c351a541a18\") " pod="openshift-multus/network-metrics-daemon-dp7h7"
Nov 02 12:26:12.576180 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:12.576016    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cws77\" (UniqueName: \"kubernetes.io/projected/089d7bd4-d019-471c-960f-51346831fe45-kube-api-access-cws77\") pod \"network-check-target-wx6zw\" (UID: \"089d7bd4-d019-471c-960f-51346831fe45\") " pod="openshift-network-diagnostics/network-check-target-wx6zw"
Nov 02 12:26:12.576245 ip-10-0-128-132 kubenswrapper[2574]: E1102 12:26:12.576192    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 02 12:26:12.576245 ip-10-0-128-132 kubenswrapper[2574]: E1102 12:26:12.576214    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 02 12:26:12.576245 ip-10-0-128-132 kubenswrapper[2574]: E1102 12:26:12.576228    2574 projected.go:196] Error preparing data for projected volume kube-api-access-cws77 for pod openshift-network-diagnostics/network-check-target-wx6zw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 12:26:12.576374 ip-10-0-128-132 kubenswrapper[2574]: E1102 12:26:12.576291    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/089d7bd4-d019-471c-960f-51346831fe45-kube-api-access-cws77 podName:089d7bd4-d019-471c-960f-51346831fe45 nodeName:}" failed. No retries permitted until 2026-11-02 12:26:20.576273618 +0000 UTC m=+18.293216380 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-cws77" (UniqueName: "kubernetes.io/projected/089d7bd4-d019-471c-960f-51346831fe45-kube-api-access-cws77") pod "network-check-target-wx6zw" (UID: "089d7bd4-d019-471c-960f-51346831fe45") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 12:26:12.576793 ip-10-0-128-132 kubenswrapper[2574]: E1102 12:26:12.576692    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 12:26:12.576793 ip-10-0-128-132 kubenswrapper[2574]: E1102 12:26:12.576759    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/09f70dd9-4dbe-46df-ac95-1c351a541a18-metrics-certs podName:09f70dd9-4dbe-46df-ac95-1c351a541a18 nodeName:}" failed. No retries permitted until 2026-11-02 12:26:20.576741155 +0000 UTC m=+18.293683905 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/09f70dd9-4dbe-46df-ac95-1c351a541a18-metrics-certs") pod "network-metrics-daemon-dp7h7" (UID: "09f70dd9-4dbe-46df-ac95-1c351a541a18") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 12:26:12.774572 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:12.774100    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dp7h7"
Nov 02 12:26:12.774572 ip-10-0-128-132 kubenswrapper[2574]: E1102 12:26:12.774238    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dp7h7" podUID="09f70dd9-4dbe-46df-ac95-1c351a541a18"
Nov 02 12:26:12.774572 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:12.774282    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6grht"
Nov 02 12:26:12.774572 ip-10-0-128-132 kubenswrapper[2574]: E1102 12:26:12.774400    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6grht" podUID="83964713-5989-4b35-91e8-3d3993688ccd"
Nov 02 12:26:12.851835 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:12.850940    2574 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-02T12:26:12.240094092Z","UUID":"425860d6-281a-4ebe-9d5f-1ca27bc607ab","Handler":null,"Name":"","Endpoint":""}
Nov 02 12:26:12.853093 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:12.852815    2574 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 02 12:26:12.853093 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:12.852847    2574 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 02 12:26:12.969975 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:12.969184    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-z5nxc" event={"ID":"993b825c-e7c4-445e-b8e8-5e92eb205f3e","Type":"ContainerStarted","Data":"dde43b17ab4f6a4afccaa402db4ef33c1284a725f649100fca0cd6ef204dcce6"}
Nov 02 12:26:12.974745 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:12.974713    2574 generic.go:356] "Generic (PLEG): container finished" podID="feba123c-eeb3-412f-911e-a66da1981358" containerID="3cc7793c56966c4d01fc5047a415e1150692626b7ceeca318108f6e998ab344b" exitCode=0
Nov 02 12:26:12.974869 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:12.974825    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v7bcl" event={"ID":"feba123c-eeb3-412f-911e-a66da1981358","Type":"ContainerDied","Data":"3cc7793c56966c4d01fc5047a415e1150692626b7ceeca318108f6e998ab344b"}
Nov 02 12:26:12.980954 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:12.979090    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kbhhk" event={"ID":"af96531f-d9a0-4acc-ab22-62949e7400aa","Type":"ContainerStarted","Data":"befba6ccfb3b86f240b8a7fdd1ebdf86160ca13e1fdb2d0dbbdcbb2465087231"}
Nov 02 12:26:13.066028 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:13.065961    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-z5nxc" podStartSLOduration=3.44656992 podStartE2EDuration="11.06593941s" totalImagesPullingTime="7.61936949s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 12:26:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:26:13.01778513 +0000 UTC m=+10.734727901" watchObservedRunningTime="2026-11-02 12:26:13.06593941 +0000 UTC m=+10.782882181"
Nov 02 12:26:13.773696 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:13.773435    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wx6zw"
Nov 02 12:26:13.774196 ip-10-0-128-132 kubenswrapper[2574]: E1102 12:26:13.773790    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wx6zw" podUID="089d7bd4-d019-471c-960f-51346831fe45"
Nov 02 12:26:13.983708 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:13.983677    2574 generic.go:356] "Generic (PLEG): container finished" podID="feba123c-eeb3-412f-911e-a66da1981358" containerID="587504a512b9673d4c6881ac1ff480e9713626eb335d4c657be8a76739125313" exitCode=0
Nov 02 12:26:13.983881 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:13.983765    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v7bcl" event={"ID":"feba123c-eeb3-412f-911e-a66da1981358","Type":"ContainerDied","Data":"587504a512b9673d4c6881ac1ff480e9713626eb335d4c657be8a76739125313"}
Nov 02 12:26:13.986498 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:13.986470    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kbhhk" event={"ID":"af96531f-d9a0-4acc-ab22-62949e7400aa","Type":"ContainerStarted","Data":"c1540b880f66d3d25eea6b16dedc6b09e582197074d0fd48e02182185f8eb7ce"}
Nov 02 12:26:13.989031 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:13.988994    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-7trhz" event={"ID":"5507fd15-57b9-422a-af1d-5daf266bd9ca","Type":"ContainerStarted","Data":"39b8764ff838622849cf2d1c9a9e8d83b8c867af111c3772cfe2065c30f74e8d"}
Nov 02 12:26:14.077450 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:14.077379    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-7trhz" podStartSLOduration=3.153542492 podStartE2EDuration="11.077360821s" totalImagesPullingTime="7.923818329s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 12:26:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:26:14.077114355 +0000 UTC m=+11.794057124" watchObservedRunningTime="2026-11-02 12:26:14.077360821 +0000 UTC m=+11.794303592"
Nov 02 12:26:14.120651 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:14.120583    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kbhhk" podStartSLOduration=4.212888094 podStartE2EDuration="12.120564271s" totalImagesPullingTime="7.907676177s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 12:26:02 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:26:14.119908969 +0000 UTC m=+11.836851739" watchObservedRunningTime="2026-11-02 12:26:14.120564271 +0000 UTC m=+11.837507042"
Nov 02 12:26:14.773979 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:14.773942    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dp7h7"
Nov 02 12:26:14.773979 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:14.773986    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6grht"
Nov 02 12:26:14.774505 ip-10-0-128-132 kubenswrapper[2574]: E1102 12:26:14.774084    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dp7h7" podUID="09f70dd9-4dbe-46df-ac95-1c351a541a18"
Nov 02 12:26:14.774505 ip-10-0-128-132 kubenswrapper[2574]: E1102 12:26:14.774215    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6grht" podUID="83964713-5989-4b35-91e8-3d3993688ccd"
Nov 02 12:26:14.993500 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:14.993465    2574 generic.go:356] "Generic (PLEG): container finished" podID="feba123c-eeb3-412f-911e-a66da1981358" containerID="eb8eb17bc09c67980e4f820babc4b369fc57ec52359decceaa2321eee619ef3c" exitCode=0
Nov 02 12:26:14.993659 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:14.993550    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v7bcl" event={"ID":"feba123c-eeb3-412f-911e-a66da1981358","Type":"ContainerDied","Data":"eb8eb17bc09c67980e4f820babc4b369fc57ec52359decceaa2321eee619ef3c"}
Nov 02 12:26:14.995376 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:14.995320    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-7gc6b" event={"ID":"af2e7d8e-b1df-499c-9a92-9ebe9d7bc7b2","Type":"ContainerStarted","Data":"3bc90822ee060557ed9112f0c61493dcdfec90144af14a99bdb26e8bdfbef655"}
Nov 02 12:26:15.096771 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:15.096720    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-7gc6b" podStartSLOduration=4.425250059 podStartE2EDuration="12.096704437s" totalImagesPullingTime="7.671454378s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 12:26:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:26:15.095999261 +0000 UTC m=+12.812942027" watchObservedRunningTime="2026-11-02 12:26:15.096704437 +0000 UTC m=+12.813647205"
Nov 02 12:26:15.772918 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:15.772879    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wx6zw"
Nov 02 12:26:15.773109 ip-10-0-128-132 kubenswrapper[2574]: E1102 12:26:15.773011    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wx6zw" podUID="089d7bd4-d019-471c-960f-51346831fe45"
Nov 02 12:26:15.999890 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:15.999848    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-vnv9x" event={"ID":"8a2e896a-489b-43fa-ba40-b2e3c0d56176","Type":"ContainerStarted","Data":"bd3db3b22ba7ab922707f66d89341c9cd2a0a358f67aae4e4d12c2d275b071ae"}
Nov 02 12:26:16.043355 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:16.043265    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-vnv9x" podStartSLOduration=4.068438106 podStartE2EDuration="14.043245966s" totalImagesPullingTime="9.97480786s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 12:26:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:26:16.042763583 +0000 UTC m=+13.759706353" watchObservedRunningTime="2026-11-02 12:26:16.043245966 +0000 UTC m=+13.760188738"
Nov 02 12:26:16.775571 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:16.775547    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dp7h7"
Nov 02 12:26:16.775718 ip-10-0-128-132 kubenswrapper[2574]: E1102 12:26:16.775696    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dp7h7" podUID="09f70dd9-4dbe-46df-ac95-1c351a541a18"
Nov 02 12:26:16.776016 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:16.775996    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6grht"
Nov 02 12:26:16.776095 ip-10-0-128-132 kubenswrapper[2574]: E1102 12:26:16.776081    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6grht" podUID="83964713-5989-4b35-91e8-3d3993688ccd"
Nov 02 12:26:17.005322 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:17.005231    2574 generic.go:356] "Generic (PLEG): container finished" podID="feba123c-eeb3-412f-911e-a66da1981358" containerID="ce7e5524e139dc45495c109a251c71316f9573dc377678ac1a035d9b2510ece1" exitCode=0
Nov 02 12:26:17.005875 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:17.005358    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v7bcl" event={"ID":"feba123c-eeb3-412f-911e-a66da1981358","Type":"ContainerDied","Data":"ce7e5524e139dc45495c109a251c71316f9573dc377678ac1a035d9b2510ece1"}
Nov 02 12:26:17.773751 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:17.773506    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wx6zw"
Nov 02 12:26:17.773933 ip-10-0-128-132 kubenswrapper[2574]: E1102 12:26:17.773868    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wx6zw" podUID="089d7bd4-d019-471c-960f-51346831fe45"
Nov 02 12:26:18.011012 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:18.010979    2574 generic.go:356] "Generic (PLEG): container finished" podID="feba123c-eeb3-412f-911e-a66da1981358" containerID="bb1c3990d6341c422630a4a27b7c219a308be1c7a114d0b93bb977ad7c4ac2f8" exitCode=0
Nov 02 12:26:18.011505 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:18.011047    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v7bcl" event={"ID":"feba123c-eeb3-412f-911e-a66da1981358","Type":"ContainerDied","Data":"bb1c3990d6341c422630a4a27b7c219a308be1c7a114d0b93bb977ad7c4ac2f8"}
Nov 02 12:26:18.773401 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:18.773367    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6grht"
Nov 02 12:26:18.773540 ip-10-0-128-132 kubenswrapper[2574]: E1102 12:26:18.773478    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6grht" podUID="83964713-5989-4b35-91e8-3d3993688ccd"
Nov 02 12:26:18.773598 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:18.773535    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dp7h7"
Nov 02 12:26:18.773683 ip-10-0-128-132 kubenswrapper[2574]: E1102 12:26:18.773661    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dp7h7" podUID="09f70dd9-4dbe-46df-ac95-1c351a541a18"
Nov 02 12:26:19.015750 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:19.015715    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v7bcl" event={"ID":"feba123c-eeb3-412f-911e-a66da1981358","Type":"ContainerStarted","Data":"83a49e406d0e15af5506b1645db1cdeec8adc20e2eb4da335999d2ba8537972d"}
Nov 02 12:26:19.088018 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:19.087926    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-v7bcl" podStartSLOduration=6.078247135 podStartE2EDuration="16.087912387s" totalImagesPullingTime="10.009665252s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 12:26:03 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:26:19.087603867 +0000 UTC m=+16.804546636" watchObservedRunningTime="2026-11-02 12:26:19.087912387 +0000 UTC m=+16.804855174"
Nov 02 12:26:19.773857 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:19.773677    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wx6zw"
Nov 02 12:26:19.774006 ip-10-0-128-132 kubenswrapper[2574]: E1102 12:26:19.773932    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wx6zw" podUID="089d7bd4-d019-471c-960f-51346831fe45"
Nov 02 12:26:19.834433 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:19.834402    2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-dnm8d"
Nov 02 12:26:19.834600 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:19.834511    2574 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 02 12:26:19.835137 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:19.835114    2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-dnm8d"
Nov 02 12:26:20.020587 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:20.020549    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qgv46" event={"ID":"a81d60e5-2049-4600-9b3f-d3826aee59fb","Type":"ContainerStarted","Data":"d846bb4ef5e7c934d352544da913c51e4e407cd720950ac76d33995c6eeebc58"}
Nov 02 12:26:20.021007 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:20.020634    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qgv46" event={"ID":"a81d60e5-2049-4600-9b3f-d3826aee59fb","Type":"ContainerStarted","Data":"30e1df9d6641332ab00ff8caf3513221724dfeeda85f8bce3defcd4a73d6bb51"}
Nov 02 12:26:20.021007 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:20.020702    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qgv46" event={"ID":"a81d60e5-2049-4600-9b3f-d3826aee59fb","Type":"ContainerStarted","Data":"12b7c976bc7def49156bb4c61d3cdb46fb02c2d32991beb927cb13f12dd0f46d"}
Nov 02 12:26:20.021007 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:20.020751    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qgv46" event={"ID":"a81d60e5-2049-4600-9b3f-d3826aee59fb","Type":"ContainerStarted","Data":"66bcda272294223a14b3ba215c50ad78f03742fcefac93135ed1f9b98bd007c9"}
Nov 02 12:26:20.021007 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:20.020796    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qgv46" event={"ID":"a81d60e5-2049-4600-9b3f-d3826aee59fb","Type":"ContainerStarted","Data":"9e3b4eb08b3347813bf7842f38d18ef041abdea2932709354a4dd5cecf0d3a21"}
Nov 02 12:26:20.021007 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:20.020837    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qgv46" event={"ID":"a81d60e5-2049-4600-9b3f-d3826aee59fb","Type":"ContainerStarted","Data":"efa23a9bda76519c467615474e401dfa7e5ce54db3185fc318a876d675da3311"}
Nov 02 12:26:20.529573 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:20.529534    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/83964713-5989-4b35-91e8-3d3993688ccd-original-pull-secret\") pod \"global-pull-secret-syncer-6grht\" (UID: \"83964713-5989-4b35-91e8-3d3993688ccd\") " pod="kube-system/global-pull-secret-syncer-6grht"
Nov 02 12:26:20.529773 ip-10-0-128-132 kubenswrapper[2574]: E1102 12:26:20.529694    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 02 12:26:20.529773 ip-10-0-128-132 kubenswrapper[2574]: E1102 12:26:20.529765    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/83964713-5989-4b35-91e8-3d3993688ccd-original-pull-secret podName:83964713-5989-4b35-91e8-3d3993688ccd nodeName:}" failed. No retries permitted until 2026-11-02 12:26:36.52974942 +0000 UTC m=+34.246692184 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/83964713-5989-4b35-91e8-3d3993688ccd-original-pull-secret") pod "global-pull-secret-syncer-6grht" (UID: "83964713-5989-4b35-91e8-3d3993688ccd") : object "kube-system"/"original-pull-secret" not registered
Nov 02 12:26:20.630457 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:20.630415    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cws77\" (UniqueName: \"kubernetes.io/projected/089d7bd4-d019-471c-960f-51346831fe45-kube-api-access-cws77\") pod \"network-check-target-wx6zw\" (UID: \"089d7bd4-d019-471c-960f-51346831fe45\") " pod="openshift-network-diagnostics/network-check-target-wx6zw"
Nov 02 12:26:20.630663 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:20.630491    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/09f70dd9-4dbe-46df-ac95-1c351a541a18-metrics-certs\") pod \"network-metrics-daemon-dp7h7\" (UID: \"09f70dd9-4dbe-46df-ac95-1c351a541a18\") " pod="openshift-multus/network-metrics-daemon-dp7h7"
Nov 02 12:26:20.630663 ip-10-0-128-132 kubenswrapper[2574]: E1102 12:26:20.630581    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 12:26:20.630663 ip-10-0-128-132 kubenswrapper[2574]: E1102 12:26:20.630600    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 02 12:26:20.630663 ip-10-0-128-132 kubenswrapper[2574]: E1102 12:26:20.630623    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 02 12:26:20.630663 ip-10-0-128-132 kubenswrapper[2574]: E1102 12:26:20.630635    2574 projected.go:196] Error preparing data for projected volume kube-api-access-cws77 for pod openshift-network-diagnostics/network-check-target-wx6zw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 12:26:20.630663 ip-10-0-128-132 kubenswrapper[2574]: E1102 12:26:20.630659    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/09f70dd9-4dbe-46df-ac95-1c351a541a18-metrics-certs podName:09f70dd9-4dbe-46df-ac95-1c351a541a18 nodeName:}" failed. No retries permitted until 2026-11-02 12:26:36.630625896 +0000 UTC m=+34.347568644 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/09f70dd9-4dbe-46df-ac95-1c351a541a18-metrics-certs") pod "network-metrics-daemon-dp7h7" (UID: "09f70dd9-4dbe-46df-ac95-1c351a541a18") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 12:26:20.630894 ip-10-0-128-132 kubenswrapper[2574]: E1102 12:26:20.630704    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/089d7bd4-d019-471c-960f-51346831fe45-kube-api-access-cws77 podName:089d7bd4-d019-471c-960f-51346831fe45 nodeName:}" failed. No retries permitted until 2026-11-02 12:26:36.63068835 +0000 UTC m=+34.347631115 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-cws77" (UniqueName: "kubernetes.io/projected/089d7bd4-d019-471c-960f-51346831fe45-kube-api-access-cws77") pod "network-check-target-wx6zw" (UID: "089d7bd4-d019-471c-960f-51346831fe45") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 12:26:20.773462 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:20.773427    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dp7h7"
Nov 02 12:26:20.773685 ip-10-0-128-132 kubenswrapper[2574]: E1102 12:26:20.773566    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dp7h7" podUID="09f70dd9-4dbe-46df-ac95-1c351a541a18"
Nov 02 12:26:20.773685 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:20.773611    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6grht"
Nov 02 12:26:20.773796 ip-10-0-128-132 kubenswrapper[2574]: E1102 12:26:20.773753    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6grht" podUID="83964713-5989-4b35-91e8-3d3993688ccd"
Nov 02 12:26:21.773809 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:21.773784    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wx6zw"
Nov 02 12:26:21.774308 ip-10-0-128-132 kubenswrapper[2574]: E1102 12:26:21.773877    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wx6zw" podUID="089d7bd4-d019-471c-960f-51346831fe45"
Nov 02 12:26:22.028319 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:22.028233    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qgv46" event={"ID":"a81d60e5-2049-4600-9b3f-d3826aee59fb","Type":"ContainerStarted","Data":"95439275d5cdf1640b586ec9b1289182d79de5e4c9c916e5cee1fc1d58903d05"}
Nov 02 12:26:22.774514 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:22.774483    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dp7h7"
Nov 02 12:26:22.774955 ip-10-0-128-132 kubenswrapper[2574]: E1102 12:26:22.774571    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dp7h7" podUID="09f70dd9-4dbe-46df-ac95-1c351a541a18"
Nov 02 12:26:22.774955 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:22.774672    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6grht"
Nov 02 12:26:22.774955 ip-10-0-128-132 kubenswrapper[2574]: E1102 12:26:22.774753    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6grht" podUID="83964713-5989-4b35-91e8-3d3993688ccd"
Nov 02 12:26:23.773924 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:23.773887    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wx6zw"
Nov 02 12:26:23.774094 ip-10-0-128-132 kubenswrapper[2574]: E1102 12:26:23.773998    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wx6zw" podUID="089d7bd4-d019-471c-960f-51346831fe45"
Nov 02 12:26:24.036218 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:24.036135    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qgv46" event={"ID":"a81d60e5-2049-4600-9b3f-d3826aee59fb","Type":"ContainerStarted","Data":"b7cf3e3f8d9247867de41b9301d31a69248c697aac45378de8a7ac22f98cd438"}
Nov 02 12:26:24.036971 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:24.036385    2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qgv46"
Nov 02 12:26:24.051514 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:24.051489    2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qgv46"
Nov 02 12:26:24.127433 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:24.127384    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-qgv46" podStartSLOduration=6.869351872 podStartE2EDuration="21.127372268s" totalImagesPullingTime="14.258020396s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 12:26:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:26:24.126913084 +0000 UTC m=+21.843855853" watchObservedRunningTime="2026-11-02 12:26:24.127372268 +0000 UTC m=+21.844315075"
Nov 02 12:26:24.773732 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:24.773698    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6grht"
Nov 02 12:26:24.773732 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:24.773732    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dp7h7"
Nov 02 12:26:24.773985 ip-10-0-128-132 kubenswrapper[2574]: E1102 12:26:24.773805    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6grht" podUID="83964713-5989-4b35-91e8-3d3993688ccd"
Nov 02 12:26:24.773985 ip-10-0-128-132 kubenswrapper[2574]: E1102 12:26:24.773922    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dp7h7" podUID="09f70dd9-4dbe-46df-ac95-1c351a541a18"
Nov 02 12:26:25.042702 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:25.042624    2574 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 02 12:26:25.043151 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:25.043130    2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qgv46"
Nov 02 12:26:25.067292 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:25.067264    2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qgv46"
Nov 02 12:26:25.773277 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:25.773095    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wx6zw"
Nov 02 12:26:25.773471 ip-10-0-128-132 kubenswrapper[2574]: E1102 12:26:25.773354    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wx6zw" podUID="089d7bd4-d019-471c-960f-51346831fe45"
Nov 02 12:26:25.951192 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:25.951163    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-6grht"]
Nov 02 12:26:25.951333 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:25.951267    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6grht"
Nov 02 12:26:25.951395 ip-10-0-128-132 kubenswrapper[2574]: E1102 12:26:25.951357    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6grht" podUID="83964713-5989-4b35-91e8-3d3993688ccd"
Nov 02 12:26:25.953912 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:25.953888    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-wx6zw"]
Nov 02 12:26:25.954623 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:25.954594    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-dp7h7"]
Nov 02 12:26:25.954718 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:25.954706    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dp7h7"
Nov 02 12:26:25.954832 ip-10-0-128-132 kubenswrapper[2574]: E1102 12:26:25.954801    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dp7h7" podUID="09f70dd9-4dbe-46df-ac95-1c351a541a18"
Nov 02 12:26:26.044634 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:26.044607    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wx6zw"
Nov 02 12:26:26.045068 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:26.044691    2574 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 02 12:26:26.045068 ip-10-0-128-132 kubenswrapper[2574]: E1102 12:26:26.044744    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wx6zw" podUID="089d7bd4-d019-471c-960f-51346831fe45"
Nov 02 12:26:27.047832 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:27.047800    2574 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 02 12:26:27.318110 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:27.318022    2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qgv46"
Nov 02 12:26:27.773051 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:27.773011    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6grht"
Nov 02 12:26:27.773220 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:27.773069    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dp7h7"
Nov 02 12:26:27.773220 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:27.773156    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wx6zw"
Nov 02 12:26:27.773220 ip-10-0-128-132 kubenswrapper[2574]: E1102 12:26:27.773160    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6grht" podUID="83964713-5989-4b35-91e8-3d3993688ccd"
Nov 02 12:26:27.773364 ip-10-0-128-132 kubenswrapper[2574]: E1102 12:26:27.773241    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dp7h7" podUID="09f70dd9-4dbe-46df-ac95-1c351a541a18"
Nov 02 12:26:27.773364 ip-10-0-128-132 kubenswrapper[2574]: E1102 12:26:27.773295    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wx6zw" podUID="089d7bd4-d019-471c-960f-51346831fe45"
Nov 02 12:26:28.061135 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:28.061053    2574 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-qgv46" podUID="a81d60e5-2049-4600-9b3f-d3826aee59fb" containerName="ovnkube-controller" probeResult="failure" output=""
Nov 02 12:26:29.773597 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:29.773558    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6grht"
Nov 02 12:26:29.773597 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:29.773586    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wx6zw"
Nov 02 12:26:29.774136 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:29.773684    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dp7h7"
Nov 02 12:26:29.774136 ip-10-0-128-132 kubenswrapper[2574]: E1102 12:26:29.773698    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wx6zw" podUID="089d7bd4-d019-471c-960f-51346831fe45"
Nov 02 12:26:29.774136 ip-10-0-128-132 kubenswrapper[2574]: E1102 12:26:29.773788    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dp7h7" podUID="09f70dd9-4dbe-46df-ac95-1c351a541a18"
Nov 02 12:26:29.774136 ip-10-0-128-132 kubenswrapper[2574]: E1102 12:26:29.773857    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6grht" podUID="83964713-5989-4b35-91e8-3d3993688ccd"
Nov 02 12:26:31.773653 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:31.773603    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6grht"
Nov 02 12:26:31.774153 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:31.773635    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dp7h7"
Nov 02 12:26:31.774153 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:31.773604    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wx6zw"
Nov 02 12:26:31.774153 ip-10-0-128-132 kubenswrapper[2574]: E1102 12:26:31.773751    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6grht" podUID="83964713-5989-4b35-91e8-3d3993688ccd"
Nov 02 12:26:31.774153 ip-10-0-128-132 kubenswrapper[2574]: E1102 12:26:31.773884    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wx6zw" podUID="089d7bd4-d019-471c-960f-51346831fe45"
Nov 02 12:26:31.774153 ip-10-0-128-132 kubenswrapper[2574]: E1102 12:26:31.773998    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dp7h7" podUID="09f70dd9-4dbe-46df-ac95-1c351a541a18"
Nov 02 12:26:32.114972 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:32.114948    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-132.ec2.internal" event="NodeReady"
Nov 02 12:26:32.115129 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:32.115078    2574 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Nov 02 12:26:32.189126 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:32.189093    2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-bg5xh"]
Nov 02 12:26:32.192102 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:32.192083    2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-9x7t5"]
Nov 02 12:26:32.192245 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:32.192224    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-bg5xh"
Nov 02 12:26:32.195445 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:32.195427    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-9x7t5"
Nov 02 12:26:32.208144 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:32.208122    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 02 12:26:32.208144 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:32.208120    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 02 12:26:32.208283 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:32.208239    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-rbwvz\""
Nov 02 12:26:32.213086 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:32.213061    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-bg5xh"]
Nov 02 12:26:32.242278 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:32.242250    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-9x7t5"]
Nov 02 12:26:32.307549 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:32.307465    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7b688512-ffc4-4b6a-b9ab-8fcf8ea0afe2-config-volume\") pod \"dns-default-bg5xh\" (UID: \"7b688512-ffc4-4b6a-b9ab-8fcf8ea0afe2\") " pod="openshift-dns/dns-default-bg5xh"
Nov 02 12:26:32.307549 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:32.307506    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f9be3b9a-c86a-473e-8a58-83d62a0cf1d7-cert\") pod \"ingress-canary-9x7t5\" (UID: \"f9be3b9a-c86a-473e-8a58-83d62a0cf1d7\") " pod="openshift-ingress-canary/ingress-canary-9x7t5"
Nov 02 12:26:32.307771 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:32.307551    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4txl\" (UniqueName: \"kubernetes.io/projected/f9be3b9a-c86a-473e-8a58-83d62a0cf1d7-kube-api-access-j4txl\") pod \"ingress-canary-9x7t5\" (UID: \"f9be3b9a-c86a-473e-8a58-83d62a0cf1d7\") " pod="openshift-ingress-canary/ingress-canary-9x7t5"
Nov 02 12:26:32.307771 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:32.307663    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-828sv\" (UniqueName: \"kubernetes.io/projected/7b688512-ffc4-4b6a-b9ab-8fcf8ea0afe2-kube-api-access-828sv\") pod \"dns-default-bg5xh\" (UID: \"7b688512-ffc4-4b6a-b9ab-8fcf8ea0afe2\") " pod="openshift-dns/dns-default-bg5xh"
Nov 02 12:26:32.307771 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:32.307715    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7b688512-ffc4-4b6a-b9ab-8fcf8ea0afe2-metrics-tls\") pod \"dns-default-bg5xh\" (UID: \"7b688512-ffc4-4b6a-b9ab-8fcf8ea0afe2\") " pod="openshift-dns/dns-default-bg5xh"
Nov 02 12:26:32.307771 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:32.307749    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b688512-ffc4-4b6a-b9ab-8fcf8ea0afe2-tmp-dir\") pod \"dns-default-bg5xh\" (UID: \"7b688512-ffc4-4b6a-b9ab-8fcf8ea0afe2\") " pod="openshift-dns/dns-default-bg5xh"
Nov 02 12:26:32.308064 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:32.308037    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 02 12:26:32.308401 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:32.308384    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 02 12:26:32.308401 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:32.308408    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-tlsnx\""
Nov 02 12:26:32.308544 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:32.308473    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 02 12:26:32.408045 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:32.408013    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f9be3b9a-c86a-473e-8a58-83d62a0cf1d7-cert\") pod \"ingress-canary-9x7t5\" (UID: \"f9be3b9a-c86a-473e-8a58-83d62a0cf1d7\") " pod="openshift-ingress-canary/ingress-canary-9x7t5"
Nov 02 12:26:32.408045 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:32.408050    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4txl\" (UniqueName: \"kubernetes.io/projected/f9be3b9a-c86a-473e-8a58-83d62a0cf1d7-kube-api-access-j4txl\") pod \"ingress-canary-9x7t5\" (UID: \"f9be3b9a-c86a-473e-8a58-83d62a0cf1d7\") " pod="openshift-ingress-canary/ingress-canary-9x7t5"
Nov 02 12:26:32.408260 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:32.408236    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-828sv\" (UniqueName: \"kubernetes.io/projected/7b688512-ffc4-4b6a-b9ab-8fcf8ea0afe2-kube-api-access-828sv\") pod \"dns-default-bg5xh\" (UID: \"7b688512-ffc4-4b6a-b9ab-8fcf8ea0afe2\") " pod="openshift-dns/dns-default-bg5xh"
Nov 02 12:26:32.408302 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:32.408287    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7b688512-ffc4-4b6a-b9ab-8fcf8ea0afe2-metrics-tls\") pod \"dns-default-bg5xh\" (UID: \"7b688512-ffc4-4b6a-b9ab-8fcf8ea0afe2\") " pod="openshift-dns/dns-default-bg5xh"
Nov 02 12:26:32.408359 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:32.408345    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b688512-ffc4-4b6a-b9ab-8fcf8ea0afe2-tmp-dir\") pod \"dns-default-bg5xh\" (UID: \"7b688512-ffc4-4b6a-b9ab-8fcf8ea0afe2\") " pod="openshift-dns/dns-default-bg5xh"
Nov 02 12:26:32.408411 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:32.408397    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7b688512-ffc4-4b6a-b9ab-8fcf8ea0afe2-config-volume\") pod \"dns-default-bg5xh\" (UID: \"7b688512-ffc4-4b6a-b9ab-8fcf8ea0afe2\") " pod="openshift-dns/dns-default-bg5xh"
Nov 02 12:26:32.408453 ip-10-0-128-132 kubenswrapper[2574]: E1102 12:26:32.408415    2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 02 12:26:32.408493 ip-10-0-128-132 kubenswrapper[2574]: E1102 12:26:32.408475    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7b688512-ffc4-4b6a-b9ab-8fcf8ea0afe2-metrics-tls podName:7b688512-ffc4-4b6a-b9ab-8fcf8ea0afe2 nodeName:}" failed. No retries permitted until 2026-11-02 12:26:32.908456973 +0000 UTC m=+30.625399741 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7b688512-ffc4-4b6a-b9ab-8fcf8ea0afe2-metrics-tls") pod "dns-default-bg5xh" (UID: "7b688512-ffc4-4b6a-b9ab-8fcf8ea0afe2") : secret "dns-default-metrics-tls" not found
Nov 02 12:26:32.408701 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:32.408686    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b688512-ffc4-4b6a-b9ab-8fcf8ea0afe2-tmp-dir\") pod \"dns-default-bg5xh\" (UID: \"7b688512-ffc4-4b6a-b9ab-8fcf8ea0afe2\") " pod="openshift-dns/dns-default-bg5xh"
Nov 02 12:26:32.408963 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:32.408948    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7b688512-ffc4-4b6a-b9ab-8fcf8ea0afe2-config-volume\") pod \"dns-default-bg5xh\" (UID: \"7b688512-ffc4-4b6a-b9ab-8fcf8ea0afe2\") " pod="openshift-dns/dns-default-bg5xh"
Nov 02 12:26:32.411956 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:32.411929    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f9be3b9a-c86a-473e-8a58-83d62a0cf1d7-cert\") pod \"ingress-canary-9x7t5\" (UID: \"f9be3b9a-c86a-473e-8a58-83d62a0cf1d7\") " pod="openshift-ingress-canary/ingress-canary-9x7t5"
Nov 02 12:26:32.431852 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:32.431825    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-828sv\" (UniqueName: \"kubernetes.io/projected/7b688512-ffc4-4b6a-b9ab-8fcf8ea0afe2-kube-api-access-828sv\") pod \"dns-default-bg5xh\" (UID: \"7b688512-ffc4-4b6a-b9ab-8fcf8ea0afe2\") " pod="openshift-dns/dns-default-bg5xh"
Nov 02 12:26:32.432019 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:32.432001    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4txl\" (UniqueName: \"kubernetes.io/projected/f9be3b9a-c86a-473e-8a58-83d62a0cf1d7-kube-api-access-j4txl\") pod \"ingress-canary-9x7t5\" (UID: \"f9be3b9a-c86a-473e-8a58-83d62a0cf1d7\") " pod="openshift-ingress-canary/ingress-canary-9x7t5"
Nov 02 12:26:32.513310 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:32.513266    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-9x7t5"
Nov 02 12:26:32.835505 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:32.835471    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-9x7t5"]
Nov 02 12:26:32.836049 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:32.836025    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9be3b9a_c86a_473e_8a58_83d62a0cf1d7.slice/crio-26ecac6199a071ff2dcd94f0d1d4d938473fe2b4001d4b0e61ecb1e3b6938acf WatchSource:0}: Error finding container 26ecac6199a071ff2dcd94f0d1d4d938473fe2b4001d4b0e61ecb1e3b6938acf: Status 404 returned error can't find the container with id 26ecac6199a071ff2dcd94f0d1d4d938473fe2b4001d4b0e61ecb1e3b6938acf
Nov 02 12:26:32.913843 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:32.913814    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7b688512-ffc4-4b6a-b9ab-8fcf8ea0afe2-metrics-tls\") pod \"dns-default-bg5xh\" (UID: \"7b688512-ffc4-4b6a-b9ab-8fcf8ea0afe2\") " pod="openshift-dns/dns-default-bg5xh"
Nov 02 12:26:32.914012 ip-10-0-128-132 kubenswrapper[2574]: E1102 12:26:32.913940    2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 02 12:26:32.914012 ip-10-0-128-132 kubenswrapper[2574]: E1102 12:26:32.913995    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7b688512-ffc4-4b6a-b9ab-8fcf8ea0afe2-metrics-tls podName:7b688512-ffc4-4b6a-b9ab-8fcf8ea0afe2 nodeName:}" failed. No retries permitted until 2026-11-02 12:26:33.913981957 +0000 UTC m=+31.630924704 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7b688512-ffc4-4b6a-b9ab-8fcf8ea0afe2-metrics-tls") pod "dns-default-bg5xh" (UID: "7b688512-ffc4-4b6a-b9ab-8fcf8ea0afe2") : secret "dns-default-metrics-tls" not found
Nov 02 12:26:33.064974 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:33.064939    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-9x7t5" event={"ID":"f9be3b9a-c86a-473e-8a58-83d62a0cf1d7","Type":"ContainerStarted","Data":"26ecac6199a071ff2dcd94f0d1d4d938473fe2b4001d4b0e61ecb1e3b6938acf"}
Nov 02 12:26:33.773833 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:33.773797    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6grht"
Nov 02 12:26:33.774022 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:33.773845    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dp7h7"
Nov 02 12:26:33.774182 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:33.774170    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wx6zw"
Nov 02 12:26:33.790067 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:33.790034    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 02 12:26:33.790213 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:33.790157    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 02 12:26:33.790312 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:33.790292    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 02 12:26:33.790368 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:33.790338    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-2lqbm\""
Nov 02 12:26:33.803079 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:33.803049    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 02 12:26:33.803079 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:33.803058    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-gr8bh\""
Nov 02 12:26:33.803223 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:33.803124    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-l6cfl\""
Nov 02 12:26:33.921372 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:33.921337    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7b688512-ffc4-4b6a-b9ab-8fcf8ea0afe2-metrics-tls\") pod \"dns-default-bg5xh\" (UID: \"7b688512-ffc4-4b6a-b9ab-8fcf8ea0afe2\") " pod="openshift-dns/dns-default-bg5xh"
Nov 02 12:26:33.921876 ip-10-0-128-132 kubenswrapper[2574]: E1102 12:26:33.921500    2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 02 12:26:33.921876 ip-10-0-128-132 kubenswrapper[2574]: E1102 12:26:33.921575    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7b688512-ffc4-4b6a-b9ab-8fcf8ea0afe2-metrics-tls podName:7b688512-ffc4-4b6a-b9ab-8fcf8ea0afe2 nodeName:}" failed. No retries permitted until 2026-11-02 12:26:35.9215479 +0000 UTC m=+33.638490647 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7b688512-ffc4-4b6a-b9ab-8fcf8ea0afe2-metrics-tls") pod "dns-default-bg5xh" (UID: "7b688512-ffc4-4b6a-b9ab-8fcf8ea0afe2") : secret "dns-default-metrics-tls" not found
Nov 02 12:26:35.938430 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:35.938389    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7b688512-ffc4-4b6a-b9ab-8fcf8ea0afe2-metrics-tls\") pod \"dns-default-bg5xh\" (UID: \"7b688512-ffc4-4b6a-b9ab-8fcf8ea0afe2\") " pod="openshift-dns/dns-default-bg5xh"
Nov 02 12:26:35.938846 ip-10-0-128-132 kubenswrapper[2574]: E1102 12:26:35.938503    2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 02 12:26:35.938846 ip-10-0-128-132 kubenswrapper[2574]: E1102 12:26:35.938556    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7b688512-ffc4-4b6a-b9ab-8fcf8ea0afe2-metrics-tls podName:7b688512-ffc4-4b6a-b9ab-8fcf8ea0afe2 nodeName:}" failed. No retries permitted until 2026-11-02 12:26:39.938543282 +0000 UTC m=+37.655486030 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7b688512-ffc4-4b6a-b9ab-8fcf8ea0afe2-metrics-tls") pod "dns-default-bg5xh" (UID: "7b688512-ffc4-4b6a-b9ab-8fcf8ea0afe2") : secret "dns-default-metrics-tls" not found
Nov 02 12:26:36.072927 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:36.072891    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-9x7t5" event={"ID":"f9be3b9a-c86a-473e-8a58-83d62a0cf1d7","Type":"ContainerStarted","Data":"cee67e82366e9758e9a8f01699b7e0578cc21ba5217258401231551414e858fb"}
Nov 02 12:26:36.114530 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:36.114478    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-9x7t5" podStartSLOduration=1.4009306590000001 podStartE2EDuration="4.114464323s" totalImagesPullingTime="2.713533664s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:26:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:26:36.113614686 +0000 UTC m=+33.830557456" watchObservedRunningTime="2026-11-02 12:26:36.114464323 +0000 UTC m=+33.831407089"
Nov 02 12:26:36.542465 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:36.542416    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/83964713-5989-4b35-91e8-3d3993688ccd-original-pull-secret\") pod \"global-pull-secret-syncer-6grht\" (UID: \"83964713-5989-4b35-91e8-3d3993688ccd\") " pod="kube-system/global-pull-secret-syncer-6grht"
Nov 02 12:26:36.544768 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:36.544741    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/83964713-5989-4b35-91e8-3d3993688ccd-original-pull-secret\") pod \"global-pull-secret-syncer-6grht\" (UID: \"83964713-5989-4b35-91e8-3d3993688ccd\") " pod="kube-system/global-pull-secret-syncer-6grht"
Nov 02 12:26:36.643376 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:36.643338    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/09f70dd9-4dbe-46df-ac95-1c351a541a18-metrics-certs\") pod \"network-metrics-daemon-dp7h7\" (UID: \"09f70dd9-4dbe-46df-ac95-1c351a541a18\") " pod="openshift-multus/network-metrics-daemon-dp7h7"
Nov 02 12:26:36.643376 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:36.643380    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cws77\" (UniqueName: \"kubernetes.io/projected/089d7bd4-d019-471c-960f-51346831fe45-kube-api-access-cws77\") pod \"network-check-target-wx6zw\" (UID: \"089d7bd4-d019-471c-960f-51346831fe45\") " pod="openshift-network-diagnostics/network-check-target-wx6zw"
Nov 02 12:26:36.643560 ip-10-0-128-132 kubenswrapper[2574]: E1102 12:26:36.643487    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 02 12:26:36.643560 ip-10-0-128-132 kubenswrapper[2574]: E1102 12:26:36.643552    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/09f70dd9-4dbe-46df-ac95-1c351a541a18-metrics-certs podName:09f70dd9-4dbe-46df-ac95-1c351a541a18 nodeName:}" failed. No retries permitted until 2026-11-02 12:27:08.643537187 +0000 UTC m=+66.360479939 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/09f70dd9-4dbe-46df-ac95-1c351a541a18-metrics-certs") pod "network-metrics-daemon-dp7h7" (UID: "09f70dd9-4dbe-46df-ac95-1c351a541a18") : secret "metrics-daemon-secret" not found
Nov 02 12:26:36.645844 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:36.645823    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cws77\" (UniqueName: \"kubernetes.io/projected/089d7bd4-d019-471c-960f-51346831fe45-kube-api-access-cws77\") pod \"network-check-target-wx6zw\" (UID: \"089d7bd4-d019-471c-960f-51346831fe45\") " pod="openshift-network-diagnostics/network-check-target-wx6zw"
Nov 02 12:26:36.832738 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:36.832635    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6grht"
Nov 02 12:26:36.872223 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:36.872195    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wx6zw"
Nov 02 12:26:37.191429 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:37.191382    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-6grht"]
Nov 02 12:26:37.191774 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:37.191719    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83964713_5989_4b35_91e8_3d3993688ccd.slice/crio-bf051a0b50b073a69b414eb0f82aeecd094209a84b66181af9c36023d9d06908 WatchSource:0}: Error finding container bf051a0b50b073a69b414eb0f82aeecd094209a84b66181af9c36023d9d06908: Status 404 returned error can't find the container with id bf051a0b50b073a69b414eb0f82aeecd094209a84b66181af9c36023d9d06908
Nov 02 12:26:37.216044 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:37.216019    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-wx6zw"]
Nov 02 12:26:37.216145 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:37.216063    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod089d7bd4_d019_471c_960f_51346831fe45.slice/crio-57ec3259bc4c3b9c41429b5c2a5b3db208605386478d98cc19462e00975e2e76 WatchSource:0}: Error finding container 57ec3259bc4c3b9c41429b5c2a5b3db208605386478d98cc19462e00975e2e76: Status 404 returned error can't find the container with id 57ec3259bc4c3b9c41429b5c2a5b3db208605386478d98cc19462e00975e2e76
Nov 02 12:26:37.463693 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:37.463589    2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-wqcmj"]
Nov 02 12:26:37.503544 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:37.503514    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-wqcmj"]
Nov 02 12:26:37.503707 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:37.503655    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-wqcmj"
Nov 02 12:26:37.529348 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:37.529326    2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-x27dt-dzqvx"]
Nov 02 12:26:37.548827 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:37.548801    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-x27dt-dzqvx"
Nov 02 12:26:37.548954 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:37.548921    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/973dc2ad-34ed-4364-b01b-9abbf4f2d43b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wqcmj\" (UID: \"973dc2ad-34ed-4364-b01b-9abbf4f2d43b\") " pod="openshift-insights/insights-runtime-extractor-wqcmj"
Nov 02 12:26:37.549011 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:37.548977    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/973dc2ad-34ed-4364-b01b-9abbf4f2d43b-data-volume\") pod \"insights-runtime-extractor-wqcmj\" (UID: \"973dc2ad-34ed-4364-b01b-9abbf4f2d43b\") " pod="openshift-insights/insights-runtime-extractor-wqcmj"
Nov 02 12:26:37.549101 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:37.549078    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/973dc2ad-34ed-4364-b01b-9abbf4f2d43b-crio-socket\") pod \"insights-runtime-extractor-wqcmj\" (UID: \"973dc2ad-34ed-4364-b01b-9abbf4f2d43b\") " pod="openshift-insights/insights-runtime-extractor-wqcmj"
Nov 02 12:26:37.549192 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:37.549162    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4n6d\" (UniqueName: \"kubernetes.io/projected/973dc2ad-34ed-4364-b01b-9abbf4f2d43b-kube-api-access-d4n6d\") pod \"insights-runtime-extractor-wqcmj\" (UID: \"973dc2ad-34ed-4364-b01b-9abbf4f2d43b\") " pod="openshift-insights/insights-runtime-extractor-wqcmj"
Nov 02 12:26:37.549277 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:37.549199    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/973dc2ad-34ed-4364-b01b-9abbf4f2d43b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wqcmj\" (UID: \"973dc2ad-34ed-4364-b01b-9abbf4f2d43b\") " pod="openshift-insights/insights-runtime-extractor-wqcmj"
Nov 02 12:26:37.552260 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:37.552224    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-x27dt-dzqvx"]
Nov 02 12:26:37.587497 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:37.587454    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 02 12:26:37.587497 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:37.587471    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 02 12:26:37.587721 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:37.587512    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 02 12:26:37.587721 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:37.587528    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 02 12:26:37.587721 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:37.587597    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Nov 02 12:26:37.587721 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:37.587632    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-7sfkq\""
Nov 02 12:26:37.587721 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:37.587702    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-wx6z7\""
Nov 02 12:26:37.613736 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:37.613711    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Nov 02 12:26:37.650229 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:37.650196    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4n6d\" (UniqueName: \"kubernetes.io/projected/973dc2ad-34ed-4364-b01b-9abbf4f2d43b-kube-api-access-d4n6d\") pod \"insights-runtime-extractor-wqcmj\" (UID: \"973dc2ad-34ed-4364-b01b-9abbf4f2d43b\") " pod="openshift-insights/insights-runtime-extractor-wqcmj"
Nov 02 12:26:37.650394 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:37.650238    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f25513a9-4a23-4cd1-9ef0-b5612826c157-trusted-ca-bundle\") pod \"periodic-gathering-x27dt-dzqvx\" (UID: \"f25513a9-4a23-4cd1-9ef0-b5612826c157\") " pod="openshift-insights/periodic-gathering-x27dt-dzqvx"
Nov 02 12:26:37.650394 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:37.650288    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/973dc2ad-34ed-4364-b01b-9abbf4f2d43b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wqcmj\" (UID: \"973dc2ad-34ed-4364-b01b-9abbf4f2d43b\") " pod="openshift-insights/insights-runtime-extractor-wqcmj"
Nov 02 12:26:37.650394 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:37.650337    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/973dc2ad-34ed-4364-b01b-9abbf4f2d43b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wqcmj\" (UID: \"973dc2ad-34ed-4364-b01b-9abbf4f2d43b\") " pod="openshift-insights/insights-runtime-extractor-wqcmj"
Nov 02 12:26:37.650394 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:37.650391    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/973dc2ad-34ed-4364-b01b-9abbf4f2d43b-data-volume\") pod \"insights-runtime-extractor-wqcmj\" (UID: \"973dc2ad-34ed-4364-b01b-9abbf4f2d43b\") " pod="openshift-insights/insights-runtime-extractor-wqcmj"
Nov 02 12:26:37.650594 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:37.650411    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7slf8\" (UniqueName: \"kubernetes.io/projected/f25513a9-4a23-4cd1-9ef0-b5612826c157-kube-api-access-7slf8\") pod \"periodic-gathering-x27dt-dzqvx\" (UID: \"f25513a9-4a23-4cd1-9ef0-b5612826c157\") " pod="openshift-insights/periodic-gathering-x27dt-dzqvx"
Nov 02 12:26:37.650594 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:37.650437    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/973dc2ad-34ed-4364-b01b-9abbf4f2d43b-crio-socket\") pod \"insights-runtime-extractor-wqcmj\" (UID: \"973dc2ad-34ed-4364-b01b-9abbf4f2d43b\") " pod="openshift-insights/insights-runtime-extractor-wqcmj"
Nov 02 12:26:37.650594 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:37.650464    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/f25513a9-4a23-4cd1-9ef0-b5612826c157-archives-path\") pod \"periodic-gathering-x27dt-dzqvx\" (UID: \"f25513a9-4a23-4cd1-9ef0-b5612826c157\") " pod="openshift-insights/periodic-gathering-x27dt-dzqvx"
Nov 02 12:26:37.650594 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:37.650487    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f25513a9-4a23-4cd1-9ef0-b5612826c157-service-ca-bundle\") pod \"periodic-gathering-x27dt-dzqvx\" (UID: \"f25513a9-4a23-4cd1-9ef0-b5612826c157\") " pod="openshift-insights/periodic-gathering-x27dt-dzqvx"
Nov 02 12:26:37.650890 ip-10-0-128-132 kubenswrapper[2574]: E1102 12:26:37.650871    2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 02 12:26:37.650943 ip-10-0-128-132 kubenswrapper[2574]: E1102 12:26:37.650928    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/973dc2ad-34ed-4364-b01b-9abbf4f2d43b-kube-rbac-proxy-cm podName:973dc2ad-34ed-4364-b01b-9abbf4f2d43b nodeName:}" failed. No retries permitted until 2026-11-02 12:26:38.150914694 +0000 UTC m=+35.867857442 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/973dc2ad-34ed-4364-b01b-9abbf4f2d43b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-wqcmj" (UID: "973dc2ad-34ed-4364-b01b-9abbf4f2d43b") : configmap "kube-rbac-proxy" not found
Nov 02 12:26:37.651146 ip-10-0-128-132 kubenswrapper[2574]: E1102 12:26:37.651133    2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 02 12:26:37.651197 ip-10-0-128-132 kubenswrapper[2574]: E1102 12:26:37.651167    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/973dc2ad-34ed-4364-b01b-9abbf4f2d43b-insights-runtime-extractor-tls podName:973dc2ad-34ed-4364-b01b-9abbf4f2d43b nodeName:}" failed. No retries permitted until 2026-11-02 12:26:38.151157706 +0000 UTC m=+35.868100452 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/973dc2ad-34ed-4364-b01b-9abbf4f2d43b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wqcmj" (UID: "973dc2ad-34ed-4364-b01b-9abbf4f2d43b") : secret "insights-runtime-extractor-tls" not found
Nov 02 12:26:37.651426 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:37.651408    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/973dc2ad-34ed-4364-b01b-9abbf4f2d43b-data-volume\") pod \"insights-runtime-extractor-wqcmj\" (UID: \"973dc2ad-34ed-4364-b01b-9abbf4f2d43b\") " pod="openshift-insights/insights-runtime-extractor-wqcmj"
Nov 02 12:26:37.651600 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:37.651586    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/973dc2ad-34ed-4364-b01b-9abbf4f2d43b-crio-socket\") pod \"insights-runtime-extractor-wqcmj\" (UID: \"973dc2ad-34ed-4364-b01b-9abbf4f2d43b\") " pod="openshift-insights/insights-runtime-extractor-wqcmj"
Nov 02 12:26:37.679063 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:37.679032    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4n6d\" (UniqueName: \"kubernetes.io/projected/973dc2ad-34ed-4364-b01b-9abbf4f2d43b-kube-api-access-d4n6d\") pod \"insights-runtime-extractor-wqcmj\" (UID: \"973dc2ad-34ed-4364-b01b-9abbf4f2d43b\") " pod="openshift-insights/insights-runtime-extractor-wqcmj"
Nov 02 12:26:37.751126 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:37.751038    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7slf8\" (UniqueName: \"kubernetes.io/projected/f25513a9-4a23-4cd1-9ef0-b5612826c157-kube-api-access-7slf8\") pod \"periodic-gathering-x27dt-dzqvx\" (UID: \"f25513a9-4a23-4cd1-9ef0-b5612826c157\") " pod="openshift-insights/periodic-gathering-x27dt-dzqvx"
Nov 02 12:26:37.751126 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:37.751095    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/f25513a9-4a23-4cd1-9ef0-b5612826c157-archives-path\") pod \"periodic-gathering-x27dt-dzqvx\" (UID: \"f25513a9-4a23-4cd1-9ef0-b5612826c157\") " pod="openshift-insights/periodic-gathering-x27dt-dzqvx"
Nov 02 12:26:37.751126 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:37.751123    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f25513a9-4a23-4cd1-9ef0-b5612826c157-service-ca-bundle\") pod \"periodic-gathering-x27dt-dzqvx\" (UID: \"f25513a9-4a23-4cd1-9ef0-b5612826c157\") " pod="openshift-insights/periodic-gathering-x27dt-dzqvx"
Nov 02 12:26:37.751368 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:37.751161    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f25513a9-4a23-4cd1-9ef0-b5612826c157-trusted-ca-bundle\") pod \"periodic-gathering-x27dt-dzqvx\" (UID: \"f25513a9-4a23-4cd1-9ef0-b5612826c157\") " pod="openshift-insights/periodic-gathering-x27dt-dzqvx"
Nov 02 12:26:37.752219 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:37.752164    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f25513a9-4a23-4cd1-9ef0-b5612826c157-trusted-ca-bundle\") pod \"periodic-gathering-x27dt-dzqvx\" (UID: \"f25513a9-4a23-4cd1-9ef0-b5612826c157\") " pod="openshift-insights/periodic-gathering-x27dt-dzqvx"
Nov 02 12:26:37.752451 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:37.752429    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/f25513a9-4a23-4cd1-9ef0-b5612826c157-archives-path\") pod \"periodic-gathering-x27dt-dzqvx\" (UID: \"f25513a9-4a23-4cd1-9ef0-b5612826c157\") " pod="openshift-insights/periodic-gathering-x27dt-dzqvx"
Nov 02 12:26:37.752621 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:37.752603    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f25513a9-4a23-4cd1-9ef0-b5612826c157-service-ca-bundle\") pod \"periodic-gathering-x27dt-dzqvx\" (UID: \"f25513a9-4a23-4cd1-9ef0-b5612826c157\") " pod="openshift-insights/periodic-gathering-x27dt-dzqvx"
Nov 02 12:26:37.777285 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:37.777248    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7slf8\" (UniqueName: \"kubernetes.io/projected/f25513a9-4a23-4cd1-9ef0-b5612826c157-kube-api-access-7slf8\") pod \"periodic-gathering-x27dt-dzqvx\" (UID: \"f25513a9-4a23-4cd1-9ef0-b5612826c157\") " pod="openshift-insights/periodic-gathering-x27dt-dzqvx"
Nov 02 12:26:37.880232 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:37.880190    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-x27dt-dzqvx"
Nov 02 12:26:38.079564 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:38.079471    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-6grht" event={"ID":"83964713-5989-4b35-91e8-3d3993688ccd","Type":"ContainerStarted","Data":"bf051a0b50b073a69b414eb0f82aeecd094209a84b66181af9c36023d9d06908"}
Nov 02 12:26:38.081276 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:38.081216    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-wx6zw" event={"ID":"089d7bd4-d019-471c-960f-51346831fe45","Type":"ContainerStarted","Data":"57ec3259bc4c3b9c41429b5c2a5b3db208605386478d98cc19462e00975e2e76"}
Nov 02 12:26:38.156227 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:38.156182    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/973dc2ad-34ed-4364-b01b-9abbf4f2d43b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wqcmj\" (UID: \"973dc2ad-34ed-4364-b01b-9abbf4f2d43b\") " pod="openshift-insights/insights-runtime-extractor-wqcmj"
Nov 02 12:26:38.156436 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:38.156298    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/973dc2ad-34ed-4364-b01b-9abbf4f2d43b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wqcmj\" (UID: \"973dc2ad-34ed-4364-b01b-9abbf4f2d43b\") " pod="openshift-insights/insights-runtime-extractor-wqcmj"
Nov 02 12:26:38.156436 ip-10-0-128-132 kubenswrapper[2574]: E1102 12:26:38.156363    2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 02 12:26:38.156436 ip-10-0-128-132 kubenswrapper[2574]: E1102 12:26:38.156385    2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 02 12:26:38.156587 ip-10-0-128-132 kubenswrapper[2574]: E1102 12:26:38.156457    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/973dc2ad-34ed-4364-b01b-9abbf4f2d43b-insights-runtime-extractor-tls podName:973dc2ad-34ed-4364-b01b-9abbf4f2d43b nodeName:}" failed. No retries permitted until 2026-11-02 12:26:39.156427157 +0000 UTC m=+36.873369905 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/973dc2ad-34ed-4364-b01b-9abbf4f2d43b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wqcmj" (UID: "973dc2ad-34ed-4364-b01b-9abbf4f2d43b") : secret "insights-runtime-extractor-tls" not found
Nov 02 12:26:38.156587 ip-10-0-128-132 kubenswrapper[2574]: E1102 12:26:38.156479    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/973dc2ad-34ed-4364-b01b-9abbf4f2d43b-kube-rbac-proxy-cm podName:973dc2ad-34ed-4364-b01b-9abbf4f2d43b nodeName:}" failed. No retries permitted until 2026-11-02 12:26:39.156469908 +0000 UTC m=+36.873412657 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/973dc2ad-34ed-4364-b01b-9abbf4f2d43b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-wqcmj" (UID: "973dc2ad-34ed-4364-b01b-9abbf4f2d43b") : configmap "kube-rbac-proxy" not found
Nov 02 12:26:38.203034 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:38.203003    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-x27dt-dzqvx"]
Nov 02 12:26:38.203697 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:38.203665    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf25513a9_4a23_4cd1_9ef0_b5612826c157.slice/crio-8b3046c39b2696c2009e1ddc2fb1a50edc8fb908c8ff8f6d16198bd9b0c86980 WatchSource:0}: Error finding container 8b3046c39b2696c2009e1ddc2fb1a50edc8fb908c8ff8f6d16198bd9b0c86980: Status 404 returned error can't find the container with id 8b3046c39b2696c2009e1ddc2fb1a50edc8fb908c8ff8f6d16198bd9b0c86980
Nov 02 12:26:38.408427 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:38.408348    2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-fmzqj"]
Nov 02 12:26:38.440952 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:38.440908    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-fmzqj"]
Nov 02 12:26:38.441136 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:38.441033    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-fmzqj"
Nov 02 12:26:38.456912 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:38.456880    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-r2cgm\""
Nov 02 12:26:38.504823 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:38.504789    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Nov 02 12:26:38.505103 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:38.505085    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Nov 02 12:26:38.559886 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:38.559852    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxjdr\" (UniqueName: \"kubernetes.io/projected/a5142bd8-95b7-499e-9885-d867d915f3a8-kube-api-access-nxjdr\") pod \"migrator-66df5d496b-fmzqj\" (UID: \"a5142bd8-95b7-499e-9885-d867d915f3a8\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-fmzqj"
Nov 02 12:26:38.660765 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:38.660344    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxjdr\" (UniqueName: \"kubernetes.io/projected/a5142bd8-95b7-499e-9885-d867d915f3a8-kube-api-access-nxjdr\") pod \"migrator-66df5d496b-fmzqj\" (UID: \"a5142bd8-95b7-499e-9885-d867d915f3a8\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-fmzqj"
Nov 02 12:26:38.701520 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:38.701489    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxjdr\" (UniqueName: \"kubernetes.io/projected/a5142bd8-95b7-499e-9885-d867d915f3a8-kube-api-access-nxjdr\") pod \"migrator-66df5d496b-fmzqj\" (UID: \"a5142bd8-95b7-499e-9885-d867d915f3a8\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-fmzqj"
Nov 02 12:26:38.767035 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:38.766964    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-fmzqj"
Nov 02 12:26:39.087521 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:39.087485    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-x27dt-dzqvx" event={"ID":"f25513a9-4a23-4cd1-9ef0-b5612826c157","Type":"ContainerStarted","Data":"8b3046c39b2696c2009e1ddc2fb1a50edc8fb908c8ff8f6d16198bd9b0c86980"}
Nov 02 12:26:39.089729 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:39.089671    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-fmzqj"]
Nov 02 12:26:39.090979 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:39.090949    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5142bd8_95b7_499e_9885_d867d915f3a8.slice/crio-a066542f5f5c41564795f0bcdfb4c2cb56babdd0144196ea9dea2384eb067da9 WatchSource:0}: Error finding container a066542f5f5c41564795f0bcdfb4c2cb56babdd0144196ea9dea2384eb067da9: Status 404 returned error can't find the container with id a066542f5f5c41564795f0bcdfb4c2cb56babdd0144196ea9dea2384eb067da9
Nov 02 12:26:39.121380 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:39.120317    2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-g7hcq"]
Nov 02 12:26:39.143056 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:39.142995    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-g7hcq"
Nov 02 12:26:39.149924 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:39.149839    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-g7hcq"]
Nov 02 12:26:39.160268 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:39.160044    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Nov 02 12:26:39.160268 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:39.160074    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Nov 02 12:26:39.160268 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:39.160168    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Nov 02 12:26:39.160268 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:39.160208    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Nov 02 12:26:39.160268 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:39.160233    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-vtb4x\""
Nov 02 12:26:39.164940 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:39.164519    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/973dc2ad-34ed-4364-b01b-9abbf4f2d43b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wqcmj\" (UID: \"973dc2ad-34ed-4364-b01b-9abbf4f2d43b\") " pod="openshift-insights/insights-runtime-extractor-wqcmj"
Nov 02 12:26:39.164940 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:39.164584    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/973dc2ad-34ed-4364-b01b-9abbf4f2d43b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wqcmj\" (UID: \"973dc2ad-34ed-4364-b01b-9abbf4f2d43b\") " pod="openshift-insights/insights-runtime-extractor-wqcmj"
Nov 02 12:26:39.164940 ip-10-0-128-132 kubenswrapper[2574]: E1102 12:26:39.164598    2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 02 12:26:39.164940 ip-10-0-128-132 kubenswrapper[2574]: E1102 12:26:39.164676    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/973dc2ad-34ed-4364-b01b-9abbf4f2d43b-kube-rbac-proxy-cm podName:973dc2ad-34ed-4364-b01b-9abbf4f2d43b nodeName:}" failed. No retries permitted until 2026-11-02 12:26:41.164659079 +0000 UTC m=+38.881601844 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/973dc2ad-34ed-4364-b01b-9abbf4f2d43b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-wqcmj" (UID: "973dc2ad-34ed-4364-b01b-9abbf4f2d43b") : configmap "kube-rbac-proxy" not found
Nov 02 12:26:39.164940 ip-10-0-128-132 kubenswrapper[2574]: E1102 12:26:39.164809    2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 02 12:26:39.164940 ip-10-0-128-132 kubenswrapper[2574]: E1102 12:26:39.164912    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/973dc2ad-34ed-4364-b01b-9abbf4f2d43b-insights-runtime-extractor-tls podName:973dc2ad-34ed-4364-b01b-9abbf4f2d43b nodeName:}" failed. No retries permitted until 2026-11-02 12:26:41.164895379 +0000 UTC m=+38.881838137 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/973dc2ad-34ed-4364-b01b-9abbf4f2d43b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wqcmj" (UID: "973dc2ad-34ed-4364-b01b-9abbf4f2d43b") : secret "insights-runtime-extractor-tls" not found
Nov 02 12:26:39.265469 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:39.265438    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/ef387e86-702c-454e-825a-23cbace64952-signing-key\") pod \"service-ca-5fffc97bdf-g7hcq\" (UID: \"ef387e86-702c-454e-825a-23cbace64952\") " pod="openshift-service-ca/service-ca-5fffc97bdf-g7hcq"
Nov 02 12:26:39.265970 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:39.265573    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twvc7\" (UniqueName: \"kubernetes.io/projected/ef387e86-702c-454e-825a-23cbace64952-kube-api-access-twvc7\") pod \"service-ca-5fffc97bdf-g7hcq\" (UID: \"ef387e86-702c-454e-825a-23cbace64952\") " pod="openshift-service-ca/service-ca-5fffc97bdf-g7hcq"
Nov 02 12:26:39.265970 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:39.265619    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/ef387e86-702c-454e-825a-23cbace64952-signing-cabundle\") pod \"service-ca-5fffc97bdf-g7hcq\" (UID: \"ef387e86-702c-454e-825a-23cbace64952\") " pod="openshift-service-ca/service-ca-5fffc97bdf-g7hcq"
Nov 02 12:26:39.366208 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:39.366125    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/ef387e86-702c-454e-825a-23cbace64952-signing-key\") pod \"service-ca-5fffc97bdf-g7hcq\" (UID: \"ef387e86-702c-454e-825a-23cbace64952\") " pod="openshift-service-ca/service-ca-5fffc97bdf-g7hcq"
Nov 02 12:26:39.366208 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:39.366193    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twvc7\" (UniqueName: \"kubernetes.io/projected/ef387e86-702c-454e-825a-23cbace64952-kube-api-access-twvc7\") pod \"service-ca-5fffc97bdf-g7hcq\" (UID: \"ef387e86-702c-454e-825a-23cbace64952\") " pod="openshift-service-ca/service-ca-5fffc97bdf-g7hcq"
Nov 02 12:26:39.366419 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:39.366227    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/ef387e86-702c-454e-825a-23cbace64952-signing-cabundle\") pod \"service-ca-5fffc97bdf-g7hcq\" (UID: \"ef387e86-702c-454e-825a-23cbace64952\") " pod="openshift-service-ca/service-ca-5fffc97bdf-g7hcq"
Nov 02 12:26:39.367009 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:39.366984    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/ef387e86-702c-454e-825a-23cbace64952-signing-cabundle\") pod \"service-ca-5fffc97bdf-g7hcq\" (UID: \"ef387e86-702c-454e-825a-23cbace64952\") " pod="openshift-service-ca/service-ca-5fffc97bdf-g7hcq"
Nov 02 12:26:39.369519 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:39.369475    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/ef387e86-702c-454e-825a-23cbace64952-signing-key\") pod \"service-ca-5fffc97bdf-g7hcq\" (UID: \"ef387e86-702c-454e-825a-23cbace64952\") " pod="openshift-service-ca/service-ca-5fffc97bdf-g7hcq"
Nov 02 12:26:39.393505 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:39.393456    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-twvc7\" (UniqueName: \"kubernetes.io/projected/ef387e86-702c-454e-825a-23cbace64952-kube-api-access-twvc7\") pod \"service-ca-5fffc97bdf-g7hcq\" (UID: \"ef387e86-702c-454e-825a-23cbace64952\") " pod="openshift-service-ca/service-ca-5fffc97bdf-g7hcq"
Nov 02 12:26:39.476521 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:39.476477    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-g7hcq"
Nov 02 12:26:39.830452 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:39.830415    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-g7hcq"]
Nov 02 12:26:39.837010 ip-10-0-128-132 kubenswrapper[2574]: W1102 12:26:39.836975    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef387e86_702c_454e_825a_23cbace64952.slice/crio-3a8c12aa0fbb41c2b4a84b506554dd0f39221224ca2d8fdfe45007c8ef78c18b WatchSource:0}: Error finding container 3a8c12aa0fbb41c2b4a84b506554dd0f39221224ca2d8fdfe45007c8ef78c18b: Status 404 returned error can't find the container with id 3a8c12aa0fbb41c2b4a84b506554dd0f39221224ca2d8fdfe45007c8ef78c18b
Nov 02 12:26:39.973354 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:39.972699    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7b688512-ffc4-4b6a-b9ab-8fcf8ea0afe2-metrics-tls\") pod \"dns-default-bg5xh\" (UID: \"7b688512-ffc4-4b6a-b9ab-8fcf8ea0afe2\") " pod="openshift-dns/dns-default-bg5xh"
Nov 02 12:26:39.973354 ip-10-0-128-132 kubenswrapper[2574]: E1102 12:26:39.972930    2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 02 12:26:39.973354 ip-10-0-128-132 kubenswrapper[2574]: E1102 12:26:39.972996    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7b688512-ffc4-4b6a-b9ab-8fcf8ea0afe2-metrics-tls podName:7b688512-ffc4-4b6a-b9ab-8fcf8ea0afe2 nodeName:}" failed. No retries permitted until 2026-11-02 12:26:47.972978221 +0000 UTC m=+45.689920970 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7b688512-ffc4-4b6a-b9ab-8fcf8ea0afe2-metrics-tls") pod "dns-default-bg5xh" (UID: "7b688512-ffc4-4b6a-b9ab-8fcf8ea0afe2") : secret "dns-default-metrics-tls" not found
Nov 02 12:26:40.092982 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:40.092857    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-g7hcq" event={"ID":"ef387e86-702c-454e-825a-23cbace64952","Type":"ContainerStarted","Data":"3a8c12aa0fbb41c2b4a84b506554dd0f39221224ca2d8fdfe45007c8ef78c18b"}
Nov 02 12:26:40.094464 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:40.094410    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-fmzqj" event={"ID":"a5142bd8-95b7-499e-9885-d867d915f3a8","Type":"ContainerStarted","Data":"a066542f5f5c41564795f0bcdfb4c2cb56babdd0144196ea9dea2384eb067da9"}
Nov 02 12:26:41.099482 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:41.099016    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-fmzqj" event={"ID":"a5142bd8-95b7-499e-9885-d867d915f3a8","Type":"ContainerStarted","Data":"a197e0ce12a5beed7d451fc04906316614aec6e5a428a52e9abd90398cf1afe8"}
Nov 02 12:26:41.140073 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:41.140002    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-fmzqj" podStartSLOduration=1.386264142 podStartE2EDuration="3.139981612s" totalImagesPullingTime="1.75371747s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:26:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:26:41.139138622 +0000 UTC m=+38.856081393" watchObservedRunningTime="2026-11-02 12:26:41.139981612 +0000 UTC m=+38.856924386"
Nov 02 12:26:41.184494 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:41.183708    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/973dc2ad-34ed-4364-b01b-9abbf4f2d43b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wqcmj\" (UID: \"973dc2ad-34ed-4364-b01b-9abbf4f2d43b\") " pod="openshift-insights/insights-runtime-extractor-wqcmj"
Nov 02 12:26:41.184494 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:41.183800    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/973dc2ad-34ed-4364-b01b-9abbf4f2d43b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wqcmj\" (UID: \"973dc2ad-34ed-4364-b01b-9abbf4f2d43b\") " pod="openshift-insights/insights-runtime-extractor-wqcmj"
Nov 02 12:26:41.184494 ip-10-0-128-132 kubenswrapper[2574]: E1102 12:26:41.183835    2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 02 12:26:41.184494 ip-10-0-128-132 kubenswrapper[2574]: E1102 12:26:41.183920    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/973dc2ad-34ed-4364-b01b-9abbf4f2d43b-kube-rbac-proxy-cm podName:973dc2ad-34ed-4364-b01b-9abbf4f2d43b nodeName:}" failed. No retries permitted until 2026-11-02 12:26:45.183898948 +0000 UTC m=+42.900841700 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/973dc2ad-34ed-4364-b01b-9abbf4f2d43b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-wqcmj" (UID: "973dc2ad-34ed-4364-b01b-9abbf4f2d43b") : configmap "kube-rbac-proxy" not found
Nov 02 12:26:41.184494 ip-10-0-128-132 kubenswrapper[2574]: E1102 12:26:41.183937    2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 02 12:26:41.184494 ip-10-0-128-132 kubenswrapper[2574]: E1102 12:26:41.183996    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/973dc2ad-34ed-4364-b01b-9abbf4f2d43b-insights-runtime-extractor-tls podName:973dc2ad-34ed-4364-b01b-9abbf4f2d43b nodeName:}" failed. No retries permitted until 2026-11-02 12:26:45.183982411 +0000 UTC m=+42.900925161 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/973dc2ad-34ed-4364-b01b-9abbf4f2d43b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wqcmj" (UID: "973dc2ad-34ed-4364-b01b-9abbf4f2d43b") : secret "insights-runtime-extractor-tls" not found
Nov 02 12:26:42.108658 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:42.108599    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-x27dt-dzqvx" event={"ID":"f25513a9-4a23-4cd1-9ef0-b5612826c157","Type":"ContainerStarted","Data":"1d56a78914255a1f48d2aa755c2d56b4537c24f5d181186d3d082a8ac4a7466d"}
Nov 02 12:26:42.110757 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:42.110722    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-wx6zw" event={"ID":"089d7bd4-d019-471c-960f-51346831fe45","Type":"ContainerStarted","Data":"5241fd79019097e1fe6d70cfac5a131258338ade176714fedce54529c32b5bb9"}
Nov 02 12:26:42.158936 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:42.158870    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-x27dt-dzqvx" podStartSLOduration=1.806710619 podStartE2EDuration="5.158856613s" totalImagesPullingTime="3.352145994s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:26:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:26:42.157773428 +0000 UTC m=+39.874716198" watchObservedRunningTime="2026-11-02 12:26:42.158856613 +0000 UTC m=+39.875799382"
Nov 02 12:26:42.198827 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:42.198052    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-wx6zw" podStartSLOduration=34.845712942 podStartE2EDuration="39.198036472s" totalImagesPullingTime="4.35232353s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:26:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:26:42.197523628 +0000 UTC m=+39.914466397" watchObservedRunningTime="2026-11-02 12:26:42.198036472 +0000 UTC m=+39.914979244"
Nov 02 12:26:43.113781 ip-10-0-128-132 kubenswrapper[2574]: I1102 12:26:43.113747    2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-wx6zw"
Nov 02 12:26:43.672440 ip-10-0-128-132 systemd[1]: Stopping Kubernetes Kubelet...
Nov 02 12:26:43.688023 ip-10-0-128-132 systemd[1]: kubelet.service: Deactivated successfully.
Nov 02 12:26:43.688198 ip-10-0-128-132 systemd[1]: Stopped Kubernetes Kubelet.
Nov 02 12:26:43.688249 ip-10-0-128-132 systemd[1]: kubelet.service: Consumed 2.487s CPU time.
Nov 02 12:26:43.703257 ip-10-0-128-132 systemd[1]: Starting Kubernetes Kubelet...
Nov 02 12:26:43.805451 ip-10-0-128-132 kubenswrapper[4559]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 02 12:26:43.805451 ip-10-0-128-132 kubenswrapper[4559]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 02 12:26:43.805451 ip-10-0-128-132 kubenswrapper[4559]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 02 12:26:43.805451 ip-10-0-128-132 kubenswrapper[4559]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 02 12:26:43.807692 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.807578    4559 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 12:26:43.807692 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.807693    4559 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 12:26:43.807774 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.807697    4559 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 12:26:43.807774 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.807701    4559 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 12:26:43.807774 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.807703    4559 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 12:26:43.807774 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.807707    4559 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 12:26:43.807774 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.807709    4559 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 12:26:43.807774 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.807712    4559 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 12:26:43.807774 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.807715    4559 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 12:26:43.807774 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.807717    4559 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 12:26:43.807774 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.807720    4559 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 12:26:43.807774 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.807722    4559 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 12:26:43.807774 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.807725    4559 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 12:26:43.807774 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.807728    4559 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 12:26:43.807774 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.807731    4559 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 12:26:43.807774 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.807733    4559 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 12:26:43.807774 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.807736    4559 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 12:26:43.807774 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.807739    4559 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 12:26:43.807774 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.807742    4559 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 12:26:43.807774 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.807744    4559 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 12:26:43.807774 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.807747    4559 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 12:26:43.807774 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.807750    4559 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 12:26:43.808310 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.807753    4559 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 12:26:43.808310 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.807767    4559 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 12:26:43.808310 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.807770    4559 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 12:26:43.808310 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.807773    4559 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 12:26:43.808310 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.807776    4559 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 12:26:43.808310 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.807780    4559 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 12:26:43.808310 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.807782    4559 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 12:26:43.808310 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.807785    4559 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 12:26:43.808310 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.807788    4559 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 12:26:43.808310 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.807793    4559 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 12:26:43.808310 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.807797    4559 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 12:26:43.808310 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.807800    4559 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 12:26:43.808310 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.807803    4559 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 12:26:43.808310 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.807806    4559 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 12:26:43.808310 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.807809    4559 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 12:26:43.808310 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.807812    4559 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 12:26:43.808310 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.807815    4559 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 12:26:43.808310 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.807818    4559 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 12:26:43.808310 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.807821    4559 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 12:26:43.808918 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.807824    4559 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 12:26:43.808918 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.807826    4559 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 12:26:43.808918 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.807829    4559 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 12:26:43.808918 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.807831    4559 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 12:26:43.808918 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.807834    4559 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 12:26:43.808918 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.807836    4559 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 12:26:43.808918 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.807839    4559 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 12:26:43.808918 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.807841    4559 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 12:26:43.808918 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.807844    4559 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 12:26:43.808918 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.807847    4559 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 12:26:43.808918 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.807850    4559 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 12:26:43.808918 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.807852    4559 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 12:26:43.808918 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.807855    4559 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 12:26:43.808918 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.807859    4559 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 12:26:43.808918 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.807864    4559 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 12:26:43.808918 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.807876    4559 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 12:26:43.808918 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.807880    4559 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 12:26:43.808918 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.807883    4559 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 12:26:43.808918 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.807885    4559 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 12:26:43.809427 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.807888    4559 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 12:26:43.809427 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.807891    4559 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 12:26:43.809427 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.807893    4559 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 12:26:43.809427 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.807896    4559 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 12:26:43.809427 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.807898    4559 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 12:26:43.809427 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.807901    4559 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 12:26:43.809427 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.807904    4559 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 12:26:43.809427 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.807906    4559 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 12:26:43.809427 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.807909    4559 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 12:26:43.809427 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.807911    4559 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 12:26:43.809427 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.807914    4559 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 12:26:43.809427 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.807916    4559 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 12:26:43.809427 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.807919    4559 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 12:26:43.809427 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.807922    4559 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 12:26:43.809427 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.807924    4559 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 12:26:43.809427 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.807927    4559 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 12:26:43.809427 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.807930    4559 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 12:26:43.809427 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.807932    4559 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 12:26:43.809427 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.807935    4559 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 12:26:43.809427 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.807941    4559 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 12:26:43.809960 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.807943    4559 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 12:26:43.809960 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.807947    4559 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 12:26:43.809960 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.807950    4559 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 12:26:43.809960 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.807952    4559 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 12:26:43.809960 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.807955    4559 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 12:26:43.809960 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.807957    4559 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 12:26:43.809960 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.807960    4559 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 12:26:43.809960 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.807963    4559 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 12:26:43.809960 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.807965    4559 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 12:26:43.809960 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.807969    4559 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 12:26:43.809960 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.807972    4559 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 12:26:43.809960 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.807975    4559 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 12:26:43.809960 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.807977    4559 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 12:26:43.809960 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.807980    4559 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 12:26:43.809960 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.807983    4559 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 12:26:43.809960 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.807985    4559 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 12:26:43.809960 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.807987    4559 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 12:26:43.809960 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.807990    4559 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 12:26:43.809960 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.807993    4559 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 12:26:43.810430 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.807995    4559 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 12:26:43.810430 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.807998    4559 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 12:26:43.810430 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.808001    4559 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 12:26:43.810430 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.808003    4559 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 12:26:43.810430 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.808006    4559 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 12:26:43.810430 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.808008    4559 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 12:26:43.810430 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.808011    4559 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 12:26:43.810430 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.808807    4559 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 12:26:43.810430 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.808815    4559 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 12:26:43.810430 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.808818    4559 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 12:26:43.810430 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.808822    4559 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 12:26:43.810430 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.808824    4559 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 12:26:43.810430 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.808829    4559 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 12:26:43.810430 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.808832    4559 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 12:26:43.810430 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.808835    4559 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 12:26:43.810430 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.808840    4559 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 12:26:43.810430 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.808844    4559 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 12:26:43.810430 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.808848    4559 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 12:26:43.810430 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.808851    4559 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 12:26:43.810922 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.808854    4559 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 12:26:43.810922 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.808858    4559 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 12:26:43.810922 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.808861    4559 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 12:26:43.810922 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.808863    4559 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 12:26:43.810922 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.808867    4559 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 12:26:43.810922 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.808870    4559 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 12:26:43.810922 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.808872    4559 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 12:26:43.810922 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.808875    4559 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 12:26:43.810922 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.808878    4559 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 12:26:43.810922 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.808880    4559 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 12:26:43.810922 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.808883    4559 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 12:26:43.810922 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.808886    4559 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 12:26:43.810922 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.808888    4559 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 12:26:43.810922 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.808891    4559 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 12:26:43.810922 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.808893    4559 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 12:26:43.810922 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.808896    4559 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 12:26:43.810922 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.808898    4559 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 12:26:43.810922 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.808901    4559 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 12:26:43.810922 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.808904    4559 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 12:26:43.810922 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.808906    4559 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 12:26:43.811426 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.808909    4559 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 12:26:43.811426 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.808911    4559 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 12:26:43.811426 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.808914    4559 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 12:26:43.811426 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.808916    4559 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 12:26:43.811426 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.808919    4559 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 12:26:43.811426 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.808923    4559 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 12:26:43.811426 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.808926    4559 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 12:26:43.811426 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.808929    4559 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 12:26:43.811426 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.808931    4559 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 12:26:43.811426 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.808934    4559 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 12:26:43.811426 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.808937    4559 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 12:26:43.811426 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.808940    4559 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 12:26:43.811426 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.808943    4559 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 12:26:43.811426 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.808946    4559 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 12:26:43.811426 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.808948    4559 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 12:26:43.811426 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.808951    4559 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 12:26:43.811426 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.808954    4559 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 12:26:43.811426 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.808957    4559 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 12:26:43.811426 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.808960    4559 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 12:26:43.811426 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.808963    4559 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 12:26:43.811945 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.808965    4559 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 12:26:43.811945 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.808968    4559 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 12:26:43.811945 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.808971    4559 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 12:26:43.811945 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.808974    4559 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 12:26:43.811945 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.808977    4559 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 12:26:43.811945 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.808989    4559 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 12:26:43.811945 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.808992    4559 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 12:26:43.811945 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.808995    4559 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 12:26:43.811945 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.808998    4559 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 12:26:43.811945 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.809000    4559 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 12:26:43.811945 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.809003    4559 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 12:26:43.811945 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.809006    4559 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 12:26:43.811945 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.809008    4559 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 12:26:43.811945 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.809011    4559 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 12:26:43.811945 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.809014    4559 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 12:26:43.811945 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.809016    4559 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 12:26:43.811945 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.809019    4559 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 12:26:43.811945 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.809022    4559 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 12:26:43.811945 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.809025    4559 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 12:26:43.811945 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.809027    4559 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 12:26:43.812464 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.809030    4559 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 12:26:43.812464 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.809032    4559 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 12:26:43.812464 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.809035    4559 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 12:26:43.812464 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.809039    4559 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 12:26:43.812464 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.809042    4559 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 12:26:43.812464 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.809044    4559 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 12:26:43.812464 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.809047    4559 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 12:26:43.812464 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.809050    4559 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 12:26:43.812464 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.809052    4559 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 12:26:43.812464 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.809055    4559 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 12:26:43.812464 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.809058    4559 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 12:26:43.812464 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.809061    4559 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 12:26:43.812464 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.809064    4559 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 12:26:43.812464 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.809066    4559 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 12:26:43.812464 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.809069    4559 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 12:26:43.812464 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.809072    4559 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 12:26:43.812464 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.809074    4559 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 12:26:43.812464 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.809077    4559 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 12:26:43.812464 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.809079    4559 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 12:26:43.812940 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.809082    4559 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 12:26:43.812940 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.809084    4559 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 12:26:43.812940 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.809087    4559 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 12:26:43.812940 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.809090    4559 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 12:26:43.812940 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.809092    4559 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 12:26:43.812940 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.809095    4559 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 12:26:43.812940 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.809098    4559 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 12:26:43.812940 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.809101    4559 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 12:26:43.812940 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.809103    4559 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 12:26:43.812940 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.809106    4559 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 12:26:43.812940 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.809108    4559 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 12:26:43.812940 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.809111    4559 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 12:26:43.812940 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.809114    4559 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 12:26:43.812940 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.809118    4559 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 12:26:43.812940 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.809122    4559 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 12:26:43.812940 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.809663    4559 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 12:26:43.812940 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.809674    4559 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 12:26:43.812940 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.809681    4559 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 12:26:43.812940 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.809685    4559 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 12:26:43.813413 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.809688    4559 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 12:26:43.813413 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.809691    4559 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 12:26:43.813413 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.809694    4559 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 12:26:43.813413 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.809697    4559 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 12:26:43.813413 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.809700    4559 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 12:26:43.813413 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.809703    4559 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 12:26:43.813413 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.809706    4559 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 12:26:43.813413 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.809708    4559 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 12:26:43.813413 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.809711    4559 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 12:26:43.813413 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.809714    4559 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 12:26:43.813413 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.809716    4559 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 12:26:43.813413 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.809720    4559 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 12:26:43.813413 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.809724    4559 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 12:26:43.813413 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.809728    4559 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 12:26:43.813413 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.809732    4559 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 12:26:43.813413 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.809734    4559 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 12:26:43.813413 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.809737    4559 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 12:26:43.813413 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.809740    4559 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 12:26:43.813413 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.809742    4559 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 12:26:43.814156 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.809745    4559 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 12:26:43.814156 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.809748    4559 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 12:26:43.814156 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.809750    4559 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 12:26:43.814156 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.809753    4559 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 12:26:43.814156 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.809756    4559 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 12:26:43.814156 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.809758    4559 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 12:26:43.814156 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.809761    4559 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 12:26:43.814156 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.809764    4559 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 12:26:43.814156 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.809766    4559 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 12:26:43.814156 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.809769    4559 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 12:26:43.814156 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.809774    4559 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 12:26:43.814156 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.809777    4559 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 12:26:43.814156 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.809779    4559 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 12:26:43.814156 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.809782    4559 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 12:26:43.814156 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.809785    4559 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 12:26:43.814156 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.809787    4559 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 12:26:43.814156 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.809790    4559 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 12:26:43.814156 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.809793    4559 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 12:26:43.814156 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.809796    4559 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 12:26:43.814156 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.809799    4559 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 12:26:43.814970 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.809801    4559 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 12:26:43.814970 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.809804    4559 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 12:26:43.814970 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.809806    4559 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 12:26:43.814970 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.809809    4559 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 12:26:43.814970 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.809812    4559 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 12:26:43.814970 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.809814    4559 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 12:26:43.814970 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.809818    4559 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 12:26:43.814970 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.809820    4559 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 12:26:43.814970 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.809823    4559 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 12:26:43.814970 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.809826    4559 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 12:26:43.814970 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.809828    4559 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 12:26:43.814970 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.809831    4559 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 12:26:43.814970 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.809834    4559 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 12:26:43.814970 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.809837    4559 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 12:26:43.814970 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.809841    4559 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 12:26:43.814970 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.809844    4559 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 12:26:43.814970 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.809847    4559 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 12:26:43.814970 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.809849    4559 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 12:26:43.814970 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.809852    4559 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 12:26:43.815494 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.809855    4559 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 12:26:43.815494 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.809857    4559 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 12:26:43.815494 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.809860    4559 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 12:26:43.815494 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.809864    4559 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 12:26:43.815494 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.809867    4559 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 12:26:43.815494 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.809869    4559 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 12:26:43.815494 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.809872    4559 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 12:26:43.815494 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.809874    4559 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 12:26:43.815494 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.809877    4559 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 12:26:43.815494 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.809879    4559 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 12:26:43.815494 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.809882    4559 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 12:26:43.815494 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.809884    4559 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 12:26:43.815494 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.809888    4559 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 12:26:43.815494 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.809890    4559 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 12:26:43.815494 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.809893    4559 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 12:26:43.815494 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.809897    4559 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 12:26:43.815494 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.809899    4559 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 12:26:43.815494 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.809902    4559 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 12:26:43.815494 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.809904    4559 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 12:26:43.815494 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.809910    4559 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 12:26:43.816129 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.809912    4559 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 12:26:43.816129 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.809915    4559 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 12:26:43.816129 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.809917    4559 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 12:26:43.816129 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.809920    4559 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 12:26:43.816129 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.809923    4559 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 12:26:43.816129 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.809925    4559 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 12:26:43.816129 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.809928    4559 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 12:26:43.816129 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.809930    4559 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 12:26:43.816129 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.809933    4559 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 12:26:43.816129 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.809936    4559 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 12:26:43.816129 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.809939    4559 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 12:26:43.816129 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.809941    4559 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 12:26:43.816129 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.809944    4559 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 12:26:43.816129 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.809946    4559 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 12:26:43.816129 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.809949    4559 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 12:26:43.816129 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.809953    4559 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 12:26:43.816129 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.809956    4559 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 12:26:43.816129 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.809959    4559 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 12:26:43.816129 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.809961    4559 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 12:26:43.816129 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.809964    4559 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 12:26:43.816651 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.809967    4559 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 12:26:43.816651 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.809970    4559 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 12:26:43.816651 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.809972    4559 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 12:26:43.816651 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.809975    4559 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 12:26:43.816651 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.810065    4559 flags.go:64] FLAG: --address="0.0.0.0"
Nov 02 12:26:43.816651 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.810074    4559 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 02 12:26:43.816651 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.810080    4559 flags.go:64] FLAG: --anonymous-auth="true"
Nov 02 12:26:43.816651 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.810085    4559 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 02 12:26:43.816651 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.810090    4559 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 02 12:26:43.816651 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.810093    4559 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 02 12:26:43.816651 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.810098    4559 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 02 12:26:43.816651 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.810104    4559 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 02 12:26:43.816651 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.810107    4559 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 02 12:26:43.816651 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.810111    4559 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 02 12:26:43.816651 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.810114    4559 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 02 12:26:43.816651 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.810118    4559 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 02 12:26:43.816651 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.810121    4559 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 02 12:26:43.816651 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.810125    4559 flags.go:64] FLAG: --cgroup-root=""
Nov 02 12:26:43.816651 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.810128    4559 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 02 12:26:43.816651 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.810131    4559 flags.go:64] FLAG: --client-ca-file=""
Nov 02 12:26:43.816651 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.810134    4559 flags.go:64] FLAG: --cloud-provider="external"
Nov 02 12:26:43.816651 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.810137    4559 flags.go:64] FLAG: --cluster-dns="[]"
Nov 02 12:26:43.816651 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.810142    4559 flags.go:64] FLAG: --cluster-domain=""
Nov 02 12:26:43.817218 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.810145    4559 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 02 12:26:43.817218 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.810148    4559 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 02 12:26:43.817218 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.810151    4559 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 02 12:26:43.817218 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.810155    4559 flags.go:64] FLAG: --container-log-max-files="5"
Nov 02 12:26:43.817218 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.810161    4559 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 02 12:26:43.817218 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.810164    4559 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 02 12:26:43.817218 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.810167    4559 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 02 12:26:43.817218 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.810171    4559 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 02 12:26:43.817218 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.810174    4559 flags.go:64] FLAG: --contention-profiling="false"
Nov 02 12:26:43.817218 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.810177    4559 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 02 12:26:43.817218 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.810180    4559 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 02 12:26:43.817218 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.810184    4559 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 02 12:26:43.817218 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.810187    4559 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 02 12:26:43.817218 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.810192    4559 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 02 12:26:43.817218 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.810195    4559 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 02 12:26:43.817218 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.810198    4559 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 02 12:26:43.817218 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.810201    4559 flags.go:64] FLAG: --enable-load-reader="false"
Nov 02 12:26:43.817218 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.810204    4559 flags.go:64] FLAG: --enable-server="true"
Nov 02 12:26:43.817218 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.810207    4559 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 02 12:26:43.817218 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.810212    4559 flags.go:64] FLAG: --event-burst="100"
Nov 02 12:26:43.817218 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.810217    4559 flags.go:64] FLAG: --event-qps="50"
Nov 02 12:26:43.817218 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.810220    4559 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 02 12:26:43.817218 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.810223    4559 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 02 12:26:43.817218 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.810226    4559 flags.go:64] FLAG: --eviction-hard=""
Nov 02 12:26:43.817881 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.810231    4559 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 02 12:26:43.817881 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.810234    4559 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 02 12:26:43.817881 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.810237    4559 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 02 12:26:43.817881 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.810240    4559 flags.go:64] FLAG: --eviction-soft=""
Nov 02 12:26:43.817881 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.810244    4559 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 02 12:26:43.817881 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.810247    4559 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 02 12:26:43.817881 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.810250    4559 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 02 12:26:43.817881 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.810254    4559 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 02 12:26:43.817881 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.810257    4559 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 02 12:26:43.817881 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.810260    4559 flags.go:64] FLAG: --fail-swap-on="true"
Nov 02 12:26:43.817881 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.810263    4559 flags.go:64] FLAG: --feature-gates=""
Nov 02 12:26:43.817881 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.810267    4559 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 02 12:26:43.817881 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.810271    4559 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 02 12:26:43.817881 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.810274    4559 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 02 12:26:43.817881 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.810278    4559 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 02 12:26:43.817881 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.810281    4559 flags.go:64] FLAG: --healthz-port="10248"
Nov 02 12:26:43.817881 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.810284    4559 flags.go:64] FLAG: --help="false"
Nov 02 12:26:43.817881 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.810287    4559 flags.go:64] FLAG: --hostname-override="ip-10-0-128-132.ec2.internal"
Nov 02 12:26:43.817881 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.810291    4559 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 02 12:26:43.817881 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.810294    4559 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 02 12:26:43.817881 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.810297    4559 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 02 12:26:43.817881 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.810301    4559 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 02 12:26:43.817881 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.810304    4559 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 02 12:26:43.818485 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.810307    4559 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 02 12:26:43.818485 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.810310    4559 flags.go:64] FLAG: --image-service-endpoint=""
Nov 02 12:26:43.818485 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.810314    4559 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 02 12:26:43.818485 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.810317    4559 flags.go:64] FLAG: --kube-api-burst="100"
Nov 02 12:26:43.818485 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.810320    4559 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 02 12:26:43.818485 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.810325    4559 flags.go:64] FLAG: --kube-api-qps="50"
Nov 02 12:26:43.818485 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.810328    4559 flags.go:64] FLAG: --kube-reserved=""
Nov 02 12:26:43.818485 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.810331    4559 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 02 12:26:43.818485 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.810334    4559 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 02 12:26:43.818485 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.810337    4559 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 02 12:26:43.818485 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.810340    4559 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 02 12:26:43.818485 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.810342    4559 flags.go:64] FLAG: --lock-file=""
Nov 02 12:26:43.818485 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.810345    4559 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 02 12:26:43.818485 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.810348    4559 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 02 12:26:43.818485 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.810351    4559 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 02 12:26:43.818485 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.810357    4559 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 02 12:26:43.818485 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.810360    4559 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 02 12:26:43.818485 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.810363    4559 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 02 12:26:43.818485 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.810366    4559 flags.go:64] FLAG: --logging-format="text"
Nov 02 12:26:43.818485 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.810370    4559 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 02 12:26:43.818485 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.810373    4559 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 02 12:26:43.818485 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.810377    4559 flags.go:64] FLAG: --manifest-url=""
Nov 02 12:26:43.818485 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.810380    4559 flags.go:64] FLAG: --manifest-url-header=""
Nov 02 12:26:43.818485 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.810385    4559 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 02 12:26:43.818485 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.810388    4559 flags.go:64] FLAG: --max-open-files="1000000"
Nov 02 12:26:43.819136 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.810392    4559 flags.go:64] FLAG: --max-pods="110"
Nov 02 12:26:43.819136 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.810395    4559 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 02 12:26:43.819136 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.810399    4559 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 02 12:26:43.819136 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.810402    4559 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 02 12:26:43.819136 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.810405    4559 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 02 12:26:43.819136 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.810408    4559 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 02 12:26:43.819136 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.810411    4559 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 02 12:26:43.819136 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.810414    4559 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 02 12:26:43.819136 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.810421    4559 flags.go:64] FLAG: --node-status-max-images="50"
Nov 02 12:26:43.819136 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.810426    4559 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 02 12:26:43.819136 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.810429    4559 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 02 12:26:43.819136 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.810432    4559 flags.go:64] FLAG: --pod-cidr=""
Nov 02 12:26:43.819136 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.810437    4559 flags.go:64] FLAG: --pod-manifest-path=""
Nov 02 12:26:43.819136 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.810440    4559 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 02 12:26:43.819136 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.810444    4559 flags.go:64] FLAG: --pods-per-core="0"
Nov 02 12:26:43.819136 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.810447    4559 flags.go:64] FLAG: --port="10250"
Nov 02 12:26:43.819136 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.810450    4559 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 02 12:26:43.819136 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.810453    4559 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-082bd4ecc28470df4"
Nov 02 12:26:43.819136 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.810457    4559 flags.go:64] FLAG: --qos-reserved=""
Nov 02 12:26:43.819136 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.810460    4559 flags.go:64] FLAG: --read-only-port="10255"
Nov 02 12:26:43.819136 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.810463    4559 flags.go:64] FLAG: --register-node="true"
Nov 02 12:26:43.819136 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.810466    4559 flags.go:64] FLAG: --register-with-taints=""
Nov 02 12:26:43.819136 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.810470    4559 flags.go:64] FLAG: --registry-burst="10"
Nov 02 12:26:43.819136 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.810473    4559 flags.go:64] FLAG: --registry-qps="5"
Nov 02 12:26:43.819136 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.810476    4559 flags.go:64] FLAG: --reserved-cpus=""
Nov 02 12:26:43.819136 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.810480    4559 flags.go:64] FLAG: --reserved-memory=""
Nov 02 12:26:43.819903 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.810485    4559 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 02 12:26:43.819903 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.810488    4559 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 02 12:26:43.819903 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.810492    4559 flags.go:64] FLAG: --rotate-certificates="false"
Nov 02 12:26:43.819903 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.810495    4559 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 02 12:26:43.819903 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.810498    4559 flags.go:64] FLAG: --runonce="false"
Nov 02 12:26:43.819903 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.810501    4559 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 02 12:26:43.819903 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.810504    4559 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 02 12:26:43.819903 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.810507    4559 flags.go:64] FLAG: --seccomp-default="false"
Nov 02 12:26:43.819903 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.810510    4559 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 02 12:26:43.819903 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.810513    4559 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 02 12:26:43.819903 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.810516    4559 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 02 12:26:43.819903 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.810519    4559 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 02 12:26:43.819903 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.810522    4559 flags.go:64] FLAG: --storage-driver-password="root"
Nov 02 12:26:43.819903 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.810525    4559 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 02 12:26:43.819903 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.810528    4559 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 02 12:26:43.819903 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.810531    4559 flags.go:64] FLAG: --storage-driver-user="root"
Nov 02 12:26:43.819903 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.810535    4559 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 02 12:26:43.819903 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.810538    4559 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 02 12:26:43.819903 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.810543    4559 flags.go:64] FLAG: --system-cgroups=""
Nov 02 12:26:43.819903 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.810546    4559 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 02 12:26:43.819903 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.810551    4559 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 02 12:26:43.819903 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.810555    4559 flags.go:64] FLAG: --tls-cert-file=""
Nov 02 12:26:43.819903 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.810557    4559 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 02 12:26:43.819903 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.810562    4559 flags.go:64] FLAG: --tls-min-version=""
Nov 02 12:26:43.819903 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.810565    4559 flags.go:64] FLAG: --tls-private-key-file=""
Nov 02 12:26:43.820528 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.810568    4559 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 02 12:26:43.820528 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.810571    4559 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 02 12:26:43.820528 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.810574    4559 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 02 12:26:43.820528 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.810577    4559 flags.go:64] FLAG: --v="2"
Nov 02 12:26:43.820528 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.810582    4559 flags.go:64] FLAG: --version="false"
Nov 02 12:26:43.820528 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.810586    4559 flags.go:64] FLAG: --vmodule=""
Nov 02 12:26:43.820528 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.810590    4559 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 02 12:26:43.820528 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.810594    4559 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 02 12:26:43.820528 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.810770    4559 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 12:26:43.820528 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.810775    4559 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 12:26:43.820528 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.810779    4559 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 12:26:43.820528 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.810782    4559 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 12:26:43.820528 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.810785    4559 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 12:26:43.820528 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.810788    4559 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 12:26:43.820528 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.810791    4559 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 12:26:43.820528 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.810793    4559 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 12:26:43.820528 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.810796    4559 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 12:26:43.820528 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.810798    4559 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 12:26:43.820528 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.810801    4559 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 12:26:43.820528 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.810804    4559 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 12:26:43.820528 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.810806    4559 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 12:26:43.821154 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.810810    4559 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 12:26:43.821154 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.810814    4559 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 12:26:43.821154 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.810818    4559 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 12:26:43.821154 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.810821    4559 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 12:26:43.821154 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.810827    4559 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 12:26:43.821154 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.810830    4559 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 12:26:43.821154 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.810832    4559 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 12:26:43.821154 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.810835    4559 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 12:26:43.821154 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.810839    4559 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 12:26:43.821154 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.810842    4559 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 12:26:43.821154 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.810846    4559 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 12:26:43.821154 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.810849    4559 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 12:26:43.821154 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.810852    4559 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 12:26:43.821154 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.810855    4559 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 12:26:43.821154 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.810857    4559 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 12:26:43.821154 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.810860    4559 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 12:26:43.821154 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.810863    4559 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 12:26:43.821154 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.810865    4559 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 12:26:43.821154 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.810868    4559 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 12:26:43.821931 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.810870    4559 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 12:26:43.821931 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.810873    4559 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 12:26:43.821931 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.810876    4559 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 12:26:43.821931 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.810879    4559 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 12:26:43.821931 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.810881    4559 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 12:26:43.821931 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.810884    4559 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 12:26:43.821931 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.810886    4559 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 12:26:43.821931 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.810889    4559 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 12:26:43.821931 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.810892    4559 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 12:26:43.821931 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.810894    4559 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 12:26:43.821931 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.810897    4559 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 12:26:43.821931 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.810899    4559 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 12:26:43.821931 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.810902    4559 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 12:26:43.821931 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.810904    4559 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 12:26:43.821931 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.810907    4559 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 12:26:43.821931 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.810909    4559 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 12:26:43.821931 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.810912    4559 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 12:26:43.821931 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.810916    4559 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 12:26:43.821931 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.810919    4559 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 12:26:43.822553 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.810922    4559 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 12:26:43.822553 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.810925    4559 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 12:26:43.822553 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.810927    4559 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 12:26:43.822553 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.810930    4559 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 12:26:43.822553 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.810933    4559 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 12:26:43.822553 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.810935    4559 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 12:26:43.822553 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.810938    4559 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 12:26:43.822553 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.810941    4559 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 12:26:43.822553 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.810943    4559 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 12:26:43.822553 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.810946    4559 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 12:26:43.822553 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.810948    4559 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 12:26:43.822553 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.810951    4559 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 12:26:43.822553 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.810953    4559 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 12:26:43.822553 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.810956    4559 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 12:26:43.822553 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.810959    4559 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 12:26:43.822553 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.810961    4559 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 12:26:43.822553 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.810963    4559 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 12:26:43.822553 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.810966    4559 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 12:26:43.822553 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.810969    4559 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 12:26:43.822553 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.810971    4559 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 12:26:43.823174 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.810974    4559 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 12:26:43.823174 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.810976    4559 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 12:26:43.823174 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.810979    4559 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 12:26:43.823174 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.810981    4559 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 12:26:43.823174 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.810984    4559 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 12:26:43.823174 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.810986    4559 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 12:26:43.823174 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.810988    4559 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 12:26:43.823174 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.810991    4559 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 12:26:43.823174 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.810993    4559 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 12:26:43.823174 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.810996    4559 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 12:26:43.823174 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.811000    4559 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 12:26:43.823174 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.811003    4559 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 12:26:43.823174 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.811006    4559 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 12:26:43.823174 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.811008    4559 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 12:26:43.823174 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.811011    4559 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 12:26:43.823174 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.811013    4559 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 12:26:43.823174 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.811016    4559 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 12:26:43.823174 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.811018    4559 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 12:26:43.823174 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.811023    4559 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 12:26:43.823174 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.811025    4559 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 12:26:43.824138 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.811028    4559 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 12:26:43.824138 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.811031    4559 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 12:26:43.824138 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.811033    4559 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 12:26:43.824138 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.811036    4559 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 12:26:43.824138 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.811039    4559 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 12:26:43.824138 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.811042    4559 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 12:26:43.824138 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.811044    4559 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 12:26:43.824138 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.811047    4559 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 12:26:43.824138 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.811049    4559 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 12:26:43.824138 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.811052    4559 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 12:26:43.824138 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.811054    4559 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 12:26:43.824138 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.811057    4559 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 12:26:43.824138 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.811059    4559 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 12:26:43.824138 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.811062    4559 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 12:26:43.824138 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.811065    4559 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 12:26:43.824138 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.811090    4559 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 02 12:26:43.824718 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.811131    4559 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 12:26:43.824718 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.811136    4559 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 12:26:43.824718 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.811139    4559 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 12:26:43.824718 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.811141    4559 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 12:26:43.824718 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.811144    4559 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 12:26:43.824718 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.811147    4559 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 12:26:43.824718 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.811151    4559 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 12:26:43.824718 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.811154    4559 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 12:26:43.824718 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.811157    4559 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 12:26:43.824718 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.811159    4559 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 12:26:43.824718 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.811162    4559 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 12:26:43.824718 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.811165    4559 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 12:26:43.824718 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.811167    4559 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 12:26:43.824718 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.811170    4559 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 12:26:43.824718 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.811173    4559 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 12:26:43.824718 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.811177    4559 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 12:26:43.824718 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.811180    4559 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 12:26:43.824718 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.811182    4559 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 12:26:43.824718 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.811185    4559 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 12:26:43.824718 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.811187    4559 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 12:26:43.825305 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.811190    4559 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 12:26:43.825305 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.811193    4559 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 12:26:43.825305 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.811196    4559 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 12:26:43.825305 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.811200    4559 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 12:26:43.825305 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.811204    4559 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 12:26:43.825305 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.811208    4559 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 12:26:43.825305 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.811211    4559 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 12:26:43.825305 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.811213    4559 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 12:26:43.825305 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.811216    4559 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 12:26:43.825305 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.811218    4559 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 12:26:43.825305 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.811221    4559 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 12:26:43.825305 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.811224    4559 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 12:26:43.825305 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.811228    4559 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 12:26:43.825305 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.811231    4559 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 12:26:43.825305 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.811234    4559 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 12:26:43.825305 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.811237    4559 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 12:26:43.825305 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.811240    4559 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 12:26:43.825305 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.811242    4559 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 12:26:43.825305 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.811246    4559 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 12:26:43.825820 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.811249    4559 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 12:26:43.825820 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.811251    4559 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 12:26:43.825820 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.811255    4559 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 12:26:43.825820 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.811257    4559 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 12:26:43.825820 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.811260    4559 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 12:26:43.825820 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.811262    4559 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 12:26:43.825820 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.811265    4559 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 12:26:43.825820 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.811267    4559 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 12:26:43.825820 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.811271    4559 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 12:26:43.825820 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.811274    4559 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 12:26:43.825820 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.811277    4559 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 12:26:43.825820 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.811279    4559 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 12:26:43.825820 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.811282    4559 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 12:26:43.825820 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.811286    4559 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 12:26:43.825820 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.811289    4559 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 12:26:43.825820 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.811292    4559 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 12:26:43.825820 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.811294    4559 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 12:26:43.825820 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.811297    4559 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 12:26:43.825820 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.811299    4559 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 12:26:43.826289 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.811302    4559 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 12:26:43.826289 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.811304    4559 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 12:26:43.826289 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.811307    4559 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 12:26:43.826289 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.811310    4559 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 12:26:43.826289 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.811312    4559 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 12:26:43.826289 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.811315    4559 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 12:26:43.826289 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.811317    4559 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 12:26:43.826289 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.811320    4559 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 12:26:43.826289 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.811322    4559 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 12:26:43.826289 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.811325    4559 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 12:26:43.826289 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.811327    4559 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 12:26:43.826289 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.811330    4559 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 12:26:43.826289 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.811338    4559 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 12:26:43.826289 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.811341    4559 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 12:26:43.826289 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.811344    4559 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 12:26:43.826289 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.811347    4559 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 12:26:43.826289 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.811350    4559 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 12:26:43.826289 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.811352    4559 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 12:26:43.826289 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.811355    4559 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 12:26:43.826772 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.811358    4559 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 12:26:43.826772 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.811361    4559 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 12:26:43.826772 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.811365    4559 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 12:26:43.826772 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.811367    4559 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 12:26:43.826772 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.811370    4559 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 12:26:43.826772 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.811373    4559 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 12:26:43.826772 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.811376    4559 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 12:26:43.826772 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.811378    4559 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 12:26:43.826772 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.811381    4559 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 12:26:43.826772 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.811383    4559 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 12:26:43.826772 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.811386    4559 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 12:26:43.826772 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.811388    4559 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 12:26:43.826772 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.811391    4559 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 12:26:43.826772 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.811394    4559 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 12:26:43.826772 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.811396    4559 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 12:26:43.826772 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.811399    4559 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 12:26:43.826772 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.811402    4559 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 12:26:43.826772 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.811404    4559 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 12:26:43.826772 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.811407    4559 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 12:26:43.826772 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.811409    4559 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 12:26:43.827315 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.811412    4559 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 12:26:43.827315 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.811414    4559 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 12:26:43.827315 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.811417    4559 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 12:26:43.827315 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.811419    4559 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 12:26:43.827315 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.811422    4559 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 12:26:43.827315 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.811426    4559 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 12:26:43.827315 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.811429    4559 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 12:26:43.827315 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.811433    4559 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 12:26:43.827315 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.811436    4559 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 12:26:43.827315 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.816707    4559 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 02 12:26:43.827315 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.816724    4559 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 02 12:26:43.827315 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.816744    4559 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 02 12:26:43.827315 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.816749    4559 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 02 12:26:43.827315 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.816810    4559 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 12:26:43.827315 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.816816    4559 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 12:26:43.827315 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.816819    4559 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 12:26:43.827315 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.816823    4559 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 12:26:43.827315 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.816826    4559 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 12:26:43.827315 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.816829    4559 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 12:26:43.827315 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.816832    4559 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 12:26:43.828105 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.816835    4559 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 12:26:43.828105 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.816838    4559 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 12:26:43.828105 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.816840    4559 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 12:26:43.828105 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.816843    4559 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 12:26:43.828105 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.816846    4559 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 12:26:43.828105 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.816849    4559 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 12:26:43.828105 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.816852    4559 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 12:26:43.828105 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.816854    4559 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 12:26:43.828105 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.816857    4559 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 12:26:43.828105 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.816860    4559 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 12:26:43.828105 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.816863    4559 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 12:26:43.828105 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.816866    4559 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 12:26:43.828105 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.816869    4559 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 12:26:43.828105 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.816872    4559 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 12:26:43.828105 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.816874    4559 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 12:26:43.828105 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.816877    4559 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 12:26:43.828105 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.816880    4559 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 12:26:43.828105 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.816883    4559 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 12:26:43.828105 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.816886    4559 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 12:26:43.828105 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.816889    4559 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 12:26:43.829084 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.816892    4559 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 12:26:43.829084 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.816894    4559 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 12:26:43.829084 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.816897    4559 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 12:26:43.829084 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.816900    4559 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 12:26:43.829084 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.816902    4559 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 12:26:43.829084 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.816905    4559 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 12:26:43.829084 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.816908    4559 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 12:26:43.829084 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.816910    4559 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 12:26:43.829084 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.816913    4559 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 12:26:43.829084 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.816916    4559 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 12:26:43.829084 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.816918    4559 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 12:26:43.829084 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.816921    4559 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 12:26:43.829084 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.816924    4559 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 12:26:43.829084 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.816927    4559 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 12:26:43.829084 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.816929    4559 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 12:26:43.829084 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.816932    4559 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 12:26:43.829084 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.816934    4559 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 12:26:43.829084 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.816937    4559 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 12:26:43.829084 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.816940    4559 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 12:26:43.829084 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.816943    4559 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 12:26:43.829975 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.816945    4559 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 12:26:43.829975 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.816948    4559 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 12:26:43.829975 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.816951    4559 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 12:26:43.829975 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.816953    4559 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 12:26:43.829975 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.816956    4559 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 12:26:43.829975 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.816958    4559 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 12:26:43.829975 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.816961    4559 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 12:26:43.829975 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.816963    4559 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 12:26:43.829975 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.816967    4559 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 12:26:43.829975 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.816969    4559 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 12:26:43.829975 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.816972    4559 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 12:26:43.829975 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.816974    4559 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 12:26:43.829975 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.816977    4559 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 12:26:43.829975 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.816979    4559 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 12:26:43.829975 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.816982    4559 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 12:26:43.829975 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.816984    4559 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 12:26:43.829975 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.816987    4559 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 12:26:43.829975 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.816990    4559 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 12:26:43.829975 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.816992    4559 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 12:26:43.829975 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.816995    4559 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 12:26:43.830884 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.816997    4559 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 12:26:43.830884 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817000    4559 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 12:26:43.830884 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817002    4559 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 12:26:43.830884 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817005    4559 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 12:26:43.830884 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817008    4559 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 12:26:43.830884 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817011    4559 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 12:26:43.830884 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817013    4559 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 12:26:43.830884 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817016    4559 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 12:26:43.830884 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817019    4559 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 12:26:43.830884 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817022    4559 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 12:26:43.830884 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817024    4559 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 12:26:43.830884 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817028    4559 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 12:26:43.830884 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817033    4559 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 12:26:43.830884 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817036    4559 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 12:26:43.830884 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817040    4559 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 12:26:43.830884 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817042    4559 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 12:26:43.830884 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817045    4559 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 12:26:43.830884 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817047    4559 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 12:26:43.830884 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817050    4559 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 12:26:43.831862 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817052    4559 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 12:26:43.831862 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817056    4559 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 12:26:43.831862 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817058    4559 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 12:26:43.831862 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817061    4559 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 12:26:43.831862 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817064    4559 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 12:26:43.831862 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817066    4559 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 12:26:43.831862 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817069    4559 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 12:26:43.831862 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817071    4559 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 12:26:43.831862 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817074    4559 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 12:26:43.831862 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817076    4559 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 12:26:43.831862 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817079    4559 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 12:26:43.831862 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817082    4559 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 12:26:43.831862 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817084    4559 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 12:26:43.831862 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817088    4559 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 12:26:43.831862 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817092    4559 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 12:26:43.831862 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817096    4559 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 12:26:43.831862 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817099    4559 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 12:26:43.831862 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817102    4559 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 12:26:43.831862 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817105    4559 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 12:26:43.832746 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817108    4559 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 12:26:43.832746 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.817136    4559 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 02 12:26:43.832746 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817272    4559 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 12:26:43.832746 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817277    4559 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 12:26:43.832746 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817281    4559 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 12:26:43.832746 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817284    4559 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 12:26:43.832746 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817286    4559 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 12:26:43.832746 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817290    4559 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 12:26:43.832746 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817293    4559 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 12:26:43.832746 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817296    4559 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 12:26:43.832746 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817299    4559 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 12:26:43.832746 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817302    4559 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 12:26:43.832746 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817304    4559 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 12:26:43.832746 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817307    4559 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 12:26:43.832746 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817310    4559 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 12:26:43.833460 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817313    4559 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 12:26:43.833460 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817315    4559 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 12:26:43.833460 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817318    4559 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 12:26:43.833460 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817321    4559 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 12:26:43.833460 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817323    4559 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 12:26:43.833460 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817326    4559 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 12:26:43.833460 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817328    4559 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 12:26:43.833460 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817332    4559 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 12:26:43.833460 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817335    4559 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 12:26:43.833460 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817337    4559 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 12:26:43.833460 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817340    4559 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 12:26:43.833460 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817343    4559 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 12:26:43.833460 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817345    4559 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 12:26:43.833460 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817348    4559 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 12:26:43.833460 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817351    4559 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 12:26:43.833460 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817354    4559 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 12:26:43.833460 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817356    4559 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 12:26:43.833460 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817359    4559 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 12:26:43.833460 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817362    4559 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 12:26:43.833460 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817364    4559 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 12:26:43.834380 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817369    4559 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 12:26:43.834380 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817372    4559 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 12:26:43.834380 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817375    4559 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 12:26:43.834380 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817378    4559 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 12:26:43.834380 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817380    4559 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 12:26:43.834380 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817383    4559 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 12:26:43.834380 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817386    4559 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 12:26:43.834380 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817389    4559 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 12:26:43.834380 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817391    4559 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 12:26:43.834380 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817394    4559 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 12:26:43.834380 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817396    4559 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 12:26:43.834380 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817399    4559 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 12:26:43.834380 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817401    4559 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 12:26:43.834380 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817404    4559 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 12:26:43.834380 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817407    4559 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 12:26:43.834380 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817409    4559 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 12:26:43.834380 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817412    4559 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 12:26:43.834380 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817414    4559 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 12:26:43.834380 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817417    4559 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 12:26:43.835332 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817420    4559 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 12:26:43.835332 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817422    4559 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 12:26:43.835332 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817425    4559 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 12:26:43.835332 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817428    4559 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 12:26:43.835332 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817430    4559 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 12:26:43.835332 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817433    4559 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 12:26:43.835332 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817436    4559 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 12:26:43.835332 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817439    4559 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 12:26:43.835332 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817441    4559 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 12:26:43.835332 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817444    4559 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 12:26:43.835332 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817446    4559 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 12:26:43.835332 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817449    4559 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 12:26:43.835332 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817451    4559 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 12:26:43.835332 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817454    4559 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 12:26:43.835332 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817456    4559 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 12:26:43.835332 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817459    4559 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 12:26:43.835332 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817462    4559 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 12:26:43.835332 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817464    4559 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 12:26:43.835332 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817467    4559 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 12:26:43.835332 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817470    4559 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 12:26:43.836375 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817473    4559 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 12:26:43.836375 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817475    4559 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 12:26:43.836375 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817478    4559 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 12:26:43.836375 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817480    4559 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 12:26:43.836375 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817483    4559 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 12:26:43.836375 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817490    4559 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 12:26:43.836375 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817493    4559 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 12:26:43.836375 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817496    4559 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 12:26:43.836375 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817498    4559 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 12:26:43.836375 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817501    4559 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 12:26:43.836375 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817505    4559 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 12:26:43.836375 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817509    4559 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 12:26:43.836375 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817513    4559 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 12:26:43.836375 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817516    4559 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 12:26:43.836375 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817519    4559 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 12:26:43.836375 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817522    4559 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 12:26:43.836375 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817524    4559 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 12:26:43.836375 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817527    4559 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 12:26:43.836375 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817530    4559 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 12:26:43.836375 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817533    4559 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 12:26:43.837285 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817535    4559 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 12:26:43.837285 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817538    4559 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 12:26:43.837285 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817541    4559 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 12:26:43.837285 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817543    4559 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 12:26:43.837285 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817546    4559 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 12:26:43.837285 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817556    4559 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 12:26:43.837285 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817560    4559 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 12:26:43.837285 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817562    4559 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 12:26:43.837285 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817565    4559 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 12:26:43.837285 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817568    4559 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 12:26:43.837285 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817571    4559 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 12:26:43.837285 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817573    4559 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 12:26:43.837285 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817577    4559 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 12:26:43.837285 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817579    4559 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 12:26:43.837285 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.817603    4559 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 02 12:26:43.837285 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817656    4559 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 12:26:43.838015 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817661    4559 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 12:26:43.838015 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817665    4559 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 12:26:43.838015 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817670    4559 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 12:26:43.838015 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817673    4559 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 12:26:43.838015 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817675    4559 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 12:26:43.838015 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817678    4559 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 12:26:43.838015 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817681    4559 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 12:26:43.838015 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817684    4559 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 12:26:43.838015 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817687    4559 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 12:26:43.838015 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817689    4559 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 12:26:43.838015 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817692    4559 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 12:26:43.838015 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817695    4559 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 12:26:43.838015 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817698    4559 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 12:26:43.838015 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817700    4559 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 12:26:43.838015 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817703    4559 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 12:26:43.838015 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817705    4559 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 12:26:43.838015 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817708    4559 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 12:26:43.838015 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817711    4559 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 12:26:43.838015 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817713    4559 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 12:26:43.838015 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817715    4559 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 12:26:43.838623 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817718    4559 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 12:26:43.838623 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817720    4559 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 12:26:43.838623 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817723    4559 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 12:26:43.838623 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817726    4559 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 12:26:43.838623 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817728    4559 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 12:26:43.838623 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817730    4559 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 12:26:43.838623 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817733    4559 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 12:26:43.838623 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817736    4559 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 12:26:43.838623 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817738    4559 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 12:26:43.838623 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817742    4559 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 12:26:43.838623 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817744    4559 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 12:26:43.838623 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817747    4559 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 12:26:43.838623 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817749    4559 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 12:26:43.838623 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817752    4559 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 12:26:43.838623 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817756    4559 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 12:26:43.838623 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817759    4559 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 12:26:43.838623 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817763    4559 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 12:26:43.838623 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817767    4559 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 12:26:43.838623 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817771    4559 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 12:26:43.838623 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817775    4559 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 12:26:43.839186 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817778    4559 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 12:26:43.839186 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817780    4559 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 12:26:43.839186 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817783    4559 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 12:26:43.839186 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817786    4559 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 12:26:43.839186 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817788    4559 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 12:26:43.839186 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817791    4559 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 12:26:43.839186 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817794    4559 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 12:26:43.839186 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817796    4559 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 12:26:43.839186 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817799    4559 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 12:26:43.839186 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817801    4559 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 12:26:43.839186 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817804    4559 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 12:26:43.839186 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817806    4559 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 12:26:43.839186 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817809    4559 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 12:26:43.839186 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817812    4559 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 12:26:43.839186 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817814    4559 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 12:26:43.839186 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817817    4559 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 12:26:43.839186 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817820    4559 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 12:26:43.839186 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817823    4559 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 12:26:43.839186 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817825    4559 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 12:26:43.839994 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817828    4559 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 12:26:43.839994 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817830    4559 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 12:26:43.839994 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817833    4559 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 12:26:43.839994 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817836    4559 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 12:26:43.839994 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817839    4559 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 12:26:43.839994 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817841    4559 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 12:26:43.839994 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817844    4559 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 12:26:43.839994 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817848    4559 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 12:26:43.839994 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817850    4559 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 12:26:43.839994 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817854    4559 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 12:26:43.839994 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817857    4559 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 12:26:43.839994 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817860    4559 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 12:26:43.839994 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817862    4559 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 12:26:43.839994 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817865    4559 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 12:26:43.839994 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817867    4559 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 12:26:43.839994 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817870    4559 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 12:26:43.839994 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817873    4559 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 12:26:43.839994 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817875    4559 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 12:26:43.839994 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817877    4559 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 12:26:43.840813 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817880    4559 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 12:26:43.840813 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817883    4559 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 12:26:43.840813 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817886    4559 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 12:26:43.840813 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817889    4559 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 12:26:43.840813 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817891    4559 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 12:26:43.840813 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817894    4559 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 12:26:43.840813 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817897    4559 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 12:26:43.840813 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817900    4559 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 12:26:43.840813 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817903    4559 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 12:26:43.840813 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817905    4559 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 12:26:43.840813 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817907    4559 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 12:26:43.840813 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817910    4559 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 12:26:43.840813 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817913    4559 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 12:26:43.840813 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817916    4559 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 12:26:43.840813 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817918    4559 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 12:26:43.840813 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817920    4559 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 12:26:43.840813 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817925    4559 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 12:26:43.840813 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817930    4559 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 12:26:43.840813 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817933    4559 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 12:26:43.841661 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817936    4559 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 12:26:43.841661 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817940    4559 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 12:26:43.841661 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817942    4559 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 12:26:43.841661 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817950    4559 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 12:26:43.841661 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817953    4559 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 12:26:43.841661 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817955    4559 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 12:26:43.841661 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817958    4559 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 12:26:43.841661 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:43.817960    4559 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 12:26:43.841661 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.818013    4559 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 02 12:26:43.841661 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.818988    4559 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Nov 02 12:26:43.841661 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.819036    4559 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Nov 02 12:26:43.841661 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.819207    4559 server.go:1014] "Starting client certificate rotation"
Nov 02 12:26:43.841661 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.819366    4559 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 02 12:26:43.841661 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.819429    4559 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-01 12:21:02 +0000 UTC" deadline="2028-05-20 13:59:25.944522376 +0000 UTC"
Nov 02 12:26:43.841661 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.819471    4559 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13561h32m42.125054864s"
Nov 02 12:26:43.841661 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.819696    4559 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 02 12:26:43.842342 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.820777    4559 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 02 12:26:43.842342 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.822466    4559 log.go:25] "Validated CRI v1 runtime API"
Nov 02 12:26:43.842342 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.824435    4559 log.go:25] "Validated CRI v1 image API"
Nov 02 12:26:43.842342 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.825178    4559 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 02 12:26:43.842342 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.826746    4559 fs.go:135] Filesystem UUIDs: map[6c917a51-8570-43b9-84b7-2d38dd82c25b:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 d917a991-98d5-4d8a-9534-b799f518d4a4:/dev/nvme0n1p4]
Nov 02 12:26:43.842586 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.826767    4559 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/21395fe98db6e90d01f7b1c1b735c298221b387616a9f95e4f33a972a04ea8d5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/21395fe98db6e90d01f7b1c1b735c298221b387616a9f95e4f33a972a04ea8d5/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/26ecac6199a071ff2dcd94f0d1d4d938473fe2b4001d4b0e61ecb1e3b6938acf/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/26ecac6199a071ff2dcd94f0d1d4d938473fe2b4001d4b0e61ecb1e3b6938acf/userdata/shm major:0 minor:202 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3a8c12aa0fbb41c2b4a84b506554dd0f39221224ca2d8fdfe45007c8ef78c18b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3a8c12aa0fbb41c2b4a84b506554dd0f39221224ca2d8fdfe45007c8ef78c18b/userdata/shm major:0 minor:235 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/56c29b248f147512b03791da0b50b810e2c43849264148c426a9a6d4abd4e9cb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/56c29b248f147512b03791da0b50b810e2c43849264148c426a9a6d4abd4e9cb/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/57ec3259bc4c3b9c41429b5c2a5b3db208605386478d98cc19462e00975e2e76/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/57ec3259bc4c3b9c41429b5c2a5b3db208605386478d98cc19462e00975e2e76/userdata/shm major:0 minor:216 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/62ef2cf548134a07d3fd774b87b4967be8dc5cedd69e89d5a6d90fa52d758b32/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/62ef2cf548134a07d3fd774b87b4967be8dc5cedd69e89d5a6d90fa52d758b32/userdata/shm major:0 minor:74 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6428f691ff96f07129a9481aec168662f7b0c994b973c88e00f854d59b8c5700/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6428f691ff96f07129a9481aec168662f7b0c994b973c88e00f854d59b8c5700/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/650a2fafdbcfc0deaeea5cf5d1f9421dda4993a5b5fb3cab9d39f65189d15577/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/650a2fafdbcfc0deaeea5cf5d1f9421dda4993a5b5fb3cab9d39f65189d15577/userdata/shm major:0 minor:78 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8b3046c39b2696c2009e1ddc2fb1a50edc8fb908c8ff8f6d16198bd9b0c86980/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8b3046c39b2696c2009e1ddc2fb1a50edc8fb908c8ff8f6d16198bd9b0c86980/userdata/shm major:0 minor:224 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/947a04ccbe41c2f96116327d8affb1d55f29d958c984d2c07e166f1457ed1df4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/947a04ccbe41c2f96116327d8affb1d55f29d958c984d2c07e166f1457ed1df4/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9519b1bdf6e9646240a4c501ff13fa301b69a8b9a474cc5f445adff3e6ddd126/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9519b1bdf6e9646240a4c501ff13fa301b69a8b9a474cc5f445adff3e6ddd126/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a066542f5f5c41564795f0bcdfb4c2cb56babdd0144196ea9dea2384eb067da9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a066542f5f5c41564795f0bcdfb4c2cb56babdd0144196ea9dea2384eb067da9/userdata/shm major:0 minor:229 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bf051a0b50b073a69b414eb0f82aeecd094209a84b66181af9c36023d9d06908/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bf051a0b50b073a69b414eb0f82aeecd094209a84b66181af9c36023d9d06908/userdata/shm major:0 minor:214 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d367350c201505849466be8254a4de268d7df96d9b246672c6d288afe8b8bedb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d367350c201505849466be8254a4de268d7df96d9b246672c6d288afe8b8bedb/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d3f9609da159aded0018387be5cfe1f3a8fce539c124f377e1cd39b8d0144242/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d3f9609da159aded0018387be5cfe1f3a8fce539c124f377e1cd39b8d0144242/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/edb20a3e6fb60b89fc7006db3c3f05d04cd7418759a6fda5a329897763afc65c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/edb20a3e6fb60b89fc7006db3c3f05d04cd7418759a6fda5a329897763afc65c/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/089d7bd4-d019-471c-960f-51346831fe45/volumes/kubernetes.io~projected/kube-api-access-cws77:{mountpoint:/var/lib/kubelet/pods/089d7bd4-d019-471c-960f-51346831fe45/volumes/kubernetes.io~projected/kube-api-access-cws77 major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/09f70dd9-4dbe-46df-ac95-1c351a541a18/volumes/kubernetes.io~projected/kube-api-access-42zsx:{mountpoint:/var/lib/kubelet/pods/09f70dd9-4dbe-46df-ac95-1c351a541a18/volumes/kubernetes.io~projected/kube-api-access-42zsx major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/20cc95cd-9718-49ee-8082-e9ac00fcc147/volumes/kubernetes.io~projected/kube-api-access-c9k4x:{mountpoint:/var/lib/kubelet/pods/20cc95cd-9718-49ee-8082-e9ac00fcc147/volumes/kubernetes.io~projected/kube-api-access-c9k4x major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5507fd15-57b9-422a-af1d-5daf266bd9ca/volumes/kubernetes.io~projected/kube-api-access-qkl5z:{mountpoint:/var/lib/kubelet/pods/5507fd15-57b9-422a-af1d-5daf266bd9ca/volumes/kubernetes.io~projected/kube-api-access-qkl5z major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7b688512-ffc4-4b6a-b9ab-8fcf8ea0afe2/volumes/kubernetes.io~projected/kube-api-access-828sv:{mountpoint:/var/lib/kubelet/pods/7b688512-ffc4-4b6a-b9ab-8fcf8ea0afe2/volumes/kubernetes.io~projected/kube-api-access-828sv major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/83964713-5989-4b35-91e8-3d3993688ccd/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/83964713-5989-4b35-91e8-3d3993688ccd/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8a2e896a-489b-43fa-ba40-b2e3c0d56176/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/8a2e896a-489b-43fa-ba40-b2e3c0d56176/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8a2e896a-489b-43fa-ba40-b2e3c0d56176/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/8a2e896a-489b-43fa-ba40-b2e3c0d56176/volumes/kubernetes.io~empty-dir/tmp major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8a2e896a-489b-43fa-ba40-b2e3c0d56176/volumes/kubernetes.io~projected/kube-api-access-q9zns:{mountpoint:/var/lib/kubelet/pods/8a2e896a-489b-43fa-ba40-b2e3c0d56176/volumes/kubernetes.io~projected/kube-api-access-q9zns major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/973dc2ad-34ed-4364-b01b-9abbf4f2d43b/volumes/kubernetes.io~projected/kube-api-access-d4n6d:{mountpoint:/var/lib/kubelet/pods/973dc2ad-34ed-4364-b01b-9abbf4f2d43b/volumes/kubernetes.io~projected/kube-api-access-d4n6d major:0 minor:222 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/993b825c-e7c4-445e-b8e8-5e92eb205f3e/volumes/kubernetes.io~projected/kube-api-access-nn9gd:{mountpoint:/var/lib/kubelet/pods/993b825c-e7c4-445e-b8e8-5e92eb205f3e/volumes/kubernetes.io~projected/kube-api-access-nn9gd major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a5142bd8-95b7-499e-9885-d867d915f3a8/volumes/kubernetes.io~projected/kube-api-access-nxjdr:{mountpoint:/var/lib/kubelet/pods/a5142bd8-95b7-499e-9885-d867d915f3a8/volumes/kubernetes.io~projected/kube-api-access-nxjdr major:0 minor:228 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a81d60e5-2049-4600-9b3f-d3826aee59fb/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/a81d60e5-2049-4600-9b3f-d3826aee59fb/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a81d60e5-2049-4600-9b3f-d3826aee59fb/volumes/kubernetes.io~projected/kube-api-access-2j4q8:{mountpoint:/var/lib/kubelet/pods/a81d60e5-2049-4600-9b3f-d3826aee59fb/volumes/kubernetes.io~projected/kube-api-access-2j4q8 major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a81d60e5-2049-4600-9b3f-d3826aee59fb/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/a81d60e5-2049-4600-9b3f-d3826aee59fb/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/af2e7d8e-b1df-499c-9a92-9ebe9d7bc7b2/volumes/kubernetes.io~projected/kube-api-access-4629q:{mountpoint:/var/lib/kubelet/pods/af2e7d8e-b1df-499c-9a92-9ebe9d7bc7b2/volumes/kubernetes.io~projected/kube-api-access-4629q major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/af96531f-d9a0-4acc-ab22-62949e7400aa/volumes/kubernetes.io~projected/kube-api-access-ztjcq:{mountpoint:/var/lib/kubelet/pods/af96531f-d9a0-4acc-ab22-62949e7400aa/volumes/kubernetes.io~projected/kube-api-access-ztjcq major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ef387e86-702c-454e-825a-23cbace64952/volumes/kubernetes.io~projected/kube-api-access-twvc7:{mountpoint:/var/lib/kubelet/pods/ef387e86-702c-454e-825a-23cbace64952/volumes/kubernetes.io~projected/kube-api-access-twvc7 major:0 minor:234 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ef387e86-702c-454e-825a-23cbace64952/volumes/kubernetes.io~secret/signing-key:{mountpoint:/var/lib/kubelet/pods/ef387e86-702c-454e-825a-23cbace64952/volumes/kubernetes.io~secret/signing-key major:0 minor:233 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f25513a9-4a23-4cd1-9ef0-b5612826c157/volumes/kubernetes.io~projected/kube-api-access-7slf8:{mountpoint:/var/lib/kubelet/pods/f25513a9-4a23-4cd1-9ef0-b5612826c157/volumes/kubernetes.io~projected/kube-api-access-7slf8 major:0 minor:223 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f7c6ba1b-8808-4b6b-8965-2e4e7a067030/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/f7c6ba1b-8808-4b6b-8965-2e4e7a067030/volumes/kubernetes.io~secret/agent-certs major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f9be3b9a-c86a-473e-8a58-83d62a0cf1d7/volumes/kubernetes.io~projected/kube-api-access-j4txl:{mountpoint:/var/lib/kubelet/pods/f9be3b9a-c86a-473e-8a58-83d62a0cf1d7/volumes/kubernetes.io~projected/kube-api-access-j4txl major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f9be3b9a-c86a-473e-8a58-83d62a0cf1d7/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/f9be3b9a-c86a-473e-8a58-83d62a0cf1d7/volumes/kubernetes.io~secret/cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/feba123c-eeb3-412f-911e-a66da1981358/volumes/kubernetes.io~projected/kube-api-access-wphgd:{mountpoint:/var/lib/kubelet/pods/feba123c-eeb3-412f-911e-a66da1981358/volumes/kubernetes.io~projected/kube-api-access-wphgd major:0 minor:63 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/65e4c3b604f5549d74c04415ae785281016adb1ccbf57e4c04ae2dab0cb0f039/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/07ae4dd35c51ef6a63c552938558d1737a6fa7f96384d042ccd1558bc7e8d93a/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/d1eeb224b97ee4bbe7da10dd6cc7b543c98cd62658a423efe86161c9bc3512d6/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/525189f5977064ebe9a98829f5310c6aa6ec934d9b801f0c7b544b091c90931d/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/6410047015ad43756f64b19eddf9ef3cab3392a3adcbcb1de169686898472a51/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/9973476d314db2469d7f2c15426d4ba07807601dc5229f64959b6e2234021647/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/6690541298decf7c8ff7277e571b3c941b1da680ed8e7c612cfddb6e085524d1/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/ced689c18d8938c470973c34b9d8d7c48885d8c08fb213f090feac6983469750/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/69f73809a8775b8b8292a5476b09c1a2aa586d021f4b08c27e6ce26ce95e3e2d/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/000ba770e0adc461b5bff5631cd17eb10decc1901bbc3c23509d6aec14df9f3a/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/f7fa679c3daa048ae3a4249fcf5ae62f9e1983b9311555a85c787d34f3bf7ace/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-143:{mountpoint:/var/lib/containers/storage/overlay/b82d838d6c764a2e46bb3c72b30c78a883e6f63f1395bc9ec3444f21e9062135/merged major:0 minor:143 fsType:overlay blockSize:0} overlay_0-149:{mountpoint:/var/lib/containers/storage/overlay/50705695a1c58af3da04c3a576592bdd9a479655bc1a0c9f8cde01732e4db09d/merged major:0 minor:149 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/1abff8e2d035b309908fca8946bd5286af9994fc35e7f74dafb0eeadcdc67218/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/d2cab06fbba70994cbe3517b28d524029d4fcda954bc1c2b47c889d72e6ec596/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/e19e973933ec4410803c1d7fa65e79cd5bce4998d9f6e9467b2eed9470df561b/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/3fcda0602063e0c2d33e6feaa0689402511bf1e82e029eddb068ce598dcd3d4d/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/6fbdbae9093aae6c67109dd016a886f4019964374e58051438e009ceb842fd0f/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/dfe1c67e72811cf5825491bfedd3bdbad8746a410ea751a9aec9851337a06f5e/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/d5077324b0825db3f904a9bec00b9f2f91fecf6f6c9b228e2e8fe8a72642e502/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/3e5c6378c4614796788affa290f90242fa1a3f1966cd629ffbdb81f6f171b80e/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-204:{mountpoint:/var/lib/containers/storage/overlay/340271be5a4d35aedd324762a056007f9094aa2c1e7f7d321cdbf2253377e1aa/merged major:0 minor:204 fsType:overlay blockSize:0} overlay_0-206:{mountpoint:/var/lib/containers/storage/overlay/c26abdc71e16f2e90f19ffb10bc8f2aafb9c1d97b1f8e167f2ca0c462d2079c5/merged major:0 minor:206 fsType:overlay blockSize:0} overlay_0-218:{mountpoint:/var/lib/containers/storage/overlay/c45956f7ee49dd280dbed6d05d35eda5dc3fa64a0ac34513467c3a6727e9242d/merged major:0 minor:218 fsType:overlay blockSize:0} overlay_0-220:{mountpoint:/var/lib/containers/storage/overlay/81dfff59a901da31a418f29f875785bf731ba9a6bbd1f452c433c405532b2cb7/merged major:0 minor:220 fsType:overlay blockSize:0} overlay_0-226:{mountpoint:/var/lib/containers/storage/overlay/6f12e42a40dd439b6886687ed3e0102a22d9066053342903ea086b2d6cff3b10/merged major:0 minor:226 fsType:overlay blockSize:0} overlay_0-231:{mountpoint:/var/lib/containers/storage/overlay/5042d2b0ffc20dd8706cdbe1e11ab15cd03d786dc7b3807d4a5fe1a8bb1c712d/merged major:0 minor:231 fsType:overlay blockSize:0} overlay_0-237:{mountpoint:/var/lib/containers/storage/overlay/0a97db4d7acc019e4a268870e9c1670d032b21fcdda8aa368858a01980dd8a18/merged major:0 minor:237 fsType:overlay blockSize:0} overlay_0-239:{mountpoint:/var/lib/containers/storage/overlay/d8de0281bcfbae24a80afb7a522c91d5dab2178be9c559943032efccd5a77e5f/merged major:0 minor:239 fsType:overlay blockSize:0} overlay_0-241:{mountpoint:/var/lib/containers/storage/overlay/45513745c9b932cffe5059b64c7ac97595bbce589716ad3d59be307f361dea04/merged major:0 minor:241 fsType:overlay blockSize:0} overlay_0-247:{mountpoint:/var/lib/containers/storage/overlay/c9b31904cc758ff7695c043a875e0715ed2c0fee9a3b17b6a3d853259290bd59/merged major:0 minor:247 fsType:overlay blockSize:0} overlay_0-249:{mountpoint:/var/lib/containers/storage/overlay/0ddba8e242348c17d758a10c74467de3f2d6333839042bb071a46264946d5a5a/merged major:0 minor:249 fsType:overlay blockSize:0} overlay_0-259:{mountpoint:/var/lib/containers/storage/overlay/b04e57847ae89c0e250c5dd772e5ffc787101faa3c7f6f2e21fd2e2f66f58698/merged major:0 minor:259 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/be88fb5cded664da0e6c222ec80092a0a14d8dc15fd8330196c0b408cac13f94/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/efbb23d5607e621e02f879f91bb8136e627ce1823ea84497e7f7ff796e348547/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/a50bbf02cb225cacb662d11944d2be2b1a4835d28d520f4892b98a4caaad8755/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-72:{mountpoint:/var/lib/containers/storage/overlay/6c8dc44bbd9feb11a70ce1c994ab7e5a04a852df813669277cf0b0b3a84953f5/merged major:0 minor:72 fsType:overlay blockSize:0} overlay_0-76:{mountpoint:/var/lib/containers/storage/overlay/93f365a563837eb535f29ea22c2d2d16f347c70d50360419ed12db024c38755b/merged major:0 minor:76 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/c6df696d39816bc382fae964760fba711e2bf2c0200ddb740854a6cd15409537/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/21b3789c7244f228f14ede629c805b0b0b5f3938af4e35d99ec1bce855f4926e/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/31b5fee581cf2540fc4ec4e9ace34a52197981b1eb8bb71561c1052dddb7fec8/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/3ac785d9d111bc0fa1d956f3f6b4daa78842db498c9107050e64e257e151671d/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/82d12c6fa08aeffcd60b1094bdb3e89a1c73b9a812495c73c92a4f0ef8f8bdb7/merged major:0 minor:96 fsType:overlay blockSize:0}]
Nov 02 12:26:43.842864 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.837599    4559 manager.go:217] Machine: {Timestamp:2026-11-02 12:26:43.836792717 +0000 UTC m=+0.092191803 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3126583 MemoryCapacity:33164345344 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec25cc89266f0ccc37e35a5d281e33db SystemUUID:ec25cc89-266f-0ccc-37e3-5a5d281e33db BootID:24d2a71e-3aa4-4267-9522-01a9aacb5268 Filesystems:[{Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/bf051a0b50b073a69b414eb0f82aeecd094209a84b66181af9c36023d9d06908/userdata/shm DeviceMajor:0 DeviceMinor:214 Capacity:67108864 Type:vfs Inodes:4048382 HasInodes:true} {Device:/run/containers/storage/overlay-containers/57ec3259bc4c3b9c41429b5c2a5b3db208605386478d98cc19462e00975e2e76/userdata/shm DeviceMajor:0 DeviceMinor:216 Capacity:67108864 Type:vfs Inodes:4048382 HasInodes:true} {Device:overlay_0-239 DeviceMajor:0 DeviceMinor:239 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/09f70dd9-4dbe-46df-ac95-1c351a541a18/volumes/kubernetes.io~projected/kube-api-access-42zsx DeviceMajor:0 DeviceMinor:64 Capacity:31985745920 Type:vfs Inodes:4048382 HasInodes:true} {Device:overlay_0-72 DeviceMajor:0 DeviceMinor:72 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/62ef2cf548134a07d3fd774b87b4967be8dc5cedd69e89d5a6d90fa52d758b32/userdata/shm DeviceMajor:0 DeviceMinor:74 Capacity:67108864 Type:vfs Inodes:4048382 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d367350c201505849466be8254a4de268d7df96d9b246672c6d288afe8b8bedb/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4048382 HasInodes:true} {Device:overlay_0-149 DeviceMajor:0 DeviceMinor:149 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-237 DeviceMajor:0 DeviceMinor:237 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-241 DeviceMajor:0 DeviceMinor:241 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8a2e896a-489b-43fa-ba40-b2e3c0d56176/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:55 Capacity:31985745920 Type:vfs Inodes:4048382 HasInodes:true} {Device:/var/lib/kubelet/pods/af2e7d8e-b1df-499c-9a92-9ebe9d7bc7b2/volumes/kubernetes.io~projected/kube-api-access-4629q DeviceMajor:0 DeviceMinor:61 Capacity:31985745920 Type:vfs Inodes:4048382 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-143 DeviceMajor:0 DeviceMinor:143 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582172672 Type:vfs Inodes:4048382 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/5507fd15-57b9-422a-af1d-5daf266bd9ca/volumes/kubernetes.io~projected/kube-api-access-qkl5z DeviceMajor:0 DeviceMinor:62 Capacity:31985745920 Type:vfs Inodes:4048382 HasInodes:true} {Device:/var/lib/kubelet/pods/feba123c-eeb3-412f-911e-a66da1981358/volumes/kubernetes.io~projected/kube-api-access-wphgd DeviceMajor:0 DeviceMinor:63 Capacity:31985745920 Type:vfs Inodes:4048382 HasInodes:true} {Device:/var/lib/kubelet/pods/20cc95cd-9718-49ee-8082-e9ac00fcc147/volumes/kubernetes.io~projected/kube-api-access-c9k4x DeviceMajor:0 DeviceMinor:67 Capacity:31985745920 Type:vfs Inodes:4048382 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-204 DeviceMajor:0 DeviceMinor:204 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-259 DeviceMajor:0 DeviceMinor:259 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-220 DeviceMajor:0 DeviceMinor:220 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-247 DeviceMajor:0 DeviceMinor:247 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/8a2e896a-489b-43fa-ba40-b2e3c0d56176/volumes/kubernetes.io~projected/kube-api-access-q9zns DeviceMajor:0 DeviceMinor:60 Capacity:31985745920 Type:vfs Inodes:4048382 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a81d60e5-2049-4600-9b3f-d3826aee59fb/volumes/kubernetes.io~projected/kube-api-access-2j4q8 DeviceMajor:0 DeviceMinor:59 Capacity:31985745920 Type:vfs Inodes:4048382 HasInodes:true} {Device:overlay_0-76 DeviceMajor:0 DeviceMinor:76 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f9be3b9a-c86a-473e-8a58-83d62a0cf1d7/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:45 Capacity:31985745920 Type:vfs Inodes:4048382 HasInodes:true} {Device:/var/lib/kubelet/pods/7b688512-ffc4-4b6a-b9ab-8fcf8ea0afe2/volumes/kubernetes.io~projected/kube-api-access-828sv DeviceMajor:0 DeviceMinor:197 Capacity:31985745920 Type:vfs Inodes:4048382 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3a8c12aa0fbb41c2b4a84b506554dd0f39221224ca2d8fdfe45007c8ef78c18b/userdata/shm DeviceMajor:0 DeviceMinor:235 Capacity:67108864 Type:vfs Inodes:4048382 HasInodes:true} {Device:/run/containers/storage/overlay-containers/947a04ccbe41c2f96116327d8affb1d55f29d958c984d2c07e166f1457ed1df4/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048382 HasInodes:true} {Device:/var/lib/kubelet/pods/8a2e896a-489b-43fa-ba40-b2e3c0d56176/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:57 Capacity:31985745920 Type:vfs Inodes:4048382 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f9be3b9a-c86a-473e-8a58-83d62a0cf1d7/volumes/kubernetes.io~projected/kube-api-access-j4txl DeviceMajor:0 DeviceMinor:201 Capacity:31985745920 Type:vfs Inodes:4048382 HasInodes:true} {Device:overlay_0-231 DeviceMajor:0 DeviceMinor:231 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a81d60e5-2049-4600-9b3f-d3826aee59fb/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/089d7bd4-d019-471c-960f-51346831fe45/volumes/kubernetes.io~projected/kube-api-access-cws77 DeviceMajor:0 DeviceMinor:213 Capacity:31985745920 Type:vfs Inodes:4048382 HasInodes:true} {Device:/var/lib/kubelet/pods/f25513a9-4a23-4cd1-9ef0-b5612826c157/volumes/kubernetes.io~projected/kube-api-access-7slf8 DeviceMajor:0 DeviceMinor:223 Capacity:31985745920 Type:vfs Inodes:4048382 HasInodes:true} {Device:/var/lib/kubelet/pods/993b825c-e7c4-445e-b8e8-5e92eb205f3e/volumes/kubernetes.io~projected/kube-api-access-nn9gd DeviceMajor:0 DeviceMinor:66 Capacity:31985745920 Type:vfs Inodes:4048382 HasInodes:true} {Device:/run/containers/storage/overlay-containers/21395fe98db6e90d01f7b1c1b735c298221b387616a9f95e4f33a972a04ea8d5/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048382 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-226 DeviceMajor:0 DeviceMinor:226 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-206 DeviceMajor:0 DeviceMinor:206 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-249 DeviceMajor:0 DeviceMinor:249 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/edb20a3e6fb60b89fc7006db3c3f05d04cd7418759a6fda5a329897763afc65c/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048382 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ef387e86-702c-454e-825a-23cbace64952/volumes/kubernetes.io~secret/signing-key DeviceMajor:0 DeviceMinor:233 Capacity:31985745920 Type:vfs Inodes:4048382 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6428f691ff96f07129a9481aec168662f7b0c994b973c88e00f854d59b8c5700/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4048382 HasInodes:true} {Device:/var/lib/kubelet/pods/af96531f-d9a0-4acc-ab22-62949e7400aa/volumes/kubernetes.io~projected/kube-api-access-ztjcq DeviceMajor:0 DeviceMinor:65 Capacity:31985745920 Type:vfs Inodes:4048382 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/973dc2ad-34ed-4364-b01b-9abbf4f2d43b/volumes/kubernetes.io~projected/kube-api-access-d4n6d DeviceMajor:0 DeviceMinor:222 Capacity:31985745920 Type:vfs Inodes:4048382 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8b3046c39b2696c2009e1ddc2fb1a50edc8fb908c8ff8f6d16198bd9b0c86980/userdata/shm DeviceMajor:0 DeviceMinor:224 Capacity:67108864 Type:vfs Inodes:4048382 HasInodes:true} {Device:/var/lib/kubelet/pods/a81d60e5-2049-4600-9b3f-d3826aee59fb/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:56 Capacity:31985745920 Type:vfs Inodes:4048382 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9519b1bdf6e9646240a4c501ff13fa301b69a8b9a474cc5f445adff3e6ddd126/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048382 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/26ecac6199a071ff2dcd94f0d1d4d938473fe2b4001d4b0e61ecb1e3b6938acf/userdata/shm DeviceMajor:0 DeviceMinor:202 Capacity:67108864 Type:vfs Inodes:4048382 HasInodes:true} {Device:/var/lib/kubelet/pods/a5142bd8-95b7-499e-9885-d867d915f3a8/volumes/kubernetes.io~projected/kube-api-access-nxjdr DeviceMajor:0 DeviceMinor:228 Capacity:31985745920 Type:vfs Inodes:4048382 HasInodes:true} {Device:/var/lib/kubelet/pods/ef387e86-702c-454e-825a-23cbace64952/volumes/kubernetes.io~projected/kube-api-access-twvc7 DeviceMajor:0 DeviceMinor:234 Capacity:31985745920 Type:vfs Inodes:4048382 HasInodes:true} {Device:/run/containers/storage/overlay-containers/650a2fafdbcfc0deaeea5cf5d1f9421dda4993a5b5fb3cab9d39f65189d15577/userdata/shm DeviceMajor:0 DeviceMinor:78 Capacity:67108864 Type:vfs Inodes:4048382 HasInodes:true} {Device:/run/containers/storage/overlay-containers/56c29b248f147512b03791da0b50b810e2c43849264148c426a9a6d4abd4e9cb/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4048382 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d3f9609da159aded0018387be5cfe1f3a8fce539c124f377e1cd39b8d0144242/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048382 HasInodes:true} {Device:overlay_0-218 DeviceMajor:0 DeviceMinor:218 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a066542f5f5c41564795f0bcdfb4c2cb56babdd0144196ea9dea2384eb067da9/userdata/shm DeviceMajor:0 DeviceMinor:229 Capacity:67108864 Type:vfs Inodes:4048382 HasInodes:true} {Device:/var/lib/kubelet/pods/f7c6ba1b-8808-4b6b-8965-2e4e7a067030/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:58 Capacity:31985745920 Type:vfs Inodes:4048382 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/83964713-5989-4b35-91e8-3d3993688ccd/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:208 Capacity:31985745920 Type:vfs Inodes:4048382 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:26ecac6199a071f MacAddress:f6:63:84:ec:2e:50 Speed:10000 Mtu:8901} {Name:3a8c12aa0fbb41c MacAddress:ea:72:cb:84:ff:35 Speed:10000 Mtu:8901} {Name:57ec3259bc4c3b9 MacAddress:86:82:b2:6e:64:17 Speed:10000 Mtu:8901} {Name:8b3046c39b2696c MacAddress:be:41:da:7c:df:5e Speed:10000 Mtu:8901} {Name:a066542f5f5c415 MacAddress:c2:0c:cf:7d:7b:e8 Speed:10000 Mtu:8901} {Name:bf051a0b50b073a MacAddress:ae:12:15:eb:26:b0 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:f7:a6:90:79 Speed:0 Mtu:9001} {Name:br-int MacAddress:ee:ed:bd:8e:8f:99 Speed:0 Mtu:8901} {Name:ens5 MacAddress:02:ff:f7:a6:90:79 Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:8a:68:30:19:e7:19 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:85:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:b2:8d:24:bb:6d:49 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164345344 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 02 12:26:43.842864 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.838129    4559 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 02 12:26:43.842864 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.838195    4559 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 02 12:26:43.842864 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.838535    4559 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 02 12:26:43.842864 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.838565    4559 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-132.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 02 12:26:43.842864 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.838714    4559 topology_manager.go:143] "Creating topology manager with none policy"
Nov 02 12:26:43.842864 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.838721    4559 container_manager_linux.go:311] "Creating device plugin manager"
Nov 02 12:26:43.842864 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.838731    4559 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 02 12:26:43.842864 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.838744    4559 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 02 12:26:43.842864 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.838829    4559 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 02 12:26:43.842864 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.838983    4559 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 02 12:26:43.842864 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.839106    4559 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 02 12:26:43.842864 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.839142    4559 kubelet.go:493] "Attempting to sync node with API server"
Nov 02 12:26:43.842864 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.839153    4559 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 02 12:26:43.842864 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.839163    4559 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 02 12:26:43.842864 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.839172    4559 kubelet.go:405] "Adding apiserver pod source"
Nov 02 12:26:43.842864 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.839180    4559 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 02 12:26:43.842864 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.839805    4559 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 02 12:26:43.842864 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.839817    4559 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 02 12:26:43.842864 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.840391    4559 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 02 12:26:43.842864 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.842416    4559 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 02 12:26:43.842864 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.842426    4559 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 02 12:26:43.842864 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.842528    4559 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Nov 02 12:26:43.842864 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.842751    4559 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 02 12:26:43.842864 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.842783    4559 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 02 12:26:43.842864 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.842882    4559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 02 12:26:43.844007 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.842894    4559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 02 12:26:43.844007 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.842901    4559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 02 12:26:43.844007 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.842906    4559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 02 12:26:43.844007 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.842912    4559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 02 12:26:43.844007 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.842919    4559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 02 12:26:43.844007 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.842927    4559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 02 12:26:43.844007 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.842936    4559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 02 12:26:43.844007 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.842946    4559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 02 12:26:43.844007 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.842953    4559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 02 12:26:43.844007 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.842962    4559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 02 12:26:43.844007 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.842972    4559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 02 12:26:43.844007 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.843000    4559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 02 12:26:43.844007 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.843008    4559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 02 12:26:43.847276 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.847262    4559 server.go:1263] "Started kubelet"
Nov 02 12:26:43.847367 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.847338    4559 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 02 12:26:43.847803 ip-10-0-128-132 systemd[1]: Started Kubernetes Kubelet.
Nov 02 12:26:43.848982 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.848890    4559 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 02 12:26:43.848982 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.848979    4559 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 02 12:26:43.849803 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.849290    4559 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 02 12:26:43.851697 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.849887    4559 server.go:317] "Adding debug handlers to kubelet server"
Nov 02 12:26:43.852135 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.851857    4559 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 02 12:26:43.852135 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.852018    4559 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 02 12:26:43.852135 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.852030    4559 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 02 12:26:43.852135 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.851959    4559 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 02 12:26:43.852135 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.852123    4559 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 02 12:26:43.852388 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.852187    4559 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 12:21:04 +0000 UTC" deadline="2028-07-02 12:25:09.486891921 +0000 UTC"
Nov 02 12:26:43.852388 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.852229    4559 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14591h58m25.634666539s"
Nov 02 12:26:43.852474 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:26:43.852450    4559 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-132.ec2.internal\" not found"
Nov 02 12:26:43.854272 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.854220    4559 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="993b825c-e7c4-445e-b8e8-5e92eb205f3e" volumeName="kubernetes.io/configmap/993b825c-e7c4-445e-b8e8-5e92eb205f3e-cni-binary-copy" seLinuxMountContext=""
Nov 02 12:26:43.854272 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.854256    4559 factory.go:55] Registering systemd factory
Nov 02 12:26:43.854272 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.854272    4559 factory.go:223] Registration of the systemd container factory successfully
Nov 02 12:26:43.854511 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.854256    4559 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a81d60e5-2049-4600-9b3f-d3826aee59fb" volumeName="kubernetes.io/configmap/a81d60e5-2049-4600-9b3f-d3826aee59fb-env-overrides" seLinuxMountContext=""
Nov 02 12:26:43.854511 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.854322    4559 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a81d60e5-2049-4600-9b3f-d3826aee59fb" volumeName="kubernetes.io/configmap/a81d60e5-2049-4600-9b3f-d3826aee59fb-ovnkube-script-lib" seLinuxMountContext=""
Nov 02 12:26:43.854511 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.854337    4559 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="feba123c-eeb3-412f-911e-a66da1981358" volumeName="kubernetes.io/projected/feba123c-eeb3-412f-911e-a66da1981358-kube-api-access-wphgd" seLinuxMountContext=""
Nov 02 12:26:43.854511 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.854346    4559 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7b688512-ffc4-4b6a-b9ab-8fcf8ea0afe2" volumeName="kubernetes.io/empty-dir/7b688512-ffc4-4b6a-b9ab-8fcf8ea0afe2-tmp-dir" seLinuxMountContext=""
Nov 02 12:26:43.854511 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.854354    4559 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a81d60e5-2049-4600-9b3f-d3826aee59fb" volumeName="kubernetes.io/configmap/a81d60e5-2049-4600-9b3f-d3826aee59fb-ovnkube-config" seLinuxMountContext=""
Nov 02 12:26:43.854511 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.854362    4559 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a81d60e5-2049-4600-9b3f-d3826aee59fb" volumeName="kubernetes.io/projected/a81d60e5-2049-4600-9b3f-d3826aee59fb-kube-api-access-2j4q8" seLinuxMountContext=""
Nov 02 12:26:43.854511 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.854370    4559 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="af96531f-d9a0-4acc-ab22-62949e7400aa" volumeName="kubernetes.io/projected/af96531f-d9a0-4acc-ab22-62949e7400aa-kube-api-access-ztjcq" seLinuxMountContext=""
Nov 02 12:26:43.854511 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.854381    4559 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f25513a9-4a23-4cd1-9ef0-b5612826c157" volumeName="kubernetes.io/configmap/f25513a9-4a23-4cd1-9ef0-b5612826c157-trusted-ca-bundle" seLinuxMountContext=""
Nov 02 12:26:43.854511 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.854389    4559 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f9be3b9a-c86a-473e-8a58-83d62a0cf1d7" volumeName="kubernetes.io/projected/f9be3b9a-c86a-473e-8a58-83d62a0cf1d7-kube-api-access-j4txl" seLinuxMountContext=""
Nov 02 12:26:43.854511 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.854397    4559 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="83964713-5989-4b35-91e8-3d3993688ccd" volumeName="kubernetes.io/secret/83964713-5989-4b35-91e8-3d3993688ccd-original-pull-secret" seLinuxMountContext=""
Nov 02 12:26:43.854511 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.854407    4559 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a5142bd8-95b7-499e-9885-d867d915f3a8" volumeName="kubernetes.io/projected/a5142bd8-95b7-499e-9885-d867d915f3a8-kube-api-access-nxjdr" seLinuxMountContext=""
Nov 02 12:26:43.854511 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.854415    4559 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f9be3b9a-c86a-473e-8a58-83d62a0cf1d7" volumeName="kubernetes.io/secret/f9be3b9a-c86a-473e-8a58-83d62a0cf1d7-cert" seLinuxMountContext=""
Nov 02 12:26:43.854511 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.854423    4559 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="993b825c-e7c4-445e-b8e8-5e92eb205f3e" volumeName="kubernetes.io/projected/993b825c-e7c4-445e-b8e8-5e92eb205f3e-kube-api-access-nn9gd" seLinuxMountContext=""
Nov 02 12:26:43.854511 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.854436    4559 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f25513a9-4a23-4cd1-9ef0-b5612826c157" volumeName="kubernetes.io/projected/f25513a9-4a23-4cd1-9ef0-b5612826c157-kube-api-access-7slf8" seLinuxMountContext=""
Nov 02 12:26:43.854511 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.854448    4559 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7b688512-ffc4-4b6a-b9ab-8fcf8ea0afe2" volumeName="kubernetes.io/projected/7b688512-ffc4-4b6a-b9ab-8fcf8ea0afe2-kube-api-access-828sv" seLinuxMountContext=""
Nov 02 12:26:43.854511 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.854460    4559 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="993b825c-e7c4-445e-b8e8-5e92eb205f3e" volumeName="kubernetes.io/configmap/993b825c-e7c4-445e-b8e8-5e92eb205f3e-multus-daemon-config" seLinuxMountContext=""
Nov 02 12:26:43.854511 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.854473    4559 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f7c6ba1b-8808-4b6b-8965-2e4e7a067030" volumeName="kubernetes.io/secret/f7c6ba1b-8808-4b6b-8965-2e4e7a067030-agent-certs" seLinuxMountContext=""
Nov 02 12:26:43.854511 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.854485    4559 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="feba123c-eeb3-412f-911e-a66da1981358" volumeName="kubernetes.io/configmap/feba123c-eeb3-412f-911e-a66da1981358-whereabouts-flatfile-configmap" seLinuxMountContext=""
Nov 02 12:26:43.854511 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.854499    4559 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="089d7bd4-d019-471c-960f-51346831fe45" volumeName="kubernetes.io/projected/089d7bd4-d019-471c-960f-51346831fe45-kube-api-access-cws77" seLinuxMountContext=""
Nov 02 12:26:43.854511 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.854512    4559 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5507fd15-57b9-422a-af1d-5daf266bd9ca" volumeName="kubernetes.io/empty-dir/5507fd15-57b9-422a-af1d-5daf266bd9ca-tmp-dir" seLinuxMountContext=""
Nov 02 12:26:43.854511 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.854524    4559 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="973dc2ad-34ed-4364-b01b-9abbf4f2d43b" volumeName="kubernetes.io/empty-dir/973dc2ad-34ed-4364-b01b-9abbf4f2d43b-data-volume" seLinuxMountContext=""
Nov 02 12:26:43.855568 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.854536    4559 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ef387e86-702c-454e-825a-23cbace64952" volumeName="kubernetes.io/configmap/ef387e86-702c-454e-825a-23cbace64952-signing-cabundle" seLinuxMountContext=""
Nov 02 12:26:43.855568 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.854547    4559 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ef387e86-702c-454e-825a-23cbace64952" volumeName="kubernetes.io/projected/ef387e86-702c-454e-825a-23cbace64952-kube-api-access-twvc7" seLinuxMountContext=""
Nov 02 12:26:43.855568 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.854559    4559 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f25513a9-4a23-4cd1-9ef0-b5612826c157" volumeName="kubernetes.io/empty-dir/f25513a9-4a23-4cd1-9ef0-b5612826c157-archives-path" seLinuxMountContext=""
Nov 02 12:26:43.855568 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.854571    4559 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5507fd15-57b9-422a-af1d-5daf266bd9ca" volumeName="kubernetes.io/projected/5507fd15-57b9-422a-af1d-5daf266bd9ca-kube-api-access-qkl5z" seLinuxMountContext=""
Nov 02 12:26:43.855568 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.854585    4559 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7b688512-ffc4-4b6a-b9ab-8fcf8ea0afe2" volumeName="kubernetes.io/configmap/7b688512-ffc4-4b6a-b9ab-8fcf8ea0afe2-config-volume" seLinuxMountContext=""
Nov 02 12:26:43.855568 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.854597    4559 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8a2e896a-489b-43fa-ba40-b2e3c0d56176" volumeName="kubernetes.io/empty-dir/8a2e896a-489b-43fa-ba40-b2e3c0d56176-etc-tuned" seLinuxMountContext=""
Nov 02 12:26:43.855568 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.854614    4559 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8a2e896a-489b-43fa-ba40-b2e3c0d56176" volumeName="kubernetes.io/projected/8a2e896a-489b-43fa-ba40-b2e3c0d56176-kube-api-access-q9zns" seLinuxMountContext=""
Nov 02 12:26:43.855568 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.854625    4559 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="af2e7d8e-b1df-499c-9a92-9ebe9d7bc7b2" volumeName="kubernetes.io/projected/af2e7d8e-b1df-499c-9a92-9ebe9d7bc7b2-kube-api-access-4629q" seLinuxMountContext=""
Nov 02 12:26:43.855568 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.854661    4559 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f7c6ba1b-8808-4b6b-8965-2e4e7a067030" volumeName="kubernetes.io/configmap/f7c6ba1b-8808-4b6b-8965-2e4e7a067030-konnectivity-ca" seLinuxMountContext=""
Nov 02 12:26:43.855568 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.854672    4559 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="feba123c-eeb3-412f-911e-a66da1981358" volumeName="kubernetes.io/configmap/feba123c-eeb3-412f-911e-a66da1981358-cni-sysctl-allowlist" seLinuxMountContext=""
Nov 02 12:26:43.855568 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.854682    4559 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="feba123c-eeb3-412f-911e-a66da1981358" volumeName="kubernetes.io/configmap/feba123c-eeb3-412f-911e-a66da1981358-cni-binary-copy" seLinuxMountContext=""
Nov 02 12:26:43.855568 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.854693    4559 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="09f70dd9-4dbe-46df-ac95-1c351a541a18" volumeName="kubernetes.io/projected/09f70dd9-4dbe-46df-ac95-1c351a541a18-kube-api-access-42zsx" seLinuxMountContext=""
Nov 02 12:26:43.855568 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.854704    4559 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="20cc95cd-9718-49ee-8082-e9ac00fcc147" volumeName="kubernetes.io/configmap/20cc95cd-9718-49ee-8082-e9ac00fcc147-serviceca" seLinuxMountContext=""
Nov 02 12:26:43.855568 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.854714    4559 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="973dc2ad-34ed-4364-b01b-9abbf4f2d43b" volumeName="kubernetes.io/projected/973dc2ad-34ed-4364-b01b-9abbf4f2d43b-kube-api-access-d4n6d" seLinuxMountContext=""
Nov 02 12:26:43.855568 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.854724    4559 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a81d60e5-2049-4600-9b3f-d3826aee59fb" volumeName="kubernetes.io/secret/a81d60e5-2049-4600-9b3f-d3826aee59fb-ovn-node-metrics-cert" seLinuxMountContext=""
Nov 02 12:26:43.855568 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.854734    4559 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="af2e7d8e-b1df-499c-9a92-9ebe9d7bc7b2" volumeName="kubernetes.io/configmap/af2e7d8e-b1df-499c-9a92-9ebe9d7bc7b2-iptables-alerter-script" seLinuxMountContext=""
Nov 02 12:26:43.855568 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.854747    4559 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ef387e86-702c-454e-825a-23cbace64952" volumeName="kubernetes.io/secret/ef387e86-702c-454e-825a-23cbace64952-signing-key" seLinuxMountContext=""
Nov 02 12:26:43.855568 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.854758    4559 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f25513a9-4a23-4cd1-9ef0-b5612826c157" volumeName="kubernetes.io/configmap/f25513a9-4a23-4cd1-9ef0-b5612826c157-service-ca-bundle" seLinuxMountContext=""
Nov 02 12:26:43.855568 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.854769    4559 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="20cc95cd-9718-49ee-8082-e9ac00fcc147" volumeName="kubernetes.io/projected/20cc95cd-9718-49ee-8082-e9ac00fcc147-kube-api-access-c9k4x" seLinuxMountContext=""
Nov 02 12:26:43.855568 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.854781    4559 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8a2e896a-489b-43fa-ba40-b2e3c0d56176" volumeName="kubernetes.io/empty-dir/8a2e896a-489b-43fa-ba40-b2e3c0d56176-tmp" seLinuxMountContext=""
Nov 02 12:26:43.855568 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.854793    4559 reconstruct.go:97] "Volume reconstruction finished"
Nov 02 12:26:43.855568 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.854800    4559 reconciler.go:29] "Reconciler: start to sync state"
Nov 02 12:26:43.855568 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.854934    4559 factory.go:153] Registering CRI-O factory
Nov 02 12:26:43.855568 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.854946    4559 factory.go:223] Registration of the crio container factory successfully
Nov 02 12:26:43.855568 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.855009    4559 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 02 12:26:43.855568 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.855027    4559 factory.go:103] Registering Raw factory
Nov 02 12:26:43.855568 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.855041    4559 manager.go:1199] Started watching for new ooms in manager
Nov 02 12:26:43.855568 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.855412    4559 manager.go:319] Starting recovery of all containers
Nov 02 12:26:43.882766 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.882555    4559 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 02 12:26:43.884944 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.884920    4559 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 02 12:26:43.884944 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.884940    4559 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 02 12:26:43.885188 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.884960    4559 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 02 12:26:43.885188 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:26:43.885005    4559 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Nov 02 12:26:43.901118 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.901063    4559 generic.go:356] "Generic (PLEG): container finished" podID="6e3a47dd3a64bdc7ae97c7ee36b68697" containerID="02cdf71f74b8f9e80b67e03511cf60785517f161c05b664e0190b920d570cd06" exitCode=0
Nov 02 12:26:43.909361 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.909340    4559 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Nov 02 12:26:43.909866 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.909835    4559 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 12:26:43.910063 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.910045    4559 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 12:26:43.918542 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.918514    4559 generic.go:356] "Generic (PLEG): container finished" podID="feba123c-eeb3-412f-911e-a66da1981358" containerID="bb1c3990d6341c422630a4a27b7c219a308be1c7a114d0b93bb977ad7c4ac2f8" exitCode=0
Nov 02 12:26:43.918542 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.918541    4559 generic.go:356] "Generic (PLEG): container finished" podID="feba123c-eeb3-412f-911e-a66da1981358" containerID="ce7e5524e139dc45495c109a251c71316f9573dc377678ac1a035d9b2510ece1" exitCode=0
Nov 02 12:26:43.918734 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.918551    4559 generic.go:356] "Generic (PLEG): container finished" podID="feba123c-eeb3-412f-911e-a66da1981358" containerID="eb8eb17bc09c67980e4f820babc4b369fc57ec52359decceaa2321eee619ef3c" exitCode=0
Nov 02 12:26:43.918734 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.918559    4559 generic.go:356] "Generic (PLEG): container finished" podID="feba123c-eeb3-412f-911e-a66da1981358" containerID="587504a512b9673d4c6881ac1ff480e9713626eb335d4c657be8a76739125313" exitCode=0
Nov 02 12:26:43.918734 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.918568    4559 generic.go:356] "Generic (PLEG): container finished" podID="feba123c-eeb3-412f-911e-a66da1981358" containerID="3cc7793c56966c4d01fc5047a415e1150692626b7ceeca318108f6e998ab344b" exitCode=0
Nov 02 12:26:43.918734 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.918575    4559 generic.go:356] "Generic (PLEG): container finished" podID="feba123c-eeb3-412f-911e-a66da1981358" containerID="3f3d71f0353934ab8912213a609d87e77f16aec3544799bdaa34c10d0c88ef60" exitCode=0
Nov 02 12:26:43.923412 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.923392    4559 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 12:26:43.923536 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.923504    4559 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 12:26:43.934841 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.934787    4559 manager.go:324] Recovery completed
Nov 02 12:26:43.952907 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.952887    4559 cpu_manager.go:226] "Starting" policy="none"
Nov 02 12:26:43.952907 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.952903    4559 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 02 12:26:43.953060 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.952924    4559 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 02 12:26:43.953092 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.953073    4559 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Nov 02 12:26:43.953134 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.953088    4559 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Nov 02 12:26:43.953134 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.953112    4559 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Nov 02 12:26:43.953134 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.953121    4559 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Nov 02 12:26:43.953134 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.953129    4559 policy_none.go:50] "Start"
Nov 02 12:26:43.953244 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.953141    4559 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 02 12:26:43.953244 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.953153    4559 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 02 12:26:43.953297 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.953287    4559 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Nov 02 12:26:43.953297 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.953294    4559 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Nov 02 12:26:43.953352 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.953300    4559 policy_none.go:44] "Start"
Nov 02 12:26:43.957417 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.957385    4559 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 02 12:26:43.961050 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.961032    4559 manager.go:341] "Starting Device Plugin manager"
Nov 02 12:26:43.961590 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:26:43.961058    4559 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 02 12:26:43.961590 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.961587    4559 server.go:87] "Starting device plugin registration server"
Nov 02 12:26:43.961874 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.961861    4559 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 02 12:26:43.961939 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.961872    4559 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 02 12:26:43.962004 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.961972    4559 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 02 12:26:43.962133 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.962095    4559 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 02 12:26:43.962133 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.962142    4559 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 02 12:26:43.962133 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.962149    4559 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 02 12:26:43.962314 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.962209    4559 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-02T12:26:43.962129149Z","UUID":"8773b1d8-d914-40ee-86e0-2cd91c75f494","Handler":null,"Name":"","Endpoint":""}
Nov 02 12:26:43.963456 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:26:43.963438    4559 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 02 12:26:43.963655 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.963616    4559 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 02 12:26:43.963765 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.963752    4559 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 02 12:26:43.985152 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.985089    4559 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-128-132.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-132.ec2.internal"]
Nov 02 12:26:43.985768 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.985731    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-132.ec2.internal" event={"ID":"6e3a47dd3a64bdc7ae97c7ee36b68697","Type":"ContainerStarted","Data":"448d328d2ca223f172b36ed842c2880b47dc762e3748c61f4b9dfa3544509dac"}
Nov 02 12:26:43.985896 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.985778    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-132.ec2.internal" event={"ID":"6e3a47dd3a64bdc7ae97c7ee36b68697","Type":"ContainerDied","Data":"02cdf71f74b8f9e80b67e03511cf60785517f161c05b664e0190b920d570cd06"}
Nov 02 12:26:43.985896 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.985801    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-132.ec2.internal" event={"ID":"6e3a47dd3a64bdc7ae97c7ee36b68697","Type":"ContainerStarted","Data":"6428f691ff96f07129a9481aec168662f7b0c994b973c88e00f854d59b8c5700"}
Nov 02 12:26:43.985896 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.985803    4559 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-132.ec2.internal"
Nov 02 12:26:43.985896 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.985834    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-132.ec2.internal" event={"ID":"9f29c1002c42911f93b255b0f3495fd0","Type":"ContainerStarted","Data":"7d401c8eda83f0557bccbe07a60d0d9ec9a79aa79285b1ab14d186ee9da4e4e6"}
Nov 02 12:26:43.985896 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.985839    4559 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-132.ec2.internal"
Nov 02 12:26:43.985896 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:43.985853    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-132.ec2.internal" event={"ID":"9f29c1002c42911f93b255b0f3495fd0","Type":"ContainerStarted","Data":"947a04ccbe41c2f96116327d8affb1d55f29d958c984d2c07e166f1457ed1df4"}
Nov 02 12:26:44.012956 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:44.012929    4559 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 02 12:26:44.013120 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:26:44.012983    4559 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-128-132.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-132.ec2.internal"
Nov 02 12:26:44.013420 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:44.013403    4559 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 02 12:26:44.013470 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:26:44.013440    4559 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-128-132.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-132.ec2.internal"
Nov 02 12:26:44.062092 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:44.062065    4559 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 12:26:44.063488 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:44.063463    4559 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-132.ec2.internal" event="NodeHasSufficientMemory"
Nov 02 12:26:44.063615 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:44.063499    4559 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-132.ec2.internal" event="NodeHasNoDiskPressure"
Nov 02 12:26:44.063615 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:44.063515    4559 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-132.ec2.internal" event="NodeHasSufficientPID"
Nov 02 12:26:44.063615 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:44.063561    4559 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-128-132.ec2.internal"
Nov 02 12:26:44.101941 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:44.101908    4559 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-128-132.ec2.internal"
Nov 02 12:26:44.102099 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:44.101992    4559 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-128-132.ec2.internal"
Nov 02 12:26:44.110750 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:44.110724    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9f29c1002c42911f93b255b0f3495fd0-config\") pod \"kube-apiserver-proxy-ip-10-0-128-132.ec2.internal\" (UID: \"9f29c1002c42911f93b255b0f3495fd0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-132.ec2.internal"
Nov 02 12:26:44.110882 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:44.110755    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6e3a47dd3a64bdc7ae97c7ee36b68697-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-132.ec2.internal\" (UID: \"6e3a47dd3a64bdc7ae97c7ee36b68697\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-132.ec2.internal"
Nov 02 12:26:44.110882 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:44.110774    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6e3a47dd3a64bdc7ae97c7ee36b68697-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-132.ec2.internal\" (UID: \"6e3a47dd3a64bdc7ae97c7ee36b68697\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-132.ec2.internal"
Nov 02 12:26:44.211827 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:44.211753    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9f29c1002c42911f93b255b0f3495fd0-config\") pod \"kube-apiserver-proxy-ip-10-0-128-132.ec2.internal\" (UID: \"9f29c1002c42911f93b255b0f3495fd0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-132.ec2.internal"
Nov 02 12:26:44.211827 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:44.211790    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6e3a47dd3a64bdc7ae97c7ee36b68697-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-132.ec2.internal\" (UID: \"6e3a47dd3a64bdc7ae97c7ee36b68697\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-132.ec2.internal"
Nov 02 12:26:44.211827 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:44.211815    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6e3a47dd3a64bdc7ae97c7ee36b68697-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-132.ec2.internal\" (UID: \"6e3a47dd3a64bdc7ae97c7ee36b68697\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-132.ec2.internal"
Nov 02 12:26:44.212020 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:44.211858    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9f29c1002c42911f93b255b0f3495fd0-config\") pod \"kube-apiserver-proxy-ip-10-0-128-132.ec2.internal\" (UID: \"9f29c1002c42911f93b255b0f3495fd0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-132.ec2.internal"
Nov 02 12:26:44.212020 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:44.211878    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6e3a47dd3a64bdc7ae97c7ee36b68697-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-132.ec2.internal\" (UID: \"6e3a47dd3a64bdc7ae97c7ee36b68697\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-132.ec2.internal"
Nov 02 12:26:44.212020 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:44.211858    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6e3a47dd3a64bdc7ae97c7ee36b68697-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-132.ec2.internal\" (UID: \"6e3a47dd3a64bdc7ae97c7ee36b68697\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-132.ec2.internal"
Nov 02 12:26:44.839338 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:44.839310    4559 apiserver.go:52] "Watching apiserver"
Nov 02 12:26:44.884215 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:44.884191    4559 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 02 12:26:44.884687 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:44.884666    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/node-ca-9t4f5","openshift-multus/network-metrics-daemon-dp7h7","openshift-network-diagnostics/network-check-target-wx6zw","openshift-dns/node-resolver-7trhz","openshift-ovn-kubernetes/ovnkube-node-qgv46","kube-system/konnectivity-agent-dnm8d","openshift-cluster-node-tuning-operator/tuned-vnv9x","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-132.ec2.internal","kube-system/global-pull-secret-syncer-6grht","openshift-multus/multus-z5nxc","openshift-ingress-canary/ingress-canary-9x7t5","openshift-insights/insights-runtime-extractor-wqcmj","openshift-insights/periodic-gathering-x27dt-dzqvx","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kbhhk","openshift-kube-storage-version-migrator/migrator-66df5d496b-fmzqj","openshift-multus/multus-additional-cni-plugins-v7bcl","openshift-network-operator/iptables-alerter-7gc6b","openshift-service-ca/service-ca-5fffc97bdf-g7hcq","kube-system/kube-apiserver-proxy-ip-10-0-128-132.ec2.internal","openshift-dns/dns-default-bg5xh"]
Nov 02 12:26:44.885477 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:44.885450    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dp7h7"
Nov 02 12:26:44.891532 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:44.891510    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-bg5xh"
Nov 02 12:26:44.892737 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:44.892716    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-wqcmj"
Nov 02 12:26:44.900837 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:44.900816    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 02 12:26:44.900837 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:44.900816    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 02 12:26:44.900973 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:44.900844    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 02 12:26:44.901375 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:44.901359    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 02 12:26:44.901453 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:44.901411    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-7btt2\""
Nov 02 12:26:44.901453 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:44.901445    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-bbkt9\""
Nov 02 12:26:44.901524 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:44.901513    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 02 12:26:44.901561 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:44.901554    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-2lqbm\""
Nov 02 12:26:44.914244 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:44.914219    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 02 12:26:44.914244 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:44.914223    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-tlsnx\""
Nov 02 12:26:44.914375 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:44.914255    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Nov 02 12:26:44.914723 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:44.914701    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 02 12:26:44.914723 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:44.914710    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 02 12:26:44.914796 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:44.914710    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 02 12:26:44.915052 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:44.915037    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-fhvpr\""
Nov 02 12:26:44.915108 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:44.915054    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 02 12:26:44.915108 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:44.915075    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 02 12:26:44.915108 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:44.915078    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 02 12:26:44.915108 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:44.915084    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-rbwvz\""
Nov 02 12:26:44.915237 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:44.915105    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-gr8bh\""
Nov 02 12:26:44.915237 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:44.915123    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 02 12:26:44.915237 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:44.915141    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 02 12:26:44.915237 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:44.915157    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 02 12:26:44.916119 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:44.916088    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Nov 02 12:26:44.916119 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:44.916091    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-l6cfl\""
Nov 02 12:26:44.916119 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:44.916114    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 02 12:26:44.916247 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:44.916146    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 02 12:26:44.916247 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:44.916146    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 02 12:26:44.916247 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:44.916205    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 02 12:26:44.927539 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:44.927524    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 02 12:26:44.927622 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:44.927606    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 02 12:26:44.928718 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:44.928695    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 02 12:26:44.928799 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:44.928718    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 02 12:26:44.928799 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:44.928737    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 02 12:26:44.928799 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:44.928739    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 02 12:26:44.928799 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:44.928775    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Nov 02 12:26:44.928964 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:44.928822    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Nov 02 12:26:44.928964 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:44.928877    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 02 12:26:44.929098 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:44.929081    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Nov 02 12:26:44.929513 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:44.929498    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Nov 02 12:26:44.929565 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:44.929502    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Nov 02 12:26:44.929886 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:44.929870    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-wx6z7\""
Nov 02 12:26:44.929960 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:44.929884    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 02 12:26:44.929960 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:44.929885    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 02 12:26:44.929960 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:44.929899    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 02 12:26:44.929960 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:44.929924    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 02 12:26:44.930153 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:44.929970    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 02 12:26:44.930153 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:44.929978    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 02 12:26:44.930153 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:44.929999    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 02 12:26:44.930153 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:44.930082    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 02 12:26:44.930153 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:44.930097    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 02 12:26:44.930153 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:44.930110    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Nov 02 12:26:44.930153 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:44.930097    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 02 12:26:44.930153 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:44.930127    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 02 12:26:44.930427 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:44.930158    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 02 12:26:44.930427 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:44.930160    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-hrc62\""
Nov 02 12:26:44.930427 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:44.930160    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-scnn5\""
Nov 02 12:26:44.930427 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:44.930168    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-wm9fw\""
Nov 02 12:26:44.930427 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:44.930196    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 02 12:26:44.930427 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:44.930215    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 02 12:26:44.930427 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:44.930251    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-7sfkq\""
Nov 02 12:26:44.930427 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:44.930307    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-2bc7v\""
Nov 02 12:26:44.930427 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:44.930363    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-l5rh2\""
Nov 02 12:26:44.930427 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:44.930419    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-vg52x\""
Nov 02 12:26:44.930726 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:44.930469    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-vtb4x\""
Nov 02 12:26:44.930726 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:44.930569    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-r2cgm\""
Nov 02 12:26:44.944513 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:44.944491    4559 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-132.ec2.internal"
Nov 02 12:26:44.944634 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:44.944619    4559 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-132.ec2.internal"
Nov 02 12:26:44.953518 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:44.953503    4559 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 02 12:26:44.966814 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:44.966794    4559 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 02 12:26:44.966872 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:26:44.966831    4559 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-128-132.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-132.ec2.internal"
Nov 02 12:26:44.967724 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:44.967700    4559 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 02 12:26:44.967807 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:26:44.967753    4559 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-128-132.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-132.ec2.internal"
Nov 02 12:26:45.016222 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.016198    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/20cc95cd-9718-49ee-8082-e9ac00fcc147-host\") pod \"node-ca-9t4f5\" (UID: \"20cc95cd-9718-49ee-8082-e9ac00fcc147\") " pod="openshift-image-registry/node-ca-9t4f5"
Nov 02 12:26:45.016222 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.016226    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/993b825c-e7c4-445e-b8e8-5e92eb205f3e-host-run-k8s-cni-cncf-io\") pod \"multus-z5nxc\" (UID: \"993b825c-e7c4-445e-b8e8-5e92eb205f3e\") " pod="openshift-multus/multus-z5nxc"
Nov 02 12:26:45.016393 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.016241    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/993b825c-e7c4-445e-b8e8-5e92eb205f3e-host-run-multus-certs\") pod \"multus-z5nxc\" (UID: \"993b825c-e7c4-445e-b8e8-5e92eb205f3e\") " pod="openshift-multus/multus-z5nxc"
Nov 02 12:26:45.016393 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.016258    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a81d60e5-2049-4600-9b3f-d3826aee59fb-run-ovn\") pod \"ovnkube-node-qgv46\" (UID: \"a81d60e5-2049-4600-9b3f-d3826aee59fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgv46"
Nov 02 12:26:45.016393 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.016327    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/83964713-5989-4b35-91e8-3d3993688ccd-dbus\") pod \"global-pull-secret-syncer-6grht\" (UID: \"83964713-5989-4b35-91e8-3d3993688ccd\") " pod="kube-system/global-pull-secret-syncer-6grht"
Nov 02 12:26:45.016501 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.016399    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f7c6ba1b-8808-4b6b-8965-2e4e7a067030-agent-certs\") pod \"konnectivity-agent-dnm8d\" (UID: \"f7c6ba1b-8808-4b6b-8965-2e4e7a067030\") " pod="kube-system/konnectivity-agent-dnm8d"
Nov 02 12:26:45.016501 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.016426    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4629q\" (UniqueName: \"kubernetes.io/projected/af2e7d8e-b1df-499c-9a92-9ebe9d7bc7b2-kube-api-access-4629q\") pod \"iptables-alerter-7gc6b\" (UID: \"af2e7d8e-b1df-499c-9a92-9ebe9d7bc7b2\") " pod="openshift-network-operator/iptables-alerter-7gc6b"
Nov 02 12:26:45.016501 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.016482    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8a2e896a-489b-43fa-ba40-b2e3c0d56176-sys\") pod \"tuned-vnv9x\" (UID: \"8a2e896a-489b-43fa-ba40-b2e3c0d56176\") " pod="openshift-cluster-node-tuning-operator/tuned-vnv9x"
Nov 02 12:26:45.016594 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.016530    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkl5z\" (UniqueName: \"kubernetes.io/projected/5507fd15-57b9-422a-af1d-5daf266bd9ca-kube-api-access-qkl5z\") pod \"node-resolver-7trhz\" (UID: \"5507fd15-57b9-422a-af1d-5daf266bd9ca\") " pod="openshift-dns/node-resolver-7trhz"
Nov 02 12:26:45.016655 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.016620    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/993b825c-e7c4-445e-b8e8-5e92eb205f3e-cnibin\") pod \"multus-z5nxc\" (UID: \"993b825c-e7c4-445e-b8e8-5e92eb205f3e\") " pod="openshift-multus/multus-z5nxc"
Nov 02 12:26:45.016795 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.016751    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a81d60e5-2049-4600-9b3f-d3826aee59fb-host-kubelet\") pod \"ovnkube-node-qgv46\" (UID: \"a81d60e5-2049-4600-9b3f-d3826aee59fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgv46"
Nov 02 12:26:45.016849 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.016820    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4n6d\" (UniqueName: \"kubernetes.io/projected/973dc2ad-34ed-4364-b01b-9abbf4f2d43b-kube-api-access-d4n6d\") pod \"insights-runtime-extractor-wqcmj\" (UID: \"973dc2ad-34ed-4364-b01b-9abbf4f2d43b\") " pod="openshift-insights/insights-runtime-extractor-wqcmj"
Nov 02 12:26:45.016893 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.016872    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twvc7\" (UniqueName: \"kubernetes.io/projected/ef387e86-702c-454e-825a-23cbace64952-kube-api-access-twvc7\") pod \"service-ca-5fffc97bdf-g7hcq\" (UID: \"ef387e86-702c-454e-825a-23cbace64952\") " pod="openshift-service-ca/service-ca-5fffc97bdf-g7hcq"
Nov 02 12:26:45.017042 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.017022    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f7c6ba1b-8808-4b6b-8965-2e4e7a067030-agent-certs\") pod \"konnectivity-agent-dnm8d\" (UID: \"f7c6ba1b-8808-4b6b-8965-2e4e7a067030\") " pod="kube-system/konnectivity-agent-dnm8d"
Nov 02 12:26:45.017401 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.017382    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/af96531f-d9a0-4acc-ab22-62949e7400aa-kubelet-dir\") pod \"aws-ebs-csi-driver-node-kbhhk\" (UID: \"af96531f-d9a0-4acc-ab22-62949e7400aa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kbhhk"
Nov 02 12:26:45.017454 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.017408    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/af96531f-d9a0-4acc-ab22-62949e7400aa-device-dir\") pod \"aws-ebs-csi-driver-node-kbhhk\" (UID: \"af96531f-d9a0-4acc-ab22-62949e7400aa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kbhhk"
Nov 02 12:26:45.017454 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.017431    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f9be3b9a-c86a-473e-8a58-83d62a0cf1d7-cert\") pod \"ingress-canary-9x7t5\" (UID: \"f9be3b9a-c86a-473e-8a58-83d62a0cf1d7\") " pod="openshift-ingress-canary/ingress-canary-9x7t5"
Nov 02 12:26:45.017513 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.017460    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a81d60e5-2049-4600-9b3f-d3826aee59fb-ovn-node-metrics-cert\") pod \"ovnkube-node-qgv46\" (UID: \"a81d60e5-2049-4600-9b3f-d3826aee59fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgv46"
Nov 02 12:26:45.017513 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.017483    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/ef387e86-702c-454e-825a-23cbace64952-signing-key\") pod \"service-ca-5fffc97bdf-g7hcq\" (UID: \"ef387e86-702c-454e-825a-23cbace64952\") " pod="openshift-service-ca/service-ca-5fffc97bdf-g7hcq"
Nov 02 12:26:45.017513 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.017504    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wphgd\" (UniqueName: \"kubernetes.io/projected/feba123c-eeb3-412f-911e-a66da1981358-kube-api-access-wphgd\") pod \"multus-additional-cni-plugins-v7bcl\" (UID: \"feba123c-eeb3-412f-911e-a66da1981358\") " pod="openshift-multus/multus-additional-cni-plugins-v7bcl"
Nov 02 12:26:45.017627 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.017522    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8a2e896a-489b-43fa-ba40-b2e3c0d56176-host\") pod \"tuned-vnv9x\" (UID: \"8a2e896a-489b-43fa-ba40-b2e3c0d56176\") " pod="openshift-cluster-node-tuning-operator/tuned-vnv9x"
Nov 02 12:26:45.017627 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.017594    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a81d60e5-2049-4600-9b3f-d3826aee59fb-run-openvswitch\") pod \"ovnkube-node-qgv46\" (UID: \"a81d60e5-2049-4600-9b3f-d3826aee59fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgv46"
Nov 02 12:26:45.017737 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.017706    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/09f70dd9-4dbe-46df-ac95-1c351a541a18-metrics-certs\") pod \"network-metrics-daemon-dp7h7\" (UID: \"09f70dd9-4dbe-46df-ac95-1c351a541a18\") " pod="openshift-multus/network-metrics-daemon-dp7h7"
Nov 02 12:26:45.017845 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.017799    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f9be3b9a-c86a-473e-8a58-83d62a0cf1d7-cert\") pod \"ingress-canary-9x7t5\" (UID: \"f9be3b9a-c86a-473e-8a58-83d62a0cf1d7\") " pod="openshift-ingress-canary/ingress-canary-9x7t5"
Nov 02 12:26:45.017903 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.017871    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42zsx\" (UniqueName: \"kubernetes.io/projected/09f70dd9-4dbe-46df-ac95-1c351a541a18-kube-api-access-42zsx\") pod \"network-metrics-daemon-dp7h7\" (UID: \"09f70dd9-4dbe-46df-ac95-1c351a541a18\") " pod="openshift-multus/network-metrics-daemon-dp7h7"
Nov 02 12:26:45.017959 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.017909    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/af96531f-d9a0-4acc-ab22-62949e7400aa-socket-dir\") pod \"aws-ebs-csi-driver-node-kbhhk\" (UID: \"af96531f-d9a0-4acc-ab22-62949e7400aa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kbhhk"
Nov 02 12:26:45.017959 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.017941    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/feba123c-eeb3-412f-911e-a66da1981358-cni-binary-copy\") pod \"multus-additional-cni-plugins-v7bcl\" (UID: \"feba123c-eeb3-412f-911e-a66da1981358\") " pod="openshift-multus/multus-additional-cni-plugins-v7bcl"
Nov 02 12:26:45.018072 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.017964    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/ef387e86-702c-454e-825a-23cbace64952-signing-key\") pod \"service-ca-5fffc97bdf-g7hcq\" (UID: \"ef387e86-702c-454e-825a-23cbace64952\") " pod="openshift-service-ca/service-ca-5fffc97bdf-g7hcq"
Nov 02 12:26:45.018072 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.018008    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a81d60e5-2049-4600-9b3f-d3826aee59fb-ovn-node-metrics-cert\") pod \"ovnkube-node-qgv46\" (UID: \"a81d60e5-2049-4600-9b3f-d3826aee59fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgv46"
Nov 02 12:26:45.018072 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.018044    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8a2e896a-489b-43fa-ba40-b2e3c0d56176-tmp\") pod \"tuned-vnv9x\" (UID: \"8a2e896a-489b-43fa-ba40-b2e3c0d56176\") " pod="openshift-cluster-node-tuning-operator/tuned-vnv9x"
Nov 02 12:26:45.018242 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.018103    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7b688512-ffc4-4b6a-b9ab-8fcf8ea0afe2-config-volume\") pod \"dns-default-bg5xh\" (UID: \"7b688512-ffc4-4b6a-b9ab-8fcf8ea0afe2\") " pod="openshift-dns/dns-default-bg5xh"
Nov 02 12:26:45.018242 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.018137    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/993b825c-e7c4-445e-b8e8-5e92eb205f3e-system-cni-dir\") pod \"multus-z5nxc\" (UID: \"993b825c-e7c4-445e-b8e8-5e92eb205f3e\") " pod="openshift-multus/multus-z5nxc"
Nov 02 12:26:45.018242 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.018154    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8a2e896a-489b-43fa-ba40-b2e3c0d56176-tmp\") pod \"tuned-vnv9x\" (UID: \"8a2e896a-489b-43fa-ba40-b2e3c0d56176\") " pod="openshift-cluster-node-tuning-operator/tuned-vnv9x"
Nov 02 12:26:45.018242 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.018164    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a81d60e5-2049-4600-9b3f-d3826aee59fb-systemd-units\") pod \"ovnkube-node-qgv46\" (UID: \"a81d60e5-2049-4600-9b3f-d3826aee59fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgv46"
Nov 02 12:26:45.018242 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.018190    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a81d60e5-2049-4600-9b3f-d3826aee59fb-log-socket\") pod \"ovnkube-node-qgv46\" (UID: \"a81d60e5-2049-4600-9b3f-d3826aee59fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgv46"
Nov 02 12:26:45.018242 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.018218    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/973dc2ad-34ed-4364-b01b-9abbf4f2d43b-crio-socket\") pod \"insights-runtime-extractor-wqcmj\" (UID: \"973dc2ad-34ed-4364-b01b-9abbf4f2d43b\") " pod="openshift-insights/insights-runtime-extractor-wqcmj"
Nov 02 12:26:45.018442 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.018271    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/973dc2ad-34ed-4364-b01b-9abbf4f2d43b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wqcmj\" (UID: \"973dc2ad-34ed-4364-b01b-9abbf4f2d43b\") " pod="openshift-insights/insights-runtime-extractor-wqcmj"
Nov 02 12:26:45.018442 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.018303    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/feba123c-eeb3-412f-911e-a66da1981358-system-cni-dir\") pod \"multus-additional-cni-plugins-v7bcl\" (UID: \"feba123c-eeb3-412f-911e-a66da1981358\") " pod="openshift-multus/multus-additional-cni-plugins-v7bcl"
Nov 02 12:26:45.018442 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.018335    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/993b825c-e7c4-445e-b8e8-5e92eb205f3e-multus-cni-dir\") pod \"multus-z5nxc\" (UID: \"993b825c-e7c4-445e-b8e8-5e92eb205f3e\") " pod="openshift-multus/multus-z5nxc"
Nov 02 12:26:45.018442 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.018360    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a81d60e5-2049-4600-9b3f-d3826aee59fb-host-cni-netd\") pod \"ovnkube-node-qgv46\" (UID: \"a81d60e5-2049-4600-9b3f-d3826aee59fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgv46"
Nov 02 12:26:45.018442 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.018366    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7b688512-ffc4-4b6a-b9ab-8fcf8ea0afe2-config-volume\") pod \"dns-default-bg5xh\" (UID: \"7b688512-ffc4-4b6a-b9ab-8fcf8ea0afe2\") " pod="openshift-dns/dns-default-bg5xh"
Nov 02 12:26:45.018442 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.018361    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/feba123c-eeb3-412f-911e-a66da1981358-cni-binary-copy\") pod \"multus-additional-cni-plugins-v7bcl\" (UID: \"feba123c-eeb3-412f-911e-a66da1981358\") " pod="openshift-multus/multus-additional-cni-plugins-v7bcl"
Nov 02 12:26:45.018442 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.018389    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a81d60e5-2049-4600-9b3f-d3826aee59fb-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qgv46\" (UID: \"a81d60e5-2049-4600-9b3f-d3826aee59fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgv46"
Nov 02 12:26:45.018442 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.018439    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/83964713-5989-4b35-91e8-3d3993688ccd-original-pull-secret\") pod \"global-pull-secret-syncer-6grht\" (UID: \"83964713-5989-4b35-91e8-3d3993688ccd\") " pod="kube-system/global-pull-secret-syncer-6grht"
Nov 02 12:26:45.018757 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.018471    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f7c6ba1b-8808-4b6b-8965-2e4e7a067030-konnectivity-ca\") pod \"konnectivity-agent-dnm8d\" (UID: \"f7c6ba1b-8808-4b6b-8965-2e4e7a067030\") " pod="kube-system/konnectivity-agent-dnm8d"
Nov 02 12:26:45.018757 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.018618    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/ef387e86-702c-454e-825a-23cbace64952-signing-cabundle\") pod \"service-ca-5fffc97bdf-g7hcq\" (UID: \"ef387e86-702c-454e-825a-23cbace64952\") " pod="openshift-service-ca/service-ca-5fffc97bdf-g7hcq"
Nov 02 12:26:45.018831 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.018751    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/83964713-5989-4b35-91e8-3d3993688ccd-original-pull-secret\") pod \"global-pull-secret-syncer-6grht\" (UID: \"83964713-5989-4b35-91e8-3d3993688ccd\") " pod="kube-system/global-pull-secret-syncer-6grht"
Nov 02 12:26:45.018831 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.018757    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/feba123c-eeb3-412f-911e-a66da1981358-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-v7bcl\" (UID: \"feba123c-eeb3-412f-911e-a66da1981358\") " pod="openshift-multus/multus-additional-cni-plugins-v7bcl"
Nov 02 12:26:45.018831 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.018777    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f7c6ba1b-8808-4b6b-8965-2e4e7a067030-konnectivity-ca\") pod \"konnectivity-agent-dnm8d\" (UID: \"f7c6ba1b-8808-4b6b-8965-2e4e7a067030\") " pod="kube-system/konnectivity-agent-dnm8d"
Nov 02 12:26:45.018831 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.018817    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/993b825c-e7c4-445e-b8e8-5e92eb205f3e-hostroot\") pod \"multus-z5nxc\" (UID: \"993b825c-e7c4-445e-b8e8-5e92eb205f3e\") " pod="openshift-multus/multus-z5nxc"
Nov 02 12:26:45.018964 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.018853    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/993b825c-e7c4-445e-b8e8-5e92eb205f3e-host-var-lib-cni-bin\") pod \"multus-z5nxc\" (UID: \"993b825c-e7c4-445e-b8e8-5e92eb205f3e\") " pod="openshift-multus/multus-z5nxc"
Nov 02 12:26:45.019004 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.018957    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/ef387e86-702c-454e-825a-23cbace64952-signing-cabundle\") pod \"service-ca-5fffc97bdf-g7hcq\" (UID: \"ef387e86-702c-454e-825a-23cbace64952\") " pod="openshift-service-ca/service-ca-5fffc97bdf-g7hcq"
Nov 02 12:26:45.019004 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.018968    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/feba123c-eeb3-412f-911e-a66da1981358-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-v7bcl\" (UID: \"feba123c-eeb3-412f-911e-a66da1981358\") " pod="openshift-multus/multus-additional-cni-plugins-v7bcl"
Nov 02 12:26:45.019004 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.018977    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/993b825c-e7c4-445e-b8e8-5e92eb205f3e-multus-daemon-config\") pod \"multus-z5nxc\" (UID: \"993b825c-e7c4-445e-b8e8-5e92eb205f3e\") " pod="openshift-multus/multus-z5nxc"
Nov 02 12:26:45.019139 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.019087    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/f25513a9-4a23-4cd1-9ef0-b5612826c157-archives-path\") pod \"periodic-gathering-x27dt-dzqvx\" (UID: \"f25513a9-4a23-4cd1-9ef0-b5612826c157\") " pod="openshift-insights/periodic-gathering-x27dt-dzqvx"
Nov 02 12:26:45.019188 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.019152    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7slf8\" (UniqueName: \"kubernetes.io/projected/f25513a9-4a23-4cd1-9ef0-b5612826c157-kube-api-access-7slf8\") pod \"periodic-gathering-x27dt-dzqvx\" (UID: \"f25513a9-4a23-4cd1-9ef0-b5612826c157\") " pod="openshift-insights/periodic-gathering-x27dt-dzqvx"
Nov 02 12:26:45.019240 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.019199    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/f25513a9-4a23-4cd1-9ef0-b5612826c157-archives-path\") pod \"periodic-gathering-x27dt-dzqvx\" (UID: \"f25513a9-4a23-4cd1-9ef0-b5612826c157\") " pod="openshift-insights/periodic-gathering-x27dt-dzqvx"
Nov 02 12:26:45.019240 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.019212    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8a2e896a-489b-43fa-ba40-b2e3c0d56176-etc-systemd\") pod \"tuned-vnv9x\" (UID: \"8a2e896a-489b-43fa-ba40-b2e3c0d56176\") " pod="openshift-cluster-node-tuning-operator/tuned-vnv9x"
Nov 02 12:26:45.019307 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.019240    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/993b825c-e7c4-445e-b8e8-5e92eb205f3e-multus-daemon-config\") pod \"multus-z5nxc\" (UID: \"993b825c-e7c4-445e-b8e8-5e92eb205f3e\") " pod="openshift-multus/multus-z5nxc"
Nov 02 12:26:45.019307 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.019238    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8a2e896a-489b-43fa-ba40-b2e3c0d56176-run\") pod \"tuned-vnv9x\" (UID: \"8a2e896a-489b-43fa-ba40-b2e3c0d56176\") " pod="openshift-cluster-node-tuning-operator/tuned-vnv9x"
Nov 02 12:26:45.019307 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.019295    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8a2e896a-489b-43fa-ba40-b2e3c0d56176-var-lib-kubelet\") pod \"tuned-vnv9x\" (UID: \"8a2e896a-489b-43fa-ba40-b2e3c0d56176\") " pod="openshift-cluster-node-tuning-operator/tuned-vnv9x"
Nov 02 12:26:45.019445 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.019324    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a81d60e5-2049-4600-9b3f-d3826aee59fb-host-run-ovn-kubernetes\") pod \"ovnkube-node-qgv46\" (UID: \"a81d60e5-2049-4600-9b3f-d3826aee59fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgv46"
Nov 02 12:26:45.019445 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.019351    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/83964713-5989-4b35-91e8-3d3993688ccd-kubelet-config\") pod \"global-pull-secret-syncer-6grht\" (UID: \"83964713-5989-4b35-91e8-3d3993688ccd\") " pod="kube-system/global-pull-secret-syncer-6grht"
Nov 02 12:26:45.019445 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.019376    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/feba123c-eeb3-412f-911e-a66da1981358-cnibin\") pod \"multus-additional-cni-plugins-v7bcl\" (UID: \"feba123c-eeb3-412f-911e-a66da1981358\") " pod="openshift-multus/multus-additional-cni-plugins-v7bcl"
Nov 02 12:26:45.019445 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.019404    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8a2e896a-489b-43fa-ba40-b2e3c0d56176-etc-sysctl-conf\") pod \"tuned-vnv9x\" (UID: \"8a2e896a-489b-43fa-ba40-b2e3c0d56176\") " pod="openshift-cluster-node-tuning-operator/tuned-vnv9x"
Nov 02 12:26:45.019557 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.019454    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8a2e896a-489b-43fa-ba40-b2e3c0d56176-etc-tuned\") pod \"tuned-vnv9x\" (UID: \"8a2e896a-489b-43fa-ba40-b2e3c0d56176\") " pod="openshift-cluster-node-tuning-operator/tuned-vnv9x"
Nov 02 12:26:45.019557 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.019481    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5507fd15-57b9-422a-af1d-5daf266bd9ca-hosts-file\") pod \"node-resolver-7trhz\" (UID: \"5507fd15-57b9-422a-af1d-5daf266bd9ca\") " pod="openshift-dns/node-resolver-7trhz"
Nov 02 12:26:45.019557 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.019533    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/20cc95cd-9718-49ee-8082-e9ac00fcc147-serviceca\") pod \"node-ca-9t4f5\" (UID: \"20cc95cd-9718-49ee-8082-e9ac00fcc147\") " pod="openshift-image-registry/node-ca-9t4f5"
Nov 02 12:26:45.019557 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.019538    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8a2e896a-489b-43fa-ba40-b2e3c0d56176-etc-tuned\") pod \"tuned-vnv9x\" (UID: \"8a2e896a-489b-43fa-ba40-b2e3c0d56176\") " pod="openshift-cluster-node-tuning-operator/tuned-vnv9x"
Nov 02 12:26:45.019685 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.019568    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b688512-ffc4-4b6a-b9ab-8fcf8ea0afe2-tmp-dir\") pod \"dns-default-bg5xh\" (UID: \"7b688512-ffc4-4b6a-b9ab-8fcf8ea0afe2\") " pod="openshift-dns/dns-default-bg5xh"
Nov 02 12:26:45.019713 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.019683    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a81d60e5-2049-4600-9b3f-d3826aee59fb-var-lib-openvswitch\") pod \"ovnkube-node-qgv46\" (UID: \"a81d60e5-2049-4600-9b3f-d3826aee59fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgv46"
Nov 02 12:26:45.019749 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.019718    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/993b825c-e7c4-445e-b8e8-5e92eb205f3e-multus-socket-dir-parent\") pod \"multus-z5nxc\" (UID: \"993b825c-e7c4-445e-b8e8-5e92eb205f3e\") " pod="openshift-multus/multus-z5nxc"
Nov 02 12:26:45.019749 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.019719    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b688512-ffc4-4b6a-b9ab-8fcf8ea0afe2-tmp-dir\") pod \"dns-default-bg5xh\" (UID: \"7b688512-ffc4-4b6a-b9ab-8fcf8ea0afe2\") " pod="openshift-dns/dns-default-bg5xh"
Nov 02 12:26:45.019749 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.019733    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/993b825c-e7c4-445e-b8e8-5e92eb205f3e-host-run-netns\") pod \"multus-z5nxc\" (UID: \"993b825c-e7c4-445e-b8e8-5e92eb205f3e\") " pod="openshift-multus/multus-z5nxc"
Nov 02 12:26:45.019831 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.019749    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/993b825c-e7c4-445e-b8e8-5e92eb205f3e-host-var-lib-cni-multus\") pod \"multus-z5nxc\" (UID: \"993b825c-e7c4-445e-b8e8-5e92eb205f3e\") " pod="openshift-multus/multus-z5nxc"
Nov 02 12:26:45.019831 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.019761    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/20cc95cd-9718-49ee-8082-e9ac00fcc147-serviceca\") pod \"node-ca-9t4f5\" (UID: \"20cc95cd-9718-49ee-8082-e9ac00fcc147\") " pod="openshift-image-registry/node-ca-9t4f5"
Nov 02 12:26:45.019831 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.019775    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/993b825c-e7c4-445e-b8e8-5e92eb205f3e-multus-conf-dir\") pod \"multus-z5nxc\" (UID: \"993b825c-e7c4-445e-b8e8-5e92eb205f3e\") " pod="openshift-multus/multus-z5nxc"
Nov 02 12:26:45.019831 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.019810    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztjcq\" (UniqueName: \"kubernetes.io/projected/af96531f-d9a0-4acc-ab22-62949e7400aa-kube-api-access-ztjcq\") pod \"aws-ebs-csi-driver-node-kbhhk\" (UID: \"af96531f-d9a0-4acc-ab22-62949e7400aa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kbhhk"
Nov 02 12:26:45.019934 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.019866    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5507fd15-57b9-422a-af1d-5daf266bd9ca-tmp-dir\") pod \"node-resolver-7trhz\" (UID: \"5507fd15-57b9-422a-af1d-5daf266bd9ca\") " pod="openshift-dns/node-resolver-7trhz"
Nov 02 12:26:45.019963 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.019935    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9k4x\" (UniqueName: \"kubernetes.io/projected/20cc95cd-9718-49ee-8082-e9ac00fcc147-kube-api-access-c9k4x\") pod \"node-ca-9t4f5\" (UID: \"20cc95cd-9718-49ee-8082-e9ac00fcc147\") " pod="openshift-image-registry/node-ca-9t4f5"
Nov 02 12:26:45.019992 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.019978    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/993b825c-e7c4-445e-b8e8-5e92eb205f3e-host-var-lib-kubelet\") pod \"multus-z5nxc\" (UID: \"993b825c-e7c4-445e-b8e8-5e92eb205f3e\") " pod="openshift-multus/multus-z5nxc"
Nov 02 12:26:45.020024 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.020007    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a81d60e5-2049-4600-9b3f-d3826aee59fb-host-slash\") pod \"ovnkube-node-qgv46\" (UID: \"a81d60e5-2049-4600-9b3f-d3826aee59fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgv46"
Nov 02 12:26:45.020068 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.020022    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5507fd15-57b9-422a-af1d-5daf266bd9ca-tmp-dir\") pod \"node-resolver-7trhz\" (UID: \"5507fd15-57b9-422a-af1d-5daf266bd9ca\") " pod="openshift-dns/node-resolver-7trhz"
Nov 02 12:26:45.020068 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.020038    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2j4q8\" (UniqueName: \"kubernetes.io/projected/a81d60e5-2049-4600-9b3f-d3826aee59fb-kube-api-access-2j4q8\") pod \"ovnkube-node-qgv46\" (UID: \"a81d60e5-2049-4600-9b3f-d3826aee59fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgv46"
Nov 02 12:26:45.020170 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.020086    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cws77\" (UniqueName: \"kubernetes.io/projected/089d7bd4-d019-471c-960f-51346831fe45-kube-api-access-cws77\") pod \"network-check-target-wx6zw\" (UID: \"089d7bd4-d019-471c-960f-51346831fe45\") " pod="openshift-network-diagnostics/network-check-target-wx6zw"
Nov 02 12:26:45.020170 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.020128    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-828sv\" (UniqueName: \"kubernetes.io/projected/7b688512-ffc4-4b6a-b9ab-8fcf8ea0afe2-kube-api-access-828sv\") pod \"dns-default-bg5xh\" (UID: \"7b688512-ffc4-4b6a-b9ab-8fcf8ea0afe2\") " pod="openshift-dns/dns-default-bg5xh"
Nov 02 12:26:45.020170 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.020157    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a81d60e5-2049-4600-9b3f-d3826aee59fb-ovnkube-config\") pod \"ovnkube-node-qgv46\" (UID: \"a81d60e5-2049-4600-9b3f-d3826aee59fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgv46"
Nov 02 12:26:45.020327 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.020186    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a81d60e5-2049-4600-9b3f-d3826aee59fb-env-overrides\") pod \"ovnkube-node-qgv46\" (UID: \"a81d60e5-2049-4600-9b3f-d3826aee59fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgv46"
Nov 02 12:26:45.020327 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.020225    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/973dc2ad-34ed-4364-b01b-9abbf4f2d43b-data-volume\") pod \"insights-runtime-extractor-wqcmj\" (UID: \"973dc2ad-34ed-4364-b01b-9abbf4f2d43b\") " pod="openshift-insights/insights-runtime-extractor-wqcmj"
Nov 02 12:26:45.020327 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.020255    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/af96531f-d9a0-4acc-ab22-62949e7400aa-registration-dir\") pod \"aws-ebs-csi-driver-node-kbhhk\" (UID: \"af96531f-d9a0-4acc-ab22-62949e7400aa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kbhhk"
Nov 02 12:26:45.020327 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.020286    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/af96531f-d9a0-4acc-ab22-62949e7400aa-sys-fs\") pod \"aws-ebs-csi-driver-node-kbhhk\" (UID: \"af96531f-d9a0-4acc-ab22-62949e7400aa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kbhhk"
Nov 02 12:26:45.020327 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.020313    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/973dc2ad-34ed-4364-b01b-9abbf4f2d43b-data-volume\") pod \"insights-runtime-extractor-wqcmj\" (UID: \"973dc2ad-34ed-4364-b01b-9abbf4f2d43b\") " pod="openshift-insights/insights-runtime-extractor-wqcmj"
Nov 02 12:26:45.020513 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.020314    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8a2e896a-489b-43fa-ba40-b2e3c0d56176-etc-modprobe-d\") pod \"tuned-vnv9x\" (UID: \"8a2e896a-489b-43fa-ba40-b2e3c0d56176\") " pod="openshift-cluster-node-tuning-operator/tuned-vnv9x"
Nov 02 12:26:45.020513 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.020374    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a81d60e5-2049-4600-9b3f-d3826aee59fb-ovnkube-config\") pod \"ovnkube-node-qgv46\" (UID: \"a81d60e5-2049-4600-9b3f-d3826aee59fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgv46"
Nov 02 12:26:45.020513 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.020382    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8a2e896a-489b-43fa-ba40-b2e3c0d56176-etc-sysconfig\") pod \"tuned-vnv9x\" (UID: \"8a2e896a-489b-43fa-ba40-b2e3c0d56176\") " pod="openshift-cluster-node-tuning-operator/tuned-vnv9x"
Nov 02 12:26:45.020513 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.020378    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a81d60e5-2049-4600-9b3f-d3826aee59fb-env-overrides\") pod \"ovnkube-node-qgv46\" (UID: \"a81d60e5-2049-4600-9b3f-d3826aee59fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgv46"
Nov 02 12:26:45.020513 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.020417    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/993b825c-e7c4-445e-b8e8-5e92eb205f3e-etc-kubernetes\") pod \"multus-z5nxc\" (UID: \"993b825c-e7c4-445e-b8e8-5e92eb205f3e\") " pod="openshift-multus/multus-z5nxc"
Nov 02 12:26:45.020513 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.020434    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a81d60e5-2049-4600-9b3f-d3826aee59fb-node-log\") pod \"ovnkube-node-qgv46\" (UID: \"a81d60e5-2049-4600-9b3f-d3826aee59fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgv46"
Nov 02 12:26:45.020513 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.020460    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f25513a9-4a23-4cd1-9ef0-b5612826c157-trusted-ca-bundle\") pod \"periodic-gathering-x27dt-dzqvx\" (UID: \"f25513a9-4a23-4cd1-9ef0-b5612826c157\") " pod="openshift-insights/periodic-gathering-x27dt-dzqvx"
Nov 02 12:26:45.020513 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.020504    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a81d60e5-2049-4600-9b3f-d3826aee59fb-run-systemd\") pod \"ovnkube-node-qgv46\" (UID: \"a81d60e5-2049-4600-9b3f-d3826aee59fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgv46"
Nov 02 12:26:45.020766 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.020563    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a81d60e5-2049-4600-9b3f-d3826aee59fb-ovnkube-script-lib\") pod \"ovnkube-node-qgv46\" (UID: \"a81d60e5-2049-4600-9b3f-d3826aee59fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgv46"
Nov 02 12:26:45.020766 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.020605    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/af96531f-d9a0-4acc-ab22-62949e7400aa-etc-selinux\") pod \"aws-ebs-csi-driver-node-kbhhk\" (UID: \"af96531f-d9a0-4acc-ab22-62949e7400aa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kbhhk"
Nov 02 12:26:45.020766 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.020695    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/feba123c-eeb3-412f-911e-a66da1981358-tuning-conf-dir\") pod \"multus-additional-cni-plugins-v7bcl\" (UID: \"feba123c-eeb3-412f-911e-a66da1981358\") " pod="openshift-multus/multus-additional-cni-plugins-v7bcl"
Nov 02 12:26:45.020766 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.020729    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/af2e7d8e-b1df-499c-9a92-9ebe9d7bc7b2-iptables-alerter-script\") pod \"iptables-alerter-7gc6b\" (UID: \"af2e7d8e-b1df-499c-9a92-9ebe9d7bc7b2\") " pod="openshift-network-operator/iptables-alerter-7gc6b"
Nov 02 12:26:45.020766 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.020752    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/af2e7d8e-b1df-499c-9a92-9ebe9d7bc7b2-host-slash\") pod \"iptables-alerter-7gc6b\" (UID: \"af2e7d8e-b1df-499c-9a92-9ebe9d7bc7b2\") " pod="openshift-network-operator/iptables-alerter-7gc6b"
Nov 02 12:26:45.020997 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.020792    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f25513a9-4a23-4cd1-9ef0-b5612826c157-trusted-ca-bundle\") pod \"periodic-gathering-x27dt-dzqvx\" (UID: \"f25513a9-4a23-4cd1-9ef0-b5612826c157\") " pod="openshift-insights/periodic-gathering-x27dt-dzqvx"
Nov 02 12:26:45.020997 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.020792    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f25513a9-4a23-4cd1-9ef0-b5612826c157-service-ca-bundle\") pod \"periodic-gathering-x27dt-dzqvx\" (UID: \"f25513a9-4a23-4cd1-9ef0-b5612826c157\") " pod="openshift-insights/periodic-gathering-x27dt-dzqvx"
Nov 02 12:26:45.020997 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.020842    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a81d60e5-2049-4600-9b3f-d3826aee59fb-ovnkube-script-lib\") pod \"ovnkube-node-qgv46\" (UID: \"a81d60e5-2049-4600-9b3f-d3826aee59fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgv46"
Nov 02 12:26:45.020997 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.020861    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8a2e896a-489b-43fa-ba40-b2e3c0d56176-lib-modules\") pod \"tuned-vnv9x\" (UID: \"8a2e896a-489b-43fa-ba40-b2e3c0d56176\") " pod="openshift-cluster-node-tuning-operator/tuned-vnv9x"
Nov 02 12:26:45.020997 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.020893    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nn9gd\" (UniqueName: \"kubernetes.io/projected/993b825c-e7c4-445e-b8e8-5e92eb205f3e-kube-api-access-nn9gd\") pod \"multus-z5nxc\" (UID: \"993b825c-e7c4-445e-b8e8-5e92eb205f3e\") " pod="openshift-multus/multus-z5nxc"
Nov 02 12:26:45.020997 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.020931    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8a2e896a-489b-43fa-ba40-b2e3c0d56176-etc-kubernetes\") pod \"tuned-vnv9x\" (UID: \"8a2e896a-489b-43fa-ba40-b2e3c0d56176\") " pod="openshift-cluster-node-tuning-operator/tuned-vnv9x"
Nov 02 12:26:45.021305 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.021015    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxjdr\" (UniqueName: \"kubernetes.io/projected/a5142bd8-95b7-499e-9885-d867d915f3a8-kube-api-access-nxjdr\") pod \"migrator-66df5d496b-fmzqj\" (UID: \"a5142bd8-95b7-499e-9885-d867d915f3a8\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-fmzqj"
Nov 02 12:26:45.021305 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.021040    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4txl\" (UniqueName: \"kubernetes.io/projected/f9be3b9a-c86a-473e-8a58-83d62a0cf1d7-kube-api-access-j4txl\") pod \"ingress-canary-9x7t5\" (UID: \"f9be3b9a-c86a-473e-8a58-83d62a0cf1d7\") " pod="openshift-ingress-canary/ingress-canary-9x7t5"
Nov 02 12:26:45.021305 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.021019    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/af2e7d8e-b1df-499c-9a92-9ebe9d7bc7b2-iptables-alerter-script\") pod \"iptables-alerter-7gc6b\" (UID: \"af2e7d8e-b1df-499c-9a92-9ebe9d7bc7b2\") " pod="openshift-network-operator/iptables-alerter-7gc6b"
Nov 02 12:26:45.021305 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.021134    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a81d60e5-2049-4600-9b3f-d3826aee59fb-host-run-netns\") pod \"ovnkube-node-qgv46\" (UID: \"a81d60e5-2049-4600-9b3f-d3826aee59fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgv46"
Nov 02 12:26:45.021305 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.021204    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a81d60e5-2049-4600-9b3f-d3826aee59fb-etc-openvswitch\") pod \"ovnkube-node-qgv46\" (UID: \"a81d60e5-2049-4600-9b3f-d3826aee59fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgv46"
Nov 02 12:26:45.021305 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.021242    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a81d60e5-2049-4600-9b3f-d3826aee59fb-host-cni-bin\") pod \"ovnkube-node-qgv46\" (UID: \"a81d60e5-2049-4600-9b3f-d3826aee59fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgv46"
Nov 02 12:26:45.021305 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.021276    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/973dc2ad-34ed-4364-b01b-9abbf4f2d43b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wqcmj\" (UID: \"973dc2ad-34ed-4364-b01b-9abbf4f2d43b\") " pod="openshift-insights/insights-runtime-extractor-wqcmj"
Nov 02 12:26:45.021564 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.021310    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9zns\" (UniqueName: \"kubernetes.io/projected/8a2e896a-489b-43fa-ba40-b2e3c0d56176-kube-api-access-q9zns\") pod \"tuned-vnv9x\" (UID: \"8a2e896a-489b-43fa-ba40-b2e3c0d56176\") " pod="openshift-cluster-node-tuning-operator/tuned-vnv9x"
Nov 02 12:26:45.021564 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.021345    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7b688512-ffc4-4b6a-b9ab-8fcf8ea0afe2-metrics-tls\") pod \"dns-default-bg5xh\" (UID: \"7b688512-ffc4-4b6a-b9ab-8fcf8ea0afe2\") " pod="openshift-dns/dns-default-bg5xh"
Nov 02 12:26:45.021826 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.021796    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f25513a9-4a23-4cd1-9ef0-b5612826c157-service-ca-bundle\") pod \"periodic-gathering-x27dt-dzqvx\" (UID: \"f25513a9-4a23-4cd1-9ef0-b5612826c157\") " pod="openshift-insights/periodic-gathering-x27dt-dzqvx"
Nov 02 12:26:45.021926 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.021905    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/993b825c-e7c4-445e-b8e8-5e92eb205f3e-cni-binary-copy\") pod \"multus-z5nxc\" (UID: \"993b825c-e7c4-445e-b8e8-5e92eb205f3e\") " pod="openshift-multus/multus-z5nxc"
Nov 02 12:26:45.021996 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.021943    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/feba123c-eeb3-412f-911e-a66da1981358-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-v7bcl\" (UID: \"feba123c-eeb3-412f-911e-a66da1981358\") " pod="openshift-multus/multus-additional-cni-plugins-v7bcl"
Nov 02 12:26:45.022064 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.022014    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8a2e896a-489b-43fa-ba40-b2e3c0d56176-etc-sysctl-d\") pod \"tuned-vnv9x\" (UID: \"8a2e896a-489b-43fa-ba40-b2e3c0d56176\") " pod="openshift-cluster-node-tuning-operator/tuned-vnv9x"
Nov 02 12:26:45.025023 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.022435    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/feba123c-eeb3-412f-911e-a66da1981358-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-v7bcl\" (UID: \"feba123c-eeb3-412f-911e-a66da1981358\") " pod="openshift-multus/multus-additional-cni-plugins-v7bcl"
Nov 02 12:26:45.025023 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.022670    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/993b825c-e7c4-445e-b8e8-5e92eb205f3e-cni-binary-copy\") pod \"multus-z5nxc\" (UID: \"993b825c-e7c4-445e-b8e8-5e92eb205f3e\") " pod="openshift-multus/multus-z5nxc"
Nov 02 12:26:45.041977 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.041954    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4629q\" (UniqueName: \"kubernetes.io/projected/af2e7d8e-b1df-499c-9a92-9ebe9d7bc7b2-kube-api-access-4629q\") pod \"iptables-alerter-7gc6b\" (UID: \"af2e7d8e-b1df-499c-9a92-9ebe9d7bc7b2\") " pod="openshift-network-operator/iptables-alerter-7gc6b"
Nov 02 12:26:45.043612 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.043597    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-twvc7\" (UniqueName: \"kubernetes.io/projected/ef387e86-702c-454e-825a-23cbace64952-kube-api-access-twvc7\") pod \"service-ca-5fffc97bdf-g7hcq\" (UID: \"ef387e86-702c-454e-825a-23cbace64952\") " pod="openshift-service-ca/service-ca-5fffc97bdf-g7hcq"
Nov 02 12:26:45.044592 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.044575    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4n6d\" (UniqueName: \"kubernetes.io/projected/973dc2ad-34ed-4364-b01b-9abbf4f2d43b-kube-api-access-d4n6d\") pod \"insights-runtime-extractor-wqcmj\" (UID: \"973dc2ad-34ed-4364-b01b-9abbf4f2d43b\") " pod="openshift-insights/insights-runtime-extractor-wqcmj"
Nov 02 12:26:45.044938 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.044921    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkl5z\" (UniqueName: \"kubernetes.io/projected/5507fd15-57b9-422a-af1d-5daf266bd9ca-kube-api-access-qkl5z\") pod \"node-resolver-7trhz\" (UID: \"5507fd15-57b9-422a-af1d-5daf266bd9ca\") " pod="openshift-dns/node-resolver-7trhz"
Nov 02 12:26:45.045828 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.045812    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-42zsx\" (UniqueName: \"kubernetes.io/projected/09f70dd9-4dbe-46df-ac95-1c351a541a18-kube-api-access-42zsx\") pod \"network-metrics-daemon-dp7h7\" (UID: \"09f70dd9-4dbe-46df-ac95-1c351a541a18\") " pod="openshift-multus/network-metrics-daemon-dp7h7"
Nov 02 12:26:45.048055 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.048040    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7slf8\" (UniqueName: \"kubernetes.io/projected/f25513a9-4a23-4cd1-9ef0-b5612826c157-kube-api-access-7slf8\") pod \"periodic-gathering-x27dt-dzqvx\" (UID: \"f25513a9-4a23-4cd1-9ef0-b5612826c157\") " pod="openshift-insights/periodic-gathering-x27dt-dzqvx"
Nov 02 12:26:45.048942 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.048927    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nn9gd\" (UniqueName: \"kubernetes.io/projected/993b825c-e7c4-445e-b8e8-5e92eb205f3e-kube-api-access-nn9gd\") pod \"multus-z5nxc\" (UID: \"993b825c-e7c4-445e-b8e8-5e92eb205f3e\") " pod="openshift-multus/multus-z5nxc"
Nov 02 12:26:45.048978 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.048928    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2j4q8\" (UniqueName: \"kubernetes.io/projected/a81d60e5-2049-4600-9b3f-d3826aee59fb-kube-api-access-2j4q8\") pod \"ovnkube-node-qgv46\" (UID: \"a81d60e5-2049-4600-9b3f-d3826aee59fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgv46"
Nov 02 12:26:45.050796 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.050779    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztjcq\" (UniqueName: \"kubernetes.io/projected/af96531f-d9a0-4acc-ab22-62949e7400aa-kube-api-access-ztjcq\") pod \"aws-ebs-csi-driver-node-kbhhk\" (UID: \"af96531f-d9a0-4acc-ab22-62949e7400aa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kbhhk"
Nov 02 12:26:45.051157 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.051139    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-828sv\" (UniqueName: \"kubernetes.io/projected/7b688512-ffc4-4b6a-b9ab-8fcf8ea0afe2-kube-api-access-828sv\") pod \"dns-default-bg5xh\" (UID: \"7b688512-ffc4-4b6a-b9ab-8fcf8ea0afe2\") " pod="openshift-dns/dns-default-bg5xh"
Nov 02 12:26:45.051188 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.051148    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9k4x\" (UniqueName: \"kubernetes.io/projected/20cc95cd-9718-49ee-8082-e9ac00fcc147-kube-api-access-c9k4x\") pod \"node-ca-9t4f5\" (UID: \"20cc95cd-9718-49ee-8082-e9ac00fcc147\") " pod="openshift-image-registry/node-ca-9t4f5"
Nov 02 12:26:45.055209 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.055189    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wphgd\" (UniqueName: \"kubernetes.io/projected/feba123c-eeb3-412f-911e-a66da1981358-kube-api-access-wphgd\") pod \"multus-additional-cni-plugins-v7bcl\" (UID: \"feba123c-eeb3-412f-911e-a66da1981358\") " pod="openshift-multus/multus-additional-cni-plugins-v7bcl"
Nov 02 12:26:45.070170 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.070146    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxjdr\" (UniqueName: \"kubernetes.io/projected/a5142bd8-95b7-499e-9885-d867d915f3a8-kube-api-access-nxjdr\") pod \"migrator-66df5d496b-fmzqj\" (UID: \"a5142bd8-95b7-499e-9885-d867d915f3a8\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-fmzqj"
Nov 02 12:26:45.070299 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.070222    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9zns\" (UniqueName: \"kubernetes.io/projected/8a2e896a-489b-43fa-ba40-b2e3c0d56176-kube-api-access-q9zns\") pod \"tuned-vnv9x\" (UID: \"8a2e896a-489b-43fa-ba40-b2e3c0d56176\") " pod="openshift-cluster-node-tuning-operator/tuned-vnv9x"
Nov 02 12:26:45.070422 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.070406    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cws77\" (UniqueName: \"kubernetes.io/projected/089d7bd4-d019-471c-960f-51346831fe45-kube-api-access-cws77\") pod \"network-check-target-wx6zw\" (UID: \"089d7bd4-d019-471c-960f-51346831fe45\") " pod="openshift-network-diagnostics/network-check-target-wx6zw"
Nov 02 12:26:45.070555 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.070533    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4txl\" (UniqueName: \"kubernetes.io/projected/f9be3b9a-c86a-473e-8a58-83d62a0cf1d7-kube-api-access-j4txl\") pod \"ingress-canary-9x7t5\" (UID: \"f9be3b9a-c86a-473e-8a58-83d62a0cf1d7\") " pod="openshift-ingress-canary/ingress-canary-9x7t5"
Nov 02 12:26:45.122989 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.122916    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8a2e896a-489b-43fa-ba40-b2e3c0d56176-etc-sysctl-d\") pod \"tuned-vnv9x\" (UID: \"8a2e896a-489b-43fa-ba40-b2e3c0d56176\") " pod="openshift-cluster-node-tuning-operator/tuned-vnv9x"
Nov 02 12:26:45.122989 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.122945    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/20cc95cd-9718-49ee-8082-e9ac00fcc147-host\") pod \"node-ca-9t4f5\" (UID: \"20cc95cd-9718-49ee-8082-e9ac00fcc147\") " pod="openshift-image-registry/node-ca-9t4f5"
Nov 02 12:26:45.122989 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.122960    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/993b825c-e7c4-445e-b8e8-5e92eb205f3e-host-run-k8s-cni-cncf-io\") pod \"multus-z5nxc\" (UID: \"993b825c-e7c4-445e-b8e8-5e92eb205f3e\") " pod="openshift-multus/multus-z5nxc"
Nov 02 12:26:45.122989 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.122975    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/993b825c-e7c4-445e-b8e8-5e92eb205f3e-host-run-multus-certs\") pod \"multus-z5nxc\" (UID: \"993b825c-e7c4-445e-b8e8-5e92eb205f3e\") " pod="openshift-multus/multus-z5nxc"
Nov 02 12:26:45.122989 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.122992    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a81d60e5-2049-4600-9b3f-d3826aee59fb-run-ovn\") pod \"ovnkube-node-qgv46\" (UID: \"a81d60e5-2049-4600-9b3f-d3826aee59fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgv46"
Nov 02 12:26:45.123281 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.123018    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/20cc95cd-9718-49ee-8082-e9ac00fcc147-host\") pod \"node-ca-9t4f5\" (UID: \"20cc95cd-9718-49ee-8082-e9ac00fcc147\") " pod="openshift-image-registry/node-ca-9t4f5"
Nov 02 12:26:45.123281 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.123035    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a81d60e5-2049-4600-9b3f-d3826aee59fb-run-ovn\") pod \"ovnkube-node-qgv46\" (UID: \"a81d60e5-2049-4600-9b3f-d3826aee59fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgv46"
Nov 02 12:26:45.123281 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.123044    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/993b825c-e7c4-445e-b8e8-5e92eb205f3e-host-run-k8s-cni-cncf-io\") pod \"multus-z5nxc\" (UID: \"993b825c-e7c4-445e-b8e8-5e92eb205f3e\") " pod="openshift-multus/multus-z5nxc"
Nov 02 12:26:45.123281 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.123056    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/993b825c-e7c4-445e-b8e8-5e92eb205f3e-host-run-multus-certs\") pod \"multus-z5nxc\" (UID: \"993b825c-e7c4-445e-b8e8-5e92eb205f3e\") " pod="openshift-multus/multus-z5nxc"
Nov 02 12:26:45.123281 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.123072    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/83964713-5989-4b35-91e8-3d3993688ccd-dbus\") pod \"global-pull-secret-syncer-6grht\" (UID: \"83964713-5989-4b35-91e8-3d3993688ccd\") " pod="kube-system/global-pull-secret-syncer-6grht"
Nov 02 12:26:45.123281 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.123106    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8a2e896a-489b-43fa-ba40-b2e3c0d56176-sys\") pod \"tuned-vnv9x\" (UID: \"8a2e896a-489b-43fa-ba40-b2e3c0d56176\") " pod="openshift-cluster-node-tuning-operator/tuned-vnv9x"
Nov 02 12:26:45.123281 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.123116    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/83964713-5989-4b35-91e8-3d3993688ccd-dbus\") pod \"global-pull-secret-syncer-6grht\" (UID: \"83964713-5989-4b35-91e8-3d3993688ccd\") " pod="kube-system/global-pull-secret-syncer-6grht"
Nov 02 12:26:45.123281 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.123130    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/993b825c-e7c4-445e-b8e8-5e92eb205f3e-cnibin\") pod \"multus-z5nxc\" (UID: \"993b825c-e7c4-445e-b8e8-5e92eb205f3e\") " pod="openshift-multus/multus-z5nxc"
Nov 02 12:26:45.123281 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.123152    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8a2e896a-489b-43fa-ba40-b2e3c0d56176-sys\") pod \"tuned-vnv9x\" (UID: \"8a2e896a-489b-43fa-ba40-b2e3c0d56176\") " pod="openshift-cluster-node-tuning-operator/tuned-vnv9x"
Nov 02 12:26:45.123281 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.123154    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a81d60e5-2049-4600-9b3f-d3826aee59fb-host-kubelet\") pod \"ovnkube-node-qgv46\" (UID: \"a81d60e5-2049-4600-9b3f-d3826aee59fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgv46"
Nov 02 12:26:45.123281 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.123174    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8a2e896a-489b-43fa-ba40-b2e3c0d56176-etc-sysctl-d\") pod \"tuned-vnv9x\" (UID: \"8a2e896a-489b-43fa-ba40-b2e3c0d56176\") " pod="openshift-cluster-node-tuning-operator/tuned-vnv9x"
Nov 02 12:26:45.123281 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.123182    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a81d60e5-2049-4600-9b3f-d3826aee59fb-host-kubelet\") pod \"ovnkube-node-qgv46\" (UID: \"a81d60e5-2049-4600-9b3f-d3826aee59fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgv46"
Nov 02 12:26:45.123281 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.123192    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/af96531f-d9a0-4acc-ab22-62949e7400aa-kubelet-dir\") pod \"aws-ebs-csi-driver-node-kbhhk\" (UID: \"af96531f-d9a0-4acc-ab22-62949e7400aa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kbhhk"
Nov 02 12:26:45.123281 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.123220    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/993b825c-e7c4-445e-b8e8-5e92eb205f3e-cnibin\") pod \"multus-z5nxc\" (UID: \"993b825c-e7c4-445e-b8e8-5e92eb205f3e\") " pod="openshift-multus/multus-z5nxc"
Nov 02 12:26:45.123281 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.123223    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/af96531f-d9a0-4acc-ab22-62949e7400aa-kubelet-dir\") pod \"aws-ebs-csi-driver-node-kbhhk\" (UID: \"af96531f-d9a0-4acc-ab22-62949e7400aa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kbhhk"
Nov 02 12:26:45.123281 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.123233    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/af96531f-d9a0-4acc-ab22-62949e7400aa-device-dir\") pod \"aws-ebs-csi-driver-node-kbhhk\" (UID: \"af96531f-d9a0-4acc-ab22-62949e7400aa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kbhhk"
Nov 02 12:26:45.123281 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.123262    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8a2e896a-489b-43fa-ba40-b2e3c0d56176-host\") pod \"tuned-vnv9x\" (UID: \"8a2e896a-489b-43fa-ba40-b2e3c0d56176\") " pod="openshift-cluster-node-tuning-operator/tuned-vnv9x"
Nov 02 12:26:45.123281 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.123277    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a81d60e5-2049-4600-9b3f-d3826aee59fb-run-openvswitch\") pod \"ovnkube-node-qgv46\" (UID: \"a81d60e5-2049-4600-9b3f-d3826aee59fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgv46"
Nov 02 12:26:45.123281 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.123282    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/af96531f-d9a0-4acc-ab22-62949e7400aa-device-dir\") pod \"aws-ebs-csi-driver-node-kbhhk\" (UID: \"af96531f-d9a0-4acc-ab22-62949e7400aa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kbhhk"
Nov 02 12:26:45.123281 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.123293    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/09f70dd9-4dbe-46df-ac95-1c351a541a18-metrics-certs\") pod \"network-metrics-daemon-dp7h7\" (UID: \"09f70dd9-4dbe-46df-ac95-1c351a541a18\") " pod="openshift-multus/network-metrics-daemon-dp7h7"
Nov 02 12:26:45.124116 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.123307    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a81d60e5-2049-4600-9b3f-d3826aee59fb-run-openvswitch\") pod \"ovnkube-node-qgv46\" (UID: \"a81d60e5-2049-4600-9b3f-d3826aee59fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgv46"
Nov 02 12:26:45.124116 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.123310    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/af96531f-d9a0-4acc-ab22-62949e7400aa-socket-dir\") pod \"aws-ebs-csi-driver-node-kbhhk\" (UID: \"af96531f-d9a0-4acc-ab22-62949e7400aa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kbhhk"
Nov 02 12:26:45.124116 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.123328    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/993b825c-e7c4-445e-b8e8-5e92eb205f3e-system-cni-dir\") pod \"multus-z5nxc\" (UID: \"993b825c-e7c4-445e-b8e8-5e92eb205f3e\") " pod="openshift-multus/multus-z5nxc"
Nov 02 12:26:45.124116 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.123342    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a81d60e5-2049-4600-9b3f-d3826aee59fb-systemd-units\") pod \"ovnkube-node-qgv46\" (UID: \"a81d60e5-2049-4600-9b3f-d3826aee59fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgv46"
Nov 02 12:26:45.124116 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.123312    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8a2e896a-489b-43fa-ba40-b2e3c0d56176-host\") pod \"tuned-vnv9x\" (UID: \"8a2e896a-489b-43fa-ba40-b2e3c0d56176\") " pod="openshift-cluster-node-tuning-operator/tuned-vnv9x"
Nov 02 12:26:45.124116 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.123362    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a81d60e5-2049-4600-9b3f-d3826aee59fb-log-socket\") pod \"ovnkube-node-qgv46\" (UID: \"a81d60e5-2049-4600-9b3f-d3826aee59fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgv46"
Nov 02 12:26:45.124116 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.123384    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/993b825c-e7c4-445e-b8e8-5e92eb205f3e-system-cni-dir\") pod \"multus-z5nxc\" (UID: \"993b825c-e7c4-445e-b8e8-5e92eb205f3e\") " pod="openshift-multus/multus-z5nxc"
Nov 02 12:26:45.124116 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.123386    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/973dc2ad-34ed-4364-b01b-9abbf4f2d43b-crio-socket\") pod \"insights-runtime-extractor-wqcmj\" (UID: \"973dc2ad-34ed-4364-b01b-9abbf4f2d43b\") " pod="openshift-insights/insights-runtime-extractor-wqcmj"
Nov 02 12:26:45.124116 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.123393    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a81d60e5-2049-4600-9b3f-d3826aee59fb-systemd-units\") pod \"ovnkube-node-qgv46\" (UID: \"a81d60e5-2049-4600-9b3f-d3826aee59fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgv46"
Nov 02 12:26:45.124116 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.123397    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/af96531f-d9a0-4acc-ab22-62949e7400aa-socket-dir\") pod \"aws-ebs-csi-driver-node-kbhhk\" (UID: \"af96531f-d9a0-4acc-ab22-62949e7400aa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kbhhk"
Nov 02 12:26:45.124116 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.123412    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a81d60e5-2049-4600-9b3f-d3826aee59fb-log-socket\") pod \"ovnkube-node-qgv46\" (UID: \"a81d60e5-2049-4600-9b3f-d3826aee59fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgv46"
Nov 02 12:26:45.124116 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.123413    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/973dc2ad-34ed-4364-b01b-9abbf4f2d43b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wqcmj\" (UID: \"973dc2ad-34ed-4364-b01b-9abbf4f2d43b\") " pod="openshift-insights/insights-runtime-extractor-wqcmj"
Nov 02 12:26:45.124116 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.123440    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/feba123c-eeb3-412f-911e-a66da1981358-system-cni-dir\") pod \"multus-additional-cni-plugins-v7bcl\" (UID: \"feba123c-eeb3-412f-911e-a66da1981358\") " pod="openshift-multus/multus-additional-cni-plugins-v7bcl"
Nov 02 12:26:45.124116 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.123441    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/973dc2ad-34ed-4364-b01b-9abbf4f2d43b-crio-socket\") pod \"insights-runtime-extractor-wqcmj\" (UID: \"973dc2ad-34ed-4364-b01b-9abbf4f2d43b\") " pod="openshift-insights/insights-runtime-extractor-wqcmj"
Nov 02 12:26:45.124116 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.123455    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/993b825c-e7c4-445e-b8e8-5e92eb205f3e-multus-cni-dir\") pod \"multus-z5nxc\" (UID: \"993b825c-e7c4-445e-b8e8-5e92eb205f3e\") " pod="openshift-multus/multus-z5nxc"
Nov 02 12:26:45.124116 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.123480    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a81d60e5-2049-4600-9b3f-d3826aee59fb-host-cni-netd\") pod \"ovnkube-node-qgv46\" (UID: \"a81d60e5-2049-4600-9b3f-d3826aee59fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgv46"
Nov 02 12:26:45.124116 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.123481    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/feba123c-eeb3-412f-911e-a66da1981358-system-cni-dir\") pod \"multus-additional-cni-plugins-v7bcl\" (UID: \"feba123c-eeb3-412f-911e-a66da1981358\") " pod="openshift-multus/multus-additional-cni-plugins-v7bcl"
Nov 02 12:26:45.124116 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.123502    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a81d60e5-2049-4600-9b3f-d3826aee59fb-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qgv46\" (UID: \"a81d60e5-2049-4600-9b3f-d3826aee59fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgv46"
Nov 02 12:26:45.124116 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.123521    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/993b825c-e7c4-445e-b8e8-5e92eb205f3e-hostroot\") pod \"multus-z5nxc\" (UID: \"993b825c-e7c4-445e-b8e8-5e92eb205f3e\") " pod="openshift-multus/multus-z5nxc"
Nov 02 12:26:45.124116 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.123535    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/993b825c-e7c4-445e-b8e8-5e92eb205f3e-host-var-lib-cni-bin\") pod \"multus-z5nxc\" (UID: \"993b825c-e7c4-445e-b8e8-5e92eb205f3e\") " pod="openshift-multus/multus-z5nxc"
Nov 02 12:26:45.124116 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.123533    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/993b825c-e7c4-445e-b8e8-5e92eb205f3e-multus-cni-dir\") pod \"multus-z5nxc\" (UID: \"993b825c-e7c4-445e-b8e8-5e92eb205f3e\") " pod="openshift-multus/multus-z5nxc"
Nov 02 12:26:45.124116 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.123550    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8a2e896a-489b-43fa-ba40-b2e3c0d56176-etc-systemd\") pod \"tuned-vnv9x\" (UID: \"8a2e896a-489b-43fa-ba40-b2e3c0d56176\") " pod="openshift-cluster-node-tuning-operator/tuned-vnv9x"
Nov 02 12:26:45.124116 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.123565    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8a2e896a-489b-43fa-ba40-b2e3c0d56176-run\") pod \"tuned-vnv9x\" (UID: \"8a2e896a-489b-43fa-ba40-b2e3c0d56176\") " pod="openshift-cluster-node-tuning-operator/tuned-vnv9x"
Nov 02 12:26:45.124116 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.123577    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/993b825c-e7c4-445e-b8e8-5e92eb205f3e-hostroot\") pod \"multus-z5nxc\" (UID: \"993b825c-e7c4-445e-b8e8-5e92eb205f3e\") " pod="openshift-multus/multus-z5nxc"
Nov 02 12:26:45.124116 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.123586    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/993b825c-e7c4-445e-b8e8-5e92eb205f3e-host-var-lib-cni-bin\") pod \"multus-z5nxc\" (UID: \"993b825c-e7c4-445e-b8e8-5e92eb205f3e\") " pod="openshift-multus/multus-z5nxc"
Nov 02 12:26:45.124116 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.123519    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a81d60e5-2049-4600-9b3f-d3826aee59fb-host-cni-netd\") pod \"ovnkube-node-qgv46\" (UID: \"a81d60e5-2049-4600-9b3f-d3826aee59fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgv46"
Nov 02 12:26:45.124116 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.123609    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8a2e896a-489b-43fa-ba40-b2e3c0d56176-run\") pod \"tuned-vnv9x\" (UID: \"8a2e896a-489b-43fa-ba40-b2e3c0d56176\") " pod="openshift-cluster-node-tuning-operator/tuned-vnv9x"
Nov 02 12:26:45.124116 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.123609    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8a2e896a-489b-43fa-ba40-b2e3c0d56176-var-lib-kubelet\") pod \"tuned-vnv9x\" (UID: \"8a2e896a-489b-43fa-ba40-b2e3c0d56176\") " pod="openshift-cluster-node-tuning-operator/tuned-vnv9x"
Nov 02 12:26:45.124116 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.123546    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a81d60e5-2049-4600-9b3f-d3826aee59fb-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qgv46\" (UID: \"a81d60e5-2049-4600-9b3f-d3826aee59fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgv46"
Nov 02 12:26:45.124116 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.123664    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a81d60e5-2049-4600-9b3f-d3826aee59fb-host-run-ovn-kubernetes\") pod \"ovnkube-node-qgv46\" (UID: \"a81d60e5-2049-4600-9b3f-d3826aee59fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgv46"
Nov 02 12:26:45.124116 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.123659    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8a2e896a-489b-43fa-ba40-b2e3c0d56176-etc-systemd\") pod \"tuned-vnv9x\" (UID: \"8a2e896a-489b-43fa-ba40-b2e3c0d56176\") " pod="openshift-cluster-node-tuning-operator/tuned-vnv9x"
Nov 02 12:26:45.124116 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.123675    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8a2e896a-489b-43fa-ba40-b2e3c0d56176-var-lib-kubelet\") pod \"tuned-vnv9x\" (UID: \"8a2e896a-489b-43fa-ba40-b2e3c0d56176\") " pod="openshift-cluster-node-tuning-operator/tuned-vnv9x"
Nov 02 12:26:45.124116 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.123668    4559 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 02 12:26:45.124116 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.123706    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/83964713-5989-4b35-91e8-3d3993688ccd-kubelet-config\") pod \"global-pull-secret-syncer-6grht\" (UID: \"83964713-5989-4b35-91e8-3d3993688ccd\") " pod="kube-system/global-pull-secret-syncer-6grht"
Nov 02 12:26:45.124116 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.123722    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a81d60e5-2049-4600-9b3f-d3826aee59fb-host-run-ovn-kubernetes\") pod \"ovnkube-node-qgv46\" (UID: \"a81d60e5-2049-4600-9b3f-d3826aee59fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgv46"
Nov 02 12:26:45.124116 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.123730    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/feba123c-eeb3-412f-911e-a66da1981358-cnibin\") pod \"multus-additional-cni-plugins-v7bcl\" (UID: \"feba123c-eeb3-412f-911e-a66da1981358\") " pod="openshift-multus/multus-additional-cni-plugins-v7bcl"
Nov 02 12:26:45.124116 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.123765    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/feba123c-eeb3-412f-911e-a66da1981358-cnibin\") pod \"multus-additional-cni-plugins-v7bcl\" (UID: \"feba123c-eeb3-412f-911e-a66da1981358\") " pod="openshift-multus/multus-additional-cni-plugins-v7bcl"
Nov 02 12:26:45.124116 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.123774    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/83964713-5989-4b35-91e8-3d3993688ccd-kubelet-config\") pod \"global-pull-secret-syncer-6grht\" (UID: \"83964713-5989-4b35-91e8-3d3993688ccd\") " pod="kube-system/global-pull-secret-syncer-6grht"
Nov 02 12:26:45.124116 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.123766    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8a2e896a-489b-43fa-ba40-b2e3c0d56176-etc-sysctl-conf\") pod \"tuned-vnv9x\" (UID: \"8a2e896a-489b-43fa-ba40-b2e3c0d56176\") " pod="openshift-cluster-node-tuning-operator/tuned-vnv9x"
Nov 02 12:26:45.124116 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.123814    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5507fd15-57b9-422a-af1d-5daf266bd9ca-hosts-file\") pod \"node-resolver-7trhz\" (UID: \"5507fd15-57b9-422a-af1d-5daf266bd9ca\") " pod="openshift-dns/node-resolver-7trhz"
Nov 02 12:26:45.124116 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.123843    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a81d60e5-2049-4600-9b3f-d3826aee59fb-var-lib-openvswitch\") pod \"ovnkube-node-qgv46\" (UID: \"a81d60e5-2049-4600-9b3f-d3826aee59fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgv46"
Nov 02 12:26:45.124116 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.123853    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5507fd15-57b9-422a-af1d-5daf266bd9ca-hosts-file\") pod \"node-resolver-7trhz\" (UID: \"5507fd15-57b9-422a-af1d-5daf266bd9ca\") " pod="openshift-dns/node-resolver-7trhz"
Nov 02 12:26:45.124116 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.123867    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/993b825c-e7c4-445e-b8e8-5e92eb205f3e-multus-socket-dir-parent\") pod \"multus-z5nxc\" (UID: \"993b825c-e7c4-445e-b8e8-5e92eb205f3e\") " pod="openshift-multus/multus-z5nxc"
Nov 02 12:26:45.124116 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.123883    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a81d60e5-2049-4600-9b3f-d3826aee59fb-var-lib-openvswitch\") pod \"ovnkube-node-qgv46\" (UID: \"a81d60e5-2049-4600-9b3f-d3826aee59fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgv46"
Nov 02 12:26:45.124116 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.123889    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8a2e896a-489b-43fa-ba40-b2e3c0d56176-etc-sysctl-conf\") pod \"tuned-vnv9x\" (UID: \"8a2e896a-489b-43fa-ba40-b2e3c0d56176\") " pod="openshift-cluster-node-tuning-operator/tuned-vnv9x"
Nov 02 12:26:45.124116 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.123901    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/993b825c-e7c4-445e-b8e8-5e92eb205f3e-host-run-netns\") pod \"multus-z5nxc\" (UID: \"993b825c-e7c4-445e-b8e8-5e92eb205f3e\") " pod="openshift-multus/multus-z5nxc"
Nov 02 12:26:45.124116 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.123916    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/993b825c-e7c4-445e-b8e8-5e92eb205f3e-multus-socket-dir-parent\") pod \"multus-z5nxc\" (UID: \"993b825c-e7c4-445e-b8e8-5e92eb205f3e\") " pod="openshift-multus/multus-z5nxc"
Nov 02 12:26:45.124116 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.123900    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/973dc2ad-34ed-4364-b01b-9abbf4f2d43b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wqcmj\" (UID: \"973dc2ad-34ed-4364-b01b-9abbf4f2d43b\") " pod="openshift-insights/insights-runtime-extractor-wqcmj"
Nov 02 12:26:45.124116 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.123927    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/993b825c-e7c4-445e-b8e8-5e92eb205f3e-host-var-lib-cni-multus\") pod \"multus-z5nxc\" (UID: \"993b825c-e7c4-445e-b8e8-5e92eb205f3e\") " pod="openshift-multus/multus-z5nxc"
Nov 02 12:26:45.124116 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.123946    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/993b825c-e7c4-445e-b8e8-5e92eb205f3e-host-run-netns\") pod \"multus-z5nxc\" (UID: \"993b825c-e7c4-445e-b8e8-5e92eb205f3e\") " pod="openshift-multus/multus-z5nxc"
Nov 02 12:26:45.124116 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.123951    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/993b825c-e7c4-445e-b8e8-5e92eb205f3e-multus-conf-dir\") pod \"multus-z5nxc\" (UID: \"993b825c-e7c4-445e-b8e8-5e92eb205f3e\") " pod="openshift-multus/multus-z5nxc"
Nov 02 12:26:45.124116 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.123993    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/993b825c-e7c4-445e-b8e8-5e92eb205f3e-host-var-lib-cni-multus\") pod \"multus-z5nxc\" (UID: \"993b825c-e7c4-445e-b8e8-5e92eb205f3e\") " pod="openshift-multus/multus-z5nxc"
Nov 02 12:26:45.124116 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.124006    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/993b825c-e7c4-445e-b8e8-5e92eb205f3e-multus-conf-dir\") pod \"multus-z5nxc\" (UID: \"993b825c-e7c4-445e-b8e8-5e92eb205f3e\") " pod="openshift-multus/multus-z5nxc"
Nov 02 12:26:45.124116 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.124056    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/993b825c-e7c4-445e-b8e8-5e92eb205f3e-host-var-lib-kubelet\") pod \"multus-z5nxc\" (UID: \"993b825c-e7c4-445e-b8e8-5e92eb205f3e\") " pod="openshift-multus/multus-z5nxc"
Nov 02 12:26:45.124116 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.124091    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/993b825c-e7c4-445e-b8e8-5e92eb205f3e-host-var-lib-kubelet\") pod \"multus-z5nxc\" (UID: \"993b825c-e7c4-445e-b8e8-5e92eb205f3e\") " pod="openshift-multus/multus-z5nxc"
Nov 02 12:26:45.124116 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.124094    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a81d60e5-2049-4600-9b3f-d3826aee59fb-host-slash\") pod \"ovnkube-node-qgv46\" (UID: \"a81d60e5-2049-4600-9b3f-d3826aee59fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgv46"
Nov 02 12:26:45.124116 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.124138    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/af96531f-d9a0-4acc-ab22-62949e7400aa-registration-dir\") pod \"aws-ebs-csi-driver-node-kbhhk\" (UID: \"af96531f-d9a0-4acc-ab22-62949e7400aa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kbhhk"
Nov 02 12:26:45.124116 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.124161    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/af96531f-d9a0-4acc-ab22-62949e7400aa-sys-fs\") pod \"aws-ebs-csi-driver-node-kbhhk\" (UID: \"af96531f-d9a0-4acc-ab22-62949e7400aa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kbhhk"
Nov 02 12:26:45.126073 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.124160    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a81d60e5-2049-4600-9b3f-d3826aee59fb-host-slash\") pod \"ovnkube-node-qgv46\" (UID: \"a81d60e5-2049-4600-9b3f-d3826aee59fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgv46"
Nov 02 12:26:45.126073 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.124193    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/af96531f-d9a0-4acc-ab22-62949e7400aa-registration-dir\") pod \"aws-ebs-csi-driver-node-kbhhk\" (UID: \"af96531f-d9a0-4acc-ab22-62949e7400aa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kbhhk"
Nov 02 12:26:45.126073 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.124215    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/af96531f-d9a0-4acc-ab22-62949e7400aa-sys-fs\") pod \"aws-ebs-csi-driver-node-kbhhk\" (UID: \"af96531f-d9a0-4acc-ab22-62949e7400aa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kbhhk"
Nov 02 12:26:45.126073 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.124223    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8a2e896a-489b-43fa-ba40-b2e3c0d56176-etc-modprobe-d\") pod \"tuned-vnv9x\" (UID: \"8a2e896a-489b-43fa-ba40-b2e3c0d56176\") " pod="openshift-cluster-node-tuning-operator/tuned-vnv9x"
Nov 02 12:26:45.126073 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.124249    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8a2e896a-489b-43fa-ba40-b2e3c0d56176-etc-sysconfig\") pod \"tuned-vnv9x\" (UID: \"8a2e896a-489b-43fa-ba40-b2e3c0d56176\") " pod="openshift-cluster-node-tuning-operator/tuned-vnv9x"
Nov 02 12:26:45.126073 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.124269    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/993b825c-e7c4-445e-b8e8-5e92eb205f3e-etc-kubernetes\") pod \"multus-z5nxc\" (UID: \"993b825c-e7c4-445e-b8e8-5e92eb205f3e\") " pod="openshift-multus/multus-z5nxc"
Nov 02 12:26:45.126073 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.124287    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a81d60e5-2049-4600-9b3f-d3826aee59fb-node-log\") pod \"ovnkube-node-qgv46\" (UID: \"a81d60e5-2049-4600-9b3f-d3826aee59fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgv46"
Nov 02 12:26:45.126073 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.124309    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a81d60e5-2049-4600-9b3f-d3826aee59fb-run-systemd\") pod \"ovnkube-node-qgv46\" (UID: \"a81d60e5-2049-4600-9b3f-d3826aee59fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgv46"
Nov 02 12:26:45.126073 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.124315    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8a2e896a-489b-43fa-ba40-b2e3c0d56176-etc-modprobe-d\") pod \"tuned-vnv9x\" (UID: \"8a2e896a-489b-43fa-ba40-b2e3c0d56176\") " pod="openshift-cluster-node-tuning-operator/tuned-vnv9x"
Nov 02 12:26:45.126073 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.124335    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/af96531f-d9a0-4acc-ab22-62949e7400aa-etc-selinux\") pod \"aws-ebs-csi-driver-node-kbhhk\" (UID: \"af96531f-d9a0-4acc-ab22-62949e7400aa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kbhhk"
Nov 02 12:26:45.126073 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.124339    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/993b825c-e7c4-445e-b8e8-5e92eb205f3e-etc-kubernetes\") pod \"multus-z5nxc\" (UID: \"993b825c-e7c4-445e-b8e8-5e92eb205f3e\") " pod="openshift-multus/multus-z5nxc"
Nov 02 12:26:45.126073 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.124351    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a81d60e5-2049-4600-9b3f-d3826aee59fb-node-log\") pod \"ovnkube-node-qgv46\" (UID: \"a81d60e5-2049-4600-9b3f-d3826aee59fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgv46"
Nov 02 12:26:45.126073 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.124353    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a81d60e5-2049-4600-9b3f-d3826aee59fb-run-systemd\") pod \"ovnkube-node-qgv46\" (UID: \"a81d60e5-2049-4600-9b3f-d3826aee59fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgv46"
Nov 02 12:26:45.126073 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.124360    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8a2e896a-489b-43fa-ba40-b2e3c0d56176-etc-sysconfig\") pod \"tuned-vnv9x\" (UID: \"8a2e896a-489b-43fa-ba40-b2e3c0d56176\") " pod="openshift-cluster-node-tuning-operator/tuned-vnv9x"
Nov 02 12:26:45.126073 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.124360    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/feba123c-eeb3-412f-911e-a66da1981358-tuning-conf-dir\") pod \"multus-additional-cni-plugins-v7bcl\" (UID: \"feba123c-eeb3-412f-911e-a66da1981358\") " pod="openshift-multus/multus-additional-cni-plugins-v7bcl"
Nov 02 12:26:45.126073 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.124418    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/feba123c-eeb3-412f-911e-a66da1981358-tuning-conf-dir\") pod \"multus-additional-cni-plugins-v7bcl\" (UID: \"feba123c-eeb3-412f-911e-a66da1981358\") " pod="openshift-multus/multus-additional-cni-plugins-v7bcl"
Nov 02 12:26:45.126073 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.124397    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/af96531f-d9a0-4acc-ab22-62949e7400aa-etc-selinux\") pod \"aws-ebs-csi-driver-node-kbhhk\" (UID: \"af96531f-d9a0-4acc-ab22-62949e7400aa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kbhhk"
Nov 02 12:26:45.126073 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.124441    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/af2e7d8e-b1df-499c-9a92-9ebe9d7bc7b2-host-slash\") pod \"iptables-alerter-7gc6b\" (UID: \"af2e7d8e-b1df-499c-9a92-9ebe9d7bc7b2\") " pod="openshift-network-operator/iptables-alerter-7gc6b"
Nov 02 12:26:45.126073 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.124419    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/af2e7d8e-b1df-499c-9a92-9ebe9d7bc7b2-host-slash\") pod \"iptables-alerter-7gc6b\" (UID: \"af2e7d8e-b1df-499c-9a92-9ebe9d7bc7b2\") " pod="openshift-network-operator/iptables-alerter-7gc6b"
Nov 02 12:26:45.126073 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.124469    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8a2e896a-489b-43fa-ba40-b2e3c0d56176-lib-modules\") pod \"tuned-vnv9x\" (UID: \"8a2e896a-489b-43fa-ba40-b2e3c0d56176\") " pod="openshift-cluster-node-tuning-operator/tuned-vnv9x"
Nov 02 12:26:45.126073 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.124488    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8a2e896a-489b-43fa-ba40-b2e3c0d56176-etc-kubernetes\") pod \"tuned-vnv9x\" (UID: \"8a2e896a-489b-43fa-ba40-b2e3c0d56176\") " pod="openshift-cluster-node-tuning-operator/tuned-vnv9x"
Nov 02 12:26:45.126073 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.124504    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a81d60e5-2049-4600-9b3f-d3826aee59fb-host-run-netns\") pod \"ovnkube-node-qgv46\" (UID: \"a81d60e5-2049-4600-9b3f-d3826aee59fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgv46"
Nov 02 12:26:45.126073 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.124518    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a81d60e5-2049-4600-9b3f-d3826aee59fb-etc-openvswitch\") pod \"ovnkube-node-qgv46\" (UID: \"a81d60e5-2049-4600-9b3f-d3826aee59fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgv46"
Nov 02 12:26:45.126073 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.124533    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a81d60e5-2049-4600-9b3f-d3826aee59fb-host-cni-bin\") pod \"ovnkube-node-qgv46\" (UID: \"a81d60e5-2049-4600-9b3f-d3826aee59fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgv46"
Nov 02 12:26:45.126073 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.124540    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8a2e896a-489b-43fa-ba40-b2e3c0d56176-etc-kubernetes\") pod \"tuned-vnv9x\" (UID: \"8a2e896a-489b-43fa-ba40-b2e3c0d56176\") " pod="openshift-cluster-node-tuning-operator/tuned-vnv9x"
Nov 02 12:26:45.126073 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.124556    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/973dc2ad-34ed-4364-b01b-9abbf4f2d43b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wqcmj\" (UID: \"973dc2ad-34ed-4364-b01b-9abbf4f2d43b\") " pod="openshift-insights/insights-runtime-extractor-wqcmj"
Nov 02 12:26:45.126073 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.124557    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a81d60e5-2049-4600-9b3f-d3826aee59fb-host-run-netns\") pod \"ovnkube-node-qgv46\" (UID: \"a81d60e5-2049-4600-9b3f-d3826aee59fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgv46"
Nov 02 12:26:45.126073 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.124572    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a81d60e5-2049-4600-9b3f-d3826aee59fb-etc-openvswitch\") pod \"ovnkube-node-qgv46\" (UID: \"a81d60e5-2049-4600-9b3f-d3826aee59fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgv46"
Nov 02 12:26:45.126073 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.124568    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8a2e896a-489b-43fa-ba40-b2e3c0d56176-lib-modules\") pod \"tuned-vnv9x\" (UID: \"8a2e896a-489b-43fa-ba40-b2e3c0d56176\") " pod="openshift-cluster-node-tuning-operator/tuned-vnv9x"
Nov 02 12:26:45.126073 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.124586    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a81d60e5-2049-4600-9b3f-d3826aee59fb-host-cni-bin\") pod \"ovnkube-node-qgv46\" (UID: \"a81d60e5-2049-4600-9b3f-d3826aee59fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-qgv46"
Nov 02 12:26:45.126073 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.124597    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7b688512-ffc4-4b6a-b9ab-8fcf8ea0afe2-metrics-tls\") pod \"dns-default-bg5xh\" (UID: \"7b688512-ffc4-4b6a-b9ab-8fcf8ea0afe2\") " pod="openshift-dns/dns-default-bg5xh"
Nov 02 12:26:45.126073 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:26:45.124672    4559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 02 12:26:45.126073 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:26:45.124754    4559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/973dc2ad-34ed-4364-b01b-9abbf4f2d43b-insights-runtime-extractor-tls podName:973dc2ad-34ed-4364-b01b-9abbf4f2d43b nodeName:}" failed. No retries permitted until 2026-11-02 12:26:45.624739404 +0000 UTC m=+1.880138477 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/973dc2ad-34ed-4364-b01b-9abbf4f2d43b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wqcmj" (UID: "973dc2ad-34ed-4364-b01b-9abbf4f2d43b") : secret "insights-runtime-extractor-tls" not found
Nov 02 12:26:45.126073 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.125866    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/09f70dd9-4dbe-46df-ac95-1c351a541a18-metrics-certs\") pod \"network-metrics-daemon-dp7h7\" (UID: \"09f70dd9-4dbe-46df-ac95-1c351a541a18\") " pod="openshift-multus/network-metrics-daemon-dp7h7"
Nov 02 12:26:45.127878 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.126706    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7b688512-ffc4-4b6a-b9ab-8fcf8ea0afe2-metrics-tls\") pod \"dns-default-bg5xh\" (UID: \"7b688512-ffc4-4b6a-b9ab-8fcf8ea0afe2\") " pod="openshift-dns/dns-default-bg5xh"
Nov 02 12:26:45.185992 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.185965    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dp7h7"
Nov 02 12:26:45.192174 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.192142    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-bg5xh"
Nov 02 12:26:45.324522 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.324494    4559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-9t4f5_20cc95cd-9718-49ee-8082-e9ac00fcc147/node-ca/0.log"
Nov 02 12:26:45.516684 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.516618    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-dp7h7"]
Nov 02 12:26:45.518488 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:45.518458    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09f70dd9_4dbe_46df_ac95_1c351a541a18.slice/crio-e2f1c058757da306662ecdb9549736dfdd7cf7da03b256c393d4737cc6bf9de4 WatchSource:0}: Error finding container e2f1c058757da306662ecdb9549736dfdd7cf7da03b256c393d4737cc6bf9de4: Status 404 returned error can't find the container with id e2f1c058757da306662ecdb9549736dfdd7cf7da03b256c393d4737cc6bf9de4
Nov 02 12:26:45.520595 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.520580    4559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 12:26:45.542535 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.542514    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-bg5xh"]
Nov 02 12:26:45.542622 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:45.542594    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b688512_ffc4_4b6a_b9ab_8fcf8ea0afe2.slice/crio-efa7fcafdbe369ce8a953070ef1180094f08401661104e1d09fb4c4b0fc077ba WatchSource:0}: Error finding container efa7fcafdbe369ce8a953070ef1180094f08401661104e1d09fb4c4b0fc077ba: Status 404 returned error can't find the container with id efa7fcafdbe369ce8a953070ef1180094f08401661104e1d09fb4c4b0fc077ba
Nov 02 12:26:45.593607 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.593582    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qgv46"
Nov 02 12:26:45.627634 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.627604    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/973dc2ad-34ed-4364-b01b-9abbf4f2d43b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wqcmj\" (UID: \"973dc2ad-34ed-4364-b01b-9abbf4f2d43b\") " pod="openshift-insights/insights-runtime-extractor-wqcmj"
Nov 02 12:26:45.627814 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:26:45.627742    4559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 02 12:26:45.627814 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:26:45.627799    4559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/973dc2ad-34ed-4364-b01b-9abbf4f2d43b-insights-runtime-extractor-tls podName:973dc2ad-34ed-4364-b01b-9abbf4f2d43b nodeName:}" failed. No retries permitted until 2026-11-02 12:26:46.627783687 +0000 UTC m=+2.883182780 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/973dc2ad-34ed-4364-b01b-9abbf4f2d43b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wqcmj" (UID: "973dc2ad-34ed-4364-b01b-9abbf4f2d43b") : secret "insights-runtime-extractor-tls" not found
Nov 02 12:26:45.948327 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.948287    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-dp7h7" event={"ID":"09f70dd9-4dbe-46df-ac95-1c351a541a18","Type":"ContainerStarted","Data":"e2f1c058757da306662ecdb9549736dfdd7cf7da03b256c393d4737cc6bf9de4"}
Nov 02 12:26:45.949469 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:45.949441    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-bg5xh" event={"ID":"7b688512-ffc4-4b6a-b9ab-8fcf8ea0afe2","Type":"ContainerStarted","Data":"efa7fcafdbe369ce8a953070ef1180094f08401661104e1d09fb4c4b0fc077ba"}
Nov 02 12:26:46.124791 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:46.124745    4559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-9x7t5_f9be3b9a-c86a-473e-8a58-83d62a0cf1d7/serve-healthcheck-canary/0.log"
Nov 02 12:26:46.526594 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:46.526568    4559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-fmzqj_a5142bd8-95b7-499e-9885-d867d915f3a8/migrator/0.log"
Nov 02 12:26:46.636248 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:46.636221    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/973dc2ad-34ed-4364-b01b-9abbf4f2d43b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wqcmj\" (UID: \"973dc2ad-34ed-4364-b01b-9abbf4f2d43b\") " pod="openshift-insights/insights-runtime-extractor-wqcmj"
Nov 02 12:26:46.636409 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:26:46.636386    4559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 02 12:26:46.636472 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:26:46.636465    4559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/973dc2ad-34ed-4364-b01b-9abbf4f2d43b-insights-runtime-extractor-tls podName:973dc2ad-34ed-4364-b01b-9abbf4f2d43b nodeName:}" failed. No retries permitted until 2026-11-02 12:26:48.636445132 +0000 UTC m=+4.891844212 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/973dc2ad-34ed-4364-b01b-9abbf4f2d43b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wqcmj" (UID: "973dc2ad-34ed-4364-b01b-9abbf4f2d43b") : secret "insights-runtime-extractor-tls" not found
Nov 02 12:26:46.955826 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:46.955731    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-dp7h7" event={"ID":"09f70dd9-4dbe-46df-ac95-1c351a541a18","Type":"ContainerStarted","Data":"e969036593b0f7b16af493ccd2a08064c8e98b74126e3528799dfa5d6b689c93"}
Nov 02 12:26:46.955826 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:46.955789    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-dp7h7" event={"ID":"09f70dd9-4dbe-46df-ac95-1c351a541a18","Type":"ContainerStarted","Data":"dbea60fcf554c9f1f28e57913f50cb35b01f9a8e422acb02e9476dc06589cb47"}
Nov 02 12:26:47.301934 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:47.301909    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-wx6zw"
Nov 02 12:26:47.959939 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:47.959909    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-bg5xh" event={"ID":"7b688512-ffc4-4b6a-b9ab-8fcf8ea0afe2","Type":"ContainerStarted","Data":"5ffa09e2efd4d20fa065c4ad3db85fda0875f57ef919cf9804bee3081000bba5"}
Nov 02 12:26:47.960423 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:47.960003    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-bg5xh"
Nov 02 12:26:47.960423 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:47.960047    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-bg5xh" event={"ID":"7b688512-ffc4-4b6a-b9ab-8fcf8ea0afe2","Type":"ContainerStarted","Data":"18f5f702370aa0fe6eb33687439c1f5d5b00f73802b9bb9b8988b01acbbbef1e"}
Nov 02 12:26:48.653518 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:48.653481    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/973dc2ad-34ed-4364-b01b-9abbf4f2d43b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wqcmj\" (UID: \"973dc2ad-34ed-4364-b01b-9abbf4f2d43b\") " pod="openshift-insights/insights-runtime-extractor-wqcmj"
Nov 02 12:26:48.653714 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:26:48.653655    4559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 02 12:26:48.653756 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:26:48.653731    4559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/973dc2ad-34ed-4364-b01b-9abbf4f2d43b-insights-runtime-extractor-tls podName:973dc2ad-34ed-4364-b01b-9abbf4f2d43b nodeName:}" failed. No retries permitted until 2026-11-02 12:26:52.65371413 +0000 UTC m=+8.909113209 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/973dc2ad-34ed-4364-b01b-9abbf4f2d43b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wqcmj" (UID: "973dc2ad-34ed-4364-b01b-9abbf4f2d43b") : secret "insights-runtime-extractor-tls" not found
Nov 02 12:26:52.678300 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:52.678266    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/973dc2ad-34ed-4364-b01b-9abbf4f2d43b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wqcmj\" (UID: \"973dc2ad-34ed-4364-b01b-9abbf4f2d43b\") " pod="openshift-insights/insights-runtime-extractor-wqcmj"
Nov 02 12:26:52.678827 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:26:52.678420    4559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 02 12:26:52.678827 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:26:52.678509    4559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/973dc2ad-34ed-4364-b01b-9abbf4f2d43b-insights-runtime-extractor-tls podName:973dc2ad-34ed-4364-b01b-9abbf4f2d43b nodeName:}" failed. No retries permitted until 2026-11-02 12:27:00.678493305 +0000 UTC m=+16.933892377 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/973dc2ad-34ed-4364-b01b-9abbf4f2d43b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wqcmj" (UID: "973dc2ad-34ed-4364-b01b-9abbf4f2d43b") : secret "insights-runtime-extractor-tls" not found
Nov 02 12:26:54.052519 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:54.052490    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qgv46"
Nov 02 12:26:54.052933 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:54.052729    4559 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 02 12:26:57.405438 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:57.405406    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-4vfzb"]
Nov 02 12:26:57.408425 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:57.408409    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-4vfzb"
Nov 02 12:26:57.433814 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:57.433775    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-4vfzb"]
Nov 02 12:26:57.446513 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:57.446486    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Nov 02 12:26:57.446513 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:57.446513    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-dxwl2\""
Nov 02 12:26:57.507614 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:57.507580    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-75fd5f49d8-7t6q5"]
Nov 02 12:26:57.510538 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:57.510521    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-75fd5f49d8-7t6q5"
Nov 02 12:26:57.511651 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:57.511617    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/7134dbc9-ca71-47f2-886f-d80d2a6ae912-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-4vfzb\" (UID: \"7134dbc9-ca71-47f2-886f-d80d2a6ae912\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-4vfzb"
Nov 02 12:26:57.555304 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:57.555271    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Nov 02 12:26:57.555304 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:57.555300    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-hnbq6\""
Nov 02 12:26:57.555520 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:57.555483    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Nov 02 12:26:57.555577 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:57.555550    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Nov 02 12:26:57.555658 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:57.555607    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Nov 02 12:26:57.556171 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:57.556154    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-75fd5f49d8-7t6q5"]
Nov 02 12:26:57.612270 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:57.612240    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/96cbbd04-22df-4987-8fd5-5ee673985f5e-registry-certificates\") pod \"image-registry-75fd5f49d8-7t6q5\" (UID: \"96cbbd04-22df-4987-8fd5-5ee673985f5e\") " pod="openshift-image-registry/image-registry-75fd5f49d8-7t6q5"
Nov 02 12:26:57.612455 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:57.612279    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/96cbbd04-22df-4987-8fd5-5ee673985f5e-bound-sa-token\") pod \"image-registry-75fd5f49d8-7t6q5\" (UID: \"96cbbd04-22df-4987-8fd5-5ee673985f5e\") " pod="openshift-image-registry/image-registry-75fd5f49d8-7t6q5"
Nov 02 12:26:57.612455 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:57.612313    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/7134dbc9-ca71-47f2-886f-d80d2a6ae912-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-4vfzb\" (UID: \"7134dbc9-ca71-47f2-886f-d80d2a6ae912\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-4vfzb"
Nov 02 12:26:57.612455 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:57.612374    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/96cbbd04-22df-4987-8fd5-5ee673985f5e-installation-pull-secrets\") pod \"image-registry-75fd5f49d8-7t6q5\" (UID: \"96cbbd04-22df-4987-8fd5-5ee673985f5e\") " pod="openshift-image-registry/image-registry-75fd5f49d8-7t6q5"
Nov 02 12:26:57.612455 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:57.612413    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/96cbbd04-22df-4987-8fd5-5ee673985f5e-ca-trust-extracted\") pod \"image-registry-75fd5f49d8-7t6q5\" (UID: \"96cbbd04-22df-4987-8fd5-5ee673985f5e\") " pod="openshift-image-registry/image-registry-75fd5f49d8-7t6q5"
Nov 02 12:26:57.612455 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:57.612444    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/96cbbd04-22df-4987-8fd5-5ee673985f5e-image-registry-private-configuration\") pod \"image-registry-75fd5f49d8-7t6q5\" (UID: \"96cbbd04-22df-4987-8fd5-5ee673985f5e\") " pod="openshift-image-registry/image-registry-75fd5f49d8-7t6q5"
Nov 02 12:26:57.612713 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:57.612473    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/96cbbd04-22df-4987-8fd5-5ee673985f5e-registry-tls\") pod \"image-registry-75fd5f49d8-7t6q5\" (UID: \"96cbbd04-22df-4987-8fd5-5ee673985f5e\") " pod="openshift-image-registry/image-registry-75fd5f49d8-7t6q5"
Nov 02 12:26:57.612713 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:57.612566    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpx5f\" (UniqueName: \"kubernetes.io/projected/96cbbd04-22df-4987-8fd5-5ee673985f5e-kube-api-access-vpx5f\") pod \"image-registry-75fd5f49d8-7t6q5\" (UID: \"96cbbd04-22df-4987-8fd5-5ee673985f5e\") " pod="openshift-image-registry/image-registry-75fd5f49d8-7t6q5"
Nov 02 12:26:57.612713 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:57.612604    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/96cbbd04-22df-4987-8fd5-5ee673985f5e-trusted-ca\") pod \"image-registry-75fd5f49d8-7t6q5\" (UID: \"96cbbd04-22df-4987-8fd5-5ee673985f5e\") " pod="openshift-image-registry/image-registry-75fd5f49d8-7t6q5"
Nov 02 12:26:57.614778 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:57.614761    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/7134dbc9-ca71-47f2-886f-d80d2a6ae912-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-4vfzb\" (UID: \"7134dbc9-ca71-47f2-886f-d80d2a6ae912\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-4vfzb"
Nov 02 12:26:57.713883 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:57.713852    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/96cbbd04-22df-4987-8fd5-5ee673985f5e-bound-sa-token\") pod \"image-registry-75fd5f49d8-7t6q5\" (UID: \"96cbbd04-22df-4987-8fd5-5ee673985f5e\") " pod="openshift-image-registry/image-registry-75fd5f49d8-7t6q5"
Nov 02 12:26:57.714076 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:57.713898    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/96cbbd04-22df-4987-8fd5-5ee673985f5e-installation-pull-secrets\") pod \"image-registry-75fd5f49d8-7t6q5\" (UID: \"96cbbd04-22df-4987-8fd5-5ee673985f5e\") " pod="openshift-image-registry/image-registry-75fd5f49d8-7t6q5"
Nov 02 12:26:57.714076 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:57.713918    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/96cbbd04-22df-4987-8fd5-5ee673985f5e-ca-trust-extracted\") pod \"image-registry-75fd5f49d8-7t6q5\" (UID: \"96cbbd04-22df-4987-8fd5-5ee673985f5e\") " pod="openshift-image-registry/image-registry-75fd5f49d8-7t6q5"
Nov 02 12:26:57.714076 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:57.713939    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/96cbbd04-22df-4987-8fd5-5ee673985f5e-image-registry-private-configuration\") pod \"image-registry-75fd5f49d8-7t6q5\" (UID: \"96cbbd04-22df-4987-8fd5-5ee673985f5e\") " pod="openshift-image-registry/image-registry-75fd5f49d8-7t6q5"
Nov 02 12:26:57.714076 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:57.713960    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/96cbbd04-22df-4987-8fd5-5ee673985f5e-registry-tls\") pod \"image-registry-75fd5f49d8-7t6q5\" (UID: \"96cbbd04-22df-4987-8fd5-5ee673985f5e\") " pod="openshift-image-registry/image-registry-75fd5f49d8-7t6q5"
Nov 02 12:26:57.714606 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:57.714568    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpx5f\" (UniqueName: \"kubernetes.io/projected/96cbbd04-22df-4987-8fd5-5ee673985f5e-kube-api-access-vpx5f\") pod \"image-registry-75fd5f49d8-7t6q5\" (UID: \"96cbbd04-22df-4987-8fd5-5ee673985f5e\") " pod="openshift-image-registry/image-registry-75fd5f49d8-7t6q5"
Nov 02 12:26:57.714819 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:57.714594    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/96cbbd04-22df-4987-8fd5-5ee673985f5e-ca-trust-extracted\") pod \"image-registry-75fd5f49d8-7t6q5\" (UID: \"96cbbd04-22df-4987-8fd5-5ee673985f5e\") " pod="openshift-image-registry/image-registry-75fd5f49d8-7t6q5"
Nov 02 12:26:57.714980 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:57.714951    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/96cbbd04-22df-4987-8fd5-5ee673985f5e-trusted-ca\") pod \"image-registry-75fd5f49d8-7t6q5\" (UID: \"96cbbd04-22df-4987-8fd5-5ee673985f5e\") " pod="openshift-image-registry/image-registry-75fd5f49d8-7t6q5"
Nov 02 12:26:57.715108 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:57.715000    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/96cbbd04-22df-4987-8fd5-5ee673985f5e-registry-certificates\") pod \"image-registry-75fd5f49d8-7t6q5\" (UID: \"96cbbd04-22df-4987-8fd5-5ee673985f5e\") " pod="openshift-image-registry/image-registry-75fd5f49d8-7t6q5"
Nov 02 12:26:57.716734 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:57.716465    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/96cbbd04-22df-4987-8fd5-5ee673985f5e-registry-certificates\") pod \"image-registry-75fd5f49d8-7t6q5\" (UID: \"96cbbd04-22df-4987-8fd5-5ee673985f5e\") " pod="openshift-image-registry/image-registry-75fd5f49d8-7t6q5"
Nov 02 12:26:57.716824 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:57.716760    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/96cbbd04-22df-4987-8fd5-5ee673985f5e-trusted-ca\") pod \"image-registry-75fd5f49d8-7t6q5\" (UID: \"96cbbd04-22df-4987-8fd5-5ee673985f5e\") " pod="openshift-image-registry/image-registry-75fd5f49d8-7t6q5"
Nov 02 12:26:57.720027 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:57.717242    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/96cbbd04-22df-4987-8fd5-5ee673985f5e-image-registry-private-configuration\") pod \"image-registry-75fd5f49d8-7t6q5\" (UID: \"96cbbd04-22df-4987-8fd5-5ee673985f5e\") " pod="openshift-image-registry/image-registry-75fd5f49d8-7t6q5"
Nov 02 12:26:57.720027 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:57.717913    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/96cbbd04-22df-4987-8fd5-5ee673985f5e-registry-tls\") pod \"image-registry-75fd5f49d8-7t6q5\" (UID: \"96cbbd04-22df-4987-8fd5-5ee673985f5e\") " pod="openshift-image-registry/image-registry-75fd5f49d8-7t6q5"
Nov 02 12:26:57.720027 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:57.718913    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-4vfzb"
Nov 02 12:26:57.720027 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:57.719264    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/96cbbd04-22df-4987-8fd5-5ee673985f5e-installation-pull-secrets\") pod \"image-registry-75fd5f49d8-7t6q5\" (UID: \"96cbbd04-22df-4987-8fd5-5ee673985f5e\") " pod="openshift-image-registry/image-registry-75fd5f49d8-7t6q5"
Nov 02 12:26:57.738407 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:57.738377    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/96cbbd04-22df-4987-8fd5-5ee673985f5e-bound-sa-token\") pod \"image-registry-75fd5f49d8-7t6q5\" (UID: \"96cbbd04-22df-4987-8fd5-5ee673985f5e\") " pod="openshift-image-registry/image-registry-75fd5f49d8-7t6q5"
Nov 02 12:26:57.739505 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:57.739479    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpx5f\" (UniqueName: \"kubernetes.io/projected/96cbbd04-22df-4987-8fd5-5ee673985f5e-kube-api-access-vpx5f\") pod \"image-registry-75fd5f49d8-7t6q5\" (UID: \"96cbbd04-22df-4987-8fd5-5ee673985f5e\") " pod="openshift-image-registry/image-registry-75fd5f49d8-7t6q5"
Nov 02 12:26:57.821396 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:57.821363    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-75fd5f49d8-7t6q5"
Nov 02 12:26:57.967368 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:57.967292    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-bg5xh"
Nov 02 12:26:58.018473 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:58.018453    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-4vfzb"]
Nov 02 12:26:58.020419 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:58.020392    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7134dbc9_ca71_47f2_886f_d80d2a6ae912.slice/crio-8e61e98378341c0521cfbcf935e586ad9bdd9de577a1e0e402bc2c0ace69de19 WatchSource:0}: Error finding container 8e61e98378341c0521cfbcf935e586ad9bdd9de577a1e0e402bc2c0ace69de19: Status 404 returned error can't find the container with id 8e61e98378341c0521cfbcf935e586ad9bdd9de577a1e0e402bc2c0ace69de19
Nov 02 12:26:58.119308 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:58.119277    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-75fd5f49d8-7t6q5"]
Nov 02 12:26:58.120177 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:26:58.120157    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96cbbd04_22df_4987_8fd5_5ee673985f5e.slice/crio-d5b321a4b71083b1eac5888bc8acd16f896f23aea5f8101e2ce3e6a1232acc28 WatchSource:0}: Error finding container d5b321a4b71083b1eac5888bc8acd16f896f23aea5f8101e2ce3e6a1232acc28: Status 404 returned error can't find the container with id d5b321a4b71083b1eac5888bc8acd16f896f23aea5f8101e2ce3e6a1232acc28
Nov 02 12:26:58.996049 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:58.996017    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-4vfzb" event={"ID":"7134dbc9-ca71-47f2-886f-d80d2a6ae912","Type":"ContainerStarted","Data":"8e61e98378341c0521cfbcf935e586ad9bdd9de577a1e0e402bc2c0ace69de19"}
Nov 02 12:26:58.997394 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:58.997372    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-75fd5f49d8-7t6q5" event={"ID":"96cbbd04-22df-4987-8fd5-5ee673985f5e","Type":"ContainerStarted","Data":"49209718b1bba238938a6f1cd3d99d198a481c35c9ac13d53d73944fb104cd22"}
Nov 02 12:26:58.997497 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:58.997413    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-75fd5f49d8-7t6q5" event={"ID":"96cbbd04-22df-4987-8fd5-5ee673985f5e","Type":"ContainerStarted","Data":"d5b321a4b71083b1eac5888bc8acd16f896f23aea5f8101e2ce3e6a1232acc28"}
Nov 02 12:26:58.997618 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:58.997599    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-75fd5f49d8-7t6q5"
Nov 02 12:26:59.046831 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:59.046737    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-75fd5f49d8-7t6q5" podStartSLOduration=2.046721778 podStartE2EDuration="2.046721778s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:26:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:26:59.046227975 +0000 UTC m=+15.301627072" watchObservedRunningTime="2026-11-02 12:26:59.046721778 +0000 UTC m=+15.302120874"
Nov 02 12:26:59.143544 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:26:59.143516    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qgv46"
Nov 02 12:27:00.001884 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:27:00.001850    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-4vfzb" event={"ID":"7134dbc9-ca71-47f2-886f-d80d2a6ae912","Type":"ContainerStarted","Data":"ca067e2401d71cf22fd793c304928e4992ba4a7080aef968b37422e6f4e86fda"}
Nov 02 12:27:00.004337 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:27:00.002682    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-4vfzb"
Nov 02 12:27:00.009095 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:27:00.009071    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-4vfzb"
Nov 02 12:27:00.043595 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:27:00.043504    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-4vfzb" podStartSLOduration=2.103996562 podStartE2EDuration="3.04348863s" totalImagesPullingTime="939.492068ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:26:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:27:00.043153419 +0000 UTC m=+16.298552514" watchObservedRunningTime="2026-11-02 12:27:00.04348863 +0000 UTC m=+16.298887726"
Nov 02 12:27:00.409084 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:27:00.409018    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-565jh"]
Nov 02 12:27:00.412219 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:27:00.412203    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-565jh"
Nov 02 12:27:00.432997 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:27:00.432971    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Nov 02 12:27:00.445620 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:27:00.445596    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Nov 02 12:27:00.446111 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:27:00.446091    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Nov 02 12:27:00.446111 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:27:00.446110    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Nov 02 12:27:00.446217 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:27:00.446123    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Nov 02 12:27:00.446691 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:27:00.446673    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-8xsjn\""
Nov 02 12:27:00.447421 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:27:00.447399    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-565jh"]
Nov 02 12:27:00.536893 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:27:00.536856    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b7da0f69-d13d-42db-96ee-32949d5bb600-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-565jh\" (UID: \"b7da0f69-d13d-42db-96ee-32949d5bb600\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-565jh"
Nov 02 12:27:00.537058 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:27:00.536903    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/b7da0f69-d13d-42db-96ee-32949d5bb600-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-565jh\" (UID: \"b7da0f69-d13d-42db-96ee-32949d5bb600\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-565jh"
Nov 02 12:27:00.537058 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:27:00.536928    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b7da0f69-d13d-42db-96ee-32949d5bb600-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-565jh\" (UID: \"b7da0f69-d13d-42db-96ee-32949d5bb600\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-565jh"
Nov 02 12:27:00.537058 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:27:00.537042    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zp2t6\" (UniqueName: \"kubernetes.io/projected/b7da0f69-d13d-42db-96ee-32949d5bb600-kube-api-access-zp2t6\") pod \"prometheus-operator-9cd89b74c-565jh\" (UID: \"b7da0f69-d13d-42db-96ee-32949d5bb600\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-565jh"
Nov 02 12:27:00.638384 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:27:00.638350    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/b7da0f69-d13d-42db-96ee-32949d5bb600-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-565jh\" (UID: \"b7da0f69-d13d-42db-96ee-32949d5bb600\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-565jh"
Nov 02 12:27:00.638384 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:27:00.638387    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b7da0f69-d13d-42db-96ee-32949d5bb600-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-565jh\" (UID: \"b7da0f69-d13d-42db-96ee-32949d5bb600\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-565jh"
Nov 02 12:27:00.638582 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:27:00.638560    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zp2t6\" (UniqueName: \"kubernetes.io/projected/b7da0f69-d13d-42db-96ee-32949d5bb600-kube-api-access-zp2t6\") pod \"prometheus-operator-9cd89b74c-565jh\" (UID: \"b7da0f69-d13d-42db-96ee-32949d5bb600\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-565jh"
Nov 02 12:27:00.638695 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:27:00.638676    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b7da0f69-d13d-42db-96ee-32949d5bb600-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-565jh\" (UID: \"b7da0f69-d13d-42db-96ee-32949d5bb600\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-565jh"
Nov 02 12:27:00.639147 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:27:00.639127    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b7da0f69-d13d-42db-96ee-32949d5bb600-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-565jh\" (UID: \"b7da0f69-d13d-42db-96ee-32949d5bb600\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-565jh"
Nov 02 12:27:00.640886 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:27:00.640863    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b7da0f69-d13d-42db-96ee-32949d5bb600-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-565jh\" (UID: \"b7da0f69-d13d-42db-96ee-32949d5bb600\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-565jh"
Nov 02 12:27:00.640963 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:27:00.640873    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/b7da0f69-d13d-42db-96ee-32949d5bb600-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-565jh\" (UID: \"b7da0f69-d13d-42db-96ee-32949d5bb600\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-565jh"
Nov 02 12:27:00.663271 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:27:00.663221    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zp2t6\" (UniqueName: \"kubernetes.io/projected/b7da0f69-d13d-42db-96ee-32949d5bb600-kube-api-access-zp2t6\") pod \"prometheus-operator-9cd89b74c-565jh\" (UID: \"b7da0f69-d13d-42db-96ee-32949d5bb600\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-565jh"
Nov 02 12:27:00.725128 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:27:00.725093    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-565jh"
Nov 02 12:27:00.739438 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:27:00.739414    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/973dc2ad-34ed-4364-b01b-9abbf4f2d43b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wqcmj\" (UID: \"973dc2ad-34ed-4364-b01b-9abbf4f2d43b\") " pod="openshift-insights/insights-runtime-extractor-wqcmj"
Nov 02 12:27:00.741657 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:27:00.741626    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/973dc2ad-34ed-4364-b01b-9abbf4f2d43b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wqcmj\" (UID: \"973dc2ad-34ed-4364-b01b-9abbf4f2d43b\") " pod="openshift-insights/insights-runtime-extractor-wqcmj"
Nov 02 12:27:00.793941 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:27:00.793914    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-wqcmj"
Nov 02 12:27:01.018253 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:27:01.018224    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-565jh"]
Nov 02 12:27:01.018863 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:27:01.018839    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7da0f69_d13d_42db_96ee_32949d5bb600.slice/crio-9c3813b18834fedcbaae598b50e8062454235bebdb7ec8af351d5729921c6098 WatchSource:0}: Error finding container 9c3813b18834fedcbaae598b50e8062454235bebdb7ec8af351d5729921c6098: Status 404 returned error can't find the container with id 9c3813b18834fedcbaae598b50e8062454235bebdb7ec8af351d5729921c6098
Nov 02 12:27:01.089090 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:27:01.089062    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-wqcmj"]
Nov 02 12:27:01.089714 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:27:01.089687    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod973dc2ad_34ed_4364_b01b_9abbf4f2d43b.slice/crio-6650adb1b59e14a6f4c86b97b3481f1f4e05cc5a9702f74d393d28592ef3f7a7 WatchSource:0}: Error finding container 6650adb1b59e14a6f4c86b97b3481f1f4e05cc5a9702f74d393d28592ef3f7a7: Status 404 returned error can't find the container with id 6650adb1b59e14a6f4c86b97b3481f1f4e05cc5a9702f74d393d28592ef3f7a7
Nov 02 12:27:02.009278 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:27:02.009209    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-565jh" event={"ID":"b7da0f69-d13d-42db-96ee-32949d5bb600","Type":"ContainerStarted","Data":"9c3813b18834fedcbaae598b50e8062454235bebdb7ec8af351d5729921c6098"}
Nov 02 12:27:02.011354 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:27:02.011328    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wqcmj" event={"ID":"973dc2ad-34ed-4364-b01b-9abbf4f2d43b","Type":"ContainerStarted","Data":"fbd56c7bb74d4f6f2c870ff23d672a790cb4cd62e1c87345675fbf60d20d2281"}
Nov 02 12:27:02.011483 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:27:02.011389    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wqcmj" event={"ID":"973dc2ad-34ed-4364-b01b-9abbf4f2d43b","Type":"ContainerStarted","Data":"fb8da22a7e3b692b3146ad37d34115aba8f886062b82ffa9ef279872cfe0fe4b"}
Nov 02 12:27:02.011483 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:27:02.011418    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wqcmj" event={"ID":"973dc2ad-34ed-4364-b01b-9abbf4f2d43b","Type":"ContainerStarted","Data":"6650adb1b59e14a6f4c86b97b3481f1f4e05cc5a9702f74d393d28592ef3f7a7"}
Nov 02 12:27:03.016825 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:27:03.016791    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-565jh" event={"ID":"b7da0f69-d13d-42db-96ee-32949d5bb600","Type":"ContainerStarted","Data":"b7a8102c9575f5602fcfc4b3df60957c1b2fbfa36c79abb791f5dcff5da504cf"}
Nov 02 12:27:03.017312 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:27:03.016888    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-565jh" event={"ID":"b7da0f69-d13d-42db-96ee-32949d5bb600","Type":"ContainerStarted","Data":"f38db7747009ec19fc5939c500b3279d45736e5a7b847bb07c66b6973237b9b9"}
Nov 02 12:27:03.024275 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:27:03.024238    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wqcmj" event={"ID":"973dc2ad-34ed-4364-b01b-9abbf4f2d43b","Type":"ContainerStarted","Data":"235d27e68c03d3a5513dde2d8434bf1e59d904c8b2f8a6d33feb61ce42363dff"}
Nov 02 12:27:03.061120 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:27:03.061067    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-9cd89b74c-565jh" podStartSLOduration=1.844485842 podStartE2EDuration="3.061051033s" totalImagesPullingTime="1.216565191s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:27:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:27:03.060545957 +0000 UTC m=+19.315945065" watchObservedRunningTime="2026-11-02 12:27:03.061051033 +0000 UTC m=+19.316450127"
Nov 02 12:27:05.049209 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:27:05.049175    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-wt562"]
Nov 02 12:27:05.054529 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:27:05.054501    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-wt562"
Nov 02 12:27:05.091930 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:27:05.091899    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Nov 02 12:27:05.092056 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:27:05.092022    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Nov 02 12:27:05.092095 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:27:05.092057    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-89s57\""
Nov 02 12:27:05.156528 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:27:05.156506    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Nov 02 12:27:05.170781 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:27:05.170755    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/614c03cd-6d2c-4fee-8718-27a60c6a2d0d-sys\") pod \"node-exporter-wt562\" (UID: \"614c03cd-6d2c-4fee-8718-27a60c6a2d0d\") " pod="openshift-monitoring/node-exporter-wt562"
Nov 02 12:27:05.170964 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:27:05.170789    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/614c03cd-6d2c-4fee-8718-27a60c6a2d0d-root\") pod \"node-exporter-wt562\" (UID: \"614c03cd-6d2c-4fee-8718-27a60c6a2d0d\") " pod="openshift-monitoring/node-exporter-wt562"
Nov 02 12:27:05.173240 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:27:05.171280    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/614c03cd-6d2c-4fee-8718-27a60c6a2d0d-node-exporter-accelerators-collector-config\") pod \"node-exporter-wt562\" (UID: \"614c03cd-6d2c-4fee-8718-27a60c6a2d0d\") " pod="openshift-monitoring/node-exporter-wt562"
Nov 02 12:27:05.173240 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:27:05.171338    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/614c03cd-6d2c-4fee-8718-27a60c6a2d0d-node-exporter-textfile\") pod \"node-exporter-wt562\" (UID: \"614c03cd-6d2c-4fee-8718-27a60c6a2d0d\") " pod="openshift-monitoring/node-exporter-wt562"
Nov 02 12:27:05.173240 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:27:05.171381    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xsxqg\" (UniqueName: \"kubernetes.io/projected/614c03cd-6d2c-4fee-8718-27a60c6a2d0d-kube-api-access-xsxqg\") pod \"node-exporter-wt562\" (UID: \"614c03cd-6d2c-4fee-8718-27a60c6a2d0d\") " pod="openshift-monitoring/node-exporter-wt562"
Nov 02 12:27:05.173240 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:27:05.171426    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/614c03cd-6d2c-4fee-8718-27a60c6a2d0d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-wt562\" (UID: \"614c03cd-6d2c-4fee-8718-27a60c6a2d0d\") " pod="openshift-monitoring/node-exporter-wt562"
Nov 02 12:27:05.173240 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:27:05.171485    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/614c03cd-6d2c-4fee-8718-27a60c6a2d0d-node-exporter-tls\") pod \"node-exporter-wt562\" (UID: \"614c03cd-6d2c-4fee-8718-27a60c6a2d0d\") " pod="openshift-monitoring/node-exporter-wt562"
Nov 02 12:27:05.173240 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:27:05.171523    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/614c03cd-6d2c-4fee-8718-27a60c6a2d0d-metrics-client-ca\") pod \"node-exporter-wt562\" (UID: \"614c03cd-6d2c-4fee-8718-27a60c6a2d0d\") " pod="openshift-monitoring/node-exporter-wt562"
Nov 02 12:27:05.173240 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:27:05.171558    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/614c03cd-6d2c-4fee-8718-27a60c6a2d0d-node-exporter-wtmp\") pod \"node-exporter-wt562\" (UID: \"614c03cd-6d2c-4fee-8718-27a60c6a2d0d\") " pod="openshift-monitoring/node-exporter-wt562"
Nov 02 12:27:05.272555 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:27:05.272531    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xsxqg\" (UniqueName: \"kubernetes.io/projected/614c03cd-6d2c-4fee-8718-27a60c6a2d0d-kube-api-access-xsxqg\") pod \"node-exporter-wt562\" (UID: \"614c03cd-6d2c-4fee-8718-27a60c6a2d0d\") " pod="openshift-monitoring/node-exporter-wt562"
Nov 02 12:27:05.272715 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:27:05.272567    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/614c03cd-6d2c-4fee-8718-27a60c6a2d0d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-wt562\" (UID: \"614c03cd-6d2c-4fee-8718-27a60c6a2d0d\") " pod="openshift-monitoring/node-exporter-wt562"
Nov 02 12:27:05.272715 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:27:05.272597    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/614c03cd-6d2c-4fee-8718-27a60c6a2d0d-node-exporter-tls\") pod \"node-exporter-wt562\" (UID: \"614c03cd-6d2c-4fee-8718-27a60c6a2d0d\") " pod="openshift-monitoring/node-exporter-wt562"
Nov 02 12:27:05.272715 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:27:05.272615    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/614c03cd-6d2c-4fee-8718-27a60c6a2d0d-metrics-client-ca\") pod \"node-exporter-wt562\" (UID: \"614c03cd-6d2c-4fee-8718-27a60c6a2d0d\") " pod="openshift-monitoring/node-exporter-wt562"
Nov 02 12:27:05.272715 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:27:05.272632    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/614c03cd-6d2c-4fee-8718-27a60c6a2d0d-node-exporter-wtmp\") pod \"node-exporter-wt562\" (UID: \"614c03cd-6d2c-4fee-8718-27a60c6a2d0d\") " pod="openshift-monitoring/node-exporter-wt562"
Nov 02 12:27:05.272715 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:27:05.272675    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/614c03cd-6d2c-4fee-8718-27a60c6a2d0d-sys\") pod \"node-exporter-wt562\" (UID: \"614c03cd-6d2c-4fee-8718-27a60c6a2d0d\") " pod="openshift-monitoring/node-exporter-wt562"
Nov 02 12:27:05.272715 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:27:05.272698    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/614c03cd-6d2c-4fee-8718-27a60c6a2d0d-root\") pod \"node-exporter-wt562\" (UID: \"614c03cd-6d2c-4fee-8718-27a60c6a2d0d\") " pod="openshift-monitoring/node-exporter-wt562"
Nov 02 12:27:05.272968 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:27:05.272734    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/614c03cd-6d2c-4fee-8718-27a60c6a2d0d-node-exporter-accelerators-collector-config\") pod \"node-exporter-wt562\" (UID: \"614c03cd-6d2c-4fee-8718-27a60c6a2d0d\") " pod="openshift-monitoring/node-exporter-wt562"
Nov 02 12:27:05.272968 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:27:05.272763    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/614c03cd-6d2c-4fee-8718-27a60c6a2d0d-node-exporter-textfile\") pod \"node-exporter-wt562\" (UID: \"614c03cd-6d2c-4fee-8718-27a60c6a2d0d\") " pod="openshift-monitoring/node-exporter-wt562"
Nov 02 12:27:05.272968 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:27:05.272781    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/614c03cd-6d2c-4fee-8718-27a60c6a2d0d-sys\") pod \"node-exporter-wt562\" (UID: \"614c03cd-6d2c-4fee-8718-27a60c6a2d0d\") " pod="openshift-monitoring/node-exporter-wt562"
Nov 02 12:27:05.272968 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:27:05.272835    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/614c03cd-6d2c-4fee-8718-27a60c6a2d0d-root\") pod \"node-exporter-wt562\" (UID: \"614c03cd-6d2c-4fee-8718-27a60c6a2d0d\") " pod="openshift-monitoring/node-exporter-wt562"
Nov 02 12:27:05.272968 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:27:05.272860    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/614c03cd-6d2c-4fee-8718-27a60c6a2d0d-node-exporter-wtmp\") pod \"node-exporter-wt562\" (UID: \"614c03cd-6d2c-4fee-8718-27a60c6a2d0d\") " pod="openshift-monitoring/node-exporter-wt562"
Nov 02 12:27:05.273151 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:27:05.273111    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/614c03cd-6d2c-4fee-8718-27a60c6a2d0d-node-exporter-textfile\") pod \"node-exporter-wt562\" (UID: \"614c03cd-6d2c-4fee-8718-27a60c6a2d0d\") " pod="openshift-monitoring/node-exporter-wt562"
Nov 02 12:27:05.273341 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:27:05.273309    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/614c03cd-6d2c-4fee-8718-27a60c6a2d0d-metrics-client-ca\") pod \"node-exporter-wt562\" (UID: \"614c03cd-6d2c-4fee-8718-27a60c6a2d0d\") " pod="openshift-monitoring/node-exporter-wt562"
Nov 02 12:27:05.273450 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:27:05.273398    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/614c03cd-6d2c-4fee-8718-27a60c6a2d0d-node-exporter-accelerators-collector-config\") pod \"node-exporter-wt562\" (UID: \"614c03cd-6d2c-4fee-8718-27a60c6a2d0d\") " pod="openshift-monitoring/node-exporter-wt562"
Nov 02 12:27:05.275023 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:27:05.275006    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/614c03cd-6d2c-4fee-8718-27a60c6a2d0d-node-exporter-tls\") pod \"node-exporter-wt562\" (UID: \"614c03cd-6d2c-4fee-8718-27a60c6a2d0d\") " pod="openshift-monitoring/node-exporter-wt562"
Nov 02 12:27:05.275143 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:27:05.275125    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/614c03cd-6d2c-4fee-8718-27a60c6a2d0d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-wt562\" (UID: \"614c03cd-6d2c-4fee-8718-27a60c6a2d0d\") " pod="openshift-monitoring/node-exporter-wt562"
Nov 02 12:27:05.297623 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:27:05.297599    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xsxqg\" (UniqueName: \"kubernetes.io/projected/614c03cd-6d2c-4fee-8718-27a60c6a2d0d-kube-api-access-xsxqg\") pod \"node-exporter-wt562\" (UID: \"614c03cd-6d2c-4fee-8718-27a60c6a2d0d\") " pod="openshift-monitoring/node-exporter-wt562"
Nov 02 12:27:05.370914 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:27:05.370835    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-wt562"
Nov 02 12:27:05.382485 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:27:05.382457    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod614c03cd_6d2c_4fee_8718_27a60c6a2d0d.slice/crio-1ebcfeb22f3b13d822ddf0366ed6cbcecd475faa62bd58a48469a6dc6903cbef WatchSource:0}: Error finding container 1ebcfeb22f3b13d822ddf0366ed6cbcecd475faa62bd58a48469a6dc6903cbef: Status 404 returned error can't find the container with id 1ebcfeb22f3b13d822ddf0366ed6cbcecd475faa62bd58a48469a6dc6903cbef
Nov 02 12:27:06.035143 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:27:06.035105    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wt562" event={"ID":"614c03cd-6d2c-4fee-8718-27a60c6a2d0d","Type":"ContainerStarted","Data":"1ebcfeb22f3b13d822ddf0366ed6cbcecd475faa62bd58a48469a6dc6903cbef"}
Nov 02 12:27:07.039902 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:27:07.039868    4559 generic.go:356] "Generic (PLEG): container finished" podID="614c03cd-6d2c-4fee-8718-27a60c6a2d0d" containerID="250867751ed1fd1256b889c776ad4d666fe38ee19fd97368c250b1131c63a88c" exitCode=0
Nov 02 12:27:07.040263 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:27:07.039947    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wt562" event={"ID":"614c03cd-6d2c-4fee-8718-27a60c6a2d0d","Type":"ContainerDied","Data":"250867751ed1fd1256b889c776ad4d666fe38ee19fd97368c250b1131c63a88c"}
Nov 02 12:27:08.047998 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:27:08.047965    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wt562" event={"ID":"614c03cd-6d2c-4fee-8718-27a60c6a2d0d","Type":"ContainerStarted","Data":"8d68688bf820451e8b802721ab60f176a39d24919447571e042b1c9a44f4a461"}
Nov 02 12:27:08.048379 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:27:08.048021    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wt562" event={"ID":"614c03cd-6d2c-4fee-8718-27a60c6a2d0d","Type":"ContainerStarted","Data":"b0275b5348e845bc55c484a9908fec7dcdc9f715d780de241b653765d4784080"}
Nov 02 12:27:08.119719 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:27:08.119661    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-wt562" podStartSLOduration=2.009905488 podStartE2EDuration="3.119629483s" totalImagesPullingTime="1.040515995s" totalInitContainerRuntime="69.208ms" isStatefulPod=true podCreationTimestamp="2026-11-02 12:27:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:27:08.106424475 +0000 UTC m=+24.361823569" watchObservedRunningTime="2026-11-02 12:27:08.119629483 +0000 UTC m=+24.375028578"
Nov 02 12:27:10.183539 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:27:10.183507    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-dnm8d"
Nov 02 12:27:15.084216 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:27:15.084182    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqcmj" containerName="extractor"
Nov 02 12:27:17.304948 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:27:17.304919    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-wx6zw"
Nov 02 12:27:20.005764 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:27:20.005734    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-75fd5f49d8-7t6q5"
Nov 02 12:27:26.613597 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:27:26.613517    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-9lc22"]
Nov 02 12:27:26.616258 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:27:26.616242    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-9lc22"
Nov 02 12:27:26.639941 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:27:26.639914    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-lnnxr\""
Nov 02 12:27:26.646804 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:27:26.646785    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Nov 02 12:27:26.646922 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:27:26.646834    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Nov 02 12:27:26.647836 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:27:26.647813    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-9lc22"]
Nov 02 12:27:26.708852 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:27:26.708826    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29vhv\" (UniqueName: \"kubernetes.io/projected/9f83980d-62d1-448f-bdbc-95114e3cd8f3-kube-api-access-29vhv\") pod \"downloads-6dd789c8b7-9lc22\" (UID: \"9f83980d-62d1-448f-bdbc-95114e3cd8f3\") " pod="openshift-console/downloads-6dd789c8b7-9lc22"
Nov 02 12:27:26.708980 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:27:26.708864    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9f83980d-62d1-448f-bdbc-95114e3cd8f3-tmp\") pod \"downloads-6dd789c8b7-9lc22\" (UID: \"9f83980d-62d1-448f-bdbc-95114e3cd8f3\") " pod="openshift-console/downloads-6dd789c8b7-9lc22"
Nov 02 12:27:26.810088 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:27:26.810060    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29vhv\" (UniqueName: \"kubernetes.io/projected/9f83980d-62d1-448f-bdbc-95114e3cd8f3-kube-api-access-29vhv\") pod \"downloads-6dd789c8b7-9lc22\" (UID: \"9f83980d-62d1-448f-bdbc-95114e3cd8f3\") " pod="openshift-console/downloads-6dd789c8b7-9lc22"
Nov 02 12:27:26.810208 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:27:26.810099    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9f83980d-62d1-448f-bdbc-95114e3cd8f3-tmp\") pod \"downloads-6dd789c8b7-9lc22\" (UID: \"9f83980d-62d1-448f-bdbc-95114e3cd8f3\") " pod="openshift-console/downloads-6dd789c8b7-9lc22"
Nov 02 12:27:26.810428 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:27:26.810413    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9f83980d-62d1-448f-bdbc-95114e3cd8f3-tmp\") pod \"downloads-6dd789c8b7-9lc22\" (UID: \"9f83980d-62d1-448f-bdbc-95114e3cd8f3\") " pod="openshift-console/downloads-6dd789c8b7-9lc22"
Nov 02 12:27:26.834568 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:27:26.834542    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-29vhv\" (UniqueName: \"kubernetes.io/projected/9f83980d-62d1-448f-bdbc-95114e3cd8f3-kube-api-access-29vhv\") pod \"downloads-6dd789c8b7-9lc22\" (UID: \"9f83980d-62d1-448f-bdbc-95114e3cd8f3\") " pod="openshift-console/downloads-6dd789c8b7-9lc22"
Nov 02 12:27:26.927654 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:27:26.927616    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-9lc22"
Nov 02 12:27:27.245762 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:27:27.245691    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-9lc22"]
Nov 02 12:27:27.258972 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:27:27.258934    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f83980d_62d1_448f_bdbc_95114e3cd8f3.slice/crio-25ef7ebd1903d34469fca3b3306418bb7ee8e88f4711a1eac0ec106dbd651ffd WatchSource:0}: Error finding container 25ef7ebd1903d34469fca3b3306418bb7ee8e88f4711a1eac0ec106dbd651ffd: Status 404 returned error can't find the container with id 25ef7ebd1903d34469fca3b3306418bb7ee8e88f4711a1eac0ec106dbd651ffd
Nov 02 12:27:28.114045 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:27:28.114002    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-9lc22" event={"ID":"9f83980d-62d1-448f-bdbc-95114e3cd8f3","Type":"ContainerStarted","Data":"25ef7ebd1903d34469fca3b3306418bb7ee8e88f4711a1eac0ec106dbd651ffd"}
Nov 02 12:27:31.351910 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:27:31.351876    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-7964fd5c55-c8drt"]
Nov 02 12:27:31.356406 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:27:31.356380    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7964fd5c55-c8drt"
Nov 02 12:27:31.390511 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:27:31.390483    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Nov 02 12:27:31.390697 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:27:31.390563    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Nov 02 12:27:31.390697 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:27:31.390626    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-grmqf\""
Nov 02 12:27:31.390823 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:27:31.390806    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Nov 02 12:27:31.390889 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:27:31.390877    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Nov 02 12:27:31.390958 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:27:31.390931    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Nov 02 12:27:31.391848 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:27:31.391818    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7964fd5c55-c8drt"]
Nov 02 12:27:31.448472 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:27:31.448442    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/eb3facba-63fa-48af-a09b-fb6e8dc3a993-console-serving-cert\") pod \"console-7964fd5c55-c8drt\" (UID: \"eb3facba-63fa-48af-a09b-fb6e8dc3a993\") " pod="openshift-console/console-7964fd5c55-c8drt"
Nov 02 12:27:31.448662 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:27:31.448479    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/eb3facba-63fa-48af-a09b-fb6e8dc3a993-console-oauth-config\") pod \"console-7964fd5c55-c8drt\" (UID: \"eb3facba-63fa-48af-a09b-fb6e8dc3a993\") " pod="openshift-console/console-7964fd5c55-c8drt"
Nov 02 12:27:31.448662 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:27:31.448505    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/eb3facba-63fa-48af-a09b-fb6e8dc3a993-console-config\") pod \"console-7964fd5c55-c8drt\" (UID: \"eb3facba-63fa-48af-a09b-fb6e8dc3a993\") " pod="openshift-console/console-7964fd5c55-c8drt"
Nov 02 12:27:31.448662 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:27:31.448530    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/eb3facba-63fa-48af-a09b-fb6e8dc3a993-oauth-serving-cert\") pod \"console-7964fd5c55-c8drt\" (UID: \"eb3facba-63fa-48af-a09b-fb6e8dc3a993\") " pod="openshift-console/console-7964fd5c55-c8drt"
Nov 02 12:27:31.448662 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:27:31.448576    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eb3facba-63fa-48af-a09b-fb6e8dc3a993-tmp\") pod \"console-7964fd5c55-c8drt\" (UID: \"eb3facba-63fa-48af-a09b-fb6e8dc3a993\") " pod="openshift-console/console-7964fd5c55-c8drt"
Nov 02 12:27:31.448662 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:27:31.448593    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lj548\" (UniqueName: \"kubernetes.io/projected/eb3facba-63fa-48af-a09b-fb6e8dc3a993-kube-api-access-lj548\") pod \"console-7964fd5c55-c8drt\" (UID: \"eb3facba-63fa-48af-a09b-fb6e8dc3a993\") " pod="openshift-console/console-7964fd5c55-c8drt"
Nov 02 12:27:31.448662 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:27:31.448611    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/eb3facba-63fa-48af-a09b-fb6e8dc3a993-service-ca\") pod \"console-7964fd5c55-c8drt\" (UID: \"eb3facba-63fa-48af-a09b-fb6e8dc3a993\") " pod="openshift-console/console-7964fd5c55-c8drt"
Nov 02 12:27:31.549697 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:27:31.549632    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/eb3facba-63fa-48af-a09b-fb6e8dc3a993-console-oauth-config\") pod \"console-7964fd5c55-c8drt\" (UID: \"eb3facba-63fa-48af-a09b-fb6e8dc3a993\") " pod="openshift-console/console-7964fd5c55-c8drt"
Nov 02 12:27:31.549868 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:27:31.549706    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/eb3facba-63fa-48af-a09b-fb6e8dc3a993-console-config\") pod \"console-7964fd5c55-c8drt\" (UID: \"eb3facba-63fa-48af-a09b-fb6e8dc3a993\") " pod="openshift-console/console-7964fd5c55-c8drt"
Nov 02 12:27:31.549868 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:27:31.549745    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/eb3facba-63fa-48af-a09b-fb6e8dc3a993-oauth-serving-cert\") pod \"console-7964fd5c55-c8drt\" (UID: \"eb3facba-63fa-48af-a09b-fb6e8dc3a993\") " pod="openshift-console/console-7964fd5c55-c8drt"
Nov 02 12:27:31.549868 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:27:31.549797    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eb3facba-63fa-48af-a09b-fb6e8dc3a993-tmp\") pod \"console-7964fd5c55-c8drt\" (UID: \"eb3facba-63fa-48af-a09b-fb6e8dc3a993\") " pod="openshift-console/console-7964fd5c55-c8drt"
Nov 02 12:27:31.549868 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:27:31.549825    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lj548\" (UniqueName: \"kubernetes.io/projected/eb3facba-63fa-48af-a09b-fb6e8dc3a993-kube-api-access-lj548\") pod \"console-7964fd5c55-c8drt\" (UID: \"eb3facba-63fa-48af-a09b-fb6e8dc3a993\") " pod="openshift-console/console-7964fd5c55-c8drt"
Nov 02 12:27:31.549868 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:27:31.549856    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/eb3facba-63fa-48af-a09b-fb6e8dc3a993-service-ca\") pod \"console-7964fd5c55-c8drt\" (UID: \"eb3facba-63fa-48af-a09b-fb6e8dc3a993\") " pod="openshift-console/console-7964fd5c55-c8drt"
Nov 02 12:27:31.550126 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:27:31.549901    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/eb3facba-63fa-48af-a09b-fb6e8dc3a993-console-serving-cert\") pod \"console-7964fd5c55-c8drt\" (UID: \"eb3facba-63fa-48af-a09b-fb6e8dc3a993\") " pod="openshift-console/console-7964fd5c55-c8drt"
Nov 02 12:27:31.550426 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:27:31.550395    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eb3facba-63fa-48af-a09b-fb6e8dc3a993-tmp\") pod \"console-7964fd5c55-c8drt\" (UID: \"eb3facba-63fa-48af-a09b-fb6e8dc3a993\") " pod="openshift-console/console-7964fd5c55-c8drt"
Nov 02 12:27:31.550544 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:27:31.550524    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/eb3facba-63fa-48af-a09b-fb6e8dc3a993-console-config\") pod \"console-7964fd5c55-c8drt\" (UID: \"eb3facba-63fa-48af-a09b-fb6e8dc3a993\") " pod="openshift-console/console-7964fd5c55-c8drt"
Nov 02 12:27:31.550612 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:27:31.550595    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/eb3facba-63fa-48af-a09b-fb6e8dc3a993-oauth-serving-cert\") pod \"console-7964fd5c55-c8drt\" (UID: \"eb3facba-63fa-48af-a09b-fb6e8dc3a993\") " pod="openshift-console/console-7964fd5c55-c8drt"
Nov 02 12:27:31.550986 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:27:31.550966    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/eb3facba-63fa-48af-a09b-fb6e8dc3a993-service-ca\") pod \"console-7964fd5c55-c8drt\" (UID: \"eb3facba-63fa-48af-a09b-fb6e8dc3a993\") " pod="openshift-console/console-7964fd5c55-c8drt"
Nov 02 12:27:31.552714 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:27:31.552688    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/eb3facba-63fa-48af-a09b-fb6e8dc3a993-console-oauth-config\") pod \"console-7964fd5c55-c8drt\" (UID: \"eb3facba-63fa-48af-a09b-fb6e8dc3a993\") " pod="openshift-console/console-7964fd5c55-c8drt"
Nov 02 12:27:31.557121 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:27:31.557095    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/eb3facba-63fa-48af-a09b-fb6e8dc3a993-console-serving-cert\") pod \"console-7964fd5c55-c8drt\" (UID: \"eb3facba-63fa-48af-a09b-fb6e8dc3a993\") " pod="openshift-console/console-7964fd5c55-c8drt"
Nov 02 12:27:31.574392 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:27:31.574369    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lj548\" (UniqueName: \"kubernetes.io/projected/eb3facba-63fa-48af-a09b-fb6e8dc3a993-kube-api-access-lj548\") pod \"console-7964fd5c55-c8drt\" (UID: \"eb3facba-63fa-48af-a09b-fb6e8dc3a993\") " pod="openshift-console/console-7964fd5c55-c8drt"
Nov 02 12:27:31.671873 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:27:31.671839    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7964fd5c55-c8drt"
Nov 02 12:27:31.982698 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:27:31.982669    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7964fd5c55-c8drt"]
Nov 02 12:27:31.984614 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:27:31.984587    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb3facba_63fa_48af_a09b_fb6e8dc3a993.slice/crio-40df2954848cce32a6bcbed1633b5ce00d75516cbfd1678f9f9e83072fdf28c4 WatchSource:0}: Error finding container 40df2954848cce32a6bcbed1633b5ce00d75516cbfd1678f9f9e83072fdf28c4: Status 404 returned error can't find the container with id 40df2954848cce32a6bcbed1633b5ce00d75516cbfd1678f9f9e83072fdf28c4
Nov 02 12:27:32.129865 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:27:32.129825    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7964fd5c55-c8drt" event={"ID":"eb3facba-63fa-48af-a09b-fb6e8dc3a993","Type":"ContainerStarted","Data":"40df2954848cce32a6bcbed1633b5ce00d75516cbfd1678f9f9e83072fdf28c4"}
Nov 02 12:27:35.029798 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:27:35.029754    4559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dnm8d_f7c6ba1b-8808-4b6b-8965-2e4e7a067030/konnectivity-agent/0.log" limit=1024
Nov 02 12:27:36.152396 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:27:36.152354    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7964fd5c55-c8drt" event={"ID":"eb3facba-63fa-48af-a09b-fb6e8dc3a993","Type":"ContainerStarted","Data":"7d7b1ce0005aee52149f156d938c8f0cd176b2697c74e6130e8eead1db4fe77c"}
Nov 02 12:27:36.198416 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:27:36.198364    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-7964fd5c55-c8drt" podStartSLOduration=1.977898615 podStartE2EDuration="5.198345185s" totalImagesPullingTime="3.22044657s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:27:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:27:36.197216023 +0000 UTC m=+52.452615119" watchObservedRunningTime="2026-11-02 12:27:36.198345185 +0000 UTC m=+52.453744281"
Nov 02 12:27:41.672919 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:27:41.672881    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7964fd5c55-c8drt"
Nov 02 12:27:41.673489 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:27:41.672979    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7964fd5c55-c8drt"
Nov 02 12:27:41.678719 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:27:41.678693    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7964fd5c55-c8drt"
Nov 02 12:27:42.179593 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:27:42.179523    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7964fd5c55-c8drt"
Nov 02 12:27:42.356301 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:27:42.356261    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-bbb695c57-n2z5q"]
Nov 02 12:27:42.361917 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:27:42.361890    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-bbb695c57-n2z5q"
Nov 02 12:27:42.410843 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:27:42.410678    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Nov 02 12:27:42.412000 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:27:42.411971    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-bbb695c57-n2z5q"]
Nov 02 12:27:42.447408 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:27:42.447328    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c89c6465-eeec-408e-b2e3-f9c131be5730-tmp\") pod \"console-bbb695c57-n2z5q\" (UID: \"c89c6465-eeec-408e-b2e3-f9c131be5730\") " pod="openshift-console/console-bbb695c57-n2z5q"
Nov 02 12:27:42.447408 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:27:42.447372    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzcxw\" (UniqueName: \"kubernetes.io/projected/c89c6465-eeec-408e-b2e3-f9c131be5730-kube-api-access-bzcxw\") pod \"console-bbb695c57-n2z5q\" (UID: \"c89c6465-eeec-408e-b2e3-f9c131be5730\") " pod="openshift-console/console-bbb695c57-n2z5q"
Nov 02 12:27:42.447597 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:27:42.447404    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c89c6465-eeec-408e-b2e3-f9c131be5730-oauth-serving-cert\") pod \"console-bbb695c57-n2z5q\" (UID: \"c89c6465-eeec-408e-b2e3-f9c131be5730\") " pod="openshift-console/console-bbb695c57-n2z5q"
Nov 02 12:27:42.447597 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:27:42.447476    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c89c6465-eeec-408e-b2e3-f9c131be5730-service-ca\") pod \"console-bbb695c57-n2z5q\" (UID: \"c89c6465-eeec-408e-b2e3-f9c131be5730\") " pod="openshift-console/console-bbb695c57-n2z5q"
Nov 02 12:27:42.447720 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:27:42.447604    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c89c6465-eeec-408e-b2e3-f9c131be5730-console-oauth-config\") pod \"console-bbb695c57-n2z5q\" (UID: \"c89c6465-eeec-408e-b2e3-f9c131be5730\") " pod="openshift-console/console-bbb695c57-n2z5q"
Nov 02 12:27:42.447720 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:27:42.447664    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c89c6465-eeec-408e-b2e3-f9c131be5730-trusted-ca-bundle\") pod \"console-bbb695c57-n2z5q\" (UID: \"c89c6465-eeec-408e-b2e3-f9c131be5730\") " pod="openshift-console/console-bbb695c57-n2z5q"
Nov 02 12:27:42.447788 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:27:42.447715    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c89c6465-eeec-408e-b2e3-f9c131be5730-console-serving-cert\") pod \"console-bbb695c57-n2z5q\" (UID: \"c89c6465-eeec-408e-b2e3-f9c131be5730\") " pod="openshift-console/console-bbb695c57-n2z5q"
Nov 02 12:27:42.447788 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:27:42.447739    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c89c6465-eeec-408e-b2e3-f9c131be5730-console-config\") pod \"console-bbb695c57-n2z5q\" (UID: \"c89c6465-eeec-408e-b2e3-f9c131be5730\") " pod="openshift-console/console-bbb695c57-n2z5q"
Nov 02 12:27:42.548153 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:27:42.548119    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c89c6465-eeec-408e-b2e3-f9c131be5730-tmp\") pod \"console-bbb695c57-n2z5q\" (UID: \"c89c6465-eeec-408e-b2e3-f9c131be5730\") " pod="openshift-console/console-bbb695c57-n2z5q"
Nov 02 12:27:42.548346 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:27:42.548162    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzcxw\" (UniqueName: \"kubernetes.io/projected/c89c6465-eeec-408e-b2e3-f9c131be5730-kube-api-access-bzcxw\") pod \"console-bbb695c57-n2z5q\" (UID: \"c89c6465-eeec-408e-b2e3-f9c131be5730\") " pod="openshift-console/console-bbb695c57-n2z5q"
Nov 02 12:27:42.548346 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:27:42.548196    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c89c6465-eeec-408e-b2e3-f9c131be5730-oauth-serving-cert\") pod \"console-bbb695c57-n2z5q\" (UID: \"c89c6465-eeec-408e-b2e3-f9c131be5730\") " pod="openshift-console/console-bbb695c57-n2z5q"
Nov 02 12:27:42.548346 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:27:42.548242    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c89c6465-eeec-408e-b2e3-f9c131be5730-service-ca\") pod \"console-bbb695c57-n2z5q\" (UID: \"c89c6465-eeec-408e-b2e3-f9c131be5730\") " pod="openshift-console/console-bbb695c57-n2z5q"
Nov 02 12:27:42.548346 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:27:42.548270    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c89c6465-eeec-408e-b2e3-f9c131be5730-console-oauth-config\") pod \"console-bbb695c57-n2z5q\" (UID: \"c89c6465-eeec-408e-b2e3-f9c131be5730\") " pod="openshift-console/console-bbb695c57-n2z5q"
Nov 02 12:27:42.548346 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:27:42.548291    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c89c6465-eeec-408e-b2e3-f9c131be5730-trusted-ca-bundle\") pod \"console-bbb695c57-n2z5q\" (UID: \"c89c6465-eeec-408e-b2e3-f9c131be5730\") " pod="openshift-console/console-bbb695c57-n2z5q"
Nov 02 12:27:42.548346 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:27:42.548317    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c89c6465-eeec-408e-b2e3-f9c131be5730-console-serving-cert\") pod \"console-bbb695c57-n2z5q\" (UID: \"c89c6465-eeec-408e-b2e3-f9c131be5730\") " pod="openshift-console/console-bbb695c57-n2z5q"
Nov 02 12:27:42.548680 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:27:42.548348    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c89c6465-eeec-408e-b2e3-f9c131be5730-console-config\") pod \"console-bbb695c57-n2z5q\" (UID: \"c89c6465-eeec-408e-b2e3-f9c131be5730\") " pod="openshift-console/console-bbb695c57-n2z5q"
Nov 02 12:27:42.548680 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:27:42.548600    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c89c6465-eeec-408e-b2e3-f9c131be5730-tmp\") pod \"console-bbb695c57-n2z5q\" (UID: \"c89c6465-eeec-408e-b2e3-f9c131be5730\") " pod="openshift-console/console-bbb695c57-n2z5q"
Nov 02 12:27:42.549176 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:27:42.549144    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c89c6465-eeec-408e-b2e3-f9c131be5730-service-ca\") pod \"console-bbb695c57-n2z5q\" (UID: \"c89c6465-eeec-408e-b2e3-f9c131be5730\") " pod="openshift-console/console-bbb695c57-n2z5q"
Nov 02 12:27:42.549176 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:27:42.549166    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c89c6465-eeec-408e-b2e3-f9c131be5730-console-config\") pod \"console-bbb695c57-n2z5q\" (UID: \"c89c6465-eeec-408e-b2e3-f9c131be5730\") " pod="openshift-console/console-bbb695c57-n2z5q"
Nov 02 12:27:42.549366 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:27:42.549166    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c89c6465-eeec-408e-b2e3-f9c131be5730-oauth-serving-cert\") pod \"console-bbb695c57-n2z5q\" (UID: \"c89c6465-eeec-408e-b2e3-f9c131be5730\") " pod="openshift-console/console-bbb695c57-n2z5q"
Nov 02 12:27:42.549724 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:27:42.549685    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c89c6465-eeec-408e-b2e3-f9c131be5730-trusted-ca-bundle\") pod \"console-bbb695c57-n2z5q\" (UID: \"c89c6465-eeec-408e-b2e3-f9c131be5730\") " pod="openshift-console/console-bbb695c57-n2z5q"
Nov 02 12:27:42.551534 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:27:42.551509    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c89c6465-eeec-408e-b2e3-f9c131be5730-console-serving-cert\") pod \"console-bbb695c57-n2z5q\" (UID: \"c89c6465-eeec-408e-b2e3-f9c131be5730\") " pod="openshift-console/console-bbb695c57-n2z5q"
Nov 02 12:27:42.555097 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:27:42.555074    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c89c6465-eeec-408e-b2e3-f9c131be5730-console-oauth-config\") pod \"console-bbb695c57-n2z5q\" (UID: \"c89c6465-eeec-408e-b2e3-f9c131be5730\") " pod="openshift-console/console-bbb695c57-n2z5q"
Nov 02 12:27:42.572407 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:27:42.572377    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzcxw\" (UniqueName: \"kubernetes.io/projected/c89c6465-eeec-408e-b2e3-f9c131be5730-kube-api-access-bzcxw\") pod \"console-bbb695c57-n2z5q\" (UID: \"c89c6465-eeec-408e-b2e3-f9c131be5730\") " pod="openshift-console/console-bbb695c57-n2z5q"
Nov 02 12:27:42.675311 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:27:42.675073    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-bbb695c57-n2z5q"
Nov 02 12:27:42.984716 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:27:42.984691    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-bbb695c57-n2z5q"]
Nov 02 12:27:42.986098 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:27:42.986070    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc89c6465_eeec_408e_b2e3_f9c131be5730.slice/crio-091712ff9b21a5feb62741b083cf984feafc50a830d5b0644057fa57094a55a2 WatchSource:0}: Error finding container 091712ff9b21a5feb62741b083cf984feafc50a830d5b0644057fa57094a55a2: Status 404 returned error can't find the container with id 091712ff9b21a5feb62741b083cf984feafc50a830d5b0644057fa57094a55a2
Nov 02 12:27:43.180193 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:27:43.180154    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-bbb695c57-n2z5q" event={"ID":"c89c6465-eeec-408e-b2e3-f9c131be5730","Type":"ContainerStarted","Data":"c839b3936397cc620ca57b5e697337365e7055abe7151351caa3c2b87ef83d8a"}
Nov 02 12:27:43.180368 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:27:43.180214    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-bbb695c57-n2z5q" event={"ID":"c89c6465-eeec-408e-b2e3-f9c131be5730","Type":"ContainerStarted","Data":"091712ff9b21a5feb62741b083cf984feafc50a830d5b0644057fa57094a55a2"}
Nov 02 12:27:43.182795 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:27:43.182706    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-9lc22" event={"ID":"9f83980d-62d1-448f-bdbc-95114e3cd8f3","Type":"ContainerStarted","Data":"169afef8331fa75721ad23c5e22fd4ff11744dd5d5de575128e2a58e1d896886"}
Nov 02 12:27:43.182939 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:27:43.182869    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-9lc22"
Nov 02 12:27:43.184078 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:27:43.184046    4559 patch_prober.go:29] interesting pod/downloads-6dd789c8b7-9lc22 container/download-server namespace/openshift-console: Readiness probe status=failure output="Get \"http://10.133.0.15:8080/\": dial tcp 10.133.0.15:8080: connect: connection refused" start-of-body=
Nov 02 12:27:43.184190 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:27:43.184093    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console/downloads-6dd789c8b7-9lc22" podUID="9f83980d-62d1-448f-bdbc-95114e3cd8f3" containerName="download-server" probeResult="failure" output="Get \"http://10.133.0.15:8080/\": dial tcp 10.133.0.15:8080: connect: connection refused"
Nov 02 12:27:43.224981 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:27:43.224930    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-bbb695c57-n2z5q" podStartSLOduration=1.224915774 podStartE2EDuration="1.224915774s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:27:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:27:43.223990325 +0000 UTC m=+59.479389434" watchObservedRunningTime="2026-11-02 12:27:43.224915774 +0000 UTC m=+59.480314869"
Nov 02 12:27:43.266819 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:27:43.265576    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/downloads-6dd789c8b7-9lc22" podStartSLOduration=1.5207754900000001 podStartE2EDuration="17.265560593s" totalImagesPullingTime="15.744785103s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:27:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:27:43.265230989 +0000 UTC m=+59.520630109" watchObservedRunningTime="2026-11-02 12:27:43.265560593 +0000 UTC m=+59.520959691"
Nov 02 12:27:44.209479 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:27:44.209441    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-9lc22"
Nov 02 12:27:45.091236 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:27:45.091203    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqcmj" containerName="extractor"
Nov 02 12:27:52.676212 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:27:52.676170    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-bbb695c57-n2z5q"
Nov 02 12:27:52.676776 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:27:52.676235    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-bbb695c57-n2z5q"
Nov 02 12:27:52.681551 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:27:52.681530    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-bbb695c57-n2z5q"
Nov 02 12:27:53.221329 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:27:53.221303    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-bbb695c57-n2z5q"
Nov 02 12:27:53.293385 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:27:53.293347    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7964fd5c55-c8drt"]
Nov 02 12:28:05.256080 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:28:05.255997    4559 generic.go:356] "Generic (PLEG): container finished" podID="f25513a9-4a23-4cd1-9ef0-b5612826c157" containerID="1d56a78914255a1f48d2aa755c2d56b4537c24f5d181186d3d082a8ac4a7466d" exitCode=0
Nov 02 12:28:05.256482 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:28:05.256070    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-x27dt-dzqvx" event={"ID":"f25513a9-4a23-4cd1-9ef0-b5612826c157","Type":"ContainerDied","Data":"1d56a78914255a1f48d2aa755c2d56b4537c24f5d181186d3d082a8ac4a7466d"}
Nov 02 12:28:06.401965 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:28:06.401937    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-x27dt-dzqvx"
Nov 02 12:28:06.539757 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:28:06.539674    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f25513a9-4a23-4cd1-9ef0-b5612826c157-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f25513a9-4a23-4cd1-9ef0-b5612826c157-service-ca-bundle\") pod \"f25513a9-4a23-4cd1-9ef0-b5612826c157\" (UID: \"f25513a9-4a23-4cd1-9ef0-b5612826c157\") "
Nov 02 12:28:06.539929 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:28:06.539774    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f25513a9-4a23-4cd1-9ef0-b5612826c157-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f25513a9-4a23-4cd1-9ef0-b5612826c157-trusted-ca-bundle\") pod \"f25513a9-4a23-4cd1-9ef0-b5612826c157\" (UID: \"f25513a9-4a23-4cd1-9ef0-b5612826c157\") "
Nov 02 12:28:06.539929 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:28:06.539804    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f25513a9-4a23-4cd1-9ef0-b5612826c157-kube-api-access-7slf8\" (UniqueName: \"kubernetes.io/projected/f25513a9-4a23-4cd1-9ef0-b5612826c157-kube-api-access-7slf8\") pod \"f25513a9-4a23-4cd1-9ef0-b5612826c157\" (UID: \"f25513a9-4a23-4cd1-9ef0-b5612826c157\") "
Nov 02 12:28:06.539929 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:28:06.539837    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f25513a9-4a23-4cd1-9ef0-b5612826c157-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/f25513a9-4a23-4cd1-9ef0-b5612826c157-archives-path\") pod \"f25513a9-4a23-4cd1-9ef0-b5612826c157\" (UID: \"f25513a9-4a23-4cd1-9ef0-b5612826c157\") "
Nov 02 12:28:06.540142 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:28:06.540113    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f25513a9-4a23-4cd1-9ef0-b5612826c157-service-ca-bundle" pod "f25513a9-4a23-4cd1-9ef0-b5612826c157" (UID: "f25513a9-4a23-4cd1-9ef0-b5612826c157"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:28:06.540182 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:28:06.540155    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f25513a9-4a23-4cd1-9ef0-b5612826c157-trusted-ca-bundle" pod "f25513a9-4a23-4cd1-9ef0-b5612826c157" (UID: "f25513a9-4a23-4cd1-9ef0-b5612826c157"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:28:06.540223 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:28:06.540186    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f25513a9-4a23-4cd1-9ef0-b5612826c157-archives-path" pod "f25513a9-4a23-4cd1-9ef0-b5612826c157" (UID: "f25513a9-4a23-4cd1-9ef0-b5612826c157"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:28:06.542047 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:28:06.542024    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f25513a9-4a23-4cd1-9ef0-b5612826c157-kube-api-access-7slf8" pod "f25513a9-4a23-4cd1-9ef0-b5612826c157" (UID: "f25513a9-4a23-4cd1-9ef0-b5612826c157"). InnerVolumeSpecName "kube-api-access-7slf8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:28:06.640560 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:28:06.640522    4559 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f25513a9-4a23-4cd1-9ef0-b5612826c157-trusted-ca-bundle\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:28:06.640560 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:28:06.640553    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7slf8\" (UniqueName: \"kubernetes.io/projected/f25513a9-4a23-4cd1-9ef0-b5612826c157-kube-api-access-7slf8\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:28:06.640560 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:28:06.640562    4559 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/f25513a9-4a23-4cd1-9ef0-b5612826c157-archives-path\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:28:06.640560 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:28:06.640571    4559 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f25513a9-4a23-4cd1-9ef0-b5612826c157-service-ca-bundle\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:28:07.263926 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:28:07.263885    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-x27dt-dzqvx" event={"ID":"f25513a9-4a23-4cd1-9ef0-b5612826c157","Type":"ContainerDied","Data":"8b3046c39b2696c2009e1ddc2fb1a50edc8fb908c8ff8f6d16198bd9b0c86980"}
Nov 02 12:28:07.263926 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:28:07.263922    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-x27dt-dzqvx"
Nov 02 12:28:07.264126 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:28:07.263937    4559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8b3046c39b2696c2009e1ddc2fb1a50edc8fb908c8ff8f6d16198bd9b0c86980"
Nov 02 12:28:13.762373 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:28:13.762338    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-gj98f-lf5h7"]
Nov 02 12:28:13.762838 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:28:13.762592    4559 cpu_manager.go:414] "containerMap: removing container" podUID="f25513a9-4a23-4cd1-9ef0-b5612826c157" containerName="insights-gathering"
Nov 02 12:28:13.762838 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:28:13.762602    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f25513a9-4a23-4cd1-9ef0-b5612826c157" containerName="insights-gathering"
Nov 02 12:28:13.762838 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:28:13.762679    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="f25513a9-4a23-4cd1-9ef0-b5612826c157" containerName="insights-gathering"
Nov 02 12:28:13.803491 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:28:13.803459    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-gj98f-lf5h7"]
Nov 02 12:28:13.803665 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:28:13.803556    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-gj98f-lf5h7"
Nov 02 12:28:13.831961 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:28:13.831938    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Nov 02 12:28:13.832072 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:28:13.832042    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-wx6z7\""
Nov 02 12:28:13.858946 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:28:13.858917    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Nov 02 12:28:13.896868 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:28:13.896842    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/80e2d631-c27d-4629-bf4f-dec1c1e29656-trusted-ca-bundle\") pod \"periodic-gathering-gj98f-lf5h7\" (UID: \"80e2d631-c27d-4629-bf4f-dec1c1e29656\") " pod="openshift-insights/periodic-gathering-gj98f-lf5h7"
Nov 02 12:28:13.897002 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:28:13.896884    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgtz9\" (UniqueName: \"kubernetes.io/projected/80e2d631-c27d-4629-bf4f-dec1c1e29656-kube-api-access-pgtz9\") pod \"periodic-gathering-gj98f-lf5h7\" (UID: \"80e2d631-c27d-4629-bf4f-dec1c1e29656\") " pod="openshift-insights/periodic-gathering-gj98f-lf5h7"
Nov 02 12:28:13.897002 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:28:13.896907    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/80e2d631-c27d-4629-bf4f-dec1c1e29656-service-ca-bundle\") pod \"periodic-gathering-gj98f-lf5h7\" (UID: \"80e2d631-c27d-4629-bf4f-dec1c1e29656\") " pod="openshift-insights/periodic-gathering-gj98f-lf5h7"
Nov 02 12:28:13.897083 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:28:13.897028    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/80e2d631-c27d-4629-bf4f-dec1c1e29656-archives-path\") pod \"periodic-gathering-gj98f-lf5h7\" (UID: \"80e2d631-c27d-4629-bf4f-dec1c1e29656\") " pod="openshift-insights/periodic-gathering-gj98f-lf5h7"
Nov 02 12:28:13.997904 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:28:13.997876    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/80e2d631-c27d-4629-bf4f-dec1c1e29656-trusted-ca-bundle\") pod \"periodic-gathering-gj98f-lf5h7\" (UID: \"80e2d631-c27d-4629-bf4f-dec1c1e29656\") " pod="openshift-insights/periodic-gathering-gj98f-lf5h7"
Nov 02 12:28:13.998082 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:28:13.997921    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgtz9\" (UniqueName: \"kubernetes.io/projected/80e2d631-c27d-4629-bf4f-dec1c1e29656-kube-api-access-pgtz9\") pod \"periodic-gathering-gj98f-lf5h7\" (UID: \"80e2d631-c27d-4629-bf4f-dec1c1e29656\") " pod="openshift-insights/periodic-gathering-gj98f-lf5h7"
Nov 02 12:28:13.998082 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:28:13.998029    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/80e2d631-c27d-4629-bf4f-dec1c1e29656-service-ca-bundle\") pod \"periodic-gathering-gj98f-lf5h7\" (UID: \"80e2d631-c27d-4629-bf4f-dec1c1e29656\") " pod="openshift-insights/periodic-gathering-gj98f-lf5h7"
Nov 02 12:28:13.998082 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:28:13.998062    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/80e2d631-c27d-4629-bf4f-dec1c1e29656-archives-path\") pod \"periodic-gathering-gj98f-lf5h7\" (UID: \"80e2d631-c27d-4629-bf4f-dec1c1e29656\") " pod="openshift-insights/periodic-gathering-gj98f-lf5h7"
Nov 02 12:28:13.998405 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:28:13.998384    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/80e2d631-c27d-4629-bf4f-dec1c1e29656-archives-path\") pod \"periodic-gathering-gj98f-lf5h7\" (UID: \"80e2d631-c27d-4629-bf4f-dec1c1e29656\") " pod="openshift-insights/periodic-gathering-gj98f-lf5h7"
Nov 02 12:28:13.998706 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:28:13.998681    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/80e2d631-c27d-4629-bf4f-dec1c1e29656-service-ca-bundle\") pod \"periodic-gathering-gj98f-lf5h7\" (UID: \"80e2d631-c27d-4629-bf4f-dec1c1e29656\") " pod="openshift-insights/periodic-gathering-gj98f-lf5h7"
Nov 02 12:28:13.998789 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:28:13.998714    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/80e2d631-c27d-4629-bf4f-dec1c1e29656-trusted-ca-bundle\") pod \"periodic-gathering-gj98f-lf5h7\" (UID: \"80e2d631-c27d-4629-bf4f-dec1c1e29656\") " pod="openshift-insights/periodic-gathering-gj98f-lf5h7"
Nov 02 12:28:14.021996 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:28:14.021947    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgtz9\" (UniqueName: \"kubernetes.io/projected/80e2d631-c27d-4629-bf4f-dec1c1e29656-kube-api-access-pgtz9\") pod \"periodic-gathering-gj98f-lf5h7\" (UID: \"80e2d631-c27d-4629-bf4f-dec1c1e29656\") " pod="openshift-insights/periodic-gathering-gj98f-lf5h7"
Nov 02 12:28:14.129252 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:28:14.129220    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-gj98f-lf5h7"
Nov 02 12:28:14.440665 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:28:14.440620    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-gj98f-lf5h7"]
Nov 02 12:28:14.441574 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:28:14.441542    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80e2d631_c27d_4629_bf4f_dec1c1e29656.slice/crio-9f40d455f5e642541ef838fbfd41941f137bd4324543466828780c9fe8805a67 WatchSource:0}: Error finding container 9f40d455f5e642541ef838fbfd41941f137bd4324543466828780c9fe8805a67: Status 404 returned error can't find the container with id 9f40d455f5e642541ef838fbfd41941f137bd4324543466828780c9fe8805a67
Nov 02 12:28:15.085096 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:28:15.085054    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqcmj" containerName="extractor"
Nov 02 12:28:15.290238 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:28:15.290198    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-gj98f-lf5h7" event={"ID":"80e2d631-c27d-4629-bf4f-dec1c1e29656","Type":"ContainerStarted","Data":"ba47506578048294d07a0883b61af4384f0abd840811f0a4dac3d463c8ada573"}
Nov 02 12:28:15.290393 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:28:15.290287    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-gj98f-lf5h7" event={"ID":"80e2d631-c27d-4629-bf4f-dec1c1e29656","Type":"ContainerStarted","Data":"9f40d455f5e642541ef838fbfd41941f137bd4324543466828780c9fe8805a67"}
Nov 02 12:28:15.333906 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:28:15.333861    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-gj98f-lf5h7" podStartSLOduration=2.333847987 podStartE2EDuration="2.333847987s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:28:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:28:15.332980106 +0000 UTC m=+91.588379201" watchObservedRunningTime="2026-11-02 12:28:15.333847987 +0000 UTC m=+91.589247084"
Nov 02 12:28:18.312488 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:28:18.312433    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-7964fd5c55-c8drt" podUID="eb3facba-63fa-48af-a09b-fb6e8dc3a993" containerName="console" containerID="cri-o://7d7b1ce0005aee52149f156d938c8f0cd176b2697c74e6130e8eead1db4fe77c" gracePeriod=15
Nov 02 12:28:18.568205 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:28:18.568153    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7964fd5c55-c8drt"
Nov 02 12:28:18.630298 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:28:18.630264    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb3facba-63fa-48af-a09b-fb6e8dc3a993-kube-api-access-lj548\" (UniqueName: \"kubernetes.io/projected/eb3facba-63fa-48af-a09b-fb6e8dc3a993-kube-api-access-lj548\") pod \"eb3facba-63fa-48af-a09b-fb6e8dc3a993\" (UID: \"eb3facba-63fa-48af-a09b-fb6e8dc3a993\") "
Nov 02 12:28:18.630462 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:28:18.630313    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb3facba-63fa-48af-a09b-fb6e8dc3a993-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb3facba-63fa-48af-a09b-fb6e8dc3a993-service-ca\") pod \"eb3facba-63fa-48af-a09b-fb6e8dc3a993\" (UID: \"eb3facba-63fa-48af-a09b-fb6e8dc3a993\") "
Nov 02 12:28:18.630462 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:28:18.630346    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb3facba-63fa-48af-a09b-fb6e8dc3a993-console-config\" (UniqueName: \"kubernetes.io/configmap/eb3facba-63fa-48af-a09b-fb6e8dc3a993-console-config\") pod \"eb3facba-63fa-48af-a09b-fb6e8dc3a993\" (UID: \"eb3facba-63fa-48af-a09b-fb6e8dc3a993\") "
Nov 02 12:28:18.630462 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:28:18.630379    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eb3facba-63fa-48af-a09b-fb6e8dc3a993-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/eb3facba-63fa-48af-a09b-fb6e8dc3a993-console-oauth-config\") pod \"eb3facba-63fa-48af-a09b-fb6e8dc3a993\" (UID: \"eb3facba-63fa-48af-a09b-fb6e8dc3a993\") "
Nov 02 12:28:18.630462 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:28:18.630449    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb3facba-63fa-48af-a09b-fb6e8dc3a993-tmp\" (UniqueName: \"kubernetes.io/empty-dir/eb3facba-63fa-48af-a09b-fb6e8dc3a993-tmp\") pod \"eb3facba-63fa-48af-a09b-fb6e8dc3a993\" (UID: \"eb3facba-63fa-48af-a09b-fb6e8dc3a993\") "
Nov 02 12:28:18.630690 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:28:18.630481    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eb3facba-63fa-48af-a09b-fb6e8dc3a993-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/eb3facba-63fa-48af-a09b-fb6e8dc3a993-console-serving-cert\") pod \"eb3facba-63fa-48af-a09b-fb6e8dc3a993\" (UID: \"eb3facba-63fa-48af-a09b-fb6e8dc3a993\") "
Nov 02 12:28:18.630690 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:28:18.630528    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb3facba-63fa-48af-a09b-fb6e8dc3a993-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/eb3facba-63fa-48af-a09b-fb6e8dc3a993-oauth-serving-cert\") pod \"eb3facba-63fa-48af-a09b-fb6e8dc3a993\" (UID: \"eb3facba-63fa-48af-a09b-fb6e8dc3a993\") "
Nov 02 12:28:18.630832 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:28:18.630801    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb3facba-63fa-48af-a09b-fb6e8dc3a993-tmp" pod "eb3facba-63fa-48af-a09b-fb6e8dc3a993" (UID: "eb3facba-63fa-48af-a09b-fb6e8dc3a993"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:28:18.630868 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:28:18.630834    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb3facba-63fa-48af-a09b-fb6e8dc3a993-console-config" pod "eb3facba-63fa-48af-a09b-fb6e8dc3a993" (UID: "eb3facba-63fa-48af-a09b-fb6e8dc3a993"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:28:18.631071 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:28:18.631043    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb3facba-63fa-48af-a09b-fb6e8dc3a993-oauth-serving-cert" pod "eb3facba-63fa-48af-a09b-fb6e8dc3a993" (UID: "eb3facba-63fa-48af-a09b-fb6e8dc3a993"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:28:18.631166 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:28:18.631104    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb3facba-63fa-48af-a09b-fb6e8dc3a993-service-ca" pod "eb3facba-63fa-48af-a09b-fb6e8dc3a993" (UID: "eb3facba-63fa-48af-a09b-fb6e8dc3a993"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:28:18.632706 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:28:18.632680    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb3facba-63fa-48af-a09b-fb6e8dc3a993-kube-api-access-lj548" pod "eb3facba-63fa-48af-a09b-fb6e8dc3a993" (UID: "eb3facba-63fa-48af-a09b-fb6e8dc3a993"). InnerVolumeSpecName "kube-api-access-lj548". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:28:18.632706 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:28:18.632686    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb3facba-63fa-48af-a09b-fb6e8dc3a993-console-serving-cert" pod "eb3facba-63fa-48af-a09b-fb6e8dc3a993" (UID: "eb3facba-63fa-48af-a09b-fb6e8dc3a993"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:28:18.632824 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:28:18.632704    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb3facba-63fa-48af-a09b-fb6e8dc3a993-console-oauth-config" pod "eb3facba-63fa-48af-a09b-fb6e8dc3a993" (UID: "eb3facba-63fa-48af-a09b-fb6e8dc3a993"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:28:18.731216 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:28:18.731182    4559 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eb3facba-63fa-48af-a09b-fb6e8dc3a993-tmp\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:28:18.731216 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:28:18.731221    4559 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/eb3facba-63fa-48af-a09b-fb6e8dc3a993-console-serving-cert\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:28:18.731406 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:28:18.731233    4559 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/eb3facba-63fa-48af-a09b-fb6e8dc3a993-oauth-serving-cert\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:28:18.731406 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:28:18.731242    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lj548\" (UniqueName: \"kubernetes.io/projected/eb3facba-63fa-48af-a09b-fb6e8dc3a993-kube-api-access-lj548\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:28:18.731406 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:28:18.731252    4559 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/eb3facba-63fa-48af-a09b-fb6e8dc3a993-service-ca\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:28:18.731406 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:28:18.731259    4559 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/eb3facba-63fa-48af-a09b-fb6e8dc3a993-console-config\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:28:18.731406 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:28:18.731267    4559 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/eb3facba-63fa-48af-a09b-fb6e8dc3a993-console-oauth-config\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:28:19.304463 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:28:19.304430    4559 generic.go:356] "Generic (PLEG): container finished" podID="eb3facba-63fa-48af-a09b-fb6e8dc3a993" containerID="7d7b1ce0005aee52149f156d938c8f0cd176b2697c74e6130e8eead1db4fe77c" exitCode=0
Nov 02 12:28:19.304678 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:28:19.304476    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7964fd5c55-c8drt"
Nov 02 12:28:19.304678 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:28:19.304504    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7964fd5c55-c8drt" event={"ID":"eb3facba-63fa-48af-a09b-fb6e8dc3a993","Type":"ContainerDied","Data":"7d7b1ce0005aee52149f156d938c8f0cd176b2697c74e6130e8eead1db4fe77c"}
Nov 02 12:28:19.304678 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:28:19.304554    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7964fd5c55-c8drt" event={"ID":"eb3facba-63fa-48af-a09b-fb6e8dc3a993","Type":"ContainerDied","Data":"40df2954848cce32a6bcbed1633b5ce00d75516cbfd1678f9f9e83072fdf28c4"}
Nov 02 12:28:19.304678 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:28:19.304574    4559 scope.go:122] "RemoveContainer" containerID="7d7b1ce0005aee52149f156d938c8f0cd176b2697c74e6130e8eead1db4fe77c"
Nov 02 12:28:19.315996 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:28:19.315916    4559 scope.go:122] "RemoveContainer" containerID="7d7b1ce0005aee52149f156d938c8f0cd176b2697c74e6130e8eead1db4fe77c"
Nov 02 12:28:19.316244 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:28:19.316223    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d7b1ce0005aee52149f156d938c8f0cd176b2697c74e6130e8eead1db4fe77c\": container with ID starting with 7d7b1ce0005aee52149f156d938c8f0cd176b2697c74e6130e8eead1db4fe77c not found: ID does not exist" containerID="7d7b1ce0005aee52149f156d938c8f0cd176b2697c74e6130e8eead1db4fe77c"
Nov 02 12:28:19.316286 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:28:19.316254    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d7b1ce0005aee52149f156d938c8f0cd176b2697c74e6130e8eead1db4fe77c"} err="failed to get container status \"7d7b1ce0005aee52149f156d938c8f0cd176b2697c74e6130e8eead1db4fe77c\": rpc error: code = NotFound desc = could not find container \"7d7b1ce0005aee52149f156d938c8f0cd176b2697c74e6130e8eead1db4fe77c\": container with ID starting with 7d7b1ce0005aee52149f156d938c8f0cd176b2697c74e6130e8eead1db4fe77c not found: ID does not exist"
Nov 02 12:28:19.357285 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:28:19.357262    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7964fd5c55-c8drt"]
Nov 02 12:28:19.369980 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:28:19.369959    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7964fd5c55-c8drt"]
Nov 02 12:28:19.891547 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:28:19.891503    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb3facba-63fa-48af-a09b-fb6e8dc3a993" path="/var/lib/kubelet/pods/eb3facba-63fa-48af-a09b-fb6e8dc3a993/volumes"
Nov 02 12:28:45.084977 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:28:45.084936    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqcmj" containerName="extractor"
Nov 02 12:29:15.082579 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:29:15.082500    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqcmj" containerName="extractor"
Nov 02 12:29:29.770581 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:29:29.770543    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-l4gld"]
Nov 02 12:29:29.771084 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:29:29.770824    4559 cpu_manager.go:414] "containerMap: removing container" podUID="eb3facba-63fa-48af-a09b-fb6e8dc3a993" containerName="console"
Nov 02 12:29:29.771084 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:29:29.770835    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb3facba-63fa-48af-a09b-fb6e8dc3a993" containerName="console"
Nov 02 12:29:29.771084 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:29:29.770897    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb3facba-63fa-48af-a09b-fb6e8dc3a993" containerName="console"
Nov 02 12:29:29.773700 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:29:29.773684    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l4gld"
Nov 02 12:29:29.789531 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:29:29.789505    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Nov 02 12:29:29.802615 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:29:29.802588    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Nov 02 12:29:29.802615 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:29:29.802610    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Nov 02 12:29:29.802834 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:29:29.802600    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-6fgxx\""
Nov 02 12:29:29.805871 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:29:29.805842    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Nov 02 12:29:29.805871 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:29:29.805857    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Nov 02 12:29:29.805977 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:29:29.805966    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Nov 02 12:29:29.806918 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:29:29.806901    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-l4gld"]
Nov 02 12:29:29.961372 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:29:29.961341    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da8a5263-2e77-47a8-9a75-fd8ee0e5b80d-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-l4gld\" (UID: \"da8a5263-2e77-47a8-9a75-fd8ee0e5b80d\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l4gld"
Nov 02 12:29:29.961589 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:29:29.961391    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/da8a5263-2e77-47a8-9a75-fd8ee0e5b80d-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-l4gld\" (UID: \"da8a5263-2e77-47a8-9a75-fd8ee0e5b80d\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l4gld"
Nov 02 12:29:29.961589 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:29:29.961418    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/da8a5263-2e77-47a8-9a75-fd8ee0e5b80d-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-l4gld\" (UID: \"da8a5263-2e77-47a8-9a75-fd8ee0e5b80d\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l4gld"
Nov 02 12:29:29.961589 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:29:29.961508    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/da8a5263-2e77-47a8-9a75-fd8ee0e5b80d-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-l4gld\" (UID: \"da8a5263-2e77-47a8-9a75-fd8ee0e5b80d\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l4gld"
Nov 02 12:29:29.961589 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:29:29.961558    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/da8a5263-2e77-47a8-9a75-fd8ee0e5b80d-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-l4gld\" (UID: \"da8a5263-2e77-47a8-9a75-fd8ee0e5b80d\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l4gld"
Nov 02 12:29:29.961806 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:29:29.961591    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlzn4\" (UniqueName: \"kubernetes.io/projected/da8a5263-2e77-47a8-9a75-fd8ee0e5b80d-kube-api-access-vlzn4\") pod \"istiod-openshift-gateway-7876bfcb74-l4gld\" (UID: \"da8a5263-2e77-47a8-9a75-fd8ee0e5b80d\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l4gld"
Nov 02 12:29:29.961806 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:29:29.961629    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/da8a5263-2e77-47a8-9a75-fd8ee0e5b80d-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-l4gld\" (UID: \"da8a5263-2e77-47a8-9a75-fd8ee0e5b80d\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l4gld"
Nov 02 12:29:30.062707 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:29:30.062605    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da8a5263-2e77-47a8-9a75-fd8ee0e5b80d-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-l4gld\" (UID: \"da8a5263-2e77-47a8-9a75-fd8ee0e5b80d\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l4gld"
Nov 02 12:29:30.062879 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:29:30.062801    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/da8a5263-2e77-47a8-9a75-fd8ee0e5b80d-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-l4gld\" (UID: \"da8a5263-2e77-47a8-9a75-fd8ee0e5b80d\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l4gld"
Nov 02 12:29:30.062879 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:29:30.062855    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/da8a5263-2e77-47a8-9a75-fd8ee0e5b80d-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-l4gld\" (UID: \"da8a5263-2e77-47a8-9a75-fd8ee0e5b80d\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l4gld"
Nov 02 12:29:30.063011 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:29:30.062882    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/da8a5263-2e77-47a8-9a75-fd8ee0e5b80d-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-l4gld\" (UID: \"da8a5263-2e77-47a8-9a75-fd8ee0e5b80d\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l4gld"
Nov 02 12:29:30.063011 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:29:30.062933    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/da8a5263-2e77-47a8-9a75-fd8ee0e5b80d-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-l4gld\" (UID: \"da8a5263-2e77-47a8-9a75-fd8ee0e5b80d\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l4gld"
Nov 02 12:29:30.063011 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:29:30.062973    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlzn4\" (UniqueName: \"kubernetes.io/projected/da8a5263-2e77-47a8-9a75-fd8ee0e5b80d-kube-api-access-vlzn4\") pod \"istiod-openshift-gateway-7876bfcb74-l4gld\" (UID: \"da8a5263-2e77-47a8-9a75-fd8ee0e5b80d\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l4gld"
Nov 02 12:29:30.063170 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:29:30.063022    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/da8a5263-2e77-47a8-9a75-fd8ee0e5b80d-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-l4gld\" (UID: \"da8a5263-2e77-47a8-9a75-fd8ee0e5b80d\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l4gld"
Nov 02 12:29:30.063454 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:29:30.063424    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/da8a5263-2e77-47a8-9a75-fd8ee0e5b80d-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-l4gld\" (UID: \"da8a5263-2e77-47a8-9a75-fd8ee0e5b80d\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l4gld"
Nov 02 12:29:30.065318 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:29:30.065271    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/da8a5263-2e77-47a8-9a75-fd8ee0e5b80d-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-l4gld\" (UID: \"da8a5263-2e77-47a8-9a75-fd8ee0e5b80d\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l4gld"
Nov 02 12:29:30.065444 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:29:30.065319    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/da8a5263-2e77-47a8-9a75-fd8ee0e5b80d-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-l4gld\" (UID: \"da8a5263-2e77-47a8-9a75-fd8ee0e5b80d\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l4gld"
Nov 02 12:29:30.065444 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:29:30.065429    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/da8a5263-2e77-47a8-9a75-fd8ee0e5b80d-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-l4gld\" (UID: \"da8a5263-2e77-47a8-9a75-fd8ee0e5b80d\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l4gld"
Nov 02 12:29:30.065565 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:29:30.065459    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/da8a5263-2e77-47a8-9a75-fd8ee0e5b80d-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-l4gld\" (UID: \"da8a5263-2e77-47a8-9a75-fd8ee0e5b80d\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l4gld"
Nov 02 12:29:30.087058 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:29:30.087026    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da8a5263-2e77-47a8-9a75-fd8ee0e5b80d-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-l4gld\" (UID: \"da8a5263-2e77-47a8-9a75-fd8ee0e5b80d\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l4gld"
Nov 02 12:29:30.087201 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:29:30.087192    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlzn4\" (UniqueName: \"kubernetes.io/projected/da8a5263-2e77-47a8-9a75-fd8ee0e5b80d-kube-api-access-vlzn4\") pod \"istiod-openshift-gateway-7876bfcb74-l4gld\" (UID: \"da8a5263-2e77-47a8-9a75-fd8ee0e5b80d\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l4gld"
Nov 02 12:29:30.384868 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:29:30.384772    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l4gld"
Nov 02 12:29:30.694028 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:29:30.693998    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-l4gld"]
Nov 02 12:29:30.695012 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:29:30.694988    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda8a5263_2e77_47a8_9a75_fd8ee0e5b80d.slice/crio-a6f5e3dc1d7e18d0c603f59d8f4a0d08dc049a5a6c046b5e5801ea50e9588375 WatchSource:0}: Error finding container a6f5e3dc1d7e18d0c603f59d8f4a0d08dc049a5a6c046b5e5801ea50e9588375: Status 404 returned error can't find the container with id a6f5e3dc1d7e18d0c603f59d8f4a0d08dc049a5a6c046b5e5801ea50e9588375
Nov 02 12:29:31.545802 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:29:31.545757    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l4gld" event={"ID":"da8a5263-2e77-47a8-9a75-fd8ee0e5b80d","Type":"ContainerStarted","Data":"a6f5e3dc1d7e18d0c603f59d8f4a0d08dc049a5a6c046b5e5801ea50e9588375"}
Nov 02 12:29:33.143256 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:29:33.143223    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:29:33.143526 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:29:33.143291    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:29:33.555221 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:29:33.555186    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l4gld" event={"ID":"da8a5263-2e77-47a8-9a75-fd8ee0e5b80d","Type":"ContainerStarted","Data":"70dd1bdc7ae3d0cfe54a4205ffe3fbc2eabcef134df2e38b82aa3ccfa17ef456"}
Nov 02 12:29:33.555416 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:29:33.555319    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l4gld"
Nov 02 12:29:33.626267 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:29:33.626192    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l4gld" podStartSLOduration=2.180551846 podStartE2EDuration="4.626171865s" totalImagesPullingTime="2.445620019s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:29:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:29:33.61233559 +0000 UTC m=+169.867734686" watchObservedRunningTime="2026-11-02 12:29:33.626171865 +0000 UTC m=+169.881570961"
Nov 02 12:29:34.560748 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:29:34.560710    4559 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-l4gld container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 12:29:34.561197 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:29:34.560771    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l4gld" podUID="da8a5263-2e77-47a8-9a75-fd8ee0e5b80d" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 12:29:37.560341 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:29:37.560304    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l4gld"
Nov 02 12:29:38.573364 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:29:38.573261    4559 generic.go:356] "Generic (PLEG): container finished" podID="80e2d631-c27d-4629-bf4f-dec1c1e29656" containerID="ba47506578048294d07a0883b61af4384f0abd840811f0a4dac3d463c8ada573" exitCode=0
Nov 02 12:29:38.573364 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:29:38.573343    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-gj98f-lf5h7" event={"ID":"80e2d631-c27d-4629-bf4f-dec1c1e29656","Type":"ContainerDied","Data":"ba47506578048294d07a0883b61af4384f0abd840811f0a4dac3d463c8ada573"}
Nov 02 12:29:39.692838 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:29:39.692813    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-gj98f-lf5h7"
Nov 02 12:29:39.735707 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:29:39.735672    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80e2d631-c27d-4629-bf4f-dec1c1e29656-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/80e2d631-c27d-4629-bf4f-dec1c1e29656-trusted-ca-bundle\") pod \"80e2d631-c27d-4629-bf4f-dec1c1e29656\" (UID: \"80e2d631-c27d-4629-bf4f-dec1c1e29656\") "
Nov 02 12:29:39.735883 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:29:39.735726    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80e2d631-c27d-4629-bf4f-dec1c1e29656-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/80e2d631-c27d-4629-bf4f-dec1c1e29656-service-ca-bundle\") pod \"80e2d631-c27d-4629-bf4f-dec1c1e29656\" (UID: \"80e2d631-c27d-4629-bf4f-dec1c1e29656\") "
Nov 02 12:29:39.735883 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:29:39.735774    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80e2d631-c27d-4629-bf4f-dec1c1e29656-kube-api-access-pgtz9\" (UniqueName: \"kubernetes.io/projected/80e2d631-c27d-4629-bf4f-dec1c1e29656-kube-api-access-pgtz9\") pod \"80e2d631-c27d-4629-bf4f-dec1c1e29656\" (UID: \"80e2d631-c27d-4629-bf4f-dec1c1e29656\") "
Nov 02 12:29:39.735883 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:29:39.735831    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80e2d631-c27d-4629-bf4f-dec1c1e29656-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/80e2d631-c27d-4629-bf4f-dec1c1e29656-archives-path\") pod \"80e2d631-c27d-4629-bf4f-dec1c1e29656\" (UID: \"80e2d631-c27d-4629-bf4f-dec1c1e29656\") "
Nov 02 12:29:39.736155 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:29:39.736114    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80e2d631-c27d-4629-bf4f-dec1c1e29656-trusted-ca-bundle" pod "80e2d631-c27d-4629-bf4f-dec1c1e29656" (UID: "80e2d631-c27d-4629-bf4f-dec1c1e29656"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:29:39.736258 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:29:39.736231    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80e2d631-c27d-4629-bf4f-dec1c1e29656-service-ca-bundle" pod "80e2d631-c27d-4629-bf4f-dec1c1e29656" (UID: "80e2d631-c27d-4629-bf4f-dec1c1e29656"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:29:39.736320 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:29:39.736272    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80e2d631-c27d-4629-bf4f-dec1c1e29656-archives-path" pod "80e2d631-c27d-4629-bf4f-dec1c1e29656" (UID: "80e2d631-c27d-4629-bf4f-dec1c1e29656"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:29:39.738102 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:29:39.738072    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80e2d631-c27d-4629-bf4f-dec1c1e29656-kube-api-access-pgtz9" pod "80e2d631-c27d-4629-bf4f-dec1c1e29656" (UID: "80e2d631-c27d-4629-bf4f-dec1c1e29656"). InnerVolumeSpecName "kube-api-access-pgtz9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:29:39.836846 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:29:39.836743    4559 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/80e2d631-c27d-4629-bf4f-dec1c1e29656-archives-path\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:29:39.836846 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:29:39.836789    4559 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/80e2d631-c27d-4629-bf4f-dec1c1e29656-trusted-ca-bundle\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:29:39.836846 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:29:39.836800    4559 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/80e2d631-c27d-4629-bf4f-dec1c1e29656-service-ca-bundle\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:29:39.836846 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:29:39.836809    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pgtz9\" (UniqueName: \"kubernetes.io/projected/80e2d631-c27d-4629-bf4f-dec1c1e29656-kube-api-access-pgtz9\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:29:40.586764 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:29:40.586728    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-gj98f-lf5h7" event={"ID":"80e2d631-c27d-4629-bf4f-dec1c1e29656","Type":"ContainerDied","Data":"9f40d455f5e642541ef838fbfd41941f137bd4324543466828780c9fe8805a67"}
Nov 02 12:29:40.586764 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:29:40.586763    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-gj98f-lf5h7"
Nov 02 12:29:40.586980 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:29:40.586797    4559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9f40d455f5e642541ef838fbfd41941f137bd4324543466828780c9fe8805a67"
Nov 02 12:29:45.084967 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:29:45.084923    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqcmj" containerName="extractor"
Nov 02 12:30:15.084037 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:30:15.084001    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqcmj" containerName="extractor"
Nov 02 12:30:23.924621 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:30:23.924585    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-67b54f6f58-8hgzl"]
Nov 02 12:30:23.925121 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:30:23.924852    4559 cpu_manager.go:414] "containerMap: removing container" podUID="80e2d631-c27d-4629-bf4f-dec1c1e29656" containerName="insights-gathering"
Nov 02 12:30:23.925121 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:30:23.924862    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80e2d631-c27d-4629-bf4f-dec1c1e29656" containerName="insights-gathering"
Nov 02 12:30:23.925121 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:30:23.924917    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="80e2d631-c27d-4629-bf4f-dec1c1e29656" containerName="insights-gathering"
Nov 02 12:30:23.931226 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:30:23.931206    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-67b54f6f58-8hgzl"
Nov 02 12:30:23.946542 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:30:23.946519    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-67b54f6f58-8hgzl"]
Nov 02 12:30:24.003421 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:30:24.003394    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\""
Nov 02 12:30:24.003421 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:30:24.003394    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Nov 02 12:30:24.003886 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:30:24.003868    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Nov 02 12:30:24.003935 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:30:24.003871    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-wwbqf\""
Nov 02 12:30:24.006717 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:30:24.006701    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Nov 02 12:30:24.006764 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:30:24.006705    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Nov 02 12:30:24.051250 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:30:24.051225    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/65ace096-0387-4523-958c-c47a473f1055-apiservice-cert\") pod \"opendatahub-operator-controller-manager-67b54f6f58-8hgzl\" (UID: \"65ace096-0387-4523-958c-c47a473f1055\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-67b54f6f58-8hgzl"
Nov 02 12:30:24.051402 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:30:24.051259    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/65ace096-0387-4523-958c-c47a473f1055-metrics-cert\") pod \"opendatahub-operator-controller-manager-67b54f6f58-8hgzl\" (UID: \"65ace096-0387-4523-958c-c47a473f1055\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-67b54f6f58-8hgzl"
Nov 02 12:30:24.051402 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:30:24.051275    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/65ace096-0387-4523-958c-c47a473f1055-manifests\") pod \"opendatahub-operator-controller-manager-67b54f6f58-8hgzl\" (UID: \"65ace096-0387-4523-958c-c47a473f1055\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-67b54f6f58-8hgzl"
Nov 02 12:30:24.051402 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:30:24.051298    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sls9b\" (UniqueName: \"kubernetes.io/projected/65ace096-0387-4523-958c-c47a473f1055-kube-api-access-sls9b\") pod \"opendatahub-operator-controller-manager-67b54f6f58-8hgzl\" (UID: \"65ace096-0387-4523-958c-c47a473f1055\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-67b54f6f58-8hgzl"
Nov 02 12:30:24.051402 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:30:24.051388    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/65ace096-0387-4523-958c-c47a473f1055-webhook-cert\") pod \"opendatahub-operator-controller-manager-67b54f6f58-8hgzl\" (UID: \"65ace096-0387-4523-958c-c47a473f1055\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-67b54f6f58-8hgzl"
Nov 02 12:30:24.051533 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:30:24.051424    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/65ace096-0387-4523-958c-c47a473f1055-tmp\") pod \"opendatahub-operator-controller-manager-67b54f6f58-8hgzl\" (UID: \"65ace096-0387-4523-958c-c47a473f1055\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-67b54f6f58-8hgzl"
Nov 02 12:30:24.152784 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:30:24.152750    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/65ace096-0387-4523-958c-c47a473f1055-apiservice-cert\") pod \"opendatahub-operator-controller-manager-67b54f6f58-8hgzl\" (UID: \"65ace096-0387-4523-958c-c47a473f1055\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-67b54f6f58-8hgzl"
Nov 02 12:30:24.152957 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:30:24.152797    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/65ace096-0387-4523-958c-c47a473f1055-metrics-cert\") pod \"opendatahub-operator-controller-manager-67b54f6f58-8hgzl\" (UID: \"65ace096-0387-4523-958c-c47a473f1055\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-67b54f6f58-8hgzl"
Nov 02 12:30:24.152957 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:30:24.152930    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/65ace096-0387-4523-958c-c47a473f1055-manifests\") pod \"opendatahub-operator-controller-manager-67b54f6f58-8hgzl\" (UID: \"65ace096-0387-4523-958c-c47a473f1055\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-67b54f6f58-8hgzl"
Nov 02 12:30:24.153097 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:30:24.153002    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sls9b\" (UniqueName: \"kubernetes.io/projected/65ace096-0387-4523-958c-c47a473f1055-kube-api-access-sls9b\") pod \"opendatahub-operator-controller-manager-67b54f6f58-8hgzl\" (UID: \"65ace096-0387-4523-958c-c47a473f1055\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-67b54f6f58-8hgzl"
Nov 02 12:30:24.153097 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:30:24.153045    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/65ace096-0387-4523-958c-c47a473f1055-webhook-cert\") pod \"opendatahub-operator-controller-manager-67b54f6f58-8hgzl\" (UID: \"65ace096-0387-4523-958c-c47a473f1055\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-67b54f6f58-8hgzl"
Nov 02 12:30:24.153097 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:30:24.153086    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/65ace096-0387-4523-958c-c47a473f1055-tmp\") pod \"opendatahub-operator-controller-manager-67b54f6f58-8hgzl\" (UID: \"65ace096-0387-4523-958c-c47a473f1055\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-67b54f6f58-8hgzl"
Nov 02 12:30:24.153374 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:30:24.153336    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/65ace096-0387-4523-958c-c47a473f1055-manifests\") pod \"opendatahub-operator-controller-manager-67b54f6f58-8hgzl\" (UID: \"65ace096-0387-4523-958c-c47a473f1055\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-67b54f6f58-8hgzl"
Nov 02 12:30:24.155285 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:30:24.155259    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/65ace096-0387-4523-958c-c47a473f1055-metrics-cert\") pod \"opendatahub-operator-controller-manager-67b54f6f58-8hgzl\" (UID: \"65ace096-0387-4523-958c-c47a473f1055\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-67b54f6f58-8hgzl"
Nov 02 12:30:24.155285 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:30:24.155267    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/65ace096-0387-4523-958c-c47a473f1055-tmp\") pod \"opendatahub-operator-controller-manager-67b54f6f58-8hgzl\" (UID: \"65ace096-0387-4523-958c-c47a473f1055\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-67b54f6f58-8hgzl"
Nov 02 12:30:24.155413 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:30:24.155349    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/65ace096-0387-4523-958c-c47a473f1055-apiservice-cert\") pod \"opendatahub-operator-controller-manager-67b54f6f58-8hgzl\" (UID: \"65ace096-0387-4523-958c-c47a473f1055\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-67b54f6f58-8hgzl"
Nov 02 12:30:24.155413 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:30:24.155387    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/65ace096-0387-4523-958c-c47a473f1055-webhook-cert\") pod \"opendatahub-operator-controller-manager-67b54f6f58-8hgzl\" (UID: \"65ace096-0387-4523-958c-c47a473f1055\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-67b54f6f58-8hgzl"
Nov 02 12:30:24.177548 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:30:24.177492    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sls9b\" (UniqueName: \"kubernetes.io/projected/65ace096-0387-4523-958c-c47a473f1055-kube-api-access-sls9b\") pod \"opendatahub-operator-controller-manager-67b54f6f58-8hgzl\" (UID: \"65ace096-0387-4523-958c-c47a473f1055\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-67b54f6f58-8hgzl"
Nov 02 12:30:24.246869 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:30:24.246836    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-67b54f6f58-8hgzl"
Nov 02 12:30:24.593037 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:30:24.593007    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-67b54f6f58-8hgzl"]
Nov 02 12:30:24.593652 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:30:24.593610    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65ace096_0387_4523_958c_c47a473f1055.slice/crio-9abf45aa41d621dd23ed4f6b4ef984a2b6ee921664a70c19e26a60c558a68af1 WatchSource:0}: Error finding container 9abf45aa41d621dd23ed4f6b4ef984a2b6ee921664a70c19e26a60c558a68af1: Status 404 returned error can't find the container with id 9abf45aa41d621dd23ed4f6b4ef984a2b6ee921664a70c19e26a60c558a68af1
Nov 02 12:30:24.734849 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:30:24.734816    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-67b54f6f58-8hgzl" event={"ID":"65ace096-0387-4523-958c-c47a473f1055","Type":"ContainerStarted","Data":"9abf45aa41d621dd23ed4f6b4ef984a2b6ee921664a70c19e26a60c558a68af1"}
Nov 02 12:30:27.748075 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:30:27.747983    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-67b54f6f58-8hgzl" event={"ID":"65ace096-0387-4523-958c-c47a473f1055","Type":"ContainerStarted","Data":"a0e955ba297984f7219071f75977678e24277725e68c5fd86a7298dee03d5dc8"}
Nov 02 12:30:28.753013 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:30:28.752978    4559 generic.go:356] "Generic (PLEG): container finished" podID="65ace096-0387-4523-958c-c47a473f1055" containerID="a0e955ba297984f7219071f75977678e24277725e68c5fd86a7298dee03d5dc8" exitCode=0
Nov 02 12:30:28.753508 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:30:28.753047    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-67b54f6f58-8hgzl" event={"ID":"65ace096-0387-4523-958c-c47a473f1055","Type":"ContainerDied","Data":"a0e955ba297984f7219071f75977678e24277725e68c5fd86a7298dee03d5dc8"}
Nov 02 12:30:29.758526 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:30:29.758490    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-67b54f6f58-8hgzl" event={"ID":"65ace096-0387-4523-958c-c47a473f1055","Type":"ContainerStarted","Data":"9e7366f5730a54425834213fc2048c6f00780bf975301cbed2f05b98dbf4e81d"}
Nov 02 12:30:29.758972 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:30:29.758616    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-67b54f6f58-8hgzl"
Nov 02 12:30:29.853242 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:30:29.853174    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-67b54f6f58-8hgzl" podStartSLOduration=3.148228045 podStartE2EDuration="6.853155057s" totalImagesPullingTime="3.258551168s" totalInitContainerRuntime="446.375844ms" isStatefulPod=false podCreationTimestamp="2026-11-02 12:30:23 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:30:29.838947616 +0000 UTC m=+226.094346710" watchObservedRunningTime="2026-11-02 12:30:29.853155057 +0000 UTC m=+226.108554153"
Nov 02 12:30:40.765030 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:30:40.765003    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-67b54f6f58-8hgzl"
Nov 02 12:30:45.083673 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:30:45.083622    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqcmj" containerName="extractor"
Nov 02 12:30:53.591375 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:30:53.591340    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-9c9fcfcf5-9pxlh"]
Nov 02 12:30:53.602690 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:30:53.602665    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-9c9fcfcf5-9pxlh"
Nov 02 12:30:53.614081 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:30:53.614055    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-9c9fcfcf5-9pxlh"]
Nov 02 12:30:53.686890 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:30:53.686857    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64pj8\" (UniqueName: \"kubernetes.io/projected/4aea6613-d504-4ab3-80cd-a1c4010709a7-kube-api-access-64pj8\") pod \"kube-auth-proxy-9c9fcfcf5-9pxlh\" (UID: \"4aea6613-d504-4ab3-80cd-a1c4010709a7\") " pod="openshift-ingress/kube-auth-proxy-9c9fcfcf5-9pxlh"
Nov 02 12:30:53.687073 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:30:53.686903    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4aea6613-d504-4ab3-80cd-a1c4010709a7-tmp\") pod \"kube-auth-proxy-9c9fcfcf5-9pxlh\" (UID: \"4aea6613-d504-4ab3-80cd-a1c4010709a7\") " pod="openshift-ingress/kube-auth-proxy-9c9fcfcf5-9pxlh"
Nov 02 12:30:53.687073 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:30:53.686923    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4aea6613-d504-4ab3-80cd-a1c4010709a7-tls-certs\") pod \"kube-auth-proxy-9c9fcfcf5-9pxlh\" (UID: \"4aea6613-d504-4ab3-80cd-a1c4010709a7\") " pod="openshift-ingress/kube-auth-proxy-9c9fcfcf5-9pxlh"
Nov 02 12:30:53.697438 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:30:53.697410    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Nov 02 12:30:53.697590 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:30:53.697443    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Nov 02 12:30:53.697667 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:30:53.697596    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-2qsjb\""
Nov 02 12:30:53.787802 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:30:53.787764    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4aea6613-d504-4ab3-80cd-a1c4010709a7-tmp\") pod \"kube-auth-proxy-9c9fcfcf5-9pxlh\" (UID: \"4aea6613-d504-4ab3-80cd-a1c4010709a7\") " pod="openshift-ingress/kube-auth-proxy-9c9fcfcf5-9pxlh"
Nov 02 12:30:53.787985 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:30:53.787815    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4aea6613-d504-4ab3-80cd-a1c4010709a7-tls-certs\") pod \"kube-auth-proxy-9c9fcfcf5-9pxlh\" (UID: \"4aea6613-d504-4ab3-80cd-a1c4010709a7\") " pod="openshift-ingress/kube-auth-proxy-9c9fcfcf5-9pxlh"
Nov 02 12:30:53.787985 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:30:53.787913    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64pj8\" (UniqueName: \"kubernetes.io/projected/4aea6613-d504-4ab3-80cd-a1c4010709a7-kube-api-access-64pj8\") pod \"kube-auth-proxy-9c9fcfcf5-9pxlh\" (UID: \"4aea6613-d504-4ab3-80cd-a1c4010709a7\") " pod="openshift-ingress/kube-auth-proxy-9c9fcfcf5-9pxlh"
Nov 02 12:30:53.790516 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:30:53.790488    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4aea6613-d504-4ab3-80cd-a1c4010709a7-tmp\") pod \"kube-auth-proxy-9c9fcfcf5-9pxlh\" (UID: \"4aea6613-d504-4ab3-80cd-a1c4010709a7\") " pod="openshift-ingress/kube-auth-proxy-9c9fcfcf5-9pxlh"
Nov 02 12:30:53.790722 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:30:53.790503    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4aea6613-d504-4ab3-80cd-a1c4010709a7-tls-certs\") pod \"kube-auth-proxy-9c9fcfcf5-9pxlh\" (UID: \"4aea6613-d504-4ab3-80cd-a1c4010709a7\") " pod="openshift-ingress/kube-auth-proxy-9c9fcfcf5-9pxlh"
Nov 02 12:30:53.812526 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:30:53.812499    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64pj8\" (UniqueName: \"kubernetes.io/projected/4aea6613-d504-4ab3-80cd-a1c4010709a7-kube-api-access-64pj8\") pod \"kube-auth-proxy-9c9fcfcf5-9pxlh\" (UID: \"4aea6613-d504-4ab3-80cd-a1c4010709a7\") " pod="openshift-ingress/kube-auth-proxy-9c9fcfcf5-9pxlh"
Nov 02 12:30:53.915743 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:30:53.915663    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-9c9fcfcf5-9pxlh"
Nov 02 12:30:54.212697 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:30:54.212669    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-9c9fcfcf5-9pxlh"]
Nov 02 12:30:54.213358 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:30:54.213334    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4aea6613_d504_4ab3_80cd_a1c4010709a7.slice/crio-88ac025d6f4eb1596aa87acc31e8902e7339f74a40de94cd5cc3c76a868fb967 WatchSource:0}: Error finding container 88ac025d6f4eb1596aa87acc31e8902e7339f74a40de94cd5cc3c76a868fb967: Status 404 returned error can't find the container with id 88ac025d6f4eb1596aa87acc31e8902e7339f74a40de94cd5cc3c76a868fb967
Nov 02 12:30:54.855574 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:30:54.855534    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-9c9fcfcf5-9pxlh" event={"ID":"4aea6613-d504-4ab3-80cd-a1c4010709a7","Type":"ContainerStarted","Data":"88ac025d6f4eb1596aa87acc31e8902e7339f74a40de94cd5cc3c76a868fb967"}
Nov 02 12:30:55.934881 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:30:55.934845    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-mgzwt"]
Nov 02 12:30:55.943441 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:30:55.943417    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"]
Nov 02 12:30:55.943573 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:30:55.943556    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-mgzwt"
Nov 02 12:30:55.946810 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:30:55.946791    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"
Nov 02 12:30:55.957138 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:30:55.957114    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-mgzwt"]
Nov 02 12:30:55.961298 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:30:55.961274    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"]
Nov 02 12:30:55.969010 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:30:55.968987    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-x7vbt\""
Nov 02 12:30:55.972459 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:30:55.972440    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Nov 02 12:30:55.972459 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:30:55.972446    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Nov 02 12:30:55.972622 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:30:55.972447    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Nov 02 12:30:55.972622 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:30:55.972542    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-nbd9h\""
Nov 02 12:30:56.039508 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:30:56.039477    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"]
Nov 02 12:30:56.108476 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:30:56.108445    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/ae914cec-8158-4c77-9870-e0fa6573c7b6-config-file\") pod \"limitador-limitador-7c9b87d574-l9pbf\" (UID: \"ae914cec-8158-4c77-9870-e0fa6573c7b6\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"
Nov 02 12:30:56.108706 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:30:56.108488    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmgdb\" (UniqueName: \"kubernetes.io/projected/5d7f5a1b-6338-4642-85aa-d7d3307265bd-kube-api-access-pmgdb\") pod \"authorino-56c6c9ddd5-mgzwt\" (UID: \"5d7f5a1b-6338-4642-85aa-d7d3307265bd\") " pod="kuadrant-system/authorino-56c6c9ddd5-mgzwt"
Nov 02 12:30:56.108706 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:30:56.108585    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgss4\" (UniqueName: \"kubernetes.io/projected/ae914cec-8158-4c77-9870-e0fa6573c7b6-kube-api-access-bgss4\") pod \"limitador-limitador-7c9b87d574-l9pbf\" (UID: \"ae914cec-8158-4c77-9870-e0fa6573c7b6\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"
Nov 02 12:30:56.210032 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:30:56.209956    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/ae914cec-8158-4c77-9870-e0fa6573c7b6-config-file\") pod \"limitador-limitador-7c9b87d574-l9pbf\" (UID: \"ae914cec-8158-4c77-9870-e0fa6573c7b6\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"
Nov 02 12:30:56.210032 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:30:56.209998    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmgdb\" (UniqueName: \"kubernetes.io/projected/5d7f5a1b-6338-4642-85aa-d7d3307265bd-kube-api-access-pmgdb\") pod \"authorino-56c6c9ddd5-mgzwt\" (UID: \"5d7f5a1b-6338-4642-85aa-d7d3307265bd\") " pod="kuadrant-system/authorino-56c6c9ddd5-mgzwt"
Nov 02 12:30:56.210248 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:30:56.210167    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgss4\" (UniqueName: \"kubernetes.io/projected/ae914cec-8158-4c77-9870-e0fa6573c7b6-kube-api-access-bgss4\") pod \"limitador-limitador-7c9b87d574-l9pbf\" (UID: \"ae914cec-8158-4c77-9870-e0fa6573c7b6\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"
Nov 02 12:30:56.210632 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:30:56.210611    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/ae914cec-8158-4c77-9870-e0fa6573c7b6-config-file\") pod \"limitador-limitador-7c9b87d574-l9pbf\" (UID: \"ae914cec-8158-4c77-9870-e0fa6573c7b6\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"
Nov 02 12:30:56.242123 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:30:56.242095    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmgdb\" (UniqueName: \"kubernetes.io/projected/5d7f5a1b-6338-4642-85aa-d7d3307265bd-kube-api-access-pmgdb\") pod \"authorino-56c6c9ddd5-mgzwt\" (UID: \"5d7f5a1b-6338-4642-85aa-d7d3307265bd\") " pod="kuadrant-system/authorino-56c6c9ddd5-mgzwt"
Nov 02 12:30:56.262749 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:30:56.262720    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgss4\" (UniqueName: \"kubernetes.io/projected/ae914cec-8158-4c77-9870-e0fa6573c7b6-kube-api-access-bgss4\") pod \"limitador-limitador-7c9b87d574-l9pbf\" (UID: \"ae914cec-8158-4c77-9870-e0fa6573c7b6\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"
Nov 02 12:30:56.268511 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:30:56.268486    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-mgzwt"
Nov 02 12:30:56.277020 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:30:56.276997    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"
Nov 02 12:30:56.582246 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:30:56.582216    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"]
Nov 02 12:30:56.592039 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:30:56.591853    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-mgzwt"]
Nov 02 12:30:56.593109 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:30:56.593086    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5d7f5a1b_6338_4642_85aa_d7d3307265bd.slice/crio-d60579d33a9845daf0a891effeb75d449ade93cb47de652858ca59ec6df3a381 WatchSource:0}: Error finding container d60579d33a9845daf0a891effeb75d449ade93cb47de652858ca59ec6df3a381: Status 404 returned error can't find the container with id d60579d33a9845daf0a891effeb75d449ade93cb47de652858ca59ec6df3a381
Nov 02 12:30:56.864808 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:30:56.864721    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-mgzwt" event={"ID":"5d7f5a1b-6338-4642-85aa-d7d3307265bd","Type":"ContainerStarted","Data":"d60579d33a9845daf0a891effeb75d449ade93cb47de652858ca59ec6df3a381"}
Nov 02 12:30:56.866303 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:30:56.866270    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-9c9fcfcf5-9pxlh" event={"ID":"4aea6613-d504-4ab3-80cd-a1c4010709a7","Type":"ContainerStarted","Data":"c1a65a029e75e0ab612c5e4f7db39e6c01432e06e4a6e758a7f71c70cc51fdfc"}
Nov 02 12:30:56.867455 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:30:56.867432    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-l9pbf" event={"ID":"ae914cec-8158-4c77-9870-e0fa6573c7b6","Type":"ContainerStarted","Data":"4f5dac5c881feca0597c9ea6bdd28693dbbcbb395e854f40f4cabedc650e6aa7"}
Nov 02 12:30:56.908463 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:30:56.908421    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-9c9fcfcf5-9pxlh" podStartSLOduration=1.4108555680000001 podStartE2EDuration="3.908408873s" totalImagesPullingTime="2.497553305s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:30:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:30:56.907114041 +0000 UTC m=+253.162513129" watchObservedRunningTime="2026-11-02 12:30:56.908408873 +0000 UTC m=+253.163807997"
Nov 02 12:30:59.889628 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:30:59.889544    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-l9pbf" event={"ID":"ae914cec-8158-4c77-9870-e0fa6573c7b6","Type":"ContainerStarted","Data":"f70187ff8995c50778ca4ab167f94e120cde93d3e5e5935f778bf7023b507049"}
Nov 02 12:30:59.889628 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:30:59.889601    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"
Nov 02 12:30:59.889628 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:30:59.889628    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-mgzwt" event={"ID":"5d7f5a1b-6338-4642-85aa-d7d3307265bd","Type":"ContainerStarted","Data":"a14342743a5efbd969419d620b905c82abdb994956c25f6944ec08fe8d860dbc"}
Nov 02 12:30:59.929366 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:30:59.929316    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-7c9b87d574-l9pbf" podStartSLOduration=1.977345771 podStartE2EDuration="4.92930322s" totalImagesPullingTime="2.951957449s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:30:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:30:59.928292176 +0000 UTC m=+256.183691272" watchObservedRunningTime="2026-11-02 12:30:59.92930322 +0000 UTC m=+256.184702315"
Nov 02 12:30:59.969206 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:30:59.969148    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-56c6c9ddd5-mgzwt" podStartSLOduration=2.025898225 podStartE2EDuration="4.969134957s" totalImagesPullingTime="2.943236732s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:30:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:30:59.968012322 +0000 UTC m=+256.223411418" watchObservedRunningTime="2026-11-02 12:30:59.969134957 +0000 UTC m=+256.224534052"
Nov 02 12:31:10.891307 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:31:10.891271    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"
Nov 02 12:31:15.084400 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:31:15.084360    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqcmj" containerName="extractor"
Nov 02 12:31:45.083538 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:31:45.083494    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqcmj" containerName="extractor"
Nov 02 12:31:59.347183 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:31:59.347107    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lwbxm"]
Nov 02 12:31:59.350262 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:31:59.350245    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lwbxm"
Nov 02 12:31:59.385758 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:31:59.385729    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lwbxm"]
Nov 02 12:31:59.536613 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:31:59.536578    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/10370825-cfb5-4493-83bd-a883abf2d789-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lwbxm\" (UID: \"10370825-cfb5-4493-83bd-a883abf2d789\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lwbxm"
Nov 02 12:31:59.536822 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:31:59.536693    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/10370825-cfb5-4493-83bd-a883abf2d789-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lwbxm\" (UID: \"10370825-cfb5-4493-83bd-a883abf2d789\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lwbxm"
Nov 02 12:31:59.536822 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:31:59.536731    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkvpr\" (UniqueName: \"kubernetes.io/projected/10370825-cfb5-4493-83bd-a883abf2d789-kube-api-access-kkvpr\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lwbxm\" (UID: \"10370825-cfb5-4493-83bd-a883abf2d789\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lwbxm"
Nov 02 12:31:59.536822 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:31:59.536805    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/10370825-cfb5-4493-83bd-a883abf2d789-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lwbxm\" (UID: \"10370825-cfb5-4493-83bd-a883abf2d789\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lwbxm"
Nov 02 12:31:59.536927 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:31:59.536832    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/10370825-cfb5-4493-83bd-a883abf2d789-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lwbxm\" (UID: \"10370825-cfb5-4493-83bd-a883abf2d789\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lwbxm"
Nov 02 12:31:59.536927 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:31:59.536857    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10370825-cfb5-4493-83bd-a883abf2d789-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lwbxm\" (UID: \"10370825-cfb5-4493-83bd-a883abf2d789\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lwbxm"
Nov 02 12:31:59.536991 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:31:59.536960    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/10370825-cfb5-4493-83bd-a883abf2d789-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lwbxm\" (UID: \"10370825-cfb5-4493-83bd-a883abf2d789\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lwbxm"
Nov 02 12:31:59.637940 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:31:59.637852    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/10370825-cfb5-4493-83bd-a883abf2d789-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lwbxm\" (UID: \"10370825-cfb5-4493-83bd-a883abf2d789\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lwbxm"
Nov 02 12:31:59.637940 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:31:59.637891    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/10370825-cfb5-4493-83bd-a883abf2d789-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lwbxm\" (UID: \"10370825-cfb5-4493-83bd-a883abf2d789\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lwbxm"
Nov 02 12:31:59.637940 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:31:59.637915    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10370825-cfb5-4493-83bd-a883abf2d789-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lwbxm\" (UID: \"10370825-cfb5-4493-83bd-a883abf2d789\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lwbxm"
Nov 02 12:31:59.638231 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:31:59.637945    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/10370825-cfb5-4493-83bd-a883abf2d789-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lwbxm\" (UID: \"10370825-cfb5-4493-83bd-a883abf2d789\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lwbxm"
Nov 02 12:31:59.638231 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:31:59.637973    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/10370825-cfb5-4493-83bd-a883abf2d789-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lwbxm\" (UID: \"10370825-cfb5-4493-83bd-a883abf2d789\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lwbxm"
Nov 02 12:31:59.638231 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:31:59.638000    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/10370825-cfb5-4493-83bd-a883abf2d789-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lwbxm\" (UID: \"10370825-cfb5-4493-83bd-a883abf2d789\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lwbxm"
Nov 02 12:31:59.638231 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:31:59.638026    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkvpr\" (UniqueName: \"kubernetes.io/projected/10370825-cfb5-4493-83bd-a883abf2d789-kube-api-access-kkvpr\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lwbxm\" (UID: \"10370825-cfb5-4493-83bd-a883abf2d789\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lwbxm"
Nov 02 12:31:59.638759 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:31:59.638731    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/10370825-cfb5-4493-83bd-a883abf2d789-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lwbxm\" (UID: \"10370825-cfb5-4493-83bd-a883abf2d789\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lwbxm"
Nov 02 12:31:59.640260 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:31:59.640234    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/10370825-cfb5-4493-83bd-a883abf2d789-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lwbxm\" (UID: \"10370825-cfb5-4493-83bd-a883abf2d789\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lwbxm"
Nov 02 12:31:59.640438 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:31:59.640417    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/10370825-cfb5-4493-83bd-a883abf2d789-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lwbxm\" (UID: \"10370825-cfb5-4493-83bd-a883abf2d789\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lwbxm"
Nov 02 12:31:59.640521 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:31:59.640499    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/10370825-cfb5-4493-83bd-a883abf2d789-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lwbxm\" (UID: \"10370825-cfb5-4493-83bd-a883abf2d789\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lwbxm"
Nov 02 12:31:59.640578 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:31:59.640524    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/10370825-cfb5-4493-83bd-a883abf2d789-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lwbxm\" (UID: \"10370825-cfb5-4493-83bd-a883abf2d789\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lwbxm"
Nov 02 12:31:59.667748 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:31:59.667711    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10370825-cfb5-4493-83bd-a883abf2d789-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lwbxm\" (UID: \"10370825-cfb5-4493-83bd-a883abf2d789\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lwbxm"
Nov 02 12:31:59.667915 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:31:59.667897    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkvpr\" (UniqueName: \"kubernetes.io/projected/10370825-cfb5-4493-83bd-a883abf2d789-kube-api-access-kkvpr\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lwbxm\" (UID: \"10370825-cfb5-4493-83bd-a883abf2d789\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lwbxm"
Nov 02 12:31:59.669666 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:31:59.669632    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lwbxm"
Nov 02 12:31:59.982531 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:31:59.982501    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lwbxm"]
Nov 02 12:31:59.982701 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:31:59.982626    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10370825_cfb5_4493_83bd_a883abf2d789.slice/crio-765561ce3bbb63effe943a648fd27bc751d73600fd64d5562459631f50f940e4 WatchSource:0}: Error finding container 765561ce3bbb63effe943a648fd27bc751d73600fd64d5562459631f50f940e4: Status 404 returned error can't find the container with id 765561ce3bbb63effe943a648fd27bc751d73600fd64d5562459631f50f940e4
Nov 02 12:31:59.984805 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:31:59.984772    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:31:59.984876 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:31:59.984839    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:32:00.109302 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:32:00.109269    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lwbxm" event={"ID":"10370825-cfb5-4493-83bd-a883abf2d789","Type":"ContainerStarted","Data":"a9345030d784c103f676bd0f43a359131e40e62184e8ec939a1dcfd6ed400af3"}
Nov 02 12:32:00.109499 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:32:00.109325    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lwbxm" event={"ID":"10370825-cfb5-4493-83bd-a883abf2d789","Type":"ContainerStarted","Data":"765561ce3bbb63effe943a648fd27bc751d73600fd64d5562459631f50f940e4"}
Nov 02 12:32:00.109499 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:32:00.109398    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lwbxm"
Nov 02 12:32:01.115195 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:32:01.115168    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lwbxm"
Nov 02 12:32:01.169553 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:32:01.169501    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lwbxm" podStartSLOduration=2.169486949 podStartE2EDuration="2.169486949s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:31:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:32:00.15986921 +0000 UTC m=+316.415268306" watchObservedRunningTime="2026-11-02 12:32:01.169486949 +0000 UTC m=+317.424886037"
Nov 02 12:32:01.204595 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:32:01.204559    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-l4gld"]
Nov 02 12:32:01.204818 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:32:01.204797    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l4gld" podUID="da8a5263-2e77-47a8-9a75-fd8ee0e5b80d" containerName="discovery" containerID="cri-o://70dd1bdc7ae3d0cfe54a4205ffe3fbc2eabcef134df2e38b82aa3ccfa17ef456" gracePeriod=30
Nov 02 12:32:01.454579 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:32:01.454554    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l4gld"
Nov 02 12:32:01.558438 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:32:01.558405    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da8a5263-2e77-47a8-9a75-fd8ee0e5b80d-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/da8a5263-2e77-47a8-9a75-fd8ee0e5b80d-istio-kubeconfig\") pod \"da8a5263-2e77-47a8-9a75-fd8ee0e5b80d\" (UID: \"da8a5263-2e77-47a8-9a75-fd8ee0e5b80d\") "
Nov 02 12:32:01.558617 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:32:01.558471    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da8a5263-2e77-47a8-9a75-fd8ee0e5b80d-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/da8a5263-2e77-47a8-9a75-fd8ee0e5b80d-istio-csr-ca-configmap\") pod \"da8a5263-2e77-47a8-9a75-fd8ee0e5b80d\" (UID: \"da8a5263-2e77-47a8-9a75-fd8ee0e5b80d\") "
Nov 02 12:32:01.558617 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:32:01.558498    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da8a5263-2e77-47a8-9a75-fd8ee0e5b80d-kube-api-access-vlzn4\" (UniqueName: \"kubernetes.io/projected/da8a5263-2e77-47a8-9a75-fd8ee0e5b80d-kube-api-access-vlzn4\") pod \"da8a5263-2e77-47a8-9a75-fd8ee0e5b80d\" (UID: \"da8a5263-2e77-47a8-9a75-fd8ee0e5b80d\") "
Nov 02 12:32:01.558617 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:32:01.558531    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da8a5263-2e77-47a8-9a75-fd8ee0e5b80d-istio-token\" (UniqueName: \"kubernetes.io/projected/da8a5263-2e77-47a8-9a75-fd8ee0e5b80d-istio-token\") pod \"da8a5263-2e77-47a8-9a75-fd8ee0e5b80d\" (UID: \"da8a5263-2e77-47a8-9a75-fd8ee0e5b80d\") "
Nov 02 12:32:01.558617 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:32:01.558560    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da8a5263-2e77-47a8-9a75-fd8ee0e5b80d-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/da8a5263-2e77-47a8-9a75-fd8ee0e5b80d-istio-csr-dns-cert\") pod \"da8a5263-2e77-47a8-9a75-fd8ee0e5b80d\" (UID: \"da8a5263-2e77-47a8-9a75-fd8ee0e5b80d\") "
Nov 02 12:32:01.558884 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:32:01.558669    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da8a5263-2e77-47a8-9a75-fd8ee0e5b80d-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/da8a5263-2e77-47a8-9a75-fd8ee0e5b80d-local-certs\") pod \"da8a5263-2e77-47a8-9a75-fd8ee0e5b80d\" (UID: \"da8a5263-2e77-47a8-9a75-fd8ee0e5b80d\") "
Nov 02 12:32:01.558884 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:32:01.558752    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da8a5263-2e77-47a8-9a75-fd8ee0e5b80d-cacerts\" (UniqueName: \"kubernetes.io/secret/da8a5263-2e77-47a8-9a75-fd8ee0e5b80d-cacerts\") pod \"da8a5263-2e77-47a8-9a75-fd8ee0e5b80d\" (UID: \"da8a5263-2e77-47a8-9a75-fd8ee0e5b80d\") "
Nov 02 12:32:01.559035 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:32:01.559006    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da8a5263-2e77-47a8-9a75-fd8ee0e5b80d-istio-csr-ca-configmap" pod "da8a5263-2e77-47a8-9a75-fd8ee0e5b80d" (UID: "da8a5263-2e77-47a8-9a75-fd8ee0e5b80d"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:32:01.561061 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:32:01.561018    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da8a5263-2e77-47a8-9a75-fd8ee0e5b80d-istio-csr-dns-cert" pod "da8a5263-2e77-47a8-9a75-fd8ee0e5b80d" (UID: "da8a5263-2e77-47a8-9a75-fd8ee0e5b80d"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:32:01.561189 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:32:01.561140    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da8a5263-2e77-47a8-9a75-fd8ee0e5b80d-istio-token" pod "da8a5263-2e77-47a8-9a75-fd8ee0e5b80d" (UID: "da8a5263-2e77-47a8-9a75-fd8ee0e5b80d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:32:01.561189 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:32:01.561166    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da8a5263-2e77-47a8-9a75-fd8ee0e5b80d-istio-kubeconfig" pod "da8a5263-2e77-47a8-9a75-fd8ee0e5b80d" (UID: "da8a5263-2e77-47a8-9a75-fd8ee0e5b80d"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:32:01.561288 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:32:01.561204    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da8a5263-2e77-47a8-9a75-fd8ee0e5b80d-kube-api-access-vlzn4" pod "da8a5263-2e77-47a8-9a75-fd8ee0e5b80d" (UID: "da8a5263-2e77-47a8-9a75-fd8ee0e5b80d"). InnerVolumeSpecName "kube-api-access-vlzn4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:32:01.561288 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:32:01.561254    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da8a5263-2e77-47a8-9a75-fd8ee0e5b80d-cacerts" pod "da8a5263-2e77-47a8-9a75-fd8ee0e5b80d" (UID: "da8a5263-2e77-47a8-9a75-fd8ee0e5b80d"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:32:01.561288 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:32:01.561263    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da8a5263-2e77-47a8-9a75-fd8ee0e5b80d-local-certs" pod "da8a5263-2e77-47a8-9a75-fd8ee0e5b80d" (UID: "da8a5263-2e77-47a8-9a75-fd8ee0e5b80d"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:32:01.659993 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:32:01.659920    4559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da8a5263-2e77-47a8-9a75-fd8ee0e5b80d-istio-token\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:32:01.659993 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:32:01.659947    4559 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/da8a5263-2e77-47a8-9a75-fd8ee0e5b80d-istio-csr-dns-cert\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:32:01.659993 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:32:01.659956    4559 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/da8a5263-2e77-47a8-9a75-fd8ee0e5b80d-local-certs\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:32:01.659993 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:32:01.659966    4559 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/da8a5263-2e77-47a8-9a75-fd8ee0e5b80d-cacerts\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:32:01.659993 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:32:01.659973    4559 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/da8a5263-2e77-47a8-9a75-fd8ee0e5b80d-istio-kubeconfig\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:32:01.659993 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:32:01.659983    4559 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/da8a5263-2e77-47a8-9a75-fd8ee0e5b80d-istio-csr-ca-configmap\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:32:01.659993 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:32:01.659992    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vlzn4\" (UniqueName: \"kubernetes.io/projected/da8a5263-2e77-47a8-9a75-fd8ee0e5b80d-kube-api-access-vlzn4\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:32:02.117822 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:32:02.117780    4559 generic.go:356] "Generic (PLEG): container finished" podID="da8a5263-2e77-47a8-9a75-fd8ee0e5b80d" containerID="70dd1bdc7ae3d0cfe54a4205ffe3fbc2eabcef134df2e38b82aa3ccfa17ef456" exitCode=0
Nov 02 12:32:02.118262 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:32:02.117830    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l4gld"
Nov 02 12:32:02.118262 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:32:02.117865    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l4gld" event={"ID":"da8a5263-2e77-47a8-9a75-fd8ee0e5b80d","Type":"ContainerDied","Data":"70dd1bdc7ae3d0cfe54a4205ffe3fbc2eabcef134df2e38b82aa3ccfa17ef456"}
Nov 02 12:32:02.118262 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:32:02.117914    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l4gld" event={"ID":"da8a5263-2e77-47a8-9a75-fd8ee0e5b80d","Type":"ContainerDied","Data":"a6f5e3dc1d7e18d0c603f59d8f4a0d08dc049a5a6c046b5e5801ea50e9588375"}
Nov 02 12:32:02.118262 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:32:02.117934    4559 scope.go:122] "RemoveContainer" containerID="70dd1bdc7ae3d0cfe54a4205ffe3fbc2eabcef134df2e38b82aa3ccfa17ef456"
Nov 02 12:32:02.125889 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:32:02.125873    4559 scope.go:122] "RemoveContainer" containerID="70dd1bdc7ae3d0cfe54a4205ffe3fbc2eabcef134df2e38b82aa3ccfa17ef456"
Nov 02 12:32:02.126144 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:32:02.126128    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70dd1bdc7ae3d0cfe54a4205ffe3fbc2eabcef134df2e38b82aa3ccfa17ef456\": container with ID starting with 70dd1bdc7ae3d0cfe54a4205ffe3fbc2eabcef134df2e38b82aa3ccfa17ef456 not found: ID does not exist" containerID="70dd1bdc7ae3d0cfe54a4205ffe3fbc2eabcef134df2e38b82aa3ccfa17ef456"
Nov 02 12:32:02.126192 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:32:02.126154    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70dd1bdc7ae3d0cfe54a4205ffe3fbc2eabcef134df2e38b82aa3ccfa17ef456"} err="failed to get container status \"70dd1bdc7ae3d0cfe54a4205ffe3fbc2eabcef134df2e38b82aa3ccfa17ef456\": rpc error: code = NotFound desc = could not find container \"70dd1bdc7ae3d0cfe54a4205ffe3fbc2eabcef134df2e38b82aa3ccfa17ef456\": container with ID starting with 70dd1bdc7ae3d0cfe54a4205ffe3fbc2eabcef134df2e38b82aa3ccfa17ef456 not found: ID does not exist"
Nov 02 12:32:02.183962 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:32:02.183928    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-l4gld"]
Nov 02 12:32:02.186824 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:32:02.186795    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-l4gld"]
Nov 02 12:32:03.891301 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:32:03.891263    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da8a5263-2e77-47a8-9a75-fd8ee0e5b80d" path="/var/lib/kubelet/pods/da8a5263-2e77-47a8-9a75-fd8ee0e5b80d/volumes"
Nov 02 12:32:15.084481 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:32:15.084444    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqcmj" containerName="extractor"
Nov 02 12:32:23.075533 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:32:23.075498    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-f54ddc9f4-p7w9d"]
Nov 02 12:32:23.076021 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:32:23.075853    4559 cpu_manager.go:414] "containerMap: removing container" podUID="da8a5263-2e77-47a8-9a75-fd8ee0e5b80d" containerName="discovery"
Nov 02 12:32:23.076021 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:32:23.075864    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da8a5263-2e77-47a8-9a75-fd8ee0e5b80d" containerName="discovery"
Nov 02 12:32:23.076021 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:32:23.075916    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="da8a5263-2e77-47a8-9a75-fd8ee0e5b80d" containerName="discovery"
Nov 02 12:32:23.080176 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:32:23.080155    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-f54ddc9f4-p7w9d"
Nov 02 12:32:23.098419 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:32:23.098391    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 12:32:23.100322 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:32:23.100298    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-f54ddc9f4-p7w9d"]
Nov 02 12:32:23.111462 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:32:23.111437    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 12:32:23.114082 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:32:23.114063    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Nov 02 12:32:23.114188 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:32:23.114061    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Nov 02 12:32:23.114188 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:32:23.114072    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-sr5s9\""
Nov 02 12:32:23.122439 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:32:23.122416    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eafc7c38-28ba-4c46-9942-20f960bb967f-openshift-service-ca\") pod \"maas-api-f54ddc9f4-p7w9d\" (UID: \"eafc7c38-28ba-4c46-9942-20f960bb967f\") " pod="odh-ai-gateway-infra/maas-api-f54ddc9f4-p7w9d"
Nov 02 12:32:23.122513 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:32:23.122456    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eafc7c38-28ba-4c46-9942-20f960bb967f-maas-api-tls\") pod \"maas-api-f54ddc9f4-p7w9d\" (UID: \"eafc7c38-28ba-4c46-9942-20f960bb967f\") " pod="odh-ai-gateway-infra/maas-api-f54ddc9f4-p7w9d"
Nov 02 12:32:23.122607 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:32:23.122591    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eafc7c38-28ba-4c46-9942-20f960bb967f-metrics-certs\") pod \"maas-api-f54ddc9f4-p7w9d\" (UID: \"eafc7c38-28ba-4c46-9942-20f960bb967f\") " pod="odh-ai-gateway-infra/maas-api-f54ddc9f4-p7w9d"
Nov 02 12:32:23.122705 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:32:23.122690    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmp6b\" (UniqueName: \"kubernetes.io/projected/eafc7c38-28ba-4c46-9942-20f960bb967f-kube-api-access-vmp6b\") pod \"maas-api-f54ddc9f4-p7w9d\" (UID: \"eafc7c38-28ba-4c46-9942-20f960bb967f\") " pod="odh-ai-gateway-infra/maas-api-f54ddc9f4-p7w9d"
Nov 02 12:32:23.223613 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:32:23.223579    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmp6b\" (UniqueName: \"kubernetes.io/projected/eafc7c38-28ba-4c46-9942-20f960bb967f-kube-api-access-vmp6b\") pod \"maas-api-f54ddc9f4-p7w9d\" (UID: \"eafc7c38-28ba-4c46-9942-20f960bb967f\") " pod="odh-ai-gateway-infra/maas-api-f54ddc9f4-p7w9d"
Nov 02 12:32:23.223795 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:32:23.223624    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eafc7c38-28ba-4c46-9942-20f960bb967f-openshift-service-ca\") pod \"maas-api-f54ddc9f4-p7w9d\" (UID: \"eafc7c38-28ba-4c46-9942-20f960bb967f\") " pod="odh-ai-gateway-infra/maas-api-f54ddc9f4-p7w9d"
Nov 02 12:32:23.223795 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:32:23.223671    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eafc7c38-28ba-4c46-9942-20f960bb967f-maas-api-tls\") pod \"maas-api-f54ddc9f4-p7w9d\" (UID: \"eafc7c38-28ba-4c46-9942-20f960bb967f\") " pod="odh-ai-gateway-infra/maas-api-f54ddc9f4-p7w9d"
Nov 02 12:32:23.223795 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:32:23.223707    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eafc7c38-28ba-4c46-9942-20f960bb967f-metrics-certs\") pod \"maas-api-f54ddc9f4-p7w9d\" (UID: \"eafc7c38-28ba-4c46-9942-20f960bb967f\") " pod="odh-ai-gateway-infra/maas-api-f54ddc9f4-p7w9d"
Nov 02 12:32:23.224424 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:32:23.224403    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eafc7c38-28ba-4c46-9942-20f960bb967f-openshift-service-ca\") pod \"maas-api-f54ddc9f4-p7w9d\" (UID: \"eafc7c38-28ba-4c46-9942-20f960bb967f\") " pod="odh-ai-gateway-infra/maas-api-f54ddc9f4-p7w9d"
Nov 02 12:32:23.226153 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:32:23.226128    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eafc7c38-28ba-4c46-9942-20f960bb967f-maas-api-tls\") pod \"maas-api-f54ddc9f4-p7w9d\" (UID: \"eafc7c38-28ba-4c46-9942-20f960bb967f\") " pod="odh-ai-gateway-infra/maas-api-f54ddc9f4-p7w9d"
Nov 02 12:32:23.226261 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:32:23.226204    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eafc7c38-28ba-4c46-9942-20f960bb967f-metrics-certs\") pod \"maas-api-f54ddc9f4-p7w9d\" (UID: \"eafc7c38-28ba-4c46-9942-20f960bb967f\") " pod="odh-ai-gateway-infra/maas-api-f54ddc9f4-p7w9d"
Nov 02 12:32:23.248043 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:32:23.248013    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmp6b\" (UniqueName: \"kubernetes.io/projected/eafc7c38-28ba-4c46-9942-20f960bb967f-kube-api-access-vmp6b\") pod \"maas-api-f54ddc9f4-p7w9d\" (UID: \"eafc7c38-28ba-4c46-9942-20f960bb967f\") " pod="odh-ai-gateway-infra/maas-api-f54ddc9f4-p7w9d"
Nov 02 12:32:23.391975 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:32:23.391886    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-f54ddc9f4-p7w9d"
Nov 02 12:32:23.690485 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:32:23.690458    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-f54ddc9f4-p7w9d"]
Nov 02 12:32:23.691966 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:32:23.691939    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeafc7c38_28ba_4c46_9942_20f960bb967f.slice/crio-197d63ffe487d02b2d8b9414986f1524b967b451938fa076e2bea647072afcbf WatchSource:0}: Error finding container 197d63ffe487d02b2d8b9414986f1524b967b451938fa076e2bea647072afcbf: Status 404 returned error can't find the container with id 197d63ffe487d02b2d8b9414986f1524b967b451938fa076e2bea647072afcbf
Nov 02 12:32:23.693264 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:32:23.693244    4559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 12:32:24.204840 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:32:24.204806    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-f54ddc9f4-p7w9d" event={"ID":"eafc7c38-28ba-4c46-9942-20f960bb967f","Type":"ContainerStarted","Data":"197d63ffe487d02b2d8b9414986f1524b967b451938fa076e2bea647072afcbf"}
Nov 02 12:32:24.485878 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:32:24.485794    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"]
Nov 02 12:32:26.214608 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:32:26.214570    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-f54ddc9f4-p7w9d" event={"ID":"eafc7c38-28ba-4c46-9942-20f960bb967f","Type":"ContainerStarted","Data":"11baa5880d4f6c4916be55c534c0ae66ef79c5b41f4e9838a984e00e62eeda95"}
Nov 02 12:32:26.215075 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:32:26.214668    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-f54ddc9f4-p7w9d"
Nov 02 12:32:26.261099 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:32:26.261039    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-f54ddc9f4-p7w9d" podStartSLOduration=0.818936245 podStartE2EDuration="3.261024077s" totalImagesPullingTime="2.442087832s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:32:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:32:26.259654634 +0000 UTC m=+342.515053723" watchObservedRunningTime="2026-11-02 12:32:26.261024077 +0000 UTC m=+342.516423168"
Nov 02 12:32:32.228238 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:32:32.228209    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-f54ddc9f4-p7w9d"
Nov 02 12:32:33.348946 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:32:33.348914    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-7j59j"]
Nov 02 12:32:33.352261 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:32:33.352243    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-7j59j"
Nov 02 12:32:33.372144 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:32:33.372110    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-7j59j"]
Nov 02 12:32:33.409521 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:32:33.409493    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/63555d6f-3439-49ad-a1ce-b15286e25a0c-tls-cert\") pod \"authorino-6889cbd8c4-7j59j\" (UID: \"63555d6f-3439-49ad-a1ce-b15286e25a0c\") " pod="kuadrant-system/authorino-6889cbd8c4-7j59j"
Nov 02 12:32:33.409693 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:32:33.409556    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjh92\" (UniqueName: \"kubernetes.io/projected/63555d6f-3439-49ad-a1ce-b15286e25a0c-kube-api-access-bjh92\") pod \"authorino-6889cbd8c4-7j59j\" (UID: \"63555d6f-3439-49ad-a1ce-b15286e25a0c\") " pod="kuadrant-system/authorino-6889cbd8c4-7j59j"
Nov 02 12:32:33.411346 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:32:33.411324    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-7j59j"]
Nov 02 12:32:33.411490 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:32:33.411472    4559 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-bjh92 tls-cert], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6889cbd8c4-7j59j" podUID="63555d6f-3439-49ad-a1ce-b15286e25a0c"
Nov 02 12:32:33.435529 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:32:33.435509    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Nov 02 12:32:33.438444 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:32:33.438425    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-57f544d9f9-84fxt"]
Nov 02 12:32:33.441457 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:32:33.441443    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f544d9f9-84fxt"
Nov 02 12:32:33.464172 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:32:33.464145    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f544d9f9-84fxt"]
Nov 02 12:32:33.510518 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:32:33.510494    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b8f8b8cd-64a0-457d-95f3-c8f2940cd9a5-tls-cert\") pod \"authorino-57f544d9f9-84fxt\" (UID: \"b8f8b8cd-64a0-457d-95f3-c8f2940cd9a5\") " pod="kuadrant-system/authorino-57f544d9f9-84fxt"
Nov 02 12:32:33.510677 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:32:33.510532    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwfzd\" (UniqueName: \"kubernetes.io/projected/b8f8b8cd-64a0-457d-95f3-c8f2940cd9a5-kube-api-access-gwfzd\") pod \"authorino-57f544d9f9-84fxt\" (UID: \"b8f8b8cd-64a0-457d-95f3-c8f2940cd9a5\") " pod="kuadrant-system/authorino-57f544d9f9-84fxt"
Nov 02 12:32:33.510677 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:32:33.510562    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/63555d6f-3439-49ad-a1ce-b15286e25a0c-tls-cert\") pod \"authorino-6889cbd8c4-7j59j\" (UID: \"63555d6f-3439-49ad-a1ce-b15286e25a0c\") " pod="kuadrant-system/authorino-6889cbd8c4-7j59j"
Nov 02 12:32:33.510843 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:32:33.510823    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjh92\" (UniqueName: \"kubernetes.io/projected/63555d6f-3439-49ad-a1ce-b15286e25a0c-kube-api-access-bjh92\") pod \"authorino-6889cbd8c4-7j59j\" (UID: \"63555d6f-3439-49ad-a1ce-b15286e25a0c\") " pod="kuadrant-system/authorino-6889cbd8c4-7j59j"
Nov 02 12:32:33.512888 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:32:33.512865    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/63555d6f-3439-49ad-a1ce-b15286e25a0c-tls-cert\") pod \"authorino-6889cbd8c4-7j59j\" (UID: \"63555d6f-3439-49ad-a1ce-b15286e25a0c\") " pod="kuadrant-system/authorino-6889cbd8c4-7j59j"
Nov 02 12:32:33.534036 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:32:33.534016    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjh92\" (UniqueName: \"kubernetes.io/projected/63555d6f-3439-49ad-a1ce-b15286e25a0c-kube-api-access-bjh92\") pod \"authorino-6889cbd8c4-7j59j\" (UID: \"63555d6f-3439-49ad-a1ce-b15286e25a0c\") " pod="kuadrant-system/authorino-6889cbd8c4-7j59j"
Nov 02 12:32:33.612239 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:32:33.612169    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b8f8b8cd-64a0-457d-95f3-c8f2940cd9a5-tls-cert\") pod \"authorino-57f544d9f9-84fxt\" (UID: \"b8f8b8cd-64a0-457d-95f3-c8f2940cd9a5\") " pod="kuadrant-system/authorino-57f544d9f9-84fxt"
Nov 02 12:32:33.612239 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:32:33.612215    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwfzd\" (UniqueName: \"kubernetes.io/projected/b8f8b8cd-64a0-457d-95f3-c8f2940cd9a5-kube-api-access-gwfzd\") pod \"authorino-57f544d9f9-84fxt\" (UID: \"b8f8b8cd-64a0-457d-95f3-c8f2940cd9a5\") " pod="kuadrant-system/authorino-57f544d9f9-84fxt"
Nov 02 12:32:33.614504 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:32:33.614477    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b8f8b8cd-64a0-457d-95f3-c8f2940cd9a5-tls-cert\") pod \"authorino-57f544d9f9-84fxt\" (UID: \"b8f8b8cd-64a0-457d-95f3-c8f2940cd9a5\") " pod="kuadrant-system/authorino-57f544d9f9-84fxt"
Nov 02 12:32:33.636137 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:32:33.636111    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwfzd\" (UniqueName: \"kubernetes.io/projected/b8f8b8cd-64a0-457d-95f3-c8f2940cd9a5-kube-api-access-gwfzd\") pod \"authorino-57f544d9f9-84fxt\" (UID: \"b8f8b8cd-64a0-457d-95f3-c8f2940cd9a5\") " pod="kuadrant-system/authorino-57f544d9f9-84fxt"
Nov 02 12:32:33.751621 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:32:33.751592    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f544d9f9-84fxt"
Nov 02 12:32:34.046464 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:32:34.046434    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f544d9f9-84fxt"]
Nov 02 12:32:34.047998 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:32:34.047973    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb8f8b8cd_64a0_457d_95f3_c8f2940cd9a5.slice/crio-591c7040edc86764592d3ea4451f5c4af5471fe8c6e5628bc2523932dab6a95f WatchSource:0}: Error finding container 591c7040edc86764592d3ea4451f5c4af5471fe8c6e5628bc2523932dab6a95f: Status 404 returned error can't find the container with id 591c7040edc86764592d3ea4451f5c4af5471fe8c6e5628bc2523932dab6a95f
Nov 02 12:32:34.249490 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:32:34.249450    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f544d9f9-84fxt" event={"ID":"b8f8b8cd-64a0-457d-95f3-c8f2940cd9a5","Type":"ContainerStarted","Data":"591c7040edc86764592d3ea4451f5c4af5471fe8c6e5628bc2523932dab6a95f"}
Nov 02 12:32:34.249490 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:32:34.249460    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-7j59j"
Nov 02 12:32:34.259787 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:32:34.259768    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-7j59j"
Nov 02 12:32:34.318886 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:32:34.318820    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/63555d6f-3439-49ad-a1ce-b15286e25a0c-tls-cert\" (UniqueName: \"kubernetes.io/secret/63555d6f-3439-49ad-a1ce-b15286e25a0c-tls-cert\") pod \"63555d6f-3439-49ad-a1ce-b15286e25a0c\" (UID: \"63555d6f-3439-49ad-a1ce-b15286e25a0c\") "
Nov 02 12:32:34.318886 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:32:34.318867    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63555d6f-3439-49ad-a1ce-b15286e25a0c-kube-api-access-bjh92\" (UniqueName: \"kubernetes.io/projected/63555d6f-3439-49ad-a1ce-b15286e25a0c-kube-api-access-bjh92\") pod \"63555d6f-3439-49ad-a1ce-b15286e25a0c\" (UID: \"63555d6f-3439-49ad-a1ce-b15286e25a0c\") "
Nov 02 12:32:34.320990 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:32:34.320964    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63555d6f-3439-49ad-a1ce-b15286e25a0c-kube-api-access-bjh92" pod "63555d6f-3439-49ad-a1ce-b15286e25a0c" (UID: "63555d6f-3439-49ad-a1ce-b15286e25a0c"). InnerVolumeSpecName "kube-api-access-bjh92". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:32:34.321048 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:32:34.320984    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63555d6f-3439-49ad-a1ce-b15286e25a0c-tls-cert" pod "63555d6f-3439-49ad-a1ce-b15286e25a0c" (UID: "63555d6f-3439-49ad-a1ce-b15286e25a0c"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:32:34.419603 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:32:34.419579    4559 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/63555d6f-3439-49ad-a1ce-b15286e25a0c-tls-cert\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:32:34.419603 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:32:34.419600    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bjh92\" (UniqueName: \"kubernetes.io/projected/63555d6f-3439-49ad-a1ce-b15286e25a0c-kube-api-access-bjh92\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:32:35.256079 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:32:35.256041    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f544d9f9-84fxt" event={"ID":"b8f8b8cd-64a0-457d-95f3-c8f2940cd9a5","Type":"ContainerStarted","Data":"8a5ab7b173f2b39d12ab0c6a30c9d90a95abbab926b6a4203e90b1c014554b6b"}
Nov 02 12:32:35.256247 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:32:35.256159    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-7j59j"
Nov 02 12:32:35.296899 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:32:35.296848    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-57f544d9f9-84fxt" podStartSLOduration=1.7476823179999998 podStartE2EDuration="2.296832754s" totalImagesPullingTime="549.150436ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:32:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:32:35.295421737 +0000 UTC m=+351.550820832" watchObservedRunningTime="2026-11-02 12:32:35.296832754 +0000 UTC m=+351.552231856"
Nov 02 12:32:35.319744 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:32:35.319703    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-mgzwt"]
Nov 02 12:32:35.319928 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:32:35.319905    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-56c6c9ddd5-mgzwt" podUID="5d7f5a1b-6338-4642-85aa-d7d3307265bd" containerName="authorino" containerID="cri-o://a14342743a5efbd969419d620b905c82abdb994956c25f6944ec08fe8d860dbc" gracePeriod=30
Nov 02 12:32:35.363198 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:32:35.363171    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-7j59j"]
Nov 02 12:32:35.369258 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:32:35.369232    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-7j59j"]
Nov 02 12:32:35.573532 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:32:35.573508    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-mgzwt"
Nov 02 12:32:35.632026 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:32:35.631998    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d7f5a1b-6338-4642-85aa-d7d3307265bd-kube-api-access-pmgdb\" (UniqueName: \"kubernetes.io/projected/5d7f5a1b-6338-4642-85aa-d7d3307265bd-kube-api-access-pmgdb\") pod \"5d7f5a1b-6338-4642-85aa-d7d3307265bd\" (UID: \"5d7f5a1b-6338-4642-85aa-d7d3307265bd\") "
Nov 02 12:32:35.634064 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:32:35.634037    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d7f5a1b-6338-4642-85aa-d7d3307265bd-kube-api-access-pmgdb" pod "5d7f5a1b-6338-4642-85aa-d7d3307265bd" (UID: "5d7f5a1b-6338-4642-85aa-d7d3307265bd"). InnerVolumeSpecName "kube-api-access-pmgdb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:32:35.733252 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:32:35.733220    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pmgdb\" (UniqueName: \"kubernetes.io/projected/5d7f5a1b-6338-4642-85aa-d7d3307265bd-kube-api-access-pmgdb\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:32:35.890207 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:32:35.890131    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63555d6f-3439-49ad-a1ce-b15286e25a0c" path="/var/lib/kubelet/pods/63555d6f-3439-49ad-a1ce-b15286e25a0c/volumes"
Nov 02 12:32:36.260822 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:32:36.260790    4559 generic.go:356] "Generic (PLEG): container finished" podID="5d7f5a1b-6338-4642-85aa-d7d3307265bd" containerID="a14342743a5efbd969419d620b905c82abdb994956c25f6944ec08fe8d860dbc" exitCode=0
Nov 02 12:32:36.261023 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:32:36.260832    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-mgzwt"
Nov 02 12:32:36.261023 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:32:36.260834    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-mgzwt" event={"ID":"5d7f5a1b-6338-4642-85aa-d7d3307265bd","Type":"ContainerDied","Data":"a14342743a5efbd969419d620b905c82abdb994956c25f6944ec08fe8d860dbc"}
Nov 02 12:32:36.261023 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:32:36.260881    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-mgzwt" event={"ID":"5d7f5a1b-6338-4642-85aa-d7d3307265bd","Type":"ContainerDied","Data":"d60579d33a9845daf0a891effeb75d449ade93cb47de652858ca59ec6df3a381"}
Nov 02 12:32:36.261023 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:32:36.260903    4559 scope.go:122] "RemoveContainer" containerID="a14342743a5efbd969419d620b905c82abdb994956c25f6944ec08fe8d860dbc"
Nov 02 12:32:36.268729 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:32:36.268714    4559 scope.go:122] "RemoveContainer" containerID="a14342743a5efbd969419d620b905c82abdb994956c25f6944ec08fe8d860dbc"
Nov 02 12:32:36.269001 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:32:36.268983    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a14342743a5efbd969419d620b905c82abdb994956c25f6944ec08fe8d860dbc\": container with ID starting with a14342743a5efbd969419d620b905c82abdb994956c25f6944ec08fe8d860dbc not found: ID does not exist" containerID="a14342743a5efbd969419d620b905c82abdb994956c25f6944ec08fe8d860dbc"
Nov 02 12:32:36.269078 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:32:36.269013    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a14342743a5efbd969419d620b905c82abdb994956c25f6944ec08fe8d860dbc"} err="failed to get container status \"a14342743a5efbd969419d620b905c82abdb994956c25f6944ec08fe8d860dbc\": rpc error: code = NotFound desc = could not find container \"a14342743a5efbd969419d620b905c82abdb994956c25f6944ec08fe8d860dbc\": container with ID starting with a14342743a5efbd969419d620b905c82abdb994956c25f6944ec08fe8d860dbc not found: ID does not exist"
Nov 02 12:32:36.312547 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:32:36.312524    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-mgzwt"]
Nov 02 12:32:36.325046 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:32:36.325019    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-mgzwt"]
Nov 02 12:32:37.890020 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:32:37.889983    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d7f5a1b-6338-4642-85aa-d7d3307265bd" path="/var/lib/kubelet/pods/5d7f5a1b-6338-4642-85aa-d7d3307265bd/volumes"
Nov 02 12:32:42.575100 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:32:42.575061    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-f54ddc9f4-p7w9d"]
Nov 02 12:32:42.575556 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:32:42.575308    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-f54ddc9f4-p7w9d" podUID="eafc7c38-28ba-4c46-9942-20f960bb967f" containerName="maas-api" containerID="cri-o://11baa5880d4f6c4916be55c534c0ae66ef79c5b41f4e9838a984e00e62eeda95" gracePeriod=30
Nov 02 12:32:42.815142 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:32:42.815119    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-f54ddc9f4-p7w9d"
Nov 02 12:32:42.888421 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:32:42.887620    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eafc7c38-28ba-4c46-9942-20f960bb967f-kube-api-access-vmp6b\" (UniqueName: \"kubernetes.io/projected/eafc7c38-28ba-4c46-9942-20f960bb967f-kube-api-access-vmp6b\") pod \"eafc7c38-28ba-4c46-9942-20f960bb967f\" (UID: \"eafc7c38-28ba-4c46-9942-20f960bb967f\") "
Nov 02 12:32:42.888421 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:32:42.887727    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eafc7c38-28ba-4c46-9942-20f960bb967f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eafc7c38-28ba-4c46-9942-20f960bb967f-maas-api-tls\") pod \"eafc7c38-28ba-4c46-9942-20f960bb967f\" (UID: \"eafc7c38-28ba-4c46-9942-20f960bb967f\") "
Nov 02 12:32:42.888421 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:32:42.887777    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eafc7c38-28ba-4c46-9942-20f960bb967f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eafc7c38-28ba-4c46-9942-20f960bb967f-openshift-service-ca\") pod \"eafc7c38-28ba-4c46-9942-20f960bb967f\" (UID: \"eafc7c38-28ba-4c46-9942-20f960bb967f\") "
Nov 02 12:32:42.888421 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:32:42.887893    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eafc7c38-28ba-4c46-9942-20f960bb967f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/eafc7c38-28ba-4c46-9942-20f960bb967f-metrics-certs\") pod \"eafc7c38-28ba-4c46-9942-20f960bb967f\" (UID: \"eafc7c38-28ba-4c46-9942-20f960bb967f\") "
Nov 02 12:32:42.891595 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:32:42.891291    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eafc7c38-28ba-4c46-9942-20f960bb967f-metrics-certs" pod "eafc7c38-28ba-4c46-9942-20f960bb967f" (UID: "eafc7c38-28ba-4c46-9942-20f960bb967f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:32:42.893240 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:32:42.893193    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eafc7c38-28ba-4c46-9942-20f960bb967f-openshift-service-ca" pod "eafc7c38-28ba-4c46-9942-20f960bb967f" (UID: "eafc7c38-28ba-4c46-9942-20f960bb967f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:32:42.893580 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:32:42.893558    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eafc7c38-28ba-4c46-9942-20f960bb967f-kube-api-access-vmp6b" pod "eafc7c38-28ba-4c46-9942-20f960bb967f" (UID: "eafc7c38-28ba-4c46-9942-20f960bb967f"). InnerVolumeSpecName "kube-api-access-vmp6b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:32:42.894830 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:32:42.894799    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eafc7c38-28ba-4c46-9942-20f960bb967f-maas-api-tls" pod "eafc7c38-28ba-4c46-9942-20f960bb967f" (UID: "eafc7c38-28ba-4c46-9942-20f960bb967f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:32:42.988979 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:32:42.988937    4559 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eafc7c38-28ba-4c46-9942-20f960bb967f-metrics-certs\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:32:42.988979 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:32:42.988976    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vmp6b\" (UniqueName: \"kubernetes.io/projected/eafc7c38-28ba-4c46-9942-20f960bb967f-kube-api-access-vmp6b\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:32:42.989133 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:32:42.988992    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eafc7c38-28ba-4c46-9942-20f960bb967f-maas-api-tls\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:32:42.989133 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:32:42.989005    4559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eafc7c38-28ba-4c46-9942-20f960bb967f-openshift-service-ca\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:32:43.288586 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:32:43.288552    4559 generic.go:356] "Generic (PLEG): container finished" podID="eafc7c38-28ba-4c46-9942-20f960bb967f" containerID="11baa5880d4f6c4916be55c534c0ae66ef79c5b41f4e9838a984e00e62eeda95" exitCode=0
Nov 02 12:32:43.288769 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:32:43.288617    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-f54ddc9f4-p7w9d" event={"ID":"eafc7c38-28ba-4c46-9942-20f960bb967f","Type":"ContainerDied","Data":"11baa5880d4f6c4916be55c534c0ae66ef79c5b41f4e9838a984e00e62eeda95"}
Nov 02 12:32:43.288769 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:32:43.288621    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-f54ddc9f4-p7w9d"
Nov 02 12:32:43.288769 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:32:43.288706    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-f54ddc9f4-p7w9d" event={"ID":"eafc7c38-28ba-4c46-9942-20f960bb967f","Type":"ContainerDied","Data":"197d63ffe487d02b2d8b9414986f1524b967b451938fa076e2bea647072afcbf"}
Nov 02 12:32:43.288769 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:32:43.288728    4559 scope.go:122] "RemoveContainer" containerID="11baa5880d4f6c4916be55c534c0ae66ef79c5b41f4e9838a984e00e62eeda95"
Nov 02 12:32:43.297821 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:32:43.297806    4559 scope.go:122] "RemoveContainer" containerID="11baa5880d4f6c4916be55c534c0ae66ef79c5b41f4e9838a984e00e62eeda95"
Nov 02 12:32:43.298067 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:32:43.298050    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11baa5880d4f6c4916be55c534c0ae66ef79c5b41f4e9838a984e00e62eeda95\": container with ID starting with 11baa5880d4f6c4916be55c534c0ae66ef79c5b41f4e9838a984e00e62eeda95 not found: ID does not exist" containerID="11baa5880d4f6c4916be55c534c0ae66ef79c5b41f4e9838a984e00e62eeda95"
Nov 02 12:32:43.298104 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:32:43.298075    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11baa5880d4f6c4916be55c534c0ae66ef79c5b41f4e9838a984e00e62eeda95"} err="failed to get container status \"11baa5880d4f6c4916be55c534c0ae66ef79c5b41f4e9838a984e00e62eeda95\": rpc error: code = NotFound desc = could not find container \"11baa5880d4f6c4916be55c534c0ae66ef79c5b41f4e9838a984e00e62eeda95\": container with ID starting with 11baa5880d4f6c4916be55c534c0ae66ef79c5b41f4e9838a984e00e62eeda95 not found: ID does not exist"
Nov 02 12:32:43.340214 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:32:43.340188    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-f54ddc9f4-p7w9d"]
Nov 02 12:32:43.344588 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:32:43.344570    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-f54ddc9f4-p7w9d"]
Nov 02 12:32:43.890502 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:32:43.890462    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eafc7c38-28ba-4c46-9942-20f960bb967f" path="/var/lib/kubelet/pods/eafc7c38-28ba-4c46-9942-20f960bb967f/volumes"
Nov 02 12:32:45.083129 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:32:45.083094    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqcmj" containerName="extractor"
Nov 02 12:33:15.085036 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:33:15.085002    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqcmj" containerName="extractor"
Nov 02 12:33:29.839030 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:33:29.838959    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-md2qx"]
Nov 02 12:33:29.839474 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:33:29.839231    4559 cpu_manager.go:414] "containerMap: removing container" podUID="5d7f5a1b-6338-4642-85aa-d7d3307265bd" containerName="authorino"
Nov 02 12:33:29.839474 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:33:29.839241    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d7f5a1b-6338-4642-85aa-d7d3307265bd" containerName="authorino"
Nov 02 12:33:29.839474 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:33:29.839265    4559 cpu_manager.go:414] "containerMap: removing container" podUID="eafc7c38-28ba-4c46-9942-20f960bb967f" containerName="maas-api"
Nov 02 12:33:29.839474 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:33:29.839270    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eafc7c38-28ba-4c46-9942-20f960bb967f" containerName="maas-api"
Nov 02 12:33:29.839474 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:33:29.839320    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d7f5a1b-6338-4642-85aa-d7d3307265bd" containerName="authorino"
Nov 02 12:33:29.839474 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:33:29.839328    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="eafc7c38-28ba-4c46-9942-20f960bb967f" containerName="maas-api"
Nov 02 12:33:29.842280 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:33:29.842258    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-md2qx"
Nov 02 12:33:29.866360 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:33:29.866335    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-md2qx"]
Nov 02 12:33:29.873350 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:33:29.873323    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\""
Nov 02 12:33:29.873350 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:33:29.873342    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Nov 02 12:33:29.873500 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:33:29.873342    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Nov 02 12:33:29.873500 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:33:29.873344    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\""
Nov 02 12:33:29.873500 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:33:29.873448    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-v9b97\""
Nov 02 12:33:29.929293 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:33:29.929266    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2d289d4a-3712-43dc-8c9a-88ea14a7e300-metrics-certs\") pod \"ai-gateway-controller-7d85675f98-md2qx\" (UID: \"2d289d4a-3712-43dc-8c9a-88ea14a7e300\") " pod="opendatahub/ai-gateway-controller-7d85675f98-md2qx"
Nov 02 12:33:29.929423 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:33:29.929321    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqvsr\" (UniqueName: \"kubernetes.io/projected/2d289d4a-3712-43dc-8c9a-88ea14a7e300-kube-api-access-fqvsr\") pod \"ai-gateway-controller-7d85675f98-md2qx\" (UID: \"2d289d4a-3712-43dc-8c9a-88ea14a7e300\") " pod="opendatahub/ai-gateway-controller-7d85675f98-md2qx"
Nov 02 12:33:30.030230 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:33:30.030199    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqvsr\" (UniqueName: \"kubernetes.io/projected/2d289d4a-3712-43dc-8c9a-88ea14a7e300-kube-api-access-fqvsr\") pod \"ai-gateway-controller-7d85675f98-md2qx\" (UID: \"2d289d4a-3712-43dc-8c9a-88ea14a7e300\") " pod="opendatahub/ai-gateway-controller-7d85675f98-md2qx"
Nov 02 12:33:30.030396 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:33:30.030256    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2d289d4a-3712-43dc-8c9a-88ea14a7e300-metrics-certs\") pod \"ai-gateway-controller-7d85675f98-md2qx\" (UID: \"2d289d4a-3712-43dc-8c9a-88ea14a7e300\") " pod="opendatahub/ai-gateway-controller-7d85675f98-md2qx"
Nov 02 12:33:30.032542 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:33:30.032518    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2d289d4a-3712-43dc-8c9a-88ea14a7e300-metrics-certs\") pod \"ai-gateway-controller-7d85675f98-md2qx\" (UID: \"2d289d4a-3712-43dc-8c9a-88ea14a7e300\") " pod="opendatahub/ai-gateway-controller-7d85675f98-md2qx"
Nov 02 12:33:30.054039 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:33:30.054009    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqvsr\" (UniqueName: \"kubernetes.io/projected/2d289d4a-3712-43dc-8c9a-88ea14a7e300-kube-api-access-fqvsr\") pod \"ai-gateway-controller-7d85675f98-md2qx\" (UID: \"2d289d4a-3712-43dc-8c9a-88ea14a7e300\") " pod="opendatahub/ai-gateway-controller-7d85675f98-md2qx"
Nov 02 12:33:30.154901 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:33:30.154841    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-md2qx"
Nov 02 12:33:30.450700 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:33:30.450671    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-md2qx"]
Nov 02 12:33:30.451285 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:33:30.451262    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d289d4a_3712_43dc_8c9a_88ea14a7e300.slice/crio-3b326d6ea873463220d3e1c30c12a1cd1a8b4bb9590e4316005b2b2d9c9f955f WatchSource:0}: Error finding container 3b326d6ea873463220d3e1c30c12a1cd1a8b4bb9590e4316005b2b2d9c9f955f: Status 404 returned error can't find the container with id 3b326d6ea873463220d3e1c30c12a1cd1a8b4bb9590e4316005b2b2d9c9f955f
Nov 02 12:33:30.468870 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:33:30.468847    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-md2qx" event={"ID":"2d289d4a-3712-43dc-8c9a-88ea14a7e300","Type":"ContainerStarted","Data":"3b326d6ea873463220d3e1c30c12a1cd1a8b4bb9590e4316005b2b2d9c9f955f"}
Nov 02 12:33:33.484012 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:33:33.483978    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-md2qx" event={"ID":"2d289d4a-3712-43dc-8c9a-88ea14a7e300","Type":"ContainerStarted","Data":"faec67169692e6e8260c01173288cf37d261a661d63d641a617dfe7c00db46e3"}
Nov 02 12:33:33.484380 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:33:33.484127    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-7d85675f98-md2qx"
Nov 02 12:33:33.525870 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:33:33.525682    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-7d85675f98-md2qx" podStartSLOduration=2.22882466 podStartE2EDuration="4.525667983s" totalImagesPullingTime="2.296843323s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:33:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:33:33.52538841 +0000 UTC m=+409.780787502" watchObservedRunningTime="2026-11-02 12:33:33.525667983 +0000 UTC m=+409.781067080"
Nov 02 12:33:34.084495 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:33:34.084460    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-qg68n"]
Nov 02 12:33:34.087673 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:33:34.087659    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-qg68n"
Nov 02 12:33:34.110205 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:33:34.110145    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-qg68n"]
Nov 02 12:33:34.121437 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:33:34.121418    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Nov 02 12:33:34.121437 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:33:34.121421    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-x88b9\""
Nov 02 12:33:34.158145 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:33:34.158120    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15512035-7903-48f9-ba69-2b53ef921f6e-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-qg68n\" (UID: \"15512035-7903-48f9-ba69-2b53ef921f6e\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-qg68n"
Nov 02 12:33:34.158304 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:33:34.158185    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngqtz\" (UniqueName: \"kubernetes.io/projected/15512035-7903-48f9-ba69-2b53ef921f6e-kube-api-access-ngqtz\") pod \"payload-pre-processing-7b98bb678c-qg68n\" (UID: \"15512035-7903-48f9-ba69-2b53ef921f6e\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-qg68n"
Nov 02 12:33:34.171659 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:33:34.171615    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-r6rdp"]
Nov 02 12:33:34.174875 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:33:34.174860    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-r6rdp"
Nov 02 12:33:34.194052 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:33:34.194029    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-r6rdp"]
Nov 02 12:33:34.258820 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:33:34.258797    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15512035-7903-48f9-ba69-2b53ef921f6e-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-qg68n\" (UID: \"15512035-7903-48f9-ba69-2b53ef921f6e\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-qg68n"
Nov 02 12:33:34.258935 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:33:34.258853    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2fef2bd6-5793-4c34-a390-37c907bcfd19-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-r6rdp\" (UID: \"2fef2bd6-5793-4c34-a390-37c907bcfd19\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-r6rdp"
Nov 02 12:33:34.258935 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:33:34.258872    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8ntk\" (UniqueName: \"kubernetes.io/projected/2fef2bd6-5793-4c34-a390-37c907bcfd19-kube-api-access-f8ntk\") pod \"payload-processing-55fbbbf9dc-r6rdp\" (UID: \"2fef2bd6-5793-4c34-a390-37c907bcfd19\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-r6rdp"
Nov 02 12:33:34.259010 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:33:34.258975    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngqtz\" (UniqueName: \"kubernetes.io/projected/15512035-7903-48f9-ba69-2b53ef921f6e-kube-api-access-ngqtz\") pod \"payload-pre-processing-7b98bb678c-qg68n\" (UID: \"15512035-7903-48f9-ba69-2b53ef921f6e\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-qg68n"
Nov 02 12:33:34.259457 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:33:34.259439    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15512035-7903-48f9-ba69-2b53ef921f6e-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-qg68n\" (UID: \"15512035-7903-48f9-ba69-2b53ef921f6e\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-qg68n"
Nov 02 12:33:34.282674 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:33:34.282653    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngqtz\" (UniqueName: \"kubernetes.io/projected/15512035-7903-48f9-ba69-2b53ef921f6e-kube-api-access-ngqtz\") pod \"payload-pre-processing-7b98bb678c-qg68n\" (UID: \"15512035-7903-48f9-ba69-2b53ef921f6e\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-qg68n"
Nov 02 12:33:34.359591 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:33:34.359509    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2fef2bd6-5793-4c34-a390-37c907bcfd19-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-r6rdp\" (UID: \"2fef2bd6-5793-4c34-a390-37c907bcfd19\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-r6rdp"
Nov 02 12:33:34.359591 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:33:34.359556    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8ntk\" (UniqueName: \"kubernetes.io/projected/2fef2bd6-5793-4c34-a390-37c907bcfd19-kube-api-access-f8ntk\") pod \"payload-processing-55fbbbf9dc-r6rdp\" (UID: \"2fef2bd6-5793-4c34-a390-37c907bcfd19\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-r6rdp"
Nov 02 12:33:34.360202 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:33:34.360180    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2fef2bd6-5793-4c34-a390-37c907bcfd19-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-r6rdp\" (UID: \"2fef2bd6-5793-4c34-a390-37c907bcfd19\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-r6rdp"
Nov 02 12:33:34.383599 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:33:34.383573    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8ntk\" (UniqueName: \"kubernetes.io/projected/2fef2bd6-5793-4c34-a390-37c907bcfd19-kube-api-access-f8ntk\") pod \"payload-processing-55fbbbf9dc-r6rdp\" (UID: \"2fef2bd6-5793-4c34-a390-37c907bcfd19\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-r6rdp"
Nov 02 12:33:34.400283 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:33:34.400259    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-qg68n"
Nov 02 12:33:34.486631 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:33:34.486605    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-r6rdp"
Nov 02 12:33:34.694616 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:33:34.694587    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-qg68n"]
Nov 02 12:33:34.695062 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:33:34.695035    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15512035_7903_48f9_ba69_2b53ef921f6e.slice/crio-c81e10686211ac4db3fd0e51faa969d57cab203d7d35079d2056570e3d690858 WatchSource:0}: Error finding container c81e10686211ac4db3fd0e51faa969d57cab203d7d35079d2056570e3d690858: Status 404 returned error can't find the container with id c81e10686211ac4db3fd0e51faa969d57cab203d7d35079d2056570e3d690858
Nov 02 12:33:34.779758 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:33:34.779731    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-r6rdp"]
Nov 02 12:33:34.780530 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:33:34.780507    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fef2bd6_5793_4c34_a390_37c907bcfd19.slice/crio-4e85fcce870daf8e3f7085e77f6ac429898acbaca12fd8096256c0dd25d56641 WatchSource:0}: Error finding container 4e85fcce870daf8e3f7085e77f6ac429898acbaca12fd8096256c0dd25d56641: Status 404 returned error can't find the container with id 4e85fcce870daf8e3f7085e77f6ac429898acbaca12fd8096256c0dd25d56641
Nov 02 12:33:35.495921 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:33:35.495823    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-r6rdp" event={"ID":"2fef2bd6-5793-4c34-a390-37c907bcfd19","Type":"ContainerStarted","Data":"4e85fcce870daf8e3f7085e77f6ac429898acbaca12fd8096256c0dd25d56641"}
Nov 02 12:33:35.499454 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:33:35.499370    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-qg68n" event={"ID":"15512035-7903-48f9-ba69-2b53ef921f6e","Type":"ContainerStarted","Data":"c81e10686211ac4db3fd0e51faa969d57cab203d7d35079d2056570e3d690858"}
Nov 02 12:33:37.514545 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:33:37.514508    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-r6rdp" event={"ID":"2fef2bd6-5793-4c34-a390-37c907bcfd19","Type":"ContainerStarted","Data":"7d9057ec8b704504a7349a97dbd26808c8e4eb6943f94c2d75c3295fd2b72b2d"}
Nov 02 12:33:37.515040 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:33:37.514664    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-r6rdp"
Nov 02 12:33:37.516050 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:33:37.516027    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-qg68n" event={"ID":"15512035-7903-48f9-ba69-2b53ef921f6e","Type":"ContainerStarted","Data":"3cdfa9ed6a91a8f8825d974d47d3e9669ead0e68707c70255d75a0b52947f004"}
Nov 02 12:33:37.516163 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:33:37.516127    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-qg68n"
Nov 02 12:33:37.556200 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:33:37.556153    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-55fbbbf9dc-r6rdp" podStartSLOduration=1.269379093 podStartE2EDuration="3.556140243s" totalImagesPullingTime="2.28676115s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:33:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:33:37.555715776 +0000 UTC m=+413.811114870" watchObservedRunningTime="2026-11-02 12:33:37.556140243 +0000 UTC m=+413.811539338"
Nov 02 12:33:37.599874 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:33:37.599835    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7b98bb678c-qg68n" podStartSLOduration=1.196683328 podStartE2EDuration="3.59982334s" totalImagesPullingTime="2.403140012s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:33:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:33:37.598388842 +0000 UTC m=+413.853787936" watchObservedRunningTime="2026-11-02 12:33:37.59982334 +0000 UTC m=+413.855222437"
Nov 02 12:33:44.491078 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:33:44.491049    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-7d85675f98-md2qx"
Nov 02 12:33:44.770972 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:33:44.770895    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-64577f84c6-5nmh8"]
Nov 02 12:33:44.774351 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:33:44.774329    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-64577f84c6-5nmh8"
Nov 02 12:33:44.796492 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:33:44.796474    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-64577f84c6-5nmh8"]
Nov 02 12:33:44.841178 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:33:44.841152    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4af87e02-6dc0-42ef-93df-26e174ec85df-metrics-certs\") pod \"ai-gateway-controller-64577f84c6-5nmh8\" (UID: \"4af87e02-6dc0-42ef-93df-26e174ec85df\") " pod="opendatahub/ai-gateway-controller-64577f84c6-5nmh8"
Nov 02 12:33:44.841289 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:33:44.841203    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbsg6\" (UniqueName: \"kubernetes.io/projected/4af87e02-6dc0-42ef-93df-26e174ec85df-kube-api-access-vbsg6\") pod \"ai-gateway-controller-64577f84c6-5nmh8\" (UID: \"4af87e02-6dc0-42ef-93df-26e174ec85df\") " pod="opendatahub/ai-gateway-controller-64577f84c6-5nmh8"
Nov 02 12:33:44.942342 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:33:44.942314    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4af87e02-6dc0-42ef-93df-26e174ec85df-metrics-certs\") pod \"ai-gateway-controller-64577f84c6-5nmh8\" (UID: \"4af87e02-6dc0-42ef-93df-26e174ec85df\") " pod="opendatahub/ai-gateway-controller-64577f84c6-5nmh8"
Nov 02 12:33:44.942444 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:33:44.942357    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbsg6\" (UniqueName: \"kubernetes.io/projected/4af87e02-6dc0-42ef-93df-26e174ec85df-kube-api-access-vbsg6\") pod \"ai-gateway-controller-64577f84c6-5nmh8\" (UID: \"4af87e02-6dc0-42ef-93df-26e174ec85df\") " pod="opendatahub/ai-gateway-controller-64577f84c6-5nmh8"
Nov 02 12:33:44.944583 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:33:44.944565    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4af87e02-6dc0-42ef-93df-26e174ec85df-metrics-certs\") pod \"ai-gateway-controller-64577f84c6-5nmh8\" (UID: \"4af87e02-6dc0-42ef-93df-26e174ec85df\") " pod="opendatahub/ai-gateway-controller-64577f84c6-5nmh8"
Nov 02 12:33:44.966221 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:33:44.966200    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbsg6\" (UniqueName: \"kubernetes.io/projected/4af87e02-6dc0-42ef-93df-26e174ec85df-kube-api-access-vbsg6\") pod \"ai-gateway-controller-64577f84c6-5nmh8\" (UID: \"4af87e02-6dc0-42ef-93df-26e174ec85df\") " pod="opendatahub/ai-gateway-controller-64577f84c6-5nmh8"
Nov 02 12:33:45.084225 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:33:45.084150    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqcmj" containerName="extractor"
Nov 02 12:33:45.086375 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:33:45.086357    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-64577f84c6-5nmh8"
Nov 02 12:33:45.379735 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:33:45.379709    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-64577f84c6-5nmh8"]
Nov 02 12:33:45.380673 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:33:45.380630    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4af87e02_6dc0_42ef_93df_26e174ec85df.slice/crio-b62c89c7338ed8ea9e43c7386c9f4ebff2fd02a2c18d0d1061a83715bfab5712 WatchSource:0}: Error finding container b62c89c7338ed8ea9e43c7386c9f4ebff2fd02a2c18d0d1061a83715bfab5712: Status 404 returned error can't find the container with id b62c89c7338ed8ea9e43c7386c9f4ebff2fd02a2c18d0d1061a83715bfab5712
Nov 02 12:33:45.547515 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:33:45.547485    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-64577f84c6-5nmh8" event={"ID":"4af87e02-6dc0-42ef-93df-26e174ec85df","Type":"ContainerStarted","Data":"b62c89c7338ed8ea9e43c7386c9f4ebff2fd02a2c18d0d1061a83715bfab5712"}
Nov 02 12:33:46.552542 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:33:46.552508    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-64577f84c6-5nmh8" event={"ID":"4af87e02-6dc0-42ef-93df-26e174ec85df","Type":"ContainerStarted","Data":"a71a81103ef1189122a6a42aa8ac4976e8c59224458179eb3df9c50fad892f73"}
Nov 02 12:33:46.552959 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:33:46.552582    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-64577f84c6-5nmh8"
Nov 02 12:33:46.596377 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:33:46.596333    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-64577f84c6-5nmh8" podStartSLOduration=2.290981928 podStartE2EDuration="2.596320431s" totalImagesPullingTime="305.338503ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:33:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:33:46.595298899 +0000 UTC m=+422.850698005" watchObservedRunningTime="2026-11-02 12:33:46.596320431 +0000 UTC m=+422.851719525"
Nov 02 12:33:48.522619 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:33:48.522590    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-r6rdp"
Nov 02 12:33:48.523016 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:33:48.522679    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-qg68n"
Nov 02 12:33:57.559128 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:33:57.559095    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-64577f84c6-5nmh8"
Nov 02 12:33:57.628626 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:33:57.628597    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-md2qx"]
Nov 02 12:33:57.628873 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:33:57.628838    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-7d85675f98-md2qx" podUID="2d289d4a-3712-43dc-8c9a-88ea14a7e300" containerName="manager" containerID="cri-o://faec67169692e6e8260c01173288cf37d261a661d63d641a617dfe7c00db46e3" gracePeriod=30
Nov 02 12:33:57.864788 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:33:57.864763    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-md2qx"
Nov 02 12:33:57.942170 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:33:57.942141    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d289d4a-3712-43dc-8c9a-88ea14a7e300-kube-api-access-fqvsr\" (UniqueName: \"kubernetes.io/projected/2d289d4a-3712-43dc-8c9a-88ea14a7e300-kube-api-access-fqvsr\") pod \"2d289d4a-3712-43dc-8c9a-88ea14a7e300\" (UID: \"2d289d4a-3712-43dc-8c9a-88ea14a7e300\") "
Nov 02 12:33:57.942328 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:33:57.942217    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2d289d4a-3712-43dc-8c9a-88ea14a7e300-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2d289d4a-3712-43dc-8c9a-88ea14a7e300-metrics-certs\") pod \"2d289d4a-3712-43dc-8c9a-88ea14a7e300\" (UID: \"2d289d4a-3712-43dc-8c9a-88ea14a7e300\") "
Nov 02 12:33:57.944352 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:33:57.944319    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d289d4a-3712-43dc-8c9a-88ea14a7e300-kube-api-access-fqvsr" pod "2d289d4a-3712-43dc-8c9a-88ea14a7e300" (UID: "2d289d4a-3712-43dc-8c9a-88ea14a7e300"). InnerVolumeSpecName "kube-api-access-fqvsr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:33:57.944440 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:33:57.944349    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d289d4a-3712-43dc-8c9a-88ea14a7e300-metrics-certs" pod "2d289d4a-3712-43dc-8c9a-88ea14a7e300" (UID: "2d289d4a-3712-43dc-8c9a-88ea14a7e300"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:33:58.043205 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:33:58.043176    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fqvsr\" (UniqueName: \"kubernetes.io/projected/2d289d4a-3712-43dc-8c9a-88ea14a7e300-kube-api-access-fqvsr\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:33:58.043205 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:33:58.043201    4559 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2d289d4a-3712-43dc-8c9a-88ea14a7e300-metrics-certs\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:33:58.605611 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:33:58.605578    4559 generic.go:356] "Generic (PLEG): container finished" podID="2d289d4a-3712-43dc-8c9a-88ea14a7e300" containerID="faec67169692e6e8260c01173288cf37d261a661d63d641a617dfe7c00db46e3" exitCode=0
Nov 02 12:33:58.606038 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:33:58.605626    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-md2qx"
Nov 02 12:33:58.606038 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:33:58.605663    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-md2qx" event={"ID":"2d289d4a-3712-43dc-8c9a-88ea14a7e300","Type":"ContainerDied","Data":"faec67169692e6e8260c01173288cf37d261a661d63d641a617dfe7c00db46e3"}
Nov 02 12:33:58.606038 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:33:58.605717    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-md2qx" event={"ID":"2d289d4a-3712-43dc-8c9a-88ea14a7e300","Type":"ContainerDied","Data":"3b326d6ea873463220d3e1c30c12a1cd1a8b4bb9590e4316005b2b2d9c9f955f"}
Nov 02 12:33:58.606038 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:33:58.605742    4559 scope.go:122] "RemoveContainer" containerID="faec67169692e6e8260c01173288cf37d261a661d63d641a617dfe7c00db46e3"
Nov 02 12:33:58.614210 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:33:58.614194    4559 scope.go:122] "RemoveContainer" containerID="faec67169692e6e8260c01173288cf37d261a661d63d641a617dfe7c00db46e3"
Nov 02 12:33:58.614454 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:33:58.614436    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"faec67169692e6e8260c01173288cf37d261a661d63d641a617dfe7c00db46e3\": container with ID starting with faec67169692e6e8260c01173288cf37d261a661d63d641a617dfe7c00db46e3 not found: ID does not exist" containerID="faec67169692e6e8260c01173288cf37d261a661d63d641a617dfe7c00db46e3"
Nov 02 12:33:58.614496 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:33:58.614465    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"faec67169692e6e8260c01173288cf37d261a661d63d641a617dfe7c00db46e3"} err="failed to get container status \"faec67169692e6e8260c01173288cf37d261a661d63d641a617dfe7c00db46e3\": rpc error: code = NotFound desc = could not find container \"faec67169692e6e8260c01173288cf37d261a661d63d641a617dfe7c00db46e3\": container with ID starting with faec67169692e6e8260c01173288cf37d261a661d63d641a617dfe7c00db46e3 not found: ID does not exist"
Nov 02 12:33:58.669091 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:33:58.669065    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-md2qx"]
Nov 02 12:33:58.681754 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:33:58.681731    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-md2qx"]
Nov 02 12:33:59.890687 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:33:59.890629    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d289d4a-3712-43dc-8c9a-88ea14a7e300" path="/var/lib/kubelet/pods/2d289d4a-3712-43dc-8c9a-88ea14a7e300/volumes"
Nov 02 12:34:05.529829 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:05.529797    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hsqc5"]
Nov 02 12:34:05.530293 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:05.530281    4559 cpu_manager.go:414] "containerMap: removing container" podUID="2d289d4a-3712-43dc-8c9a-88ea14a7e300" containerName="manager"
Nov 02 12:34:05.530344 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:05.530297    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d289d4a-3712-43dc-8c9a-88ea14a7e300" containerName="manager"
Nov 02 12:34:05.530431 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:05.530410    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d289d4a-3712-43dc-8c9a-88ea14a7e300" containerName="manager"
Nov 02 12:34:05.535107 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:05.535088    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hsqc5"
Nov 02 12:34:05.551702 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:05.551678    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Nov 02 12:34:05.564772 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:05.564748    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Nov 02 12:34:05.564889 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:05.564859    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-lwrqm\""
Nov 02 12:34:05.564963 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:05.564948    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Nov 02 12:34:05.566168 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:05.566146    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hsqc5"]
Nov 02 12:34:05.595632 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:05.595610    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2w6bl\" (UniqueName: \"kubernetes.io/projected/b3be8164-4f0a-490f-bd56-bc2270f12bbc-kube-api-access-2w6bl\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-hsqc5\" (UID: \"b3be8164-4f0a-490f-bd56-bc2270f12bbc\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hsqc5"
Nov 02 12:34:05.595757 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:05.595654    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b3be8164-4f0a-490f-bd56-bc2270f12bbc-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-hsqc5\" (UID: \"b3be8164-4f0a-490f-bd56-bc2270f12bbc\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hsqc5"
Nov 02 12:34:05.595859 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:05.595838    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b3be8164-4f0a-490f-bd56-bc2270f12bbc-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-hsqc5\" (UID: \"b3be8164-4f0a-490f-bd56-bc2270f12bbc\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hsqc5"
Nov 02 12:34:05.595930 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:05.595873    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3be8164-4f0a-490f-bd56-bc2270f12bbc-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-hsqc5\" (UID: \"b3be8164-4f0a-490f-bd56-bc2270f12bbc\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hsqc5"
Nov 02 12:34:05.595989 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:05.595962    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b3be8164-4f0a-490f-bd56-bc2270f12bbc-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-hsqc5\" (UID: \"b3be8164-4f0a-490f-bd56-bc2270f12bbc\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hsqc5"
Nov 02 12:34:05.596033 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:05.596019    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3be8164-4f0a-490f-bd56-bc2270f12bbc-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-hsqc5\" (UID: \"b3be8164-4f0a-490f-bd56-bc2270f12bbc\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hsqc5"
Nov 02 12:34:05.697315 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:05.697280    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2w6bl\" (UniqueName: \"kubernetes.io/projected/b3be8164-4f0a-490f-bd56-bc2270f12bbc-kube-api-access-2w6bl\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-hsqc5\" (UID: \"b3be8164-4f0a-490f-bd56-bc2270f12bbc\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hsqc5"
Nov 02 12:34:05.697459 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:05.697318    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b3be8164-4f0a-490f-bd56-bc2270f12bbc-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-hsqc5\" (UID: \"b3be8164-4f0a-490f-bd56-bc2270f12bbc\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hsqc5"
Nov 02 12:34:05.697459 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:05.697388    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b3be8164-4f0a-490f-bd56-bc2270f12bbc-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-hsqc5\" (UID: \"b3be8164-4f0a-490f-bd56-bc2270f12bbc\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hsqc5"
Nov 02 12:34:05.697459 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:05.697410    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3be8164-4f0a-490f-bd56-bc2270f12bbc-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-hsqc5\" (UID: \"b3be8164-4f0a-490f-bd56-bc2270f12bbc\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hsqc5"
Nov 02 12:34:05.697459 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:05.697449    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b3be8164-4f0a-490f-bd56-bc2270f12bbc-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-hsqc5\" (UID: \"b3be8164-4f0a-490f-bd56-bc2270f12bbc\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hsqc5"
Nov 02 12:34:05.697678 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:05.697473    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3be8164-4f0a-490f-bd56-bc2270f12bbc-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-hsqc5\" (UID: \"b3be8164-4f0a-490f-bd56-bc2270f12bbc\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hsqc5"
Nov 02 12:34:05.699266 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:05.697750    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b3be8164-4f0a-490f-bd56-bc2270f12bbc-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-hsqc5\" (UID: \"b3be8164-4f0a-490f-bd56-bc2270f12bbc\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hsqc5"
Nov 02 12:34:05.699266 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:05.697833    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3be8164-4f0a-490f-bd56-bc2270f12bbc-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-hsqc5\" (UID: \"b3be8164-4f0a-490f-bd56-bc2270f12bbc\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hsqc5"
Nov 02 12:34:05.699266 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:05.697966    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b3be8164-4f0a-490f-bd56-bc2270f12bbc-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-hsqc5\" (UID: \"b3be8164-4f0a-490f-bd56-bc2270f12bbc\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hsqc5"
Nov 02 12:34:05.699865 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:05.699841    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b3be8164-4f0a-490f-bd56-bc2270f12bbc-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-hsqc5\" (UID: \"b3be8164-4f0a-490f-bd56-bc2270f12bbc\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hsqc5"
Nov 02 12:34:05.700063 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:05.700046    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3be8164-4f0a-490f-bd56-bc2270f12bbc-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-hsqc5\" (UID: \"b3be8164-4f0a-490f-bd56-bc2270f12bbc\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hsqc5"
Nov 02 12:34:05.720936 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:05.720911    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2w6bl\" (UniqueName: \"kubernetes.io/projected/b3be8164-4f0a-490f-bd56-bc2270f12bbc-kube-api-access-2w6bl\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-hsqc5\" (UID: \"b3be8164-4f0a-490f-bd56-bc2270f12bbc\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hsqc5"
Nov 02 12:34:05.848500 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:05.848421    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hsqc5"
Nov 02 12:34:06.145440 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:06.145408    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hsqc5"]
Nov 02 12:34:06.146548 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:34:06.146518    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3be8164_4f0a_490f_bd56_bc2270f12bbc.slice/crio-06d73933e5b1cbe6fef0b27abcf871da55325163fa9cb8c0557691b347894036 WatchSource:0}: Error finding container 06d73933e5b1cbe6fef0b27abcf871da55325163fa9cb8c0557691b347894036: Status 404 returned error can't find the container with id 06d73933e5b1cbe6fef0b27abcf871da55325163fa9cb8c0557691b347894036
Nov 02 12:34:06.515483 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:06.515445    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"]
Nov 02 12:34:06.645855 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:06.645818    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hsqc5" event={"ID":"b3be8164-4f0a-490f-bd56-bc2270f12bbc","Type":"ContainerStarted","Data":"06d73933e5b1cbe6fef0b27abcf871da55325163fa9cb8c0557691b347894036"}
Nov 02 12:34:07.652322 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:07.652290    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hsqc5" event={"ID":"b3be8164-4f0a-490f-bd56-bc2270f12bbc","Type":"ContainerStarted","Data":"bb66d56930657bd61662b870b218a5aa024c27c974994f759f0d370c2e7b2e71"}
Nov 02 12:34:07.696799 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:07.696746    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hsqc5" podStartSLOduration=1.313429761 podStartE2EDuration="2.69673064s" totalImagesPullingTime="1.383300879s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:34:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:34:07.69489428 +0000 UTC m=+443.950293375" watchObservedRunningTime="2026-11-02 12:34:07.69673064 +0000 UTC m=+443.952129731"
Nov 02 12:34:07.849336 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:07.849253    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hsqc5"
Nov 02 12:34:08.794943 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:08.794912    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-tz6wl"]
Nov 02 12:34:08.798263 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:08.798242    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tz6wl"
Nov 02 12:34:08.821069 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:08.821043    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-tz6wl"]
Nov 02 12:34:08.828529 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:08.828511    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Nov 02 12:34:08.928371 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:08.928336    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a48eaf5-31e6-410e-b93f-24b21ab5b871-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-tz6wl\" (UID: \"1a48eaf5-31e6-410e-b93f-24b21ab5b871\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tz6wl"
Nov 02 12:34:08.928371 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:08.928373    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a48eaf5-31e6-410e-b93f-24b21ab5b871-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-tz6wl\" (UID: \"1a48eaf5-31e6-410e-b93f-24b21ab5b871\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tz6wl"
Nov 02 12:34:08.928585 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:08.928393    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-db7zz\" (UniqueName: \"kubernetes.io/projected/1a48eaf5-31e6-410e-b93f-24b21ab5b871-kube-api-access-db7zz\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-tz6wl\" (UID: \"1a48eaf5-31e6-410e-b93f-24b21ab5b871\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tz6wl"
Nov 02 12:34:08.928585 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:08.928465    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a48eaf5-31e6-410e-b93f-24b21ab5b871-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-tz6wl\" (UID: \"1a48eaf5-31e6-410e-b93f-24b21ab5b871\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tz6wl"
Nov 02 12:34:08.928585 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:08.928493    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a48eaf5-31e6-410e-b93f-24b21ab5b871-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-tz6wl\" (UID: \"1a48eaf5-31e6-410e-b93f-24b21ab5b871\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tz6wl"
Nov 02 12:34:08.928775 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:08.928621    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a48eaf5-31e6-410e-b93f-24b21ab5b871-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-tz6wl\" (UID: \"1a48eaf5-31e6-410e-b93f-24b21ab5b871\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tz6wl"
Nov 02 12:34:09.029797 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:09.029767    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a48eaf5-31e6-410e-b93f-24b21ab5b871-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-tz6wl\" (UID: \"1a48eaf5-31e6-410e-b93f-24b21ab5b871\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tz6wl"
Nov 02 12:34:09.029941 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:09.029803    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a48eaf5-31e6-410e-b93f-24b21ab5b871-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-tz6wl\" (UID: \"1a48eaf5-31e6-410e-b93f-24b21ab5b871\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tz6wl"
Nov 02 12:34:09.029941 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:09.029916    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-db7zz\" (UniqueName: \"kubernetes.io/projected/1a48eaf5-31e6-410e-b93f-24b21ab5b871-kube-api-access-db7zz\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-tz6wl\" (UID: \"1a48eaf5-31e6-410e-b93f-24b21ab5b871\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tz6wl"
Nov 02 12:34:09.030058 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:09.029946    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a48eaf5-31e6-410e-b93f-24b21ab5b871-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-tz6wl\" (UID: \"1a48eaf5-31e6-410e-b93f-24b21ab5b871\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tz6wl"
Nov 02 12:34:09.030058 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:09.029969    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a48eaf5-31e6-410e-b93f-24b21ab5b871-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-tz6wl\" (UID: \"1a48eaf5-31e6-410e-b93f-24b21ab5b871\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tz6wl"
Nov 02 12:34:09.030058 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:09.030029    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a48eaf5-31e6-410e-b93f-24b21ab5b871-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-tz6wl\" (UID: \"1a48eaf5-31e6-410e-b93f-24b21ab5b871\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tz6wl"
Nov 02 12:34:09.030192 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:09.030161    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a48eaf5-31e6-410e-b93f-24b21ab5b871-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-tz6wl\" (UID: \"1a48eaf5-31e6-410e-b93f-24b21ab5b871\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tz6wl"
Nov 02 12:34:09.030316 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:09.030294    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a48eaf5-31e6-410e-b93f-24b21ab5b871-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-tz6wl\" (UID: \"1a48eaf5-31e6-410e-b93f-24b21ab5b871\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tz6wl"
Nov 02 12:34:09.030382 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:09.030366    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a48eaf5-31e6-410e-b93f-24b21ab5b871-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-tz6wl\" (UID: \"1a48eaf5-31e6-410e-b93f-24b21ab5b871\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tz6wl"
Nov 02 12:34:09.032117 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:09.032091    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a48eaf5-31e6-410e-b93f-24b21ab5b871-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-tz6wl\" (UID: \"1a48eaf5-31e6-410e-b93f-24b21ab5b871\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tz6wl"
Nov 02 12:34:09.032494 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:09.032472    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a48eaf5-31e6-410e-b93f-24b21ab5b871-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-tz6wl\" (UID: \"1a48eaf5-31e6-410e-b93f-24b21ab5b871\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tz6wl"
Nov 02 12:34:09.053856 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:09.053809    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-db7zz\" (UniqueName: \"kubernetes.io/projected/1a48eaf5-31e6-410e-b93f-24b21ab5b871-kube-api-access-db7zz\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-tz6wl\" (UID: \"1a48eaf5-31e6-410e-b93f-24b21ab5b871\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tz6wl"
Nov 02 12:34:09.112840 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:09.112813    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tz6wl"
Nov 02 12:34:09.413445 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:09.412324    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-tz6wl"]
Nov 02 12:34:09.661670 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:09.661621    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tz6wl" event={"ID":"1a48eaf5-31e6-410e-b93f-24b21ab5b871","Type":"ContainerStarted","Data":"6be9fc918d33edfd83501fb4cda1f1c42add40d90b648ea174ad37e8bbf33aca"}
Nov 02 12:34:09.825529 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:09.825495    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"]
Nov 02 12:34:10.668123 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:10.668083    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tz6wl" event={"ID":"1a48eaf5-31e6-410e-b93f-24b21ab5b871","Type":"ContainerStarted","Data":"4bdf2436f676531b7227e3ea6ebc7c394ab101eca8dd4dc791a14fbf7406326e"}
Nov 02 12:34:10.711651 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:10.711589    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tz6wl" podStartSLOduration=2.4680484959999998 podStartE2EDuration="2.711569728s" totalImagesPullingTime="243.521232ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:34:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:34:10.709866772 +0000 UTC m=+446.965265857" watchObservedRunningTime="2026-11-02 12:34:10.711569728 +0000 UTC m=+446.966968824"
Nov 02 12:34:11.116606 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:11.116507    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tz6wl"
Nov 02 12:34:12.828538 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:12.828496    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-9frnx"]
Nov 02 12:34:12.833658 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:12.832176    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-9frnx"
Nov 02 12:34:12.867683 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:12.851753    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Nov 02 12:34:12.883869 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:12.883829    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-9frnx"]
Nov 02 12:34:12.976030 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:12.975992    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6bda1dc-2b3a-477c-a148-1d5be4611146-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-9frnx\" (UID: \"e6bda1dc-2b3a-477c-a148-1d5be4611146\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-9frnx"
Nov 02 12:34:12.976188 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:12.976050    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6bda1dc-2b3a-477c-a148-1d5be4611146-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-9frnx\" (UID: \"e6bda1dc-2b3a-477c-a148-1d5be4611146\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-9frnx"
Nov 02 12:34:12.976188 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:12.976128    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6bda1dc-2b3a-477c-a148-1d5be4611146-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-9frnx\" (UID: \"e6bda1dc-2b3a-477c-a148-1d5be4611146\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-9frnx"
Nov 02 12:34:12.976188 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:12.976163    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6sprx\" (UniqueName: \"kubernetes.io/projected/e6bda1dc-2b3a-477c-a148-1d5be4611146-kube-api-access-6sprx\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-9frnx\" (UID: \"e6bda1dc-2b3a-477c-a148-1d5be4611146\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-9frnx"
Nov 02 12:34:12.976324 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:12.976305    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6bda1dc-2b3a-477c-a148-1d5be4611146-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-9frnx\" (UID: \"e6bda1dc-2b3a-477c-a148-1d5be4611146\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-9frnx"
Nov 02 12:34:12.976364 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:12.976355    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6bda1dc-2b3a-477c-a148-1d5be4611146-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-9frnx\" (UID: \"e6bda1dc-2b3a-477c-a148-1d5be4611146\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-9frnx"
Nov 02 12:34:13.077375 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:13.077326    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6bda1dc-2b3a-477c-a148-1d5be4611146-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-9frnx\" (UID: \"e6bda1dc-2b3a-477c-a148-1d5be4611146\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-9frnx"
Nov 02 12:34:13.077560 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:13.077411    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6bda1dc-2b3a-477c-a148-1d5be4611146-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-9frnx\" (UID: \"e6bda1dc-2b3a-477c-a148-1d5be4611146\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-9frnx"
Nov 02 12:34:13.077560 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:13.077442    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6bda1dc-2b3a-477c-a148-1d5be4611146-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-9frnx\" (UID: \"e6bda1dc-2b3a-477c-a148-1d5be4611146\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-9frnx"
Nov 02 12:34:13.077560 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:13.077484    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6bda1dc-2b3a-477c-a148-1d5be4611146-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-9frnx\" (UID: \"e6bda1dc-2b3a-477c-a148-1d5be4611146\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-9frnx"
Nov 02 12:34:13.077560 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:13.077511    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6sprx\" (UniqueName: \"kubernetes.io/projected/e6bda1dc-2b3a-477c-a148-1d5be4611146-kube-api-access-6sprx\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-9frnx\" (UID: \"e6bda1dc-2b3a-477c-a148-1d5be4611146\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-9frnx"
Nov 02 12:34:13.077804 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:13.077597    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6bda1dc-2b3a-477c-a148-1d5be4611146-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-9frnx\" (UID: \"e6bda1dc-2b3a-477c-a148-1d5be4611146\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-9frnx"
Nov 02 12:34:13.077929 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:13.077903    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6bda1dc-2b3a-477c-a148-1d5be4611146-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-9frnx\" (UID: \"e6bda1dc-2b3a-477c-a148-1d5be4611146\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-9frnx"
Nov 02 12:34:13.077993 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:13.077924    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6bda1dc-2b3a-477c-a148-1d5be4611146-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-9frnx\" (UID: \"e6bda1dc-2b3a-477c-a148-1d5be4611146\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-9frnx"
Nov 02 12:34:13.078073 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:13.077993    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6bda1dc-2b3a-477c-a148-1d5be4611146-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-9frnx\" (UID: \"e6bda1dc-2b3a-477c-a148-1d5be4611146\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-9frnx"
Nov 02 12:34:13.079743 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:13.079681    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6bda1dc-2b3a-477c-a148-1d5be4611146-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-9frnx\" (UID: \"e6bda1dc-2b3a-477c-a148-1d5be4611146\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-9frnx"
Nov 02 12:34:13.079993 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:13.079971    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6bda1dc-2b3a-477c-a148-1d5be4611146-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-9frnx\" (UID: \"e6bda1dc-2b3a-477c-a148-1d5be4611146\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-9frnx"
Nov 02 12:34:13.101974 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:13.101951    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6sprx\" (UniqueName: \"kubernetes.io/projected/e6bda1dc-2b3a-477c-a148-1d5be4611146-kube-api-access-6sprx\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-9frnx\" (UID: \"e6bda1dc-2b3a-477c-a148-1d5be4611146\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-9frnx"
Nov 02 12:34:13.152700 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:13.152674    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-9frnx"
Nov 02 12:34:13.458862 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:13.458835    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-9frnx"]
Nov 02 12:34:13.459414 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:34:13.459393    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6bda1dc_2b3a_477c_a148_1d5be4611146.slice/crio-3914a823ec87235ca0048f1027562c540658ca60757b3fce2cfe3e5d0a1cc3b0 WatchSource:0}: Error finding container 3914a823ec87235ca0048f1027562c540658ca60757b3fce2cfe3e5d0a1cc3b0: Status 404 returned error can't find the container with id 3914a823ec87235ca0048f1027562c540658ca60757b3fce2cfe3e5d0a1cc3b0
Nov 02 12:34:13.682177 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:13.682138    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-9frnx" event={"ID":"e6bda1dc-2b3a-477c-a148-1d5be4611146","Type":"ContainerStarted","Data":"3914a823ec87235ca0048f1027562c540658ca60757b3fce2cfe3e5d0a1cc3b0"}
Nov 02 12:34:14.687511 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:14.687476    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-9frnx" event={"ID":"e6bda1dc-2b3a-477c-a148-1d5be4611146","Type":"ContainerStarted","Data":"cefdb1d310df3c0c86221fc1fc0a5166cfbdc7f006bc635d5df06cf9a53ba6ba"}
Nov 02 12:34:14.734873 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:14.734828    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-9frnx" podStartSLOduration=2.167090244 podStartE2EDuration="2.734813848s" totalImagesPullingTime="567.723604ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:34:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:34:14.733221738 +0000 UTC m=+450.988620832" watchObservedRunningTime="2026-11-02 12:34:14.734813848 +0000 UTC m=+450.990212974"
Nov 02 12:34:15.090570 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:34:15.090477    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqcmj" containerName="extractor"
Nov 02 12:34:15.090810 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:15.090786    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"]
Nov 02 12:34:15.155663 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:15.153808    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-9frnx"
Nov 02 12:34:15.849022 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:15.848981    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hsqc5"
Nov 02 12:34:15.862098 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:15.862072    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hsqc5"
Nov 02 12:34:16.707160 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:16.707124    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hsqc5"
Nov 02 12:34:18.101871 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:18.101842    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-b82wj"]
Nov 02 12:34:18.105613 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:18.105592    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-b82wj"
Nov 02 12:34:18.121107 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:18.121090    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"echo-simulator-kserve-self-signed-certs\""
Nov 02 12:34:18.135041 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:18.135010    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-b82wj"]
Nov 02 12:34:18.220110 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:18.220082    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/538a0b23-da86-458d-baf6-bb864c837a8d-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-b82wj\" (UID: \"538a0b23-da86-458d-baf6-bb864c837a8d\") " pod="llm/echo-simulator-kserve-64b6f8f96c-b82wj"
Nov 02 12:34:18.220244 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:18.220139    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85xhp\" (UniqueName: \"kubernetes.io/projected/538a0b23-da86-458d-baf6-bb864c837a8d-kube-api-access-85xhp\") pod \"echo-simulator-kserve-64b6f8f96c-b82wj\" (UID: \"538a0b23-da86-458d-baf6-bb864c837a8d\") " pod="llm/echo-simulator-kserve-64b6f8f96c-b82wj"
Nov 02 12:34:18.220306 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:18.220286    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/538a0b23-da86-458d-baf6-bb864c837a8d-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-b82wj\" (UID: \"538a0b23-da86-458d-baf6-bb864c837a8d\") " pod="llm/echo-simulator-kserve-64b6f8f96c-b82wj"
Nov 02 12:34:18.220368 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:18.220323    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/538a0b23-da86-458d-baf6-bb864c837a8d-home\") pod \"echo-simulator-kserve-64b6f8f96c-b82wj\" (UID: \"538a0b23-da86-458d-baf6-bb864c837a8d\") " pod="llm/echo-simulator-kserve-64b6f8f96c-b82wj"
Nov 02 12:34:18.220368 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:18.220347    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/538a0b23-da86-458d-baf6-bb864c837a8d-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-b82wj\" (UID: \"538a0b23-da86-458d-baf6-bb864c837a8d\") " pod="llm/echo-simulator-kserve-64b6f8f96c-b82wj"
Nov 02 12:34:18.220464 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:18.220416    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/538a0b23-da86-458d-baf6-bb864c837a8d-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-b82wj\" (UID: \"538a0b23-da86-458d-baf6-bb864c837a8d\") " pod="llm/echo-simulator-kserve-64b6f8f96c-b82wj"
Nov 02 12:34:18.321007 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:18.320976    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85xhp\" (UniqueName: \"kubernetes.io/projected/538a0b23-da86-458d-baf6-bb864c837a8d-kube-api-access-85xhp\") pod \"echo-simulator-kserve-64b6f8f96c-b82wj\" (UID: \"538a0b23-da86-458d-baf6-bb864c837a8d\") " pod="llm/echo-simulator-kserve-64b6f8f96c-b82wj"
Nov 02 12:34:18.321160 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:18.321139    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/538a0b23-da86-458d-baf6-bb864c837a8d-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-b82wj\" (UID: \"538a0b23-da86-458d-baf6-bb864c837a8d\") " pod="llm/echo-simulator-kserve-64b6f8f96c-b82wj"
Nov 02 12:34:18.321222 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:18.321179    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/538a0b23-da86-458d-baf6-bb864c837a8d-home\") pod \"echo-simulator-kserve-64b6f8f96c-b82wj\" (UID: \"538a0b23-da86-458d-baf6-bb864c837a8d\") " pod="llm/echo-simulator-kserve-64b6f8f96c-b82wj"
Nov 02 12:34:18.321222 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:18.321199    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/538a0b23-da86-458d-baf6-bb864c837a8d-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-b82wj\" (UID: \"538a0b23-da86-458d-baf6-bb864c837a8d\") " pod="llm/echo-simulator-kserve-64b6f8f96c-b82wj"
Nov 02 12:34:18.321325 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:18.321224    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/538a0b23-da86-458d-baf6-bb864c837a8d-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-b82wj\" (UID: \"538a0b23-da86-458d-baf6-bb864c837a8d\") " pod="llm/echo-simulator-kserve-64b6f8f96c-b82wj"
Nov 02 12:34:18.321325 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:18.321267    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/538a0b23-da86-458d-baf6-bb864c837a8d-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-b82wj\" (UID: \"538a0b23-da86-458d-baf6-bb864c837a8d\") " pod="llm/echo-simulator-kserve-64b6f8f96c-b82wj"
Nov 02 12:34:18.321569 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:18.321551    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/538a0b23-da86-458d-baf6-bb864c837a8d-home\") pod \"echo-simulator-kserve-64b6f8f96c-b82wj\" (UID: \"538a0b23-da86-458d-baf6-bb864c837a8d\") " pod="llm/echo-simulator-kserve-64b6f8f96c-b82wj"
Nov 02 12:34:18.321676 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:18.321572    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/538a0b23-da86-458d-baf6-bb864c837a8d-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-b82wj\" (UID: \"538a0b23-da86-458d-baf6-bb864c837a8d\") " pod="llm/echo-simulator-kserve-64b6f8f96c-b82wj"
Nov 02 12:34:18.321676 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:18.321632    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/538a0b23-da86-458d-baf6-bb864c837a8d-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-b82wj\" (UID: \"538a0b23-da86-458d-baf6-bb864c837a8d\") " pod="llm/echo-simulator-kserve-64b6f8f96c-b82wj"
Nov 02 12:34:18.323529 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:18.323509    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/538a0b23-da86-458d-baf6-bb864c837a8d-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-b82wj\" (UID: \"538a0b23-da86-458d-baf6-bb864c837a8d\") " pod="llm/echo-simulator-kserve-64b6f8f96c-b82wj"
Nov 02 12:34:18.323616 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:18.323600    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/538a0b23-da86-458d-baf6-bb864c837a8d-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-b82wj\" (UID: \"538a0b23-da86-458d-baf6-bb864c837a8d\") " pod="llm/echo-simulator-kserve-64b6f8f96c-b82wj"
Nov 02 12:34:18.347519 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:18.347499    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-85xhp\" (UniqueName: \"kubernetes.io/projected/538a0b23-da86-458d-baf6-bb864c837a8d-kube-api-access-85xhp\") pod \"echo-simulator-kserve-64b6f8f96c-b82wj\" (UID: \"538a0b23-da86-458d-baf6-bb864c837a8d\") " pod="llm/echo-simulator-kserve-64b6f8f96c-b82wj"
Nov 02 12:34:18.417986 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:18.417962    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-b82wj"
Nov 02 12:34:18.712716 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:18.712687    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-b82wj"]
Nov 02 12:34:18.712843 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:34:18.712768    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod538a0b23_da86_458d_baf6_bb864c837a8d.slice/crio-26c54d5690eb91c7fa127f7f3a8399e7c47bbfb36c275e5f8f7a4b90a5a796e4 WatchSource:0}: Error finding container 26c54d5690eb91c7fa127f7f3a8399e7c47bbfb36c275e5f8f7a4b90a5a796e4: Status 404 returned error can't find the container with id 26c54d5690eb91c7fa127f7f3a8399e7c47bbfb36c275e5f8f7a4b90a5a796e4
Nov 02 12:34:19.113398 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:19.113321    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tz6wl"
Nov 02 12:34:19.125773 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:19.125750    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tz6wl"
Nov 02 12:34:19.708913 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:19.708880    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-b82wj" event={"ID":"538a0b23-da86-458d-baf6-bb864c837a8d","Type":"ContainerStarted","Data":"26c54d5690eb91c7fa127f7f3a8399e7c47bbfb36c275e5f8f7a4b90a5a796e4"}
Nov 02 12:34:19.720121 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:19.720093    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tz6wl"
Nov 02 12:34:20.007314 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:20.007233    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"]
Nov 02 12:34:20.720467 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:20.720424    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-b82wj" event={"ID":"538a0b23-da86-458d-baf6-bb864c837a8d","Type":"ContainerStarted","Data":"521f80ee6bbec88b86b236e2b833707298243e4454bda140fb7ee857547597bc"}
Nov 02 12:34:20.768810 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:20.768751    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/echo-simulator-kserve-64b6f8f96c-b82wj" podStartSLOduration=1.236480566 podStartE2EDuration="2.768730023s" totalImagesPullingTime="1.532249457s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:34:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:34:20.767028879 +0000 UTC m=+457.022427975" watchObservedRunningTime="2026-11-02 12:34:20.768730023 +0000 UTC m=+457.024129122"
Nov 02 12:34:21.418675 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:21.418624    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/echo-simulator-kserve-64b6f8f96c-b82wj"
Nov 02 12:34:23.153650 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:23.153597    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-9frnx"
Nov 02 12:34:23.167205 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:23.167178    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-9frnx"
Nov 02 12:34:23.743888 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:23.743862    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-9frnx"
Nov 02 12:34:25.301257 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:25.301223    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"]
Nov 02 12:34:28.418863 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:28.418827    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/echo-simulator-kserve-64b6f8f96c-b82wj"
Nov 02 12:34:28.431867 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:28.431841    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/echo-simulator-kserve-64b6f8f96c-b82wj"
Nov 02 12:34:28.765762 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:28.765670    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/echo-simulator-kserve-64b6f8f96c-b82wj"
Nov 02 12:34:35.822989 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:35.822948    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-66rd4"]
Nov 02 12:34:35.826429 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:35.826407    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-66rd4"
Nov 02 12:34:35.854880 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:35.854849    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-2nvrx\""
Nov 02 12:34:35.859277 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:35.859251    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e09c5fe2-c1dd-47ab-8464-facefd253549-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-66rd4\" (UID: \"e09c5fe2-c1dd-47ab-8464-facefd253549\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-66rd4"
Nov 02 12:34:35.859360 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:35.859298    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e09c5fe2-c1dd-47ab-8464-facefd253549-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-66rd4\" (UID: \"e09c5fe2-c1dd-47ab-8464-facefd253549\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-66rd4"
Nov 02 12:34:35.859360 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:35.859324    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e09c5fe2-c1dd-47ab-8464-facefd253549-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-66rd4\" (UID: \"e09c5fe2-c1dd-47ab-8464-facefd253549\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-66rd4"
Nov 02 12:34:35.859507 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:35.859453    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e09c5fe2-c1dd-47ab-8464-facefd253549-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-66rd4\" (UID: \"e09c5fe2-c1dd-47ab-8464-facefd253549\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-66rd4"
Nov 02 12:34:35.859625 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:35.859516    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e09c5fe2-c1dd-47ab-8464-facefd253549-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-66rd4\" (UID: \"e09c5fe2-c1dd-47ab-8464-facefd253549\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-66rd4"
Nov 02 12:34:35.859625 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:35.859544    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e09c5fe2-c1dd-47ab-8464-facefd253549-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-66rd4\" (UID: \"e09c5fe2-c1dd-47ab-8464-facefd253549\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-66rd4"
Nov 02 12:34:35.859625 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:35.859567    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e09c5fe2-c1dd-47ab-8464-facefd253549-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-66rd4\" (UID: \"e09c5fe2-c1dd-47ab-8464-facefd253549\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-66rd4"
Nov 02 12:34:35.859625 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:35.859623    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwnnv\" (UniqueName: \"kubernetes.io/projected/e09c5fe2-c1dd-47ab-8464-facefd253549-kube-api-access-dwnnv\") pod \"maas-default-gateway-openshift-default-5bc884dd87-66rd4\" (UID: \"e09c5fe2-c1dd-47ab-8464-facefd253549\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-66rd4"
Nov 02 12:34:35.859843 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:35.859738    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e09c5fe2-c1dd-47ab-8464-facefd253549-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-66rd4\" (UID: \"e09c5fe2-c1dd-47ab-8464-facefd253549\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-66rd4"
Nov 02 12:34:35.863241 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:35.863217    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-66rd4"]
Nov 02 12:34:35.961144 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:35.961102    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e09c5fe2-c1dd-47ab-8464-facefd253549-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-66rd4\" (UID: \"e09c5fe2-c1dd-47ab-8464-facefd253549\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-66rd4"
Nov 02 12:34:35.961144 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:35.961145    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e09c5fe2-c1dd-47ab-8464-facefd253549-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-66rd4\" (UID: \"e09c5fe2-c1dd-47ab-8464-facefd253549\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-66rd4"
Nov 02 12:34:35.961406 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:35.961185    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e09c5fe2-c1dd-47ab-8464-facefd253549-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-66rd4\" (UID: \"e09c5fe2-c1dd-47ab-8464-facefd253549\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-66rd4"
Nov 02 12:34:35.961406 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:35.961241    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e09c5fe2-c1dd-47ab-8464-facefd253549-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-66rd4\" (UID: \"e09c5fe2-c1dd-47ab-8464-facefd253549\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-66rd4"
Nov 02 12:34:35.961406 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:35.961270    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e09c5fe2-c1dd-47ab-8464-facefd253549-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-66rd4\" (UID: \"e09c5fe2-c1dd-47ab-8464-facefd253549\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-66rd4"
Nov 02 12:34:35.961563 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:35.961407    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e09c5fe2-c1dd-47ab-8464-facefd253549-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-66rd4\" (UID: \"e09c5fe2-c1dd-47ab-8464-facefd253549\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-66rd4"
Nov 02 12:34:35.961563 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:35.961462    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwnnv\" (UniqueName: \"kubernetes.io/projected/e09c5fe2-c1dd-47ab-8464-facefd253549-kube-api-access-dwnnv\") pod \"maas-default-gateway-openshift-default-5bc884dd87-66rd4\" (UID: \"e09c5fe2-c1dd-47ab-8464-facefd253549\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-66rd4"
Nov 02 12:34:35.961563 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:35.961500    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e09c5fe2-c1dd-47ab-8464-facefd253549-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-66rd4\" (UID: \"e09c5fe2-c1dd-47ab-8464-facefd253549\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-66rd4"
Nov 02 12:34:35.961749 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:35.961559    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e09c5fe2-c1dd-47ab-8464-facefd253549-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-66rd4\" (UID: \"e09c5fe2-c1dd-47ab-8464-facefd253549\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-66rd4"
Nov 02 12:34:35.961749 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:35.961583    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e09c5fe2-c1dd-47ab-8464-facefd253549-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-66rd4\" (UID: \"e09c5fe2-c1dd-47ab-8464-facefd253549\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-66rd4"
Nov 02 12:34:35.961892 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:35.961874    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e09c5fe2-c1dd-47ab-8464-facefd253549-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-66rd4\" (UID: \"e09c5fe2-c1dd-47ab-8464-facefd253549\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-66rd4"
Nov 02 12:34:35.961961 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:35.961921    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e09c5fe2-c1dd-47ab-8464-facefd253549-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-66rd4\" (UID: \"e09c5fe2-c1dd-47ab-8464-facefd253549\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-66rd4"
Nov 02 12:34:35.962119 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:35.962098    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e09c5fe2-c1dd-47ab-8464-facefd253549-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-66rd4\" (UID: \"e09c5fe2-c1dd-47ab-8464-facefd253549\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-66rd4"
Nov 02 12:34:35.962350 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:35.962332    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e09c5fe2-c1dd-47ab-8464-facefd253549-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-66rd4\" (UID: \"e09c5fe2-c1dd-47ab-8464-facefd253549\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-66rd4"
Nov 02 12:34:35.963770 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:35.963742    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e09c5fe2-c1dd-47ab-8464-facefd253549-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-66rd4\" (UID: \"e09c5fe2-c1dd-47ab-8464-facefd253549\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-66rd4"
Nov 02 12:34:35.963899 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:35.963849    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e09c5fe2-c1dd-47ab-8464-facefd253549-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-66rd4\" (UID: \"e09c5fe2-c1dd-47ab-8464-facefd253549\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-66rd4"
Nov 02 12:34:35.990363 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:35.990322    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e09c5fe2-c1dd-47ab-8464-facefd253549-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-66rd4\" (UID: \"e09c5fe2-c1dd-47ab-8464-facefd253549\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-66rd4"
Nov 02 12:34:36.002623 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:36.002587    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwnnv\" (UniqueName: \"kubernetes.io/projected/e09c5fe2-c1dd-47ab-8464-facefd253549-kube-api-access-dwnnv\") pod \"maas-default-gateway-openshift-default-5bc884dd87-66rd4\" (UID: \"e09c5fe2-c1dd-47ab-8464-facefd253549\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-66rd4"
Nov 02 12:34:36.139325 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:36.139232    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-66rd4"
Nov 02 12:34:36.452190 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:36.452156    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-66rd4"]
Nov 02 12:34:36.452604 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:34:36.452577    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode09c5fe2_c1dd_47ab_8464_facefd253549.slice/crio-33a9b2bad3363c7295f82299f896d52d18589f35bc39f80de66de1b619fbd4c9 WatchSource:0}: Error finding container 33a9b2bad3363c7295f82299f896d52d18589f35bc39f80de66de1b619fbd4c9: Status 404 returned error can't find the container with id 33a9b2bad3363c7295f82299f896d52d18589f35bc39f80de66de1b619fbd4c9
Nov 02 12:34:36.790305 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:36.790217    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-66rd4" event={"ID":"e09c5fe2-c1dd-47ab-8464-facefd253549","Type":"ContainerStarted","Data":"33a9b2bad3363c7295f82299f896d52d18589f35bc39f80de66de1b619fbd4c9"}
Nov 02 12:34:39.222627 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:39.222592    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:34:39.222939 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:39.222683    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:34:39.222939 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:39.222715    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:34:39.811944 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:39.811911    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-66rd4" event={"ID":"e09c5fe2-c1dd-47ab-8464-facefd253549","Type":"ContainerStarted","Data":"01cb3f6d6d9046ecf876e009e0c30e94347a355c0df84d6ddcc2a15171f82c94"}
Nov 02 12:34:39.873408 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:39.873355    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-66rd4" podStartSLOduration=2.106200461 podStartE2EDuration="4.873342495s" totalImagesPullingTime="2.767142034s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:34:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:34:39.872756352 +0000 UTC m=+476.128155476" watchObservedRunningTime="2026-11-02 12:34:39.873342495 +0000 UTC m=+476.128741591"
Nov 02 12:34:40.139431 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:40.139353    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-66rd4"
Nov 02 12:34:40.140877 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:40.140848    4559 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-66rd4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Nov 02 12:34:40.141022 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:40.140895    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-66rd4" podUID="e09c5fe2-c1dd-47ab-8464-facefd253549" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Nov 02 12:34:41.140408 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:41.140374    4559 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-66rd4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Nov 02 12:34:41.140849 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:41.140440    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-66rd4" podUID="e09c5fe2-c1dd-47ab-8464-facefd253549" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Nov 02 12:34:41.258508 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:41.258477    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6768bb47c6-q87t7"]
Nov 02 12:34:41.262321 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:41.262300    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6768bb47c6-q87t7"
Nov 02 12:34:41.282595 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:41.282572    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6768bb47c6-q87t7"]
Nov 02 12:34:41.315011 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:41.314981    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a00817af-1ee2-4bad-b007-b058928a12b7-tls-cert\") pod \"authorino-6768bb47c6-q87t7\" (UID: \"a00817af-1ee2-4bad-b007-b058928a12b7\") " pod="kuadrant-system/authorino-6768bb47c6-q87t7"
Nov 02 12:34:41.315209 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:41.315050    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t55gr\" (UniqueName: \"kubernetes.io/projected/a00817af-1ee2-4bad-b007-b058928a12b7-kube-api-access-t55gr\") pod \"authorino-6768bb47c6-q87t7\" (UID: \"a00817af-1ee2-4bad-b007-b058928a12b7\") " pod="kuadrant-system/authorino-6768bb47c6-q87t7"
Nov 02 12:34:41.415659 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:41.415545    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a00817af-1ee2-4bad-b007-b058928a12b7-tls-cert\") pod \"authorino-6768bb47c6-q87t7\" (UID: \"a00817af-1ee2-4bad-b007-b058928a12b7\") " pod="kuadrant-system/authorino-6768bb47c6-q87t7"
Nov 02 12:34:41.415659 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:41.415623    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t55gr\" (UniqueName: \"kubernetes.io/projected/a00817af-1ee2-4bad-b007-b058928a12b7-kube-api-access-t55gr\") pod \"authorino-6768bb47c6-q87t7\" (UID: \"a00817af-1ee2-4bad-b007-b058928a12b7\") " pod="kuadrant-system/authorino-6768bb47c6-q87t7"
Nov 02 12:34:41.417989 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:41.417966    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a00817af-1ee2-4bad-b007-b058928a12b7-tls-cert\") pod \"authorino-6768bb47c6-q87t7\" (UID: \"a00817af-1ee2-4bad-b007-b058928a12b7\") " pod="kuadrant-system/authorino-6768bb47c6-q87t7"
Nov 02 12:34:41.440164 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:41.440132    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t55gr\" (UniqueName: \"kubernetes.io/projected/a00817af-1ee2-4bad-b007-b058928a12b7-kube-api-access-t55gr\") pod \"authorino-6768bb47c6-q87t7\" (UID: \"a00817af-1ee2-4bad-b007-b058928a12b7\") " pod="kuadrant-system/authorino-6768bb47c6-q87t7"
Nov 02 12:34:41.574093 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:41.574057    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6768bb47c6-q87t7"
Nov 02 12:34:41.916098 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:41.916071    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6768bb47c6-q87t7"]
Nov 02 12:34:41.917697 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:34:41.917670    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda00817af_1ee2_4bad_b007_b058928a12b7.slice/crio-3a3c8cf66c0e9a0de089621d50343f11661c6748bc32e38f42bc065b3d7809c3 WatchSource:0}: Error finding container 3a3c8cf66c0e9a0de089621d50343f11661c6748bc32e38f42bc065b3d7809c3: Status 404 returned error can't find the container with id 3a3c8cf66c0e9a0de089621d50343f11661c6748bc32e38f42bc065b3d7809c3
Nov 02 12:34:42.139845 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:42.139806    4559 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-66rd4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Nov 02 12:34:42.140031 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:42.139864    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-66rd4" podUID="e09c5fe2-c1dd-47ab-8464-facefd253549" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Nov 02 12:34:42.826994 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:42.826859    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6768bb47c6-q87t7" event={"ID":"a00817af-1ee2-4bad-b007-b058928a12b7","Type":"ContainerStarted","Data":"31b8fba461b6766f2f293dc8bd6bc34f59c4d8f82adae11857718f0a63c89e92"}
Nov 02 12:34:42.826994 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:42.826945    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6768bb47c6-q87t7" event={"ID":"a00817af-1ee2-4bad-b007-b058928a12b7","Type":"ContainerStarted","Data":"3a3c8cf66c0e9a0de089621d50343f11661c6748bc32e38f42bc065b3d7809c3"}
Nov 02 12:34:42.877456 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:42.877395    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-6768bb47c6-q87t7" podStartSLOduration=1.268074739 podStartE2EDuration="1.877381028s" totalImagesPullingTime="609.306289ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:34:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:34:42.875795753 +0000 UTC m=+479.131194848" watchObservedRunningTime="2026-11-02 12:34:42.877381028 +0000 UTC m=+479.132780138"
Nov 02 12:34:42.917700 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:42.916482    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f544d9f9-84fxt"]
Nov 02 12:34:42.917700 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:42.916765    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-57f544d9f9-84fxt" podUID="b8f8b8cd-64a0-457d-95f3-c8f2940cd9a5" containerName="authorino" containerID="cri-o://8a5ab7b173f2b39d12ab0c6a30c9d90a95abbab926b6a4203e90b1c014554b6b" gracePeriod=30
Nov 02 12:34:42.931995 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:42.931957    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"]
Nov 02 12:34:43.140245 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:43.139774    4559 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-66rd4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Nov 02 12:34:43.140245 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:43.139852    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-66rd4" podUID="e09c5fe2-c1dd-47ab-8464-facefd253549" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Nov 02 12:34:43.220511 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:43.220480    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f544d9f9-84fxt"
Nov 02 12:34:43.236150 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:43.236066    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8f8b8cd-64a0-457d-95f3-c8f2940cd9a5-kube-api-access-gwfzd\" (UniqueName: \"kubernetes.io/projected/b8f8b8cd-64a0-457d-95f3-c8f2940cd9a5-kube-api-access-gwfzd\") pod \"b8f8b8cd-64a0-457d-95f3-c8f2940cd9a5\" (UID: \"b8f8b8cd-64a0-457d-95f3-c8f2940cd9a5\") "
Nov 02 12:34:43.236150 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:43.236142    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b8f8b8cd-64a0-457d-95f3-c8f2940cd9a5-tls-cert\" (UniqueName: \"kubernetes.io/secret/b8f8b8cd-64a0-457d-95f3-c8f2940cd9a5-tls-cert\") pod \"b8f8b8cd-64a0-457d-95f3-c8f2940cd9a5\" (UID: \"b8f8b8cd-64a0-457d-95f3-c8f2940cd9a5\") "
Nov 02 12:34:43.242097 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:43.238741    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8f8b8cd-64a0-457d-95f3-c8f2940cd9a5-kube-api-access-gwfzd" pod "b8f8b8cd-64a0-457d-95f3-c8f2940cd9a5" (UID: "b8f8b8cd-64a0-457d-95f3-c8f2940cd9a5"). InnerVolumeSpecName "kube-api-access-gwfzd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:34:43.257256 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:43.257215    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8f8b8cd-64a0-457d-95f3-c8f2940cd9a5-tls-cert" pod "b8f8b8cd-64a0-457d-95f3-c8f2940cd9a5" (UID: "b8f8b8cd-64a0-457d-95f3-c8f2940cd9a5"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:34:43.337860 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:43.337819    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gwfzd\" (UniqueName: \"kubernetes.io/projected/b8f8b8cd-64a0-457d-95f3-c8f2940cd9a5-kube-api-access-gwfzd\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:34:43.337860 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:43.337853    4559 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b8f8b8cd-64a0-457d-95f3-c8f2940cd9a5-tls-cert\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:34:43.833115 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:43.833031    4559 generic.go:356] "Generic (PLEG): container finished" podID="b8f8b8cd-64a0-457d-95f3-c8f2940cd9a5" containerID="8a5ab7b173f2b39d12ab0c6a30c9d90a95abbab926b6a4203e90b1c014554b6b" exitCode=0
Nov 02 12:34:43.833115 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:43.833081    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f544d9f9-84fxt"
Nov 02 12:34:43.833115 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:43.833113    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f544d9f9-84fxt" event={"ID":"b8f8b8cd-64a0-457d-95f3-c8f2940cd9a5","Type":"ContainerDied","Data":"8a5ab7b173f2b39d12ab0c6a30c9d90a95abbab926b6a4203e90b1c014554b6b"}
Nov 02 12:34:43.833632 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:43.833144    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f544d9f9-84fxt" event={"ID":"b8f8b8cd-64a0-457d-95f3-c8f2940cd9a5","Type":"ContainerDied","Data":"591c7040edc86764592d3ea4451f5c4af5471fe8c6e5628bc2523932dab6a95f"}
Nov 02 12:34:43.833632 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:43.833160    4559 scope.go:122] "RemoveContainer" containerID="8a5ab7b173f2b39d12ab0c6a30c9d90a95abbab926b6a4203e90b1c014554b6b"
Nov 02 12:34:43.842699 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:43.842674    4559 scope.go:122] "RemoveContainer" containerID="8a5ab7b173f2b39d12ab0c6a30c9d90a95abbab926b6a4203e90b1c014554b6b"
Nov 02 12:34:43.843021 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:34:43.842995    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a5ab7b173f2b39d12ab0c6a30c9d90a95abbab926b6a4203e90b1c014554b6b\": container with ID starting with 8a5ab7b173f2b39d12ab0c6a30c9d90a95abbab926b6a4203e90b1c014554b6b not found: ID does not exist" containerID="8a5ab7b173f2b39d12ab0c6a30c9d90a95abbab926b6a4203e90b1c014554b6b"
Nov 02 12:34:43.843107 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:43.843037    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a5ab7b173f2b39d12ab0c6a30c9d90a95abbab926b6a4203e90b1c014554b6b"} err="failed to get container status \"8a5ab7b173f2b39d12ab0c6a30c9d90a95abbab926b6a4203e90b1c014554b6b\": rpc error: code = NotFound desc = could not find container \"8a5ab7b173f2b39d12ab0c6a30c9d90a95abbab926b6a4203e90b1c014554b6b\": container with ID starting with 8a5ab7b173f2b39d12ab0c6a30c9d90a95abbab926b6a4203e90b1c014554b6b not found: ID does not exist"
Nov 02 12:34:43.915046 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:43.915012    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f544d9f9-84fxt"]
Nov 02 12:34:43.920942 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:43.920914    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-57f544d9f9-84fxt"]
Nov 02 12:34:44.140505 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:44.140421    4559 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-66rd4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Nov 02 12:34:44.140505 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:44.140477    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-66rd4" podUID="e09c5fe2-c1dd-47ab-8464-facefd253549" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Nov 02 12:34:45.103150 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:34:45.103109    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqcmj" containerName="extractor"
Nov 02 12:34:45.139965 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:45.139920    4559 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-66rd4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Nov 02 12:34:45.140148 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:45.139999    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-66rd4" podUID="e09c5fe2-c1dd-47ab-8464-facefd253549" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Nov 02 12:34:45.893585 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:45.893549    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8f8b8cd-64a0-457d-95f3-c8f2940cd9a5" path="/var/lib/kubelet/pods/b8f8b8cd-64a0-457d-95f3-c8f2940cd9a5/volumes"
Nov 02 12:34:46.140213 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:46.140171    4559 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-66rd4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Nov 02 12:34:46.140826 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:46.140221    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-66rd4" podUID="e09c5fe2-c1dd-47ab-8464-facefd253549" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Nov 02 12:34:48.140715 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:48.140668    4559 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-66rd4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 12:34:48.141239 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:48.140751    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-66rd4" podUID="e09c5fe2-c1dd-47ab-8464-facefd253549" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:34:49.181807 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:49.181760    4559 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-66rd4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 12:34:49.182209 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:49.181889    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-66rd4" podUID="e09c5fe2-c1dd-47ab-8464-facefd253549" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:34:49.984413 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:49.984369    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-66rd4"
Nov 02 12:34:49.984687 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:49.984664    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-66rd4"
Nov 02 12:34:49.985539 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:49.985515    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-66rd4"
Nov 02 12:34:50.870990 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:50.870958    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x9hwg"]
Nov 02 12:34:50.871625 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:50.871458    4559 cpu_manager.go:414] "containerMap: removing container" podUID="b8f8b8cd-64a0-457d-95f3-c8f2940cd9a5" containerName="authorino"
Nov 02 12:34:50.871625 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:50.871474    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8f8b8cd-64a0-457d-95f3-c8f2940cd9a5" containerName="authorino"
Nov 02 12:34:50.871625 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:50.871593    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8f8b8cd-64a0-457d-95f3-c8f2940cd9a5" containerName="authorino"
Nov 02 12:34:50.878729 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:50.878678    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x9hwg"
Nov 02 12:34:50.901532 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:50.901510    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x9hwg"]
Nov 02 12:34:50.911423 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:50.911401    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tz7qn"]
Nov 02 12:34:50.914893 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:50.914878    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tz7qn"
Nov 02 12:34:50.921498 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:50.921469    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eec5b49d-6cf5-4203-b386-9f9a5fa6c7fa-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-x9hwg\" (UID: \"eec5b49d-6cf5-4203-b386-9f9a5fa6c7fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x9hwg"
Nov 02 12:34:50.921594 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:50.921509    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eec5b49d-6cf5-4203-b386-9f9a5fa6c7fa-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-x9hwg\" (UID: \"eec5b49d-6cf5-4203-b386-9f9a5fa6c7fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x9hwg"
Nov 02 12:34:50.921594 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:50.921554    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eec5b49d-6cf5-4203-b386-9f9a5fa6c7fa-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-x9hwg\" (UID: \"eec5b49d-6cf5-4203-b386-9f9a5fa6c7fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x9hwg"
Nov 02 12:34:50.921865 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:50.921848    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlrqm\" (UniqueName: \"kubernetes.io/projected/eec5b49d-6cf5-4203-b386-9f9a5fa6c7fa-kube-api-access-dlrqm\") pod \"maas-default-gateway-openshift-default-5bc884dd87-x9hwg\" (UID: \"eec5b49d-6cf5-4203-b386-9f9a5fa6c7fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x9hwg"
Nov 02 12:34:50.921917 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:50.921888    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eec5b49d-6cf5-4203-b386-9f9a5fa6c7fa-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-x9hwg\" (UID: \"eec5b49d-6cf5-4203-b386-9f9a5fa6c7fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x9hwg"
Nov 02 12:34:50.921989 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:50.921967    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eec5b49d-6cf5-4203-b386-9f9a5fa6c7fa-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-x9hwg\" (UID: \"eec5b49d-6cf5-4203-b386-9f9a5fa6c7fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x9hwg"
Nov 02 12:34:50.922117 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:50.921997    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eec5b49d-6cf5-4203-b386-9f9a5fa6c7fa-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-x9hwg\" (UID: \"eec5b49d-6cf5-4203-b386-9f9a5fa6c7fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x9hwg"
Nov 02 12:34:50.922117 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:50.922014    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eec5b49d-6cf5-4203-b386-9f9a5fa6c7fa-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-x9hwg\" (UID: \"eec5b49d-6cf5-4203-b386-9f9a5fa6c7fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x9hwg"
Nov 02 12:34:50.922117 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:50.922031    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eec5b49d-6cf5-4203-b386-9f9a5fa6c7fa-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-x9hwg\" (UID: \"eec5b49d-6cf5-4203-b386-9f9a5fa6c7fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x9hwg"
Nov 02 12:34:50.939537 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:50.939494    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tz7qn"]
Nov 02 12:34:51.023382 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:51.023353    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eec5b49d-6cf5-4203-b386-9f9a5fa6c7fa-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-x9hwg\" (UID: \"eec5b49d-6cf5-4203-b386-9f9a5fa6c7fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x9hwg"
Nov 02 12:34:51.023538 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:51.023406    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eec5b49d-6cf5-4203-b386-9f9a5fa6c7fa-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-x9hwg\" (UID: \"eec5b49d-6cf5-4203-b386-9f9a5fa6c7fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x9hwg"
Nov 02 12:34:51.023538 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:51.023427    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eec5b49d-6cf5-4203-b386-9f9a5fa6c7fa-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-x9hwg\" (UID: \"eec5b49d-6cf5-4203-b386-9f9a5fa6c7fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x9hwg"
Nov 02 12:34:51.023538 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:51.023448    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b98s2\" (UniqueName: \"kubernetes.io/projected/597fdf69-8d19-46b2-b61f-bbe2ec731e61-kube-api-access-b98s2\") pod \"maas-default-gateway-openshift-default-5bc884dd87-tz7qn\" (UID: \"597fdf69-8d19-46b2-b61f-bbe2ec731e61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tz7qn"
Nov 02 12:34:51.023538 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:51.023493    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlrqm\" (UniqueName: \"kubernetes.io/projected/eec5b49d-6cf5-4203-b386-9f9a5fa6c7fa-kube-api-access-dlrqm\") pod \"maas-default-gateway-openshift-default-5bc884dd87-x9hwg\" (UID: \"eec5b49d-6cf5-4203-b386-9f9a5fa6c7fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x9hwg"
Nov 02 12:34:51.023538 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:51.023512    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/597fdf69-8d19-46b2-b61f-bbe2ec731e61-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-tz7qn\" (UID: \"597fdf69-8d19-46b2-b61f-bbe2ec731e61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tz7qn"
Nov 02 12:34:51.023872 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:51.023613    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eec5b49d-6cf5-4203-b386-9f9a5fa6c7fa-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-x9hwg\" (UID: \"eec5b49d-6cf5-4203-b386-9f9a5fa6c7fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x9hwg"
Nov 02 12:34:51.023872 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:51.023715    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/597fdf69-8d19-46b2-b61f-bbe2ec731e61-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-tz7qn\" (UID: \"597fdf69-8d19-46b2-b61f-bbe2ec731e61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tz7qn"
Nov 02 12:34:51.023872 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:51.023743    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/597fdf69-8d19-46b2-b61f-bbe2ec731e61-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-tz7qn\" (UID: \"597fdf69-8d19-46b2-b61f-bbe2ec731e61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tz7qn"
Nov 02 12:34:51.023872 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:51.023821    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eec5b49d-6cf5-4203-b386-9f9a5fa6c7fa-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-x9hwg\" (UID: \"eec5b49d-6cf5-4203-b386-9f9a5fa6c7fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x9hwg"
Nov 02 12:34:51.024080 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:51.023950    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eec5b49d-6cf5-4203-b386-9f9a5fa6c7fa-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-x9hwg\" (UID: \"eec5b49d-6cf5-4203-b386-9f9a5fa6c7fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x9hwg"
Nov 02 12:34:51.024080 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:51.023985    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eec5b49d-6cf5-4203-b386-9f9a5fa6c7fa-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-x9hwg\" (UID: \"eec5b49d-6cf5-4203-b386-9f9a5fa6c7fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x9hwg"
Nov 02 12:34:51.024080 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:51.024012    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eec5b49d-6cf5-4203-b386-9f9a5fa6c7fa-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-x9hwg\" (UID: \"eec5b49d-6cf5-4203-b386-9f9a5fa6c7fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x9hwg"
Nov 02 12:34:51.024080 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:51.024037    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eec5b49d-6cf5-4203-b386-9f9a5fa6c7fa-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-x9hwg\" (UID: \"eec5b49d-6cf5-4203-b386-9f9a5fa6c7fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x9hwg"
Nov 02 12:34:51.024080 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:51.023957    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eec5b49d-6cf5-4203-b386-9f9a5fa6c7fa-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-x9hwg\" (UID: \"eec5b49d-6cf5-4203-b386-9f9a5fa6c7fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x9hwg"
Nov 02 12:34:51.024299 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:51.024102    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/597fdf69-8d19-46b2-b61f-bbe2ec731e61-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-tz7qn\" (UID: \"597fdf69-8d19-46b2-b61f-bbe2ec731e61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tz7qn"
Nov 02 12:34:51.024299 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:51.024163    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/597fdf69-8d19-46b2-b61f-bbe2ec731e61-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-tz7qn\" (UID: \"597fdf69-8d19-46b2-b61f-bbe2ec731e61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tz7qn"
Nov 02 12:34:51.024299 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:51.024192    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/597fdf69-8d19-46b2-b61f-bbe2ec731e61-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-tz7qn\" (UID: \"597fdf69-8d19-46b2-b61f-bbe2ec731e61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tz7qn"
Nov 02 12:34:51.024299 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:51.024220    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eec5b49d-6cf5-4203-b386-9f9a5fa6c7fa-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-x9hwg\" (UID: \"eec5b49d-6cf5-4203-b386-9f9a5fa6c7fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x9hwg"
Nov 02 12:34:51.024299 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:51.024294    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eec5b49d-6cf5-4203-b386-9f9a5fa6c7fa-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-x9hwg\" (UID: \"eec5b49d-6cf5-4203-b386-9f9a5fa6c7fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x9hwg"
Nov 02 12:34:51.024527 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:51.024312    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/597fdf69-8d19-46b2-b61f-bbe2ec731e61-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-tz7qn\" (UID: \"597fdf69-8d19-46b2-b61f-bbe2ec731e61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tz7qn"
Nov 02 12:34:51.024527 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:51.024360    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/597fdf69-8d19-46b2-b61f-bbe2ec731e61-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-tz7qn\" (UID: \"597fdf69-8d19-46b2-b61f-bbe2ec731e61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tz7qn"
Nov 02 12:34:51.024527 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:51.024365    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eec5b49d-6cf5-4203-b386-9f9a5fa6c7fa-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-x9hwg\" (UID: \"eec5b49d-6cf5-4203-b386-9f9a5fa6c7fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x9hwg"
Nov 02 12:34:51.026070 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:51.026048    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eec5b49d-6cf5-4203-b386-9f9a5fa6c7fa-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-x9hwg\" (UID: \"eec5b49d-6cf5-4203-b386-9f9a5fa6c7fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x9hwg"
Nov 02 12:34:51.026346 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:51.026328    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eec5b49d-6cf5-4203-b386-9f9a5fa6c7fa-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-x9hwg\" (UID: \"eec5b49d-6cf5-4203-b386-9f9a5fa6c7fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x9hwg"
Nov 02 12:34:51.047768 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:51.047741    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eec5b49d-6cf5-4203-b386-9f9a5fa6c7fa-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-x9hwg\" (UID: \"eec5b49d-6cf5-4203-b386-9f9a5fa6c7fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x9hwg"
Nov 02 12:34:51.047888 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:51.047835    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlrqm\" (UniqueName: \"kubernetes.io/projected/eec5b49d-6cf5-4203-b386-9f9a5fa6c7fa-kube-api-access-dlrqm\") pod \"maas-default-gateway-openshift-default-5bc884dd87-x9hwg\" (UID: \"eec5b49d-6cf5-4203-b386-9f9a5fa6c7fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x9hwg"
Nov 02 12:34:51.125288 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:51.125207    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/597fdf69-8d19-46b2-b61f-bbe2ec731e61-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-tz7qn\" (UID: \"597fdf69-8d19-46b2-b61f-bbe2ec731e61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tz7qn"
Nov 02 12:34:51.125288 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:51.125259    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/597fdf69-8d19-46b2-b61f-bbe2ec731e61-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-tz7qn\" (UID: \"597fdf69-8d19-46b2-b61f-bbe2ec731e61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tz7qn"
Nov 02 12:34:51.125288 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:51.125278    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/597fdf69-8d19-46b2-b61f-bbe2ec731e61-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-tz7qn\" (UID: \"597fdf69-8d19-46b2-b61f-bbe2ec731e61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tz7qn"
Nov 02 12:34:51.125579 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:51.125303    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/597fdf69-8d19-46b2-b61f-bbe2ec731e61-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-tz7qn\" (UID: \"597fdf69-8d19-46b2-b61f-bbe2ec731e61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tz7qn"
Nov 02 12:34:51.125579 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:51.125327    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/597fdf69-8d19-46b2-b61f-bbe2ec731e61-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-tz7qn\" (UID: \"597fdf69-8d19-46b2-b61f-bbe2ec731e61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tz7qn"
Nov 02 12:34:51.125579 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:51.125370    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b98s2\" (UniqueName: \"kubernetes.io/projected/597fdf69-8d19-46b2-b61f-bbe2ec731e61-kube-api-access-b98s2\") pod \"maas-default-gateway-openshift-default-5bc884dd87-tz7qn\" (UID: \"597fdf69-8d19-46b2-b61f-bbe2ec731e61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tz7qn"
Nov 02 12:34:51.125579 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:51.125427    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/597fdf69-8d19-46b2-b61f-bbe2ec731e61-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-tz7qn\" (UID: \"597fdf69-8d19-46b2-b61f-bbe2ec731e61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tz7qn"
Nov 02 12:34:51.125579 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:51.125483    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/597fdf69-8d19-46b2-b61f-bbe2ec731e61-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-tz7qn\" (UID: \"597fdf69-8d19-46b2-b61f-bbe2ec731e61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tz7qn"
Nov 02 12:34:51.125579 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:51.125504    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/597fdf69-8d19-46b2-b61f-bbe2ec731e61-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-tz7qn\" (UID: \"597fdf69-8d19-46b2-b61f-bbe2ec731e61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tz7qn"
Nov 02 12:34:51.125883 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:51.125654    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/597fdf69-8d19-46b2-b61f-bbe2ec731e61-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-tz7qn\" (UID: \"597fdf69-8d19-46b2-b61f-bbe2ec731e61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tz7qn"
Nov 02 12:34:51.125883 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:51.125709    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/597fdf69-8d19-46b2-b61f-bbe2ec731e61-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-tz7qn\" (UID: \"597fdf69-8d19-46b2-b61f-bbe2ec731e61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tz7qn"
Nov 02 12:34:51.125883 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:51.125773    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/597fdf69-8d19-46b2-b61f-bbe2ec731e61-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-tz7qn\" (UID: \"597fdf69-8d19-46b2-b61f-bbe2ec731e61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tz7qn"
Nov 02 12:34:51.125990 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:51.125941    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/597fdf69-8d19-46b2-b61f-bbe2ec731e61-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-tz7qn\" (UID: \"597fdf69-8d19-46b2-b61f-bbe2ec731e61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tz7qn"
Nov 02 12:34:51.126131 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:51.126114    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/597fdf69-8d19-46b2-b61f-bbe2ec731e61-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-tz7qn\" (UID: \"597fdf69-8d19-46b2-b61f-bbe2ec731e61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tz7qn"
Nov 02 12:34:51.127601 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:51.127574    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/597fdf69-8d19-46b2-b61f-bbe2ec731e61-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-tz7qn\" (UID: \"597fdf69-8d19-46b2-b61f-bbe2ec731e61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tz7qn"
Nov 02 12:34:51.127721 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:51.127694    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/597fdf69-8d19-46b2-b61f-bbe2ec731e61-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-tz7qn\" (UID: \"597fdf69-8d19-46b2-b61f-bbe2ec731e61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tz7qn"
Nov 02 12:34:51.149417 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:51.149393    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/597fdf69-8d19-46b2-b61f-bbe2ec731e61-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-tz7qn\" (UID: \"597fdf69-8d19-46b2-b61f-bbe2ec731e61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tz7qn"
Nov 02 12:34:51.149667 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:51.149633    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b98s2\" (UniqueName: \"kubernetes.io/projected/597fdf69-8d19-46b2-b61f-bbe2ec731e61-kube-api-access-b98s2\") pod \"maas-default-gateway-openshift-default-5bc884dd87-tz7qn\" (UID: \"597fdf69-8d19-46b2-b61f-bbe2ec731e61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tz7qn"
Nov 02 12:34:51.191524 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:51.191500    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x9hwg"
Nov 02 12:34:51.230109 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:51.230077    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tz7qn"
Nov 02 12:34:51.502812 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:51.502773    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x9hwg"]
Nov 02 12:34:51.502990 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:34:51.502968    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeec5b49d_6cf5_4203_b386_9f9a5fa6c7fa.slice/crio-97bc16107109223ef76bb78854322cf8b07d953a1da39288cb3ac0ad5bfded8d WatchSource:0}: Error finding container 97bc16107109223ef76bb78854322cf8b07d953a1da39288cb3ac0ad5bfded8d: Status 404 returned error can't find the container with id 97bc16107109223ef76bb78854322cf8b07d953a1da39288cb3ac0ad5bfded8d
Nov 02 12:34:51.505104 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:51.505071    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:34:51.505192 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:51.505134    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:34:51.505192 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:51.505162    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:34:51.541534 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:51.541503    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tz7qn"]
Nov 02 12:34:51.542184 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:34:51.542162    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod597fdf69_8d19_46b2_b61f_bbe2ec731e61.slice/crio-4235c5a2446f6e06b8e6b6ec683cb23887d6abbe707bb4bf60e549e05f72e639 WatchSource:0}: Error finding container 4235c5a2446f6e06b8e6b6ec683cb23887d6abbe707bb4bf60e549e05f72e639: Status 404 returned error can't find the container with id 4235c5a2446f6e06b8e6b6ec683cb23887d6abbe707bb4bf60e549e05f72e639
Nov 02 12:34:51.544429 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:51.544397    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:34:51.544504 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:51.544463    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:34:51.544504 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:51.544496    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:34:51.872714 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:51.872585    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tz7qn" event={"ID":"597fdf69-8d19-46b2-b61f-bbe2ec731e61","Type":"ContainerStarted","Data":"d65388fb60ebe4470e7fdd8574df2e564aca716b71e203247453ff63961a0559"}
Nov 02 12:34:51.872714 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:51.872690    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tz7qn" event={"ID":"597fdf69-8d19-46b2-b61f-bbe2ec731e61","Type":"ContainerStarted","Data":"4235c5a2446f6e06b8e6b6ec683cb23887d6abbe707bb4bf60e549e05f72e639"}
Nov 02 12:34:51.874318 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:51.874286    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x9hwg" event={"ID":"eec5b49d-6cf5-4203-b386-9f9a5fa6c7fa","Type":"ContainerStarted","Data":"a35152e4368ed7dc8d754d70c001c11136a446cf09d66af582a531290627369a"}
Nov 02 12:34:51.874447 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:51.874359    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x9hwg" event={"ID":"eec5b49d-6cf5-4203-b386-9f9a5fa6c7fa","Type":"ContainerStarted","Data":"97bc16107109223ef76bb78854322cf8b07d953a1da39288cb3ac0ad5bfded8d"}
Nov 02 12:34:51.930295 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:51.930237    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tz7qn" podStartSLOduration=1.930216641 podStartE2EDuration="1.930216641s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:34:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:34:51.929723719 +0000 UTC m=+488.185122842" watchObservedRunningTime="2026-11-02 12:34:51.930216641 +0000 UTC m=+488.185615737"
Nov 02 12:34:51.977095 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:51.977015    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x9hwg" podStartSLOduration=1.9769900009999999 podStartE2EDuration="1.976990001s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:34:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:34:51.975745077 +0000 UTC m=+488.231144173" watchObservedRunningTime="2026-11-02 12:34:51.976990001 +0000 UTC m=+488.232389123"
Nov 02 12:34:52.192664 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:52.192006    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x9hwg"
Nov 02 12:34:52.216664 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:52.215976    4559 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-x9hwg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Nov 02 12:34:52.216664 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:52.216056    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x9hwg" podUID="eec5b49d-6cf5-4203-b386-9f9a5fa6c7fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Nov 02 12:34:52.233763 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:52.233729    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tz7qn"
Nov 02 12:34:52.234433 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:52.234402    4559 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-tz7qn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Nov 02 12:34:52.234547 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:52.234463    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tz7qn" podUID="597fdf69-8d19-46b2-b61f-bbe2ec731e61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Nov 02 12:34:53.203666 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:53.198037    4559 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-x9hwg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Nov 02 12:34:53.203666 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:53.198115    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x9hwg" podUID="eec5b49d-6cf5-4203-b386-9f9a5fa6c7fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Nov 02 12:34:53.232901 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:53.232868    4559 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-tz7qn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Nov 02 12:34:53.233174 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:53.233145    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tz7qn" podUID="597fdf69-8d19-46b2-b61f-bbe2ec731e61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Nov 02 12:34:54.203662 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:54.200572    4559 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-x9hwg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Nov 02 12:34:54.203662 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:54.200655    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x9hwg" podUID="eec5b49d-6cf5-4203-b386-9f9a5fa6c7fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Nov 02 12:34:54.232360 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:54.232191    4559 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-tz7qn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Nov 02 12:34:54.232360 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:54.232276    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tz7qn" podUID="597fdf69-8d19-46b2-b61f-bbe2ec731e61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Nov 02 12:34:55.201720 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:55.199181    4559 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-x9hwg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Nov 02 12:34:55.201720 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:55.199270    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x9hwg" podUID="eec5b49d-6cf5-4203-b386-9f9a5fa6c7fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Nov 02 12:34:55.231047 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:55.231010    4559 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-tz7qn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Nov 02 12:34:55.231611 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:55.231583    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tz7qn" podUID="597fdf69-8d19-46b2-b61f-bbe2ec731e61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Nov 02 12:34:56.204662 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:56.204085    4559 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-x9hwg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Nov 02 12:34:56.204662 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:56.204151    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x9hwg" podUID="eec5b49d-6cf5-4203-b386-9f9a5fa6c7fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Nov 02 12:34:56.231662 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:56.231280    4559 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-tz7qn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Nov 02 12:34:56.231662 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:56.231354    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tz7qn" podUID="597fdf69-8d19-46b2-b61f-bbe2ec731e61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Nov 02 12:34:57.194714 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:57.194054    4559 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-x9hwg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Nov 02 12:34:57.194714 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:57.194116    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x9hwg" podUID="eec5b49d-6cf5-4203-b386-9f9a5fa6c7fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Nov 02 12:34:57.232147 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:57.232064    4559 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-tz7qn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Nov 02 12:34:57.232147 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:57.232129    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tz7qn" podUID="597fdf69-8d19-46b2-b61f-bbe2ec731e61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Nov 02 12:34:58.192584 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:58.192538    4559 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-x9hwg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Nov 02 12:34:58.192825 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:58.192592    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x9hwg" podUID="eec5b49d-6cf5-4203-b386-9f9a5fa6c7fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Nov 02 12:34:58.230924 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:58.230893    4559 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-tz7qn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Nov 02 12:34:58.231087 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:58.230945    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tz7qn" podUID="597fdf69-8d19-46b2-b61f-bbe2ec731e61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Nov 02 12:34:59.855440 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:59.855393    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tz7qn"
Nov 02 12:34:59.856260 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:59.856235    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tz7qn"
Nov 02 12:34:59.856998 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:59.856973    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tz7qn"
Nov 02 12:34:59.909307 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:59.909269    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x9hwg"
Nov 02 12:34:59.909679 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:59.909654    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x9hwg"
Nov 02 12:34:59.910415 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:34:59.910398    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x9hwg"
Nov 02 12:35:15.086143 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:35:15.086103    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqcmj" containerName="extractor"
Nov 02 12:35:23.258840 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:23.258806    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-d9dc4ff4-gw-openshift-default-9bfdcc79c-w7gqc"]
Nov 02 12:35:23.267408 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:23.267380    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-d9dc4ff4-gw-openshift-default-9bfdcc79c-w7gqc"
Nov 02 12:35:23.288331 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:23.288303    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-d9dc4ff4-gw-openshift-default-9bfdcc79c-w7gqc"]
Nov 02 12:35:23.296919 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:23.296888    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-d9dc4ff4-gw-openshift-default-dockercfg-s4qcm\""
Nov 02 12:35:23.430327 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:23.430287    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b64510b-aa47-4a38-ac26-7c0df79188b2-workload-certs\") pod \"e2e-migrate-d9dc4ff4-gw-openshift-default-9bfdcc79c-w7gqc\" (UID: \"8b64510b-aa47-4a38-ac26-7c0df79188b2\") " pod="openshift-ingress/e2e-migrate-d9dc4ff4-gw-openshift-default-9bfdcc79c-w7gqc"
Nov 02 12:35:23.430534 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:23.430353    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b64510b-aa47-4a38-ac26-7c0df79188b2-workload-socket\") pod \"e2e-migrate-d9dc4ff4-gw-openshift-default-9bfdcc79c-w7gqc\" (UID: \"8b64510b-aa47-4a38-ac26-7c0df79188b2\") " pod="openshift-ingress/e2e-migrate-d9dc4ff4-gw-openshift-default-9bfdcc79c-w7gqc"
Nov 02 12:35:23.430610 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:23.430471    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b64510b-aa47-4a38-ac26-7c0df79188b2-credential-socket\") pod \"e2e-migrate-d9dc4ff4-gw-openshift-default-9bfdcc79c-w7gqc\" (UID: \"8b64510b-aa47-4a38-ac26-7c0df79188b2\") " pod="openshift-ingress/e2e-migrate-d9dc4ff4-gw-openshift-default-9bfdcc79c-w7gqc"
Nov 02 12:35:23.430681 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:23.430610    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b64510b-aa47-4a38-ac26-7c0df79188b2-istio-podinfo\") pod \"e2e-migrate-d9dc4ff4-gw-openshift-default-9bfdcc79c-w7gqc\" (UID: \"8b64510b-aa47-4a38-ac26-7c0df79188b2\") " pod="openshift-ingress/e2e-migrate-d9dc4ff4-gw-openshift-default-9bfdcc79c-w7gqc"
Nov 02 12:35:23.430681 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:23.430667    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b64510b-aa47-4a38-ac26-7c0df79188b2-istio-data\") pod \"e2e-migrate-d9dc4ff4-gw-openshift-default-9bfdcc79c-w7gqc\" (UID: \"8b64510b-aa47-4a38-ac26-7c0df79188b2\") " pod="openshift-ingress/e2e-migrate-d9dc4ff4-gw-openshift-default-9bfdcc79c-w7gqc"
Nov 02 12:35:23.430793 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:23.430706    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvgf8\" (UniqueName: \"kubernetes.io/projected/8b64510b-aa47-4a38-ac26-7c0df79188b2-kube-api-access-pvgf8\") pod \"e2e-migrate-d9dc4ff4-gw-openshift-default-9bfdcc79c-w7gqc\" (UID: \"8b64510b-aa47-4a38-ac26-7c0df79188b2\") " pod="openshift-ingress/e2e-migrate-d9dc4ff4-gw-openshift-default-9bfdcc79c-w7gqc"
Nov 02 12:35:23.430793 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:23.430739    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b64510b-aa47-4a38-ac26-7c0df79188b2-istio-envoy\") pod \"e2e-migrate-d9dc4ff4-gw-openshift-default-9bfdcc79c-w7gqc\" (UID: \"8b64510b-aa47-4a38-ac26-7c0df79188b2\") " pod="openshift-ingress/e2e-migrate-d9dc4ff4-gw-openshift-default-9bfdcc79c-w7gqc"
Nov 02 12:35:23.430898 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:23.430834    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b64510b-aa47-4a38-ac26-7c0df79188b2-istiod-ca-cert\") pod \"e2e-migrate-d9dc4ff4-gw-openshift-default-9bfdcc79c-w7gqc\" (UID: \"8b64510b-aa47-4a38-ac26-7c0df79188b2\") " pod="openshift-ingress/e2e-migrate-d9dc4ff4-gw-openshift-default-9bfdcc79c-w7gqc"
Nov 02 12:35:23.430898 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:23.430865    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b64510b-aa47-4a38-ac26-7c0df79188b2-istio-token\") pod \"e2e-migrate-d9dc4ff4-gw-openshift-default-9bfdcc79c-w7gqc\" (UID: \"8b64510b-aa47-4a38-ac26-7c0df79188b2\") " pod="openshift-ingress/e2e-migrate-d9dc4ff4-gw-openshift-default-9bfdcc79c-w7gqc"
Nov 02 12:35:23.532442 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:23.532358    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b64510b-aa47-4a38-ac26-7c0df79188b2-workload-socket\") pod \"e2e-migrate-d9dc4ff4-gw-openshift-default-9bfdcc79c-w7gqc\" (UID: \"8b64510b-aa47-4a38-ac26-7c0df79188b2\") " pod="openshift-ingress/e2e-migrate-d9dc4ff4-gw-openshift-default-9bfdcc79c-w7gqc"
Nov 02 12:35:23.532442 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:23.532412    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b64510b-aa47-4a38-ac26-7c0df79188b2-credential-socket\") pod \"e2e-migrate-d9dc4ff4-gw-openshift-default-9bfdcc79c-w7gqc\" (UID: \"8b64510b-aa47-4a38-ac26-7c0df79188b2\") " pod="openshift-ingress/e2e-migrate-d9dc4ff4-gw-openshift-default-9bfdcc79c-w7gqc"
Nov 02 12:35:23.532724 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:23.532452    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b64510b-aa47-4a38-ac26-7c0df79188b2-istio-podinfo\") pod \"e2e-migrate-d9dc4ff4-gw-openshift-default-9bfdcc79c-w7gqc\" (UID: \"8b64510b-aa47-4a38-ac26-7c0df79188b2\") " pod="openshift-ingress/e2e-migrate-d9dc4ff4-gw-openshift-default-9bfdcc79c-w7gqc"
Nov 02 12:35:23.532724 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:23.532477    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b64510b-aa47-4a38-ac26-7c0df79188b2-istio-data\") pod \"e2e-migrate-d9dc4ff4-gw-openshift-default-9bfdcc79c-w7gqc\" (UID: \"8b64510b-aa47-4a38-ac26-7c0df79188b2\") " pod="openshift-ingress/e2e-migrate-d9dc4ff4-gw-openshift-default-9bfdcc79c-w7gqc"
Nov 02 12:35:23.532724 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:23.532499    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvgf8\" (UniqueName: \"kubernetes.io/projected/8b64510b-aa47-4a38-ac26-7c0df79188b2-kube-api-access-pvgf8\") pod \"e2e-migrate-d9dc4ff4-gw-openshift-default-9bfdcc79c-w7gqc\" (UID: \"8b64510b-aa47-4a38-ac26-7c0df79188b2\") " pod="openshift-ingress/e2e-migrate-d9dc4ff4-gw-openshift-default-9bfdcc79c-w7gqc"
Nov 02 12:35:23.532724 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:23.532521    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b64510b-aa47-4a38-ac26-7c0df79188b2-istio-envoy\") pod \"e2e-migrate-d9dc4ff4-gw-openshift-default-9bfdcc79c-w7gqc\" (UID: \"8b64510b-aa47-4a38-ac26-7c0df79188b2\") " pod="openshift-ingress/e2e-migrate-d9dc4ff4-gw-openshift-default-9bfdcc79c-w7gqc"
Nov 02 12:35:23.532724 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:23.532537    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b64510b-aa47-4a38-ac26-7c0df79188b2-istiod-ca-cert\") pod \"e2e-migrate-d9dc4ff4-gw-openshift-default-9bfdcc79c-w7gqc\" (UID: \"8b64510b-aa47-4a38-ac26-7c0df79188b2\") " pod="openshift-ingress/e2e-migrate-d9dc4ff4-gw-openshift-default-9bfdcc79c-w7gqc"
Nov 02 12:35:23.532989 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:23.532774    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b64510b-aa47-4a38-ac26-7c0df79188b2-istio-token\") pod \"e2e-migrate-d9dc4ff4-gw-openshift-default-9bfdcc79c-w7gqc\" (UID: \"8b64510b-aa47-4a38-ac26-7c0df79188b2\") " pod="openshift-ingress/e2e-migrate-d9dc4ff4-gw-openshift-default-9bfdcc79c-w7gqc"
Nov 02 12:35:23.532989 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:23.532883    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b64510b-aa47-4a38-ac26-7c0df79188b2-workload-certs\") pod \"e2e-migrate-d9dc4ff4-gw-openshift-default-9bfdcc79c-w7gqc\" (UID: \"8b64510b-aa47-4a38-ac26-7c0df79188b2\") " pod="openshift-ingress/e2e-migrate-d9dc4ff4-gw-openshift-default-9bfdcc79c-w7gqc"
Nov 02 12:35:23.532989 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:23.532885    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b64510b-aa47-4a38-ac26-7c0df79188b2-workload-socket\") pod \"e2e-migrate-d9dc4ff4-gw-openshift-default-9bfdcc79c-w7gqc\" (UID: \"8b64510b-aa47-4a38-ac26-7c0df79188b2\") " pod="openshift-ingress/e2e-migrate-d9dc4ff4-gw-openshift-default-9bfdcc79c-w7gqc"
Nov 02 12:35:23.532989 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:23.532897    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b64510b-aa47-4a38-ac26-7c0df79188b2-istio-data\") pod \"e2e-migrate-d9dc4ff4-gw-openshift-default-9bfdcc79c-w7gqc\" (UID: \"8b64510b-aa47-4a38-ac26-7c0df79188b2\") " pod="openshift-ingress/e2e-migrate-d9dc4ff4-gw-openshift-default-9bfdcc79c-w7gqc"
Nov 02 12:35:23.533215 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:23.533189    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b64510b-aa47-4a38-ac26-7c0df79188b2-credential-socket\") pod \"e2e-migrate-d9dc4ff4-gw-openshift-default-9bfdcc79c-w7gqc\" (UID: \"8b64510b-aa47-4a38-ac26-7c0df79188b2\") " pod="openshift-ingress/e2e-migrate-d9dc4ff4-gw-openshift-default-9bfdcc79c-w7gqc"
Nov 02 12:35:23.533276 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:23.533222    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b64510b-aa47-4a38-ac26-7c0df79188b2-workload-certs\") pod \"e2e-migrate-d9dc4ff4-gw-openshift-default-9bfdcc79c-w7gqc\" (UID: \"8b64510b-aa47-4a38-ac26-7c0df79188b2\") " pod="openshift-ingress/e2e-migrate-d9dc4ff4-gw-openshift-default-9bfdcc79c-w7gqc"
Nov 02 12:35:23.533355 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:23.533334    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b64510b-aa47-4a38-ac26-7c0df79188b2-istiod-ca-cert\") pod \"e2e-migrate-d9dc4ff4-gw-openshift-default-9bfdcc79c-w7gqc\" (UID: \"8b64510b-aa47-4a38-ac26-7c0df79188b2\") " pod="openshift-ingress/e2e-migrate-d9dc4ff4-gw-openshift-default-9bfdcc79c-w7gqc"
Nov 02 12:35:23.534945 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:23.534917    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b64510b-aa47-4a38-ac26-7c0df79188b2-istio-envoy\") pod \"e2e-migrate-d9dc4ff4-gw-openshift-default-9bfdcc79c-w7gqc\" (UID: \"8b64510b-aa47-4a38-ac26-7c0df79188b2\") " pod="openshift-ingress/e2e-migrate-d9dc4ff4-gw-openshift-default-9bfdcc79c-w7gqc"
Nov 02 12:35:23.535102 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:23.535085    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b64510b-aa47-4a38-ac26-7c0df79188b2-istio-podinfo\") pod \"e2e-migrate-d9dc4ff4-gw-openshift-default-9bfdcc79c-w7gqc\" (UID: \"8b64510b-aa47-4a38-ac26-7c0df79188b2\") " pod="openshift-ingress/e2e-migrate-d9dc4ff4-gw-openshift-default-9bfdcc79c-w7gqc"
Nov 02 12:35:23.557124 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:23.557097    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b64510b-aa47-4a38-ac26-7c0df79188b2-istio-token\") pod \"e2e-migrate-d9dc4ff4-gw-openshift-default-9bfdcc79c-w7gqc\" (UID: \"8b64510b-aa47-4a38-ac26-7c0df79188b2\") " pod="openshift-ingress/e2e-migrate-d9dc4ff4-gw-openshift-default-9bfdcc79c-w7gqc"
Nov 02 12:35:23.558063 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:23.558042    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvgf8\" (UniqueName: \"kubernetes.io/projected/8b64510b-aa47-4a38-ac26-7c0df79188b2-kube-api-access-pvgf8\") pod \"e2e-migrate-d9dc4ff4-gw-openshift-default-9bfdcc79c-w7gqc\" (UID: \"8b64510b-aa47-4a38-ac26-7c0df79188b2\") " pod="openshift-ingress/e2e-migrate-d9dc4ff4-gw-openshift-default-9bfdcc79c-w7gqc"
Nov 02 12:35:23.581767 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:23.581739    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-d9dc4ff4-gw-openshift-default-9bfdcc79c-w7gqc"
Nov 02 12:35:23.896283 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:23.896256    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-d9dc4ff4-gw-openshift-default-9bfdcc79c-w7gqc"]
Nov 02 12:35:23.897164 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:35:23.897144    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b64510b_aa47_4a38_ac26_7c0df79188b2.slice/crio-4e6eeabe9ac531717c61c56cac1b4463b748c3829d7a3ec3f0066b2c5e916c4d WatchSource:0}: Error finding container 4e6eeabe9ac531717c61c56cac1b4463b748c3829d7a3ec3f0066b2c5e916c4d: Status 404 returned error can't find the container with id 4e6eeabe9ac531717c61c56cac1b4463b748c3829d7a3ec3f0066b2c5e916c4d
Nov 02 12:35:23.899152 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:23.899120    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:35:23.899238 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:23.899198    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:35:23.899281 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:23.899238    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:35:24.057387 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:24.057346    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-d9dc4ff4-gw-openshift-default-9bfdcc79c-w7gqc" event={"ID":"8b64510b-aa47-4a38-ac26-7c0df79188b2","Type":"ContainerStarted","Data":"7f777244670ed6d28a65b70700029490b9add71a7800a2a20e511bec14316af5"}
Nov 02 12:35:24.057552 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:24.057457    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-d9dc4ff4-gw-openshift-default-9bfdcc79c-w7gqc" event={"ID":"8b64510b-aa47-4a38-ac26-7c0df79188b2","Type":"ContainerStarted","Data":"4e6eeabe9ac531717c61c56cac1b4463b748c3829d7a3ec3f0066b2c5e916c4d"}
Nov 02 12:35:24.104504 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:24.104449    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-d9dc4ff4-gw-openshift-default-9bfdcc79c-w7gqc" podStartSLOduration=1.104436863 podStartE2EDuration="1.104436863s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:35:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:35:24.103594576 +0000 UTC m=+520.358993675" watchObservedRunningTime="2026-11-02 12:35:24.104436863 +0000 UTC m=+520.359835958"
Nov 02 12:35:24.467059 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:24.467025    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-628cf9bd-openshift-default-75cb8d8c97-g77zh"]
Nov 02 12:35:24.476100 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:24.470785    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-628cf9bd-openshift-default-75cb8d8c97-g77zh"
Nov 02 12:35:24.487508 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:24.487483    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-628cf9bd-openshift-default-dockercfg-pnk54\""
Nov 02 12:35:24.502719 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:24.502611    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-628cf9bd-openshift-default-75cb8d8c97-g77zh"]
Nov 02 12:35:24.582087 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:24.582055    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-d9dc4ff4-gw-openshift-default-9bfdcc79c-w7gqc"
Nov 02 12:35:24.587551 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:24.587525    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-d9dc4ff4-gw-openshift-default-9bfdcc79c-w7gqc"
Nov 02 12:35:24.644369 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:24.644338    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54f220f8-02ad-4a8b-8c3a-735b0864bdff-workload-socket\") pod \"e2e-ait-628cf9bd-openshift-default-75cb8d8c97-g77zh\" (UID: \"54f220f8-02ad-4a8b-8c3a-735b0864bdff\") " pod="openshift-ingress/e2e-ait-628cf9bd-openshift-default-75cb8d8c97-g77zh"
Nov 02 12:35:24.644545 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:24.644379    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54f220f8-02ad-4a8b-8c3a-735b0864bdff-istiod-ca-cert\") pod \"e2e-ait-628cf9bd-openshift-default-75cb8d8c97-g77zh\" (UID: \"54f220f8-02ad-4a8b-8c3a-735b0864bdff\") " pod="openshift-ingress/e2e-ait-628cf9bd-openshift-default-75cb8d8c97-g77zh"
Nov 02 12:35:24.644545 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:24.644468    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54f220f8-02ad-4a8b-8c3a-735b0864bdff-istio-podinfo\") pod \"e2e-ait-628cf9bd-openshift-default-75cb8d8c97-g77zh\" (UID: \"54f220f8-02ad-4a8b-8c3a-735b0864bdff\") " pod="openshift-ingress/e2e-ait-628cf9bd-openshift-default-75cb8d8c97-g77zh"
Nov 02 12:35:24.644545 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:24.644498    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54f220f8-02ad-4a8b-8c3a-735b0864bdff-istio-envoy\") pod \"e2e-ait-628cf9bd-openshift-default-75cb8d8c97-g77zh\" (UID: \"54f220f8-02ad-4a8b-8c3a-735b0864bdff\") " pod="openshift-ingress/e2e-ait-628cf9bd-openshift-default-75cb8d8c97-g77zh"
Nov 02 12:35:24.644545 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:24.644516    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54f220f8-02ad-4a8b-8c3a-735b0864bdff-istio-data\") pod \"e2e-ait-628cf9bd-openshift-default-75cb8d8c97-g77zh\" (UID: \"54f220f8-02ad-4a8b-8c3a-735b0864bdff\") " pod="openshift-ingress/e2e-ait-628cf9bd-openshift-default-75cb8d8c97-g77zh"
Nov 02 12:35:24.644743 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:24.644626    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6465t\" (UniqueName: \"kubernetes.io/projected/54f220f8-02ad-4a8b-8c3a-735b0864bdff-kube-api-access-6465t\") pod \"e2e-ait-628cf9bd-openshift-default-75cb8d8c97-g77zh\" (UID: \"54f220f8-02ad-4a8b-8c3a-735b0864bdff\") " pod="openshift-ingress/e2e-ait-628cf9bd-openshift-default-75cb8d8c97-g77zh"
Nov 02 12:35:24.644743 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:24.644665    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54f220f8-02ad-4a8b-8c3a-735b0864bdff-workload-certs\") pod \"e2e-ait-628cf9bd-openshift-default-75cb8d8c97-g77zh\" (UID: \"54f220f8-02ad-4a8b-8c3a-735b0864bdff\") " pod="openshift-ingress/e2e-ait-628cf9bd-openshift-default-75cb8d8c97-g77zh"
Nov 02 12:35:24.644743 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:24.644724    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54f220f8-02ad-4a8b-8c3a-735b0864bdff-credential-socket\") pod \"e2e-ait-628cf9bd-openshift-default-75cb8d8c97-g77zh\" (UID: \"54f220f8-02ad-4a8b-8c3a-735b0864bdff\") " pod="openshift-ingress/e2e-ait-628cf9bd-openshift-default-75cb8d8c97-g77zh"
Nov 02 12:35:24.644860 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:24.644771    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54f220f8-02ad-4a8b-8c3a-735b0864bdff-istio-token\") pod \"e2e-ait-628cf9bd-openshift-default-75cb8d8c97-g77zh\" (UID: \"54f220f8-02ad-4a8b-8c3a-735b0864bdff\") " pod="openshift-ingress/e2e-ait-628cf9bd-openshift-default-75cb8d8c97-g77zh"
Nov 02 12:35:24.745788 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:24.745688    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54f220f8-02ad-4a8b-8c3a-735b0864bdff-istiod-ca-cert\") pod \"e2e-ait-628cf9bd-openshift-default-75cb8d8c97-g77zh\" (UID: \"54f220f8-02ad-4a8b-8c3a-735b0864bdff\") " pod="openshift-ingress/e2e-ait-628cf9bd-openshift-default-75cb8d8c97-g77zh"
Nov 02 12:35:24.746532 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:24.746395    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54f220f8-02ad-4a8b-8c3a-735b0864bdff-istiod-ca-cert\") pod \"e2e-ait-628cf9bd-openshift-default-75cb8d8c97-g77zh\" (UID: \"54f220f8-02ad-4a8b-8c3a-735b0864bdff\") " pod="openshift-ingress/e2e-ait-628cf9bd-openshift-default-75cb8d8c97-g77zh"
Nov 02 12:35:24.746757 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:24.746735    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54f220f8-02ad-4a8b-8c3a-735b0864bdff-istio-podinfo\") pod \"e2e-ait-628cf9bd-openshift-default-75cb8d8c97-g77zh\" (UID: \"54f220f8-02ad-4a8b-8c3a-735b0864bdff\") " pod="openshift-ingress/e2e-ait-628cf9bd-openshift-default-75cb8d8c97-g77zh"
Nov 02 12:35:24.746912 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:24.746895    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54f220f8-02ad-4a8b-8c3a-735b0864bdff-istio-envoy\") pod \"e2e-ait-628cf9bd-openshift-default-75cb8d8c97-g77zh\" (UID: \"54f220f8-02ad-4a8b-8c3a-735b0864bdff\") " pod="openshift-ingress/e2e-ait-628cf9bd-openshift-default-75cb8d8c97-g77zh"
Nov 02 12:35:24.747370 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:24.747350    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54f220f8-02ad-4a8b-8c3a-735b0864bdff-istio-data\") pod \"e2e-ait-628cf9bd-openshift-default-75cb8d8c97-g77zh\" (UID: \"54f220f8-02ad-4a8b-8c3a-735b0864bdff\") " pod="openshift-ingress/e2e-ait-628cf9bd-openshift-default-75cb8d8c97-g77zh"
Nov 02 12:35:24.747850 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:24.747831    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6465t\" (UniqueName: \"kubernetes.io/projected/54f220f8-02ad-4a8b-8c3a-735b0864bdff-kube-api-access-6465t\") pod \"e2e-ait-628cf9bd-openshift-default-75cb8d8c97-g77zh\" (UID: \"54f220f8-02ad-4a8b-8c3a-735b0864bdff\") " pod="openshift-ingress/e2e-ait-628cf9bd-openshift-default-75cb8d8c97-g77zh"
Nov 02 12:35:24.747993 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:24.747976    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54f220f8-02ad-4a8b-8c3a-735b0864bdff-workload-certs\") pod \"e2e-ait-628cf9bd-openshift-default-75cb8d8c97-g77zh\" (UID: \"54f220f8-02ad-4a8b-8c3a-735b0864bdff\") " pod="openshift-ingress/e2e-ait-628cf9bd-openshift-default-75cb8d8c97-g77zh"
Nov 02 12:35:24.748131 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:24.748114    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54f220f8-02ad-4a8b-8c3a-735b0864bdff-credential-socket\") pod \"e2e-ait-628cf9bd-openshift-default-75cb8d8c97-g77zh\" (UID: \"54f220f8-02ad-4a8b-8c3a-735b0864bdff\") " pod="openshift-ingress/e2e-ait-628cf9bd-openshift-default-75cb8d8c97-g77zh"
Nov 02 12:35:24.748873 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:24.747745    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54f220f8-02ad-4a8b-8c3a-735b0864bdff-istio-data\") pod \"e2e-ait-628cf9bd-openshift-default-75cb8d8c97-g77zh\" (UID: \"54f220f8-02ad-4a8b-8c3a-735b0864bdff\") " pod="openshift-ingress/e2e-ait-628cf9bd-openshift-default-75cb8d8c97-g77zh"
Nov 02 12:35:24.748873 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:24.748574    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54f220f8-02ad-4a8b-8c3a-735b0864bdff-workload-certs\") pod \"e2e-ait-628cf9bd-openshift-default-75cb8d8c97-g77zh\" (UID: \"54f220f8-02ad-4a8b-8c3a-735b0864bdff\") " pod="openshift-ingress/e2e-ait-628cf9bd-openshift-default-75cb8d8c97-g77zh"
Nov 02 12:35:24.748873 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:24.748832    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54f220f8-02ad-4a8b-8c3a-735b0864bdff-credential-socket\") pod \"e2e-ait-628cf9bd-openshift-default-75cb8d8c97-g77zh\" (UID: \"54f220f8-02ad-4a8b-8c3a-735b0864bdff\") " pod="openshift-ingress/e2e-ait-628cf9bd-openshift-default-75cb8d8c97-g77zh"
Nov 02 12:35:24.749493 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:24.748243    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54f220f8-02ad-4a8b-8c3a-735b0864bdff-istio-token\") pod \"e2e-ait-628cf9bd-openshift-default-75cb8d8c97-g77zh\" (UID: \"54f220f8-02ad-4a8b-8c3a-735b0864bdff\") " pod="openshift-ingress/e2e-ait-628cf9bd-openshift-default-75cb8d8c97-g77zh"
Nov 02 12:35:24.749978 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:24.749958    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54f220f8-02ad-4a8b-8c3a-735b0864bdff-workload-socket\") pod \"e2e-ait-628cf9bd-openshift-default-75cb8d8c97-g77zh\" (UID: \"54f220f8-02ad-4a8b-8c3a-735b0864bdff\") " pod="openshift-ingress/e2e-ait-628cf9bd-openshift-default-75cb8d8c97-g77zh"
Nov 02 12:35:24.750519 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:24.750413    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54f220f8-02ad-4a8b-8c3a-735b0864bdff-workload-socket\") pod \"e2e-ait-628cf9bd-openshift-default-75cb8d8c97-g77zh\" (UID: \"54f220f8-02ad-4a8b-8c3a-735b0864bdff\") " pod="openshift-ingress/e2e-ait-628cf9bd-openshift-default-75cb8d8c97-g77zh"
Nov 02 12:35:24.751696 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:24.751621    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54f220f8-02ad-4a8b-8c3a-735b0864bdff-istio-podinfo\") pod \"e2e-ait-628cf9bd-openshift-default-75cb8d8c97-g77zh\" (UID: \"54f220f8-02ad-4a8b-8c3a-735b0864bdff\") " pod="openshift-ingress/e2e-ait-628cf9bd-openshift-default-75cb8d8c97-g77zh"
Nov 02 12:35:24.754907 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:24.754865    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54f220f8-02ad-4a8b-8c3a-735b0864bdff-istio-envoy\") pod \"e2e-ait-628cf9bd-openshift-default-75cb8d8c97-g77zh\" (UID: \"54f220f8-02ad-4a8b-8c3a-735b0864bdff\") " pod="openshift-ingress/e2e-ait-628cf9bd-openshift-default-75cb8d8c97-g77zh"
Nov 02 12:35:24.792681 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:24.788922    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6465t\" (UniqueName: \"kubernetes.io/projected/54f220f8-02ad-4a8b-8c3a-735b0864bdff-kube-api-access-6465t\") pod \"e2e-ait-628cf9bd-openshift-default-75cb8d8c97-g77zh\" (UID: \"54f220f8-02ad-4a8b-8c3a-735b0864bdff\") " pod="openshift-ingress/e2e-ait-628cf9bd-openshift-default-75cb8d8c97-g77zh"
Nov 02 12:35:24.801248 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:24.801180    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54f220f8-02ad-4a8b-8c3a-735b0864bdff-istio-token\") pod \"e2e-ait-628cf9bd-openshift-default-75cb8d8c97-g77zh\" (UID: \"54f220f8-02ad-4a8b-8c3a-735b0864bdff\") " pod="openshift-ingress/e2e-ait-628cf9bd-openshift-default-75cb8d8c97-g77zh"
Nov 02 12:35:25.061585 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:25.061506    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-d9dc4ff4-gw-openshift-default-9bfdcc79c-w7gqc"
Nov 02 12:35:25.062624 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:25.062606    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-d9dc4ff4-gw-openshift-default-9bfdcc79c-w7gqc"
Nov 02 12:35:25.087777 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:25.087742    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-628cf9bd-openshift-default-75cb8d8c97-g77zh"
Nov 02 12:35:25.406488 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:25.406462    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-628cf9bd-openshift-default-75cb8d8c97-g77zh"]
Nov 02 12:35:25.407456 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:35:25.407432    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54f220f8_02ad_4a8b_8c3a_735b0864bdff.slice/crio-42be404ad25c4cda44779f1b6c0f5b5106d48779d3d03e8ccb11d762095b2b3e WatchSource:0}: Error finding container 42be404ad25c4cda44779f1b6c0f5b5106d48779d3d03e8ccb11d762095b2b3e: Status 404 returned error can't find the container with id 42be404ad25c4cda44779f1b6c0f5b5106d48779d3d03e8ccb11d762095b2b3e
Nov 02 12:35:25.409878 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:25.409845    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:35:25.409965 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:25.409907    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:35:25.410006 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:25.409977    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:35:26.067756 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:26.067708    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-628cf9bd-openshift-default-75cb8d8c97-g77zh" event={"ID":"54f220f8-02ad-4a8b-8c3a-735b0864bdff","Type":"ContainerStarted","Data":"e376457369e920ce66d4a6383f4a72ac24b531aeaf6e541fdaf1ffa9e98bd0bb"}
Nov 02 12:35:26.068254 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:26.067770    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-628cf9bd-openshift-default-75cb8d8c97-g77zh" event={"ID":"54f220f8-02ad-4a8b-8c3a-735b0864bdff","Type":"ContainerStarted","Data":"42be404ad25c4cda44779f1b6c0f5b5106d48779d3d03e8ccb11d762095b2b3e"}
Nov 02 12:35:26.088202 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:26.088173    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-628cf9bd-openshift-default-75cb8d8c97-g77zh"
Nov 02 12:35:26.093262 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:26.093239    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-628cf9bd-openshift-default-75cb8d8c97-g77zh"
Nov 02 12:35:26.126294 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:26.126241    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-628cf9bd-openshift-default-75cb8d8c97-g77zh" podStartSLOduration=2.126227105 podStartE2EDuration="2.126227105s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:35:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:35:26.125383794 +0000 UTC m=+522.380782891" watchObservedRunningTime="2026-11-02 12:35:26.126227105 +0000 UTC m=+522.381626199"
Nov 02 12:35:26.507261 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:26.507226    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-d9dc4ff4-5dbf6d9d4d-l2fkz"]
Nov 02 12:35:26.510798 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:26.510783    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d9dc4ff4-5dbf6d9d4d-l2fkz"
Nov 02 12:35:26.533117 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:26.533086    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-d9dc4ff4-5dbf6d9d4d-l2fkz"]
Nov 02 12:35:26.534379 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:26.534341    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-d9dc4ff4-serving-cert\""
Nov 02 12:35:26.534551 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:26.534535    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 12:35:26.534886 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:26.534847    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 12:35:26.535009 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:26.534977    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Nov 02 12:35:26.542475 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:26.542455    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-sr5s9\""
Nov 02 12:35:26.668503 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:26.668468    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f83542d0-1d12-4105-b625-54f0764c6f8b-maas-api-tls\") pod \"maas-api-e2e-migrate-d9dc4ff4-5dbf6d9d4d-l2fkz\" (UID: \"f83542d0-1d12-4105-b625-54f0764c6f8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d9dc4ff4-5dbf6d9d4d-l2fkz"
Nov 02 12:35:26.668741 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:26.668523    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2qpx\" (UniqueName: \"kubernetes.io/projected/f83542d0-1d12-4105-b625-54f0764c6f8b-kube-api-access-b2qpx\") pod \"maas-api-e2e-migrate-d9dc4ff4-5dbf6d9d4d-l2fkz\" (UID: \"f83542d0-1d12-4105-b625-54f0764c6f8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d9dc4ff4-5dbf6d9d4d-l2fkz"
Nov 02 12:35:26.668741 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:26.668661    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f83542d0-1d12-4105-b625-54f0764c6f8b-openshift-service-ca\") pod \"maas-api-e2e-migrate-d9dc4ff4-5dbf6d9d4d-l2fkz\" (UID: \"f83542d0-1d12-4105-b625-54f0764c6f8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d9dc4ff4-5dbf6d9d4d-l2fkz"
Nov 02 12:35:26.668741 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:26.668723    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f83542d0-1d12-4105-b625-54f0764c6f8b-metrics-certs\") pod \"maas-api-e2e-migrate-d9dc4ff4-5dbf6d9d4d-l2fkz\" (UID: \"f83542d0-1d12-4105-b625-54f0764c6f8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d9dc4ff4-5dbf6d9d4d-l2fkz"
Nov 02 12:35:26.769380 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:26.769290    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f83542d0-1d12-4105-b625-54f0764c6f8b-openshift-service-ca\") pod \"maas-api-e2e-migrate-d9dc4ff4-5dbf6d9d4d-l2fkz\" (UID: \"f83542d0-1d12-4105-b625-54f0764c6f8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d9dc4ff4-5dbf6d9d4d-l2fkz"
Nov 02 12:35:26.769380 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:26.769346    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f83542d0-1d12-4105-b625-54f0764c6f8b-metrics-certs\") pod \"maas-api-e2e-migrate-d9dc4ff4-5dbf6d9d4d-l2fkz\" (UID: \"f83542d0-1d12-4105-b625-54f0764c6f8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d9dc4ff4-5dbf6d9d4d-l2fkz"
Nov 02 12:35:26.769676 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:26.769430    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f83542d0-1d12-4105-b625-54f0764c6f8b-maas-api-tls\") pod \"maas-api-e2e-migrate-d9dc4ff4-5dbf6d9d4d-l2fkz\" (UID: \"f83542d0-1d12-4105-b625-54f0764c6f8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d9dc4ff4-5dbf6d9d4d-l2fkz"
Nov 02 12:35:26.769676 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:26.769468    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2qpx\" (UniqueName: \"kubernetes.io/projected/f83542d0-1d12-4105-b625-54f0764c6f8b-kube-api-access-b2qpx\") pod \"maas-api-e2e-migrate-d9dc4ff4-5dbf6d9d4d-l2fkz\" (UID: \"f83542d0-1d12-4105-b625-54f0764c6f8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d9dc4ff4-5dbf6d9d4d-l2fkz"
Nov 02 12:35:26.770183 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:26.770148    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f83542d0-1d12-4105-b625-54f0764c6f8b-openshift-service-ca\") pod \"maas-api-e2e-migrate-d9dc4ff4-5dbf6d9d4d-l2fkz\" (UID: \"f83542d0-1d12-4105-b625-54f0764c6f8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d9dc4ff4-5dbf6d9d4d-l2fkz"
Nov 02 12:35:26.772140 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:26.772116    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f83542d0-1d12-4105-b625-54f0764c6f8b-metrics-certs\") pod \"maas-api-e2e-migrate-d9dc4ff4-5dbf6d9d4d-l2fkz\" (UID: \"f83542d0-1d12-4105-b625-54f0764c6f8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d9dc4ff4-5dbf6d9d4d-l2fkz"
Nov 02 12:35:26.772249 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:26.772119    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f83542d0-1d12-4105-b625-54f0764c6f8b-maas-api-tls\") pod \"maas-api-e2e-migrate-d9dc4ff4-5dbf6d9d4d-l2fkz\" (UID: \"f83542d0-1d12-4105-b625-54f0764c6f8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d9dc4ff4-5dbf6d9d4d-l2fkz"
Nov 02 12:35:26.796529 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:26.796508    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2qpx\" (UniqueName: \"kubernetes.io/projected/f83542d0-1d12-4105-b625-54f0764c6f8b-kube-api-access-b2qpx\") pod \"maas-api-e2e-migrate-d9dc4ff4-5dbf6d9d4d-l2fkz\" (UID: \"f83542d0-1d12-4105-b625-54f0764c6f8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d9dc4ff4-5dbf6d9d4d-l2fkz"
Nov 02 12:35:26.825362 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:26.825339    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d9dc4ff4-5dbf6d9d4d-l2fkz"
Nov 02 12:35:27.072729 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:27.072624    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-628cf9bd-openshift-default-75cb8d8c97-g77zh"
Nov 02 12:35:27.073513 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:27.073497    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-628cf9bd-openshift-default-75cb8d8c97-g77zh"
Nov 02 12:35:27.136355 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:27.136318    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-d9dc4ff4-5dbf6d9d4d-l2fkz"]
Nov 02 12:35:28.078035 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:28.077992    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d9dc4ff4-5dbf6d9d4d-l2fkz" event={"ID":"f83542d0-1d12-4105-b625-54f0764c6f8b","Type":"ContainerStarted","Data":"e3a5a2b8d9b994475784028ada55d5ebddb7bd67ad2f52e8b79d79e247c94421"}
Nov 02 12:35:28.078493 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:28.078056    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d9dc4ff4-5dbf6d9d4d-l2fkz" event={"ID":"f83542d0-1d12-4105-b625-54f0764c6f8b","Type":"ContainerStarted","Data":"44c142472fdf6e903376a7252c48e6bb86e012fd652f42ddc9ef0b07d07ba21e"}
Nov 02 12:35:28.078493 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:28.078293    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d9dc4ff4-5dbf6d9d4d-l2fkz"
Nov 02 12:35:28.121868 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:28.121825    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d9dc4ff4-5dbf6d9d4d-l2fkz" podStartSLOduration=1.740081191 podStartE2EDuration="2.121812233s" totalImagesPullingTime="381.731042ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:35:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:35:28.120936817 +0000 UTC m=+524.376335911" watchObservedRunningTime="2026-11-02 12:35:28.121812233 +0000 UTC m=+524.377211328"
Nov 02 12:35:28.308692 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:28.308657    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-53b6603d-846469fb44-gc4xc"]
Nov 02 12:35:28.312164 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:28.312148    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-53b6603d-846469fb44-gc4xc"
Nov 02 12:35:28.327368 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:28.327348    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-53b6603d-serving-cert\""
Nov 02 12:35:28.333211 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:28.333152    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-53b6603d-846469fb44-gc4xc"]
Nov 02 12:35:28.392400 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:28.392371    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwtpm\" (UniqueName: \"kubernetes.io/projected/cfa55f58-67b9-4026-b30f-2abdbb09b87c-kube-api-access-zwtpm\") pod \"maas-api-e2e-ait-53b6603d-846469fb44-gc4xc\" (UID: \"cfa55f58-67b9-4026-b30f-2abdbb09b87c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-53b6603d-846469fb44-gc4xc"
Nov 02 12:35:28.392553 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:28.392425    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfa55f58-67b9-4026-b30f-2abdbb09b87c-openshift-service-ca\") pod \"maas-api-e2e-ait-53b6603d-846469fb44-gc4xc\" (UID: \"cfa55f58-67b9-4026-b30f-2abdbb09b87c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-53b6603d-846469fb44-gc4xc"
Nov 02 12:35:28.392553 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:28.392529    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cfa55f58-67b9-4026-b30f-2abdbb09b87c-maas-api-tls\") pod \"maas-api-e2e-ait-53b6603d-846469fb44-gc4xc\" (UID: \"cfa55f58-67b9-4026-b30f-2abdbb09b87c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-53b6603d-846469fb44-gc4xc"
Nov 02 12:35:28.392630 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:28.392563    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cfa55f58-67b9-4026-b30f-2abdbb09b87c-metrics-certs\") pod \"maas-api-e2e-ait-53b6603d-846469fb44-gc4xc\" (UID: \"cfa55f58-67b9-4026-b30f-2abdbb09b87c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-53b6603d-846469fb44-gc4xc"
Nov 02 12:35:28.504333 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:28.500140    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwtpm\" (UniqueName: \"kubernetes.io/projected/cfa55f58-67b9-4026-b30f-2abdbb09b87c-kube-api-access-zwtpm\") pod \"maas-api-e2e-ait-53b6603d-846469fb44-gc4xc\" (UID: \"cfa55f58-67b9-4026-b30f-2abdbb09b87c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-53b6603d-846469fb44-gc4xc"
Nov 02 12:35:28.504333 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:28.500234    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfa55f58-67b9-4026-b30f-2abdbb09b87c-openshift-service-ca\") pod \"maas-api-e2e-ait-53b6603d-846469fb44-gc4xc\" (UID: \"cfa55f58-67b9-4026-b30f-2abdbb09b87c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-53b6603d-846469fb44-gc4xc"
Nov 02 12:35:28.504333 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:28.500287    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cfa55f58-67b9-4026-b30f-2abdbb09b87c-maas-api-tls\") pod \"maas-api-e2e-ait-53b6603d-846469fb44-gc4xc\" (UID: \"cfa55f58-67b9-4026-b30f-2abdbb09b87c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-53b6603d-846469fb44-gc4xc"
Nov 02 12:35:28.504333 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:28.500320    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cfa55f58-67b9-4026-b30f-2abdbb09b87c-metrics-certs\") pod \"maas-api-e2e-ait-53b6603d-846469fb44-gc4xc\" (UID: \"cfa55f58-67b9-4026-b30f-2abdbb09b87c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-53b6603d-846469fb44-gc4xc"
Nov 02 12:35:28.504333 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:35:28.500980    4559 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-53b6603d-serving-cert: secret "maas-api-e2e-ait-53b6603d-serving-cert" not found
Nov 02 12:35:28.504333 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:35:28.501065    4559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cfa55f58-67b9-4026-b30f-2abdbb09b87c-maas-api-tls podName:cfa55f58-67b9-4026-b30f-2abdbb09b87c nodeName:}" failed. No retries permitted until 2026-11-02 12:35:29.001043675 +0000 UTC m=+525.256442750 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/cfa55f58-67b9-4026-b30f-2abdbb09b87c-maas-api-tls") pod "maas-api-e2e-ait-53b6603d-846469fb44-gc4xc" (UID: "cfa55f58-67b9-4026-b30f-2abdbb09b87c") : secret "maas-api-e2e-ait-53b6603d-serving-cert" not found
Nov 02 12:35:28.505876 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:28.505812    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfa55f58-67b9-4026-b30f-2abdbb09b87c-openshift-service-ca\") pod \"maas-api-e2e-ait-53b6603d-846469fb44-gc4xc\" (UID: \"cfa55f58-67b9-4026-b30f-2abdbb09b87c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-53b6603d-846469fb44-gc4xc"
Nov 02 12:35:28.511566 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:28.511535    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cfa55f58-67b9-4026-b30f-2abdbb09b87c-metrics-certs\") pod \"maas-api-e2e-ait-53b6603d-846469fb44-gc4xc\" (UID: \"cfa55f58-67b9-4026-b30f-2abdbb09b87c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-53b6603d-846469fb44-gc4xc"
Nov 02 12:35:28.529406 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:28.529376    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwtpm\" (UniqueName: \"kubernetes.io/projected/cfa55f58-67b9-4026-b30f-2abdbb09b87c-kube-api-access-zwtpm\") pod \"maas-api-e2e-ait-53b6603d-846469fb44-gc4xc\" (UID: \"cfa55f58-67b9-4026-b30f-2abdbb09b87c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-53b6603d-846469fb44-gc4xc"
Nov 02 12:35:29.005121 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:29.005081    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cfa55f58-67b9-4026-b30f-2abdbb09b87c-maas-api-tls\") pod \"maas-api-e2e-ait-53b6603d-846469fb44-gc4xc\" (UID: \"cfa55f58-67b9-4026-b30f-2abdbb09b87c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-53b6603d-846469fb44-gc4xc"
Nov 02 12:35:29.007506 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:29.007485    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cfa55f58-67b9-4026-b30f-2abdbb09b87c-maas-api-tls\") pod \"maas-api-e2e-ait-53b6603d-846469fb44-gc4xc\" (UID: \"cfa55f58-67b9-4026-b30f-2abdbb09b87c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-53b6603d-846469fb44-gc4xc"
Nov 02 12:35:29.225099 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:29.225066    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-53b6603d-846469fb44-gc4xc"
Nov 02 12:35:29.548369 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:29.545902    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-53b6603d-846469fb44-gc4xc"]
Nov 02 12:35:29.550093 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:35:29.550034    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfa55f58_67b9_4026_b30f_2abdbb09b87c.slice/crio-d1855c7411e11817ff15863d8703884780661b63cc888b44cbdc3d67bd52e28c WatchSource:0}: Error finding container d1855c7411e11817ff15863d8703884780661b63cc888b44cbdc3d67bd52e28c: Status 404 returned error can't find the container with id d1855c7411e11817ff15863d8703884780661b63cc888b44cbdc3d67bd52e28c
Nov 02 12:35:29.902980 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:29.902902    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f080cb5f-76f8d88955-vn85z"]
Nov 02 12:35:29.906301 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:29.906259    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-f080cb5f-76f8d88955-vn85z"
Nov 02 12:35:29.925664 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:29.925606    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f080cb5f-76f8d88955-vn85z"]
Nov 02 12:35:29.943090 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:29.943065    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-f080cb5f\""
Nov 02 12:35:30.011124 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:30.011075    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-f080cb5f-dockercfg-jf58p\""
Nov 02 12:35:30.039868 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:30.025274    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07bca5a6-5631-49de-b344-50a8390d1fa6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-f080cb5f-76f8d88955-vn85z\" (UID: \"07bca5a6-5631-49de-b344-50a8390d1fa6\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f080cb5f-76f8d88955-vn85z"
Nov 02 12:35:30.039868 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:30.025365    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4krpl\" (UniqueName: \"kubernetes.io/projected/07bca5a6-5631-49de-b344-50a8390d1fa6-kube-api-access-4krpl\") pod \"payload-pre-processing-e2e-ait-f080cb5f-76f8d88955-vn85z\" (UID: \"07bca5a6-5631-49de-b344-50a8390d1fa6\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f080cb5f-76f8d88955-vn85z"
Nov 02 12:35:30.053658 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:30.051703    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"]
Nov 02 12:35:30.097124 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:30.097086    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-53b6603d-846469fb44-gc4xc" event={"ID":"cfa55f58-67b9-4026-b30f-2abdbb09b87c","Type":"ContainerStarted","Data":"0de29d26b955631ff6a120d1c3882c68691d73e3b1ec377effef4530ebb57953"}
Nov 02 12:35:30.097314 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:30.097144    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-53b6603d-846469fb44-gc4xc" event={"ID":"cfa55f58-67b9-4026-b30f-2abdbb09b87c","Type":"ContainerStarted","Data":"d1855c7411e11817ff15863d8703884780661b63cc888b44cbdc3d67bd52e28c"}
Nov 02 12:35:30.097314 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:30.097200    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-53b6603d-846469fb44-gc4xc"
Nov 02 12:35:30.126944 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:30.126902    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07bca5a6-5631-49de-b344-50a8390d1fa6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-f080cb5f-76f8d88955-vn85z\" (UID: \"07bca5a6-5631-49de-b344-50a8390d1fa6\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f080cb5f-76f8d88955-vn85z"
Nov 02 12:35:30.127119 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:30.126961    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4krpl\" (UniqueName: \"kubernetes.io/projected/07bca5a6-5631-49de-b344-50a8390d1fa6-kube-api-access-4krpl\") pod \"payload-pre-processing-e2e-ait-f080cb5f-76f8d88955-vn85z\" (UID: \"07bca5a6-5631-49de-b344-50a8390d1fa6\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f080cb5f-76f8d88955-vn85z"
Nov 02 12:35:30.127752 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:30.127723    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07bca5a6-5631-49de-b344-50a8390d1fa6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-f080cb5f-76f8d88955-vn85z\" (UID: \"07bca5a6-5631-49de-b344-50a8390d1fa6\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f080cb5f-76f8d88955-vn85z"
Nov 02 12:35:30.198952 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:30.198903    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-53b6603d-846469fb44-gc4xc" podStartSLOduration=1.828786472 podStartE2EDuration="2.198886844s" totalImagesPullingTime="370.100372ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:35:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:35:30.197718537 +0000 UTC m=+526.453117633" watchObservedRunningTime="2026-11-02 12:35:30.198886844 +0000 UTC m=+526.454285938"
Nov 02 12:35:30.265562 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:30.252308    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4krpl\" (UniqueName: \"kubernetes.io/projected/07bca5a6-5631-49de-b344-50a8390d1fa6-kube-api-access-4krpl\") pod \"payload-pre-processing-e2e-ait-f080cb5f-76f8d88955-vn85z\" (UID: \"07bca5a6-5631-49de-b344-50a8390d1fa6\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f080cb5f-76f8d88955-vn85z"
Nov 02 12:35:30.281320 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:30.280945    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f080cb5f-6486999cf7-lw2zl"]
Nov 02 12:35:30.288257 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:30.288216    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f080cb5f-6486999cf7-lw2zl"]
Nov 02 12:35:30.288435 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:30.288343    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f080cb5f-6486999cf7-lw2zl"
Nov 02 12:35:30.306900 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:30.306873    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-f080cb5f-serving-cert\""
Nov 02 12:35:30.439332 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:30.438996    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzfwm\" (UniqueName: \"kubernetes.io/projected/33114556-fa3f-4168-9da6-9ca4a74802f3-kube-api-access-fzfwm\") pod \"maas-api-e2e-ait-f080cb5f-6486999cf7-lw2zl\" (UID: \"33114556-fa3f-4168-9da6-9ca4a74802f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f080cb5f-6486999cf7-lw2zl"
Nov 02 12:35:30.439332 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:30.439126    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33114556-fa3f-4168-9da6-9ca4a74802f3-openshift-service-ca\") pod \"maas-api-e2e-ait-f080cb5f-6486999cf7-lw2zl\" (UID: \"33114556-fa3f-4168-9da6-9ca4a74802f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f080cb5f-6486999cf7-lw2zl"
Nov 02 12:35:30.439332 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:30.439155    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33114556-fa3f-4168-9da6-9ca4a74802f3-metrics-certs\") pod \"maas-api-e2e-ait-f080cb5f-6486999cf7-lw2zl\" (UID: \"33114556-fa3f-4168-9da6-9ca4a74802f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f080cb5f-6486999cf7-lw2zl"
Nov 02 12:35:30.439332 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:30.439192    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33114556-fa3f-4168-9da6-9ca4a74802f3-maas-api-tls\") pod \"maas-api-e2e-ait-f080cb5f-6486999cf7-lw2zl\" (UID: \"33114556-fa3f-4168-9da6-9ca4a74802f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f080cb5f-6486999cf7-lw2zl"
Nov 02 12:35:30.522673 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:30.522574    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-f080cb5f-76f8d88955-vn85z"
Nov 02 12:35:30.540144 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:30.540112    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzfwm\" (UniqueName: \"kubernetes.io/projected/33114556-fa3f-4168-9da6-9ca4a74802f3-kube-api-access-fzfwm\") pod \"maas-api-e2e-ait-f080cb5f-6486999cf7-lw2zl\" (UID: \"33114556-fa3f-4168-9da6-9ca4a74802f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f080cb5f-6486999cf7-lw2zl"
Nov 02 12:35:30.540267 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:30.540232    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33114556-fa3f-4168-9da6-9ca4a74802f3-openshift-service-ca\") pod \"maas-api-e2e-ait-f080cb5f-6486999cf7-lw2zl\" (UID: \"33114556-fa3f-4168-9da6-9ca4a74802f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f080cb5f-6486999cf7-lw2zl"
Nov 02 12:35:30.540340 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:30.540263    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33114556-fa3f-4168-9da6-9ca4a74802f3-metrics-certs\") pod \"maas-api-e2e-ait-f080cb5f-6486999cf7-lw2zl\" (UID: \"33114556-fa3f-4168-9da6-9ca4a74802f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f080cb5f-6486999cf7-lw2zl"
Nov 02 12:35:30.540340 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:30.540298    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33114556-fa3f-4168-9da6-9ca4a74802f3-maas-api-tls\") pod \"maas-api-e2e-ait-f080cb5f-6486999cf7-lw2zl\" (UID: \"33114556-fa3f-4168-9da6-9ca4a74802f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f080cb5f-6486999cf7-lw2zl"
Nov 02 12:35:30.540998 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:30.540970    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33114556-fa3f-4168-9da6-9ca4a74802f3-openshift-service-ca\") pod \"maas-api-e2e-ait-f080cb5f-6486999cf7-lw2zl\" (UID: \"33114556-fa3f-4168-9da6-9ca4a74802f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f080cb5f-6486999cf7-lw2zl"
Nov 02 12:35:30.543124 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:30.543093    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33114556-fa3f-4168-9da6-9ca4a74802f3-metrics-certs\") pod \"maas-api-e2e-ait-f080cb5f-6486999cf7-lw2zl\" (UID: \"33114556-fa3f-4168-9da6-9ca4a74802f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f080cb5f-6486999cf7-lw2zl"
Nov 02 12:35:30.543252 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:30.543204    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33114556-fa3f-4168-9da6-9ca4a74802f3-maas-api-tls\") pod \"maas-api-e2e-ait-f080cb5f-6486999cf7-lw2zl\" (UID: \"33114556-fa3f-4168-9da6-9ca4a74802f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f080cb5f-6486999cf7-lw2zl"
Nov 02 12:35:30.597115 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:30.597083    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzfwm\" (UniqueName: \"kubernetes.io/projected/33114556-fa3f-4168-9da6-9ca4a74802f3-kube-api-access-fzfwm\") pod \"maas-api-e2e-ait-f080cb5f-6486999cf7-lw2zl\" (UID: \"33114556-fa3f-4168-9da6-9ca4a74802f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f080cb5f-6486999cf7-lw2zl"
Nov 02 12:35:30.607805 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:30.607775    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f080cb5f-6486999cf7-lw2zl"
Nov 02 12:35:30.831226 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:30.831198    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f080cb5f-76f8d88955-vn85z"]
Nov 02 12:35:30.832879 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:35:30.832851    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07bca5a6_5631_49de_b344_50a8390d1fa6.slice/crio-9169a7eb7c509592c505a4c98bb122bddceb51c96f558cdf48f943799d886cb1 WatchSource:0}: Error finding container 9169a7eb7c509592c505a4c98bb122bddceb51c96f558cdf48f943799d886cb1: Status 404 returned error can't find the container with id 9169a7eb7c509592c505a4c98bb122bddceb51c96f558cdf48f943799d886cb1
Nov 02 12:35:30.926250 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:30.926224    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f080cb5f-6486999cf7-lw2zl"]
Nov 02 12:35:30.926688 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:35:30.926660    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33114556_fa3f_4168_9da6_9ca4a74802f3.slice/crio-c9787efce920e7e516db12c89a947cc29958efec3326dd002577b090462d6816 WatchSource:0}: Error finding container c9787efce920e7e516db12c89a947cc29958efec3326dd002577b090462d6816: Status 404 returned error can't find the container with id c9787efce920e7e516db12c89a947cc29958efec3326dd002577b090462d6816
Nov 02 12:35:31.112680 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:31.112544    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f080cb5f-6486999cf7-lw2zl" event={"ID":"33114556-fa3f-4168-9da6-9ca4a74802f3","Type":"ContainerStarted","Data":"c9787efce920e7e516db12c89a947cc29958efec3326dd002577b090462d6816"}
Nov 02 12:35:31.115509 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:31.115475    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-f080cb5f-76f8d88955-vn85z" event={"ID":"07bca5a6-5631-49de-b344-50a8390d1fa6","Type":"ContainerStarted","Data":"b07831d10f4ca8b015e192cc46d451e3d95e481ccdaca902301036f18f11b469"}
Nov 02 12:35:31.115678 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:31.115529    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-f080cb5f-76f8d88955-vn85z" event={"ID":"07bca5a6-5631-49de-b344-50a8390d1fa6","Type":"ContainerStarted","Data":"9169a7eb7c509592c505a4c98bb122bddceb51c96f558cdf48f943799d886cb1"}
Nov 02 12:35:31.115678 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:31.115578    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-f080cb5f-76f8d88955-vn85z"
Nov 02 12:35:31.171006 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:31.170945    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-f080cb5f-76f8d88955-vn85z" podStartSLOduration=2.170925875 podStartE2EDuration="2.170925875s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:35:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:35:31.168615008 +0000 UTC m=+527.424014128" watchObservedRunningTime="2026-11-02 12:35:31.170925875 +0000 UTC m=+527.426324971"
Nov 02 12:35:32.130933 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:32.130894    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f080cb5f-6486999cf7-lw2zl" event={"ID":"33114556-fa3f-4168-9da6-9ca4a74802f3","Type":"ContainerStarted","Data":"1d59509153523e0c0eef638d9038fab237c89a4e64e3c3828511116b4516bc52"}
Nov 02 12:35:32.131299 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:32.131056    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f080cb5f-6486999cf7-lw2zl"
Nov 02 12:35:32.174465 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:32.174419    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f080cb5f-6486999cf7-lw2zl" podStartSLOduration=1.673355229 podStartE2EDuration="2.174405178s" totalImagesPullingTime="501.049949ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:35:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:35:32.172669412 +0000 UTC m=+528.428068508" watchObservedRunningTime="2026-11-02 12:35:32.174405178 +0000 UTC m=+528.429804272"
Nov 02 12:35:34.129739 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:34.129438    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d9dc4ff4-5dbf6d9d4d-l2fkz"
Nov 02 12:35:35.813365 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:35.813327    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-628cf9bd-f5bbdc6cf-49rc4"]
Nov 02 12:35:35.817215 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:35.817194    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-628cf9bd-f5bbdc6cf-49rc4"
Nov 02 12:35:35.843693 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:35.842294    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-628cf9bd-serving-cert\""
Nov 02 12:35:35.848692 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:35.845770    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-628cf9bd-f5bbdc6cf-49rc4"]
Nov 02 12:35:35.929057 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:35.929029    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be21bfc9-2d60-4cfd-8ded-41ed733000bb-maas-api-tls\") pod \"maas-api-e2e-ait-628cf9bd-f5bbdc6cf-49rc4\" (UID: \"be21bfc9-2d60-4cfd-8ded-41ed733000bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-628cf9bd-f5bbdc6cf-49rc4"
Nov 02 12:35:35.929292 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:35.929082    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/be21bfc9-2d60-4cfd-8ded-41ed733000bb-metrics-certs\") pod \"maas-api-e2e-ait-628cf9bd-f5bbdc6cf-49rc4\" (UID: \"be21bfc9-2d60-4cfd-8ded-41ed733000bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-628cf9bd-f5bbdc6cf-49rc4"
Nov 02 12:35:35.929292 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:35.929122    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be21bfc9-2d60-4cfd-8ded-41ed733000bb-openshift-service-ca\") pod \"maas-api-e2e-ait-628cf9bd-f5bbdc6cf-49rc4\" (UID: \"be21bfc9-2d60-4cfd-8ded-41ed733000bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-628cf9bd-f5bbdc6cf-49rc4"
Nov 02 12:35:35.929292 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:35.929158    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbb76\" (UniqueName: \"kubernetes.io/projected/be21bfc9-2d60-4cfd-8ded-41ed733000bb-kube-api-access-jbb76\") pod \"maas-api-e2e-ait-628cf9bd-f5bbdc6cf-49rc4\" (UID: \"be21bfc9-2d60-4cfd-8ded-41ed733000bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-628cf9bd-f5bbdc6cf-49rc4"
Nov 02 12:35:36.035345 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:36.034184    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/be21bfc9-2d60-4cfd-8ded-41ed733000bb-metrics-certs\") pod \"maas-api-e2e-ait-628cf9bd-f5bbdc6cf-49rc4\" (UID: \"be21bfc9-2d60-4cfd-8ded-41ed733000bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-628cf9bd-f5bbdc6cf-49rc4"
Nov 02 12:35:36.035345 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:36.034259    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be21bfc9-2d60-4cfd-8ded-41ed733000bb-openshift-service-ca\") pod \"maas-api-e2e-ait-628cf9bd-f5bbdc6cf-49rc4\" (UID: \"be21bfc9-2d60-4cfd-8ded-41ed733000bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-628cf9bd-f5bbdc6cf-49rc4"
Nov 02 12:35:36.035345 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:36.034331    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbb76\" (UniqueName: \"kubernetes.io/projected/be21bfc9-2d60-4cfd-8ded-41ed733000bb-kube-api-access-jbb76\") pod \"maas-api-e2e-ait-628cf9bd-f5bbdc6cf-49rc4\" (UID: \"be21bfc9-2d60-4cfd-8ded-41ed733000bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-628cf9bd-f5bbdc6cf-49rc4"
Nov 02 12:35:36.035345 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:36.034436    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be21bfc9-2d60-4cfd-8ded-41ed733000bb-maas-api-tls\") pod \"maas-api-e2e-ait-628cf9bd-f5bbdc6cf-49rc4\" (UID: \"be21bfc9-2d60-4cfd-8ded-41ed733000bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-628cf9bd-f5bbdc6cf-49rc4"
Nov 02 12:35:36.036789 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:36.036764    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be21bfc9-2d60-4cfd-8ded-41ed733000bb-openshift-service-ca\") pod \"maas-api-e2e-ait-628cf9bd-f5bbdc6cf-49rc4\" (UID: \"be21bfc9-2d60-4cfd-8ded-41ed733000bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-628cf9bd-f5bbdc6cf-49rc4"
Nov 02 12:35:36.038554 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:36.038524    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/be21bfc9-2d60-4cfd-8ded-41ed733000bb-metrics-certs\") pod \"maas-api-e2e-ait-628cf9bd-f5bbdc6cf-49rc4\" (UID: \"be21bfc9-2d60-4cfd-8ded-41ed733000bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-628cf9bd-f5bbdc6cf-49rc4"
Nov 02 12:35:36.041222 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:36.040720    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be21bfc9-2d60-4cfd-8ded-41ed733000bb-maas-api-tls\") pod \"maas-api-e2e-ait-628cf9bd-f5bbdc6cf-49rc4\" (UID: \"be21bfc9-2d60-4cfd-8ded-41ed733000bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-628cf9bd-f5bbdc6cf-49rc4"
Nov 02 12:35:36.061240 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:36.061207    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbb76\" (UniqueName: \"kubernetes.io/projected/be21bfc9-2d60-4cfd-8ded-41ed733000bb-kube-api-access-jbb76\") pod \"maas-api-e2e-ait-628cf9bd-f5bbdc6cf-49rc4\" (UID: \"be21bfc9-2d60-4cfd-8ded-41ed733000bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-628cf9bd-f5bbdc6cf-49rc4"
Nov 02 12:35:36.120840 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:36.120772    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-53b6603d-846469fb44-gc4xc"
Nov 02 12:35:36.130825 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:36.130800    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-628cf9bd-f5bbdc6cf-49rc4"
Nov 02 12:35:36.485186 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:36.485071    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-628cf9bd-f5bbdc6cf-49rc4"]
Nov 02 12:35:37.168300 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:37.168261    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-628cf9bd-f5bbdc6cf-49rc4" event={"ID":"be21bfc9-2d60-4cfd-8ded-41ed733000bb","Type":"ContainerStarted","Data":"d9dd371672cad6551c5ea8312529b7b6deebe8b16f875e248afb70338c038bef"}
Nov 02 12:35:37.168723 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:37.168361    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-628cf9bd-f5bbdc6cf-49rc4" event={"ID":"be21bfc9-2d60-4cfd-8ded-41ed733000bb","Type":"ContainerStarted","Data":"dacb02c65dfd75eb4883b5c925b10bb4639e489c4c47016c2e8404dc7ad4e8d8"}
Nov 02 12:35:37.168723 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:37.168393    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-628cf9bd-f5bbdc6cf-49rc4"
Nov 02 12:35:37.212217 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:37.212163    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-628cf9bd-f5bbdc6cf-49rc4" podStartSLOduration=1.889209168 podStartE2EDuration="2.212147048s" totalImagesPullingTime="322.93788ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:35:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:35:37.210787497 +0000 UTC m=+533.466186591" watchObservedRunningTime="2026-11-02 12:35:37.212147048 +0000 UTC m=+533.467546142"
Nov 02 12:35:38.141355 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:38.141320    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f080cb5f-6486999cf7-lw2zl"
Nov 02 12:35:40.063676 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:40.063579    4559 patch_prober.go:29] interesting pod/e2e-migrate-d9dc4ff4-gw-openshift-default-9bfdcc79c-w7gqc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Nov 02 12:35:40.064090 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:40.063673    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-d9dc4ff4-gw-openshift-default-9bfdcc79c-w7gqc" podUID="8b64510b-aa47-4a38-ac26-7c0df79188b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Nov 02 12:35:40.183947 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:40.183903    4559 generic.go:356] "Generic (PLEG): container finished" podID="8b64510b-aa47-4a38-ac26-7c0df79188b2" containerID="7f777244670ed6d28a65b70700029490b9add71a7800a2a20e511bec14316af5" exitCode=0
Nov 02 12:35:40.184159 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:40.184008    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-d9dc4ff4-gw-openshift-default-9bfdcc79c-w7gqc" event={"ID":"8b64510b-aa47-4a38-ac26-7c0df79188b2","Type":"ContainerDied","Data":"7f777244670ed6d28a65b70700029490b9add71a7800a2a20e511bec14316af5"}
Nov 02 12:35:40.184456 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:40.184429    4559 scope.go:122] "RemoveContainer" containerID="7f777244670ed6d28a65b70700029490b9add71a7800a2a20e511bec14316af5"
Nov 02 12:35:40.185972 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:40.185925    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:35:40.186098 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:40.186022    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:35:40.186098 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:40.186069    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:35:40.240718 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:40.231158    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"]
Nov 02 12:35:41.212190 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:41.212158    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-d9dc4ff4-gw-openshift-default-9bfdcc79c-w7gqc" event={"ID":"8b64510b-aa47-4a38-ac26-7c0df79188b2","Type":"ContainerStarted","Data":"ef3e4263fb8bfb9a56fb7780ee2a8a52c04f4b659059288983a7fada9ffa8f75"}
Nov 02 12:35:41.582263 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:41.582180    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-d9dc4ff4-gw-openshift-default-9bfdcc79c-w7gqc"
Nov 02 12:35:41.582406 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:41.582365    4559 patch_prober.go:29] interesting pod/e2e-migrate-d9dc4ff4-gw-openshift-default-9bfdcc79c-w7gqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Nov 02 12:35:41.582447 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:41.582424    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-d9dc4ff4-gw-openshift-default-9bfdcc79c-w7gqc" podUID="8b64510b-aa47-4a38-ac26-7c0df79188b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Nov 02 12:35:42.582722 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:42.582688    4559 patch_prober.go:29] interesting pod/e2e-migrate-d9dc4ff4-gw-openshift-default-9bfdcc79c-w7gqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Nov 02 12:35:42.583274 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:42.583243    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-d9dc4ff4-gw-openshift-default-9bfdcc79c-w7gqc" podUID="8b64510b-aa47-4a38-ac26-7c0df79188b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Nov 02 12:35:43.000836 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:43.000799    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-2f0bef09-68f9f765c8-zngcp"]
Nov 02 12:35:43.041435 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:43.041264    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-2f0bef09-68f9f765c8-zngcp"]
Nov 02 12:35:43.041609 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:43.041445    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2f0bef09-68f9f765c8-zngcp"
Nov 02 12:35:43.056743 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:43.056718    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-2f0bef09-serving-cert\""
Nov 02 12:35:43.139746 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:43.139704    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-f080cb5f-76f8d88955-vn85z"
Nov 02 12:35:43.152261 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:43.152230    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d393c034-7597-427f-af11-aed585377518-maas-api-tls\") pod \"maas-api-e2e-derive-2f0bef09-68f9f765c8-zngcp\" (UID: \"d393c034-7597-427f-af11-aed585377518\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2f0bef09-68f9f765c8-zngcp"
Nov 02 12:35:43.152426 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:43.152269    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d393c034-7597-427f-af11-aed585377518-openshift-service-ca\") pod \"maas-api-e2e-derive-2f0bef09-68f9f765c8-zngcp\" (UID: \"d393c034-7597-427f-af11-aed585377518\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2f0bef09-68f9f765c8-zngcp"
Nov 02 12:35:43.152426 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:43.152295    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d393c034-7597-427f-af11-aed585377518-metrics-certs\") pod \"maas-api-e2e-derive-2f0bef09-68f9f765c8-zngcp\" (UID: \"d393c034-7597-427f-af11-aed585377518\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2f0bef09-68f9f765c8-zngcp"
Nov 02 12:35:43.152426 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:43.152310    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92hvr\" (UniqueName: \"kubernetes.io/projected/d393c034-7597-427f-af11-aed585377518-kube-api-access-92hvr\") pod \"maas-api-e2e-derive-2f0bef09-68f9f765c8-zngcp\" (UID: \"d393c034-7597-427f-af11-aed585377518\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2f0bef09-68f9f765c8-zngcp"
Nov 02 12:35:43.179652 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:43.179605    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-628cf9bd-f5bbdc6cf-49rc4"
Nov 02 12:35:43.253870 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:43.253790    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d393c034-7597-427f-af11-aed585377518-maas-api-tls\") pod \"maas-api-e2e-derive-2f0bef09-68f9f765c8-zngcp\" (UID: \"d393c034-7597-427f-af11-aed585377518\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2f0bef09-68f9f765c8-zngcp"
Nov 02 12:35:43.253870 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:43.253834    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d393c034-7597-427f-af11-aed585377518-openshift-service-ca\") pod \"maas-api-e2e-derive-2f0bef09-68f9f765c8-zngcp\" (UID: \"d393c034-7597-427f-af11-aed585377518\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2f0bef09-68f9f765c8-zngcp"
Nov 02 12:35:43.254973 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:43.253879    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d393c034-7597-427f-af11-aed585377518-metrics-certs\") pod \"maas-api-e2e-derive-2f0bef09-68f9f765c8-zngcp\" (UID: \"d393c034-7597-427f-af11-aed585377518\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2f0bef09-68f9f765c8-zngcp"
Nov 02 12:35:43.254973 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:43.253904    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92hvr\" (UniqueName: \"kubernetes.io/projected/d393c034-7597-427f-af11-aed585377518-kube-api-access-92hvr\") pod \"maas-api-e2e-derive-2f0bef09-68f9f765c8-zngcp\" (UID: \"d393c034-7597-427f-af11-aed585377518\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2f0bef09-68f9f765c8-zngcp"
Nov 02 12:35:43.254973 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:43.254772    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d393c034-7597-427f-af11-aed585377518-openshift-service-ca\") pod \"maas-api-e2e-derive-2f0bef09-68f9f765c8-zngcp\" (UID: \"d393c034-7597-427f-af11-aed585377518\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2f0bef09-68f9f765c8-zngcp"
Nov 02 12:35:43.257593 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:43.257564    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d393c034-7597-427f-af11-aed585377518-metrics-certs\") pod \"maas-api-e2e-derive-2f0bef09-68f9f765c8-zngcp\" (UID: \"d393c034-7597-427f-af11-aed585377518\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2f0bef09-68f9f765c8-zngcp"
Nov 02 12:35:43.257779 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:43.257760    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d393c034-7597-427f-af11-aed585377518-maas-api-tls\") pod \"maas-api-e2e-derive-2f0bef09-68f9f765c8-zngcp\" (UID: \"d393c034-7597-427f-af11-aed585377518\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2f0bef09-68f9f765c8-zngcp"
Nov 02 12:35:43.277962 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:43.277928    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-92hvr\" (UniqueName: \"kubernetes.io/projected/d393c034-7597-427f-af11-aed585377518-kube-api-access-92hvr\") pod \"maas-api-e2e-derive-2f0bef09-68f9f765c8-zngcp\" (UID: \"d393c034-7597-427f-af11-aed585377518\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2f0bef09-68f9f765c8-zngcp"
Nov 02 12:35:43.359404 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:43.359357    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2f0bef09-68f9f765c8-zngcp"
Nov 02 12:35:43.582106 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:43.582028    4559 patch_prober.go:29] interesting pod/e2e-migrate-d9dc4ff4-gw-openshift-default-9bfdcc79c-w7gqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Nov 02 12:35:43.582106 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:43.582080    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-d9dc4ff4-gw-openshift-default-9bfdcc79c-w7gqc" podUID="8b64510b-aa47-4a38-ac26-7c0df79188b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Nov 02 12:35:43.741860 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:43.740483    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-2f0bef09-68f9f765c8-zngcp"]
Nov 02 12:35:43.744601 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:35:43.744557    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd393c034_7597_427f_af11_aed585377518.slice/crio-e6cd6b2891d2b19e81f3b45ff235c4f867b1ce6cc94d868d0c5818c82113fa53 WatchSource:0}: Error finding container e6cd6b2891d2b19e81f3b45ff235c4f867b1ce6cc94d868d0c5818c82113fa53: Status 404 returned error can't find the container with id e6cd6b2891d2b19e81f3b45ff235c4f867b1ce6cc94d868d0c5818c82113fa53
Nov 02 12:35:44.254725 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:44.254685    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2f0bef09-68f9f765c8-zngcp" event={"ID":"d393c034-7597-427f-af11-aed585377518","Type":"ContainerStarted","Data":"2a3e0d9a1650ae5a2fd3d395c920f9b4b97eedecaf3fec239b6a18c9b5839ca4"}
Nov 02 12:35:44.254882 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:44.254776    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2f0bef09-68f9f765c8-zngcp" event={"ID":"d393c034-7597-427f-af11-aed585377518","Type":"ContainerStarted","Data":"e6cd6b2891d2b19e81f3b45ff235c4f867b1ce6cc94d868d0c5818c82113fa53"}
Nov 02 12:35:44.254882 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:44.254820    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2f0bef09-68f9f765c8-zngcp"
Nov 02 12:35:44.601665 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:44.591017    4559 patch_prober.go:29] interesting pod/e2e-migrate-d9dc4ff4-gw-openshift-default-9bfdcc79c-w7gqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Nov 02 12:35:44.601665 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:44.591080    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-d9dc4ff4-gw-openshift-default-9bfdcc79c-w7gqc" podUID="8b64510b-aa47-4a38-ac26-7c0df79188b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Nov 02 12:35:45.114498 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:35:45.114408    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqcmj" containerName="extractor"
Nov 02 12:35:45.590682 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:45.582383    4559 patch_prober.go:29] interesting pod/e2e-migrate-d9dc4ff4-gw-openshift-default-9bfdcc79c-w7gqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Nov 02 12:35:45.590682 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:45.582436    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-d9dc4ff4-gw-openshift-default-9bfdcc79c-w7gqc" podUID="8b64510b-aa47-4a38-ac26-7c0df79188b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Nov 02 12:35:46.235689 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:46.228461    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2f0bef09-68f9f765c8-zngcp" podStartSLOduration=3.895492483 podStartE2EDuration="4.228437434s" totalImagesPullingTime="332.944951ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:35:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:35:44.303538865 +0000 UTC m=+540.558937962" watchObservedRunningTime="2026-11-02 12:35:46.228437434 +0000 UTC m=+542.483836530"
Nov 02 12:35:46.235689 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:46.228877    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"]
Nov 02 12:35:46.602032 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:46.601946    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-d9dc4ff4-gw-openshift-default-9bfdcc79c-w7gqc"
Nov 02 12:35:46.603535 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:46.603512    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-d9dc4ff4-gw-openshift-default-9bfdcc79c-w7gqc"
Nov 02 12:35:46.610400 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:46.608993    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-d9dc4ff4-gw-openshift-default-9bfdcc79c-w7gqc"
Nov 02 12:35:50.264716 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:50.264689    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2f0bef09-68f9f765c8-zngcp"
Nov 02 12:35:51.314738 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:51.314703    4559 generic.go:356] "Generic (PLEG): container finished" podID="54f220f8-02ad-4a8b-8c3a-735b0864bdff" containerID="e376457369e920ce66d4a6383f4a72ac24b531aeaf6e541fdaf1ffa9e98bd0bb" exitCode=0
Nov 02 12:35:51.315154 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:51.314776    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-628cf9bd-openshift-default-75cb8d8c97-g77zh" event={"ID":"54f220f8-02ad-4a8b-8c3a-735b0864bdff","Type":"ContainerDied","Data":"e376457369e920ce66d4a6383f4a72ac24b531aeaf6e541fdaf1ffa9e98bd0bb"}
Nov 02 12:35:51.315207 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:51.315165    4559 scope.go:122] "RemoveContainer" containerID="e376457369e920ce66d4a6383f4a72ac24b531aeaf6e541fdaf1ffa9e98bd0bb"
Nov 02 12:35:51.316402 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:51.316374    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:35:51.316457 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:51.316444    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:35:51.316500 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:51.316479    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:35:52.321728 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:52.321690    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-628cf9bd-openshift-default-75cb8d8c97-g77zh" event={"ID":"54f220f8-02ad-4a8b-8c3a-735b0864bdff","Type":"ContainerStarted","Data":"b09e7499c573a6f4c04b42a3ae75e4892fc57acb825884bb78f5c54606e67046"}
Nov 02 12:35:53.101665 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:53.097929    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-628cf9bd-openshift-default-75cb8d8c97-g77zh"
Nov 02 12:35:53.101665 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:53.098101    4559 patch_prober.go:29] interesting pod/e2e-ait-628cf9bd-openshift-default-75cb8d8c97-g77zh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body=
Nov 02 12:35:53.101665 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:53.098146    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-628cf9bd-openshift-default-75cb8d8c97-g77zh" podUID="54f220f8-02ad-4a8b-8c3a-735b0864bdff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused"
Nov 02 12:35:54.105924 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:54.105128    4559 patch_prober.go:29] interesting pod/e2e-ait-628cf9bd-openshift-default-75cb8d8c97-g77zh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body=
Nov 02 12:35:54.105924 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:54.105206    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-628cf9bd-openshift-default-75cb8d8c97-g77zh" podUID="54f220f8-02ad-4a8b-8c3a-735b0864bdff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused"
Nov 02 12:35:55.082533 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:55.082495    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-d9dc4ff4-5dbf6d9d4d-l2fkz"]
Nov 02 12:35:55.082813 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:55.082781    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d9dc4ff4-5dbf6d9d4d-l2fkz" podUID="f83542d0-1d12-4105-b625-54f0764c6f8b" containerName="maas-api" containerID="cri-o://e3a5a2b8d9b994475784028ada55d5ebddb7bd67ad2f52e8b79d79e247c94421" gracePeriod=30
Nov 02 12:35:55.099132 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:55.098787    4559 patch_prober.go:29] interesting pod/e2e-ait-628cf9bd-openshift-default-75cb8d8c97-g77zh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body=
Nov 02 12:35:55.100802 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:55.100767    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-628cf9bd-openshift-default-75cb8d8c97-g77zh" podUID="54f220f8-02ad-4a8b-8c3a-735b0864bdff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused"
Nov 02 12:35:55.363615 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:55.363527    4559 generic.go:356] "Generic (PLEG): container finished" podID="f83542d0-1d12-4105-b625-54f0764c6f8b" containerID="e3a5a2b8d9b994475784028ada55d5ebddb7bd67ad2f52e8b79d79e247c94421" exitCode=0
Nov 02 12:35:55.364249 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:55.364217    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d9dc4ff4-5dbf6d9d4d-l2fkz" event={"ID":"f83542d0-1d12-4105-b625-54f0764c6f8b","Type":"ContainerDied","Data":"e3a5a2b8d9b994475784028ada55d5ebddb7bd67ad2f52e8b79d79e247c94421"}
Nov 02 12:35:55.446621 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:55.446596    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d9dc4ff4-5dbf6d9d4d-l2fkz"
Nov 02 12:35:55.466304 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:55.466281    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f83542d0-1d12-4105-b625-54f0764c6f8b-kube-api-access-b2qpx\" (UniqueName: \"kubernetes.io/projected/f83542d0-1d12-4105-b625-54f0764c6f8b-kube-api-access-b2qpx\") pod \"f83542d0-1d12-4105-b625-54f0764c6f8b\" (UID: \"f83542d0-1d12-4105-b625-54f0764c6f8b\") "
Nov 02 12:35:55.466432 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:55.466352    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f83542d0-1d12-4105-b625-54f0764c6f8b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f83542d0-1d12-4105-b625-54f0764c6f8b-maas-api-tls\") pod \"f83542d0-1d12-4105-b625-54f0764c6f8b\" (UID: \"f83542d0-1d12-4105-b625-54f0764c6f8b\") "
Nov 02 12:35:55.466432 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:55.466378    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f83542d0-1d12-4105-b625-54f0764c6f8b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f83542d0-1d12-4105-b625-54f0764c6f8b-openshift-service-ca\") pod \"f83542d0-1d12-4105-b625-54f0764c6f8b\" (UID: \"f83542d0-1d12-4105-b625-54f0764c6f8b\") "
Nov 02 12:35:55.466432 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:55.466427    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f83542d0-1d12-4105-b625-54f0764c6f8b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f83542d0-1d12-4105-b625-54f0764c6f8b-metrics-certs\") pod \"f83542d0-1d12-4105-b625-54f0764c6f8b\" (UID: \"f83542d0-1d12-4105-b625-54f0764c6f8b\") "
Nov 02 12:35:55.466795 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:55.466762    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f83542d0-1d12-4105-b625-54f0764c6f8b-openshift-service-ca" pod "f83542d0-1d12-4105-b625-54f0764c6f8b" (UID: "f83542d0-1d12-4105-b625-54f0764c6f8b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:35:55.466924 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:55.466907    4559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f83542d0-1d12-4105-b625-54f0764c6f8b-openshift-service-ca\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:35:55.468310 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:55.468281    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f83542d0-1d12-4105-b625-54f0764c6f8b-kube-api-access-b2qpx" pod "f83542d0-1d12-4105-b625-54f0764c6f8b" (UID: "f83542d0-1d12-4105-b625-54f0764c6f8b"). InnerVolumeSpecName "kube-api-access-b2qpx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:35:55.468410 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:55.468363    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f83542d0-1d12-4105-b625-54f0764c6f8b-metrics-certs" pod "f83542d0-1d12-4105-b625-54f0764c6f8b" (UID: "f83542d0-1d12-4105-b625-54f0764c6f8b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:35:55.468410 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:55.468380    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f83542d0-1d12-4105-b625-54f0764c6f8b-maas-api-tls" pod "f83542d0-1d12-4105-b625-54f0764c6f8b" (UID: "f83542d0-1d12-4105-b625-54f0764c6f8b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:35:55.567378 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:55.567344    4559 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f83542d0-1d12-4105-b625-54f0764c6f8b-metrics-certs\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:35:55.567378 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:55.567371    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b2qpx\" (UniqueName: \"kubernetes.io/projected/f83542d0-1d12-4105-b625-54f0764c6f8b-kube-api-access-b2qpx\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:35:55.567378 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:55.567382    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f83542d0-1d12-4105-b625-54f0764c6f8b-maas-api-tls\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:35:55.938745 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:55.938705    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-d9dc4ff4-gw-openshift-default-9bfdcc79c-w7gqc"]
Nov 02 12:35:55.938951 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:55.938924    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-d9dc4ff4-gw-openshift-default-9bfdcc79c-w7gqc" podUID="8b64510b-aa47-4a38-ac26-7c0df79188b2" containerName="istio-proxy" containerID="cri-o://ef3e4263fb8bfb9a56fb7780ee2a8a52c04f4b659059288983a7fada9ffa8f75" gracePeriod=30
Nov 02 12:35:56.102964 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:56.102929    4559 patch_prober.go:29] interesting pod/e2e-ait-628cf9bd-openshift-default-75cb8d8c97-g77zh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body=
Nov 02 12:35:56.103232 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:56.103203    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-628cf9bd-openshift-default-75cb8d8c97-g77zh" podUID="54f220f8-02ad-4a8b-8c3a-735b0864bdff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused"
Nov 02 12:35:56.370100 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:56.370014    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d9dc4ff4-5dbf6d9d4d-l2fkz" event={"ID":"f83542d0-1d12-4105-b625-54f0764c6f8b","Type":"ContainerDied","Data":"44c142472fdf6e903376a7252c48e6bb86e012fd652f42ddc9ef0b07d07ba21e"}
Nov 02 12:35:56.370100 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:56.370046    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d9dc4ff4-5dbf6d9d4d-l2fkz"
Nov 02 12:35:56.370100 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:56.370078    4559 scope.go:122] "RemoveContainer" containerID="e3a5a2b8d9b994475784028ada55d5ebddb7bd67ad2f52e8b79d79e247c94421"
Nov 02 12:35:56.467227 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:56.467193    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-d9dc4ff4-5dbf6d9d4d-l2fkz"]
Nov 02 12:35:56.504664 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:56.499192    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-d9dc4ff4-5dbf6d9d4d-l2fkz"]
Nov 02 12:35:56.865361 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:56.865323    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f080cb5f-76f8d88955-vn85z"]
Nov 02 12:35:57.089689 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:57.074144    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-628cf9bd-openshift-default-75cb8d8c97-g77zh"
Nov 02 12:35:57.118724 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:57.103028    4559 patch_prober.go:29] interesting pod/e2e-ait-628cf9bd-openshift-default-75cb8d8c97-g77zh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body=
Nov 02 12:35:57.118724 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:57.103089    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-628cf9bd-openshift-default-75cb8d8c97-g77zh" podUID="54f220f8-02ad-4a8b-8c3a-735b0864bdff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused"
Nov 02 12:35:57.317991 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:57.317956    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-53b6603d-846469fb44-gc4xc"]
Nov 02 12:35:57.320424 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:57.320156    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-53b6603d-846469fb44-gc4xc" podUID="cfa55f58-67b9-4026-b30f-2abdbb09b87c" containerName="maas-api" containerID="cri-o://0de29d26b955631ff6a120d1c3882c68691d73e3b1ec377effef4530ebb57953" gracePeriod=30
Nov 02 12:35:57.619384 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:57.619352    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-53b6603d-846469fb44-gc4xc"
Nov 02 12:35:57.695860 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:57.695292    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f080cb5f-6486999cf7-lw2zl"]
Nov 02 12:35:57.695860 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:57.695788    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f080cb5f-6486999cf7-lw2zl" podUID="33114556-fa3f-4168-9da6-9ca4a74802f3" containerName="maas-api" containerID="cri-o://1d59509153523e0c0eef638d9038fab237c89a4e64e3c3828511116b4516bc52" gracePeriod=30
Nov 02 12:35:57.714460 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:57.714425    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cfa55f58-67b9-4026-b30f-2abdbb09b87c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cfa55f58-67b9-4026-b30f-2abdbb09b87c-maas-api-tls\") pod \"cfa55f58-67b9-4026-b30f-2abdbb09b87c\" (UID: \"cfa55f58-67b9-4026-b30f-2abdbb09b87c\") "
Nov 02 12:35:57.714460 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:57.714484    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cfa55f58-67b9-4026-b30f-2abdbb09b87c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfa55f58-67b9-4026-b30f-2abdbb09b87c-openshift-service-ca\") pod \"cfa55f58-67b9-4026-b30f-2abdbb09b87c\" (UID: \"cfa55f58-67b9-4026-b30f-2abdbb09b87c\") "
Nov 02 12:35:57.714763 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:57.714628    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cfa55f58-67b9-4026-b30f-2abdbb09b87c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/cfa55f58-67b9-4026-b30f-2abdbb09b87c-metrics-certs\") pod \"cfa55f58-67b9-4026-b30f-2abdbb09b87c\" (UID: \"cfa55f58-67b9-4026-b30f-2abdbb09b87c\") "
Nov 02 12:35:57.714763 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:57.714745    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cfa55f58-67b9-4026-b30f-2abdbb09b87c-kube-api-access-zwtpm\" (UniqueName: \"kubernetes.io/projected/cfa55f58-67b9-4026-b30f-2abdbb09b87c-kube-api-access-zwtpm\") pod \"cfa55f58-67b9-4026-b30f-2abdbb09b87c\" (UID: \"cfa55f58-67b9-4026-b30f-2abdbb09b87c\") "
Nov 02 12:35:57.718873 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:57.717306    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cfa55f58-67b9-4026-b30f-2abdbb09b87c-maas-api-tls" pod "cfa55f58-67b9-4026-b30f-2abdbb09b87c" (UID: "cfa55f58-67b9-4026-b30f-2abdbb09b87c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:35:57.719197 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:57.719158    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfa55f58-67b9-4026-b30f-2abdbb09b87c-kube-api-access-zwtpm" pod "cfa55f58-67b9-4026-b30f-2abdbb09b87c" (UID: "cfa55f58-67b9-4026-b30f-2abdbb09b87c"). InnerVolumeSpecName "kube-api-access-zwtpm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:35:57.720161 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:57.719455    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfa55f58-67b9-4026-b30f-2abdbb09b87c-openshift-service-ca" pod "cfa55f58-67b9-4026-b30f-2abdbb09b87c" (UID: "cfa55f58-67b9-4026-b30f-2abdbb09b87c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:35:57.720161 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:57.719889    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cfa55f58-67b9-4026-b30f-2abdbb09b87c-metrics-certs" pod "cfa55f58-67b9-4026-b30f-2abdbb09b87c" (UID: "cfa55f58-67b9-4026-b30f-2abdbb09b87c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:35:57.815881 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:57.815785    4559 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cfa55f58-67b9-4026-b30f-2abdbb09b87c-metrics-certs\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:35:57.815881 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:57.815815    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zwtpm\" (UniqueName: \"kubernetes.io/projected/cfa55f58-67b9-4026-b30f-2abdbb09b87c-kube-api-access-zwtpm\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:35:57.815881 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:57.815829    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cfa55f58-67b9-4026-b30f-2abdbb09b87c-maas-api-tls\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:35:57.815881 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:57.815841    4559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfa55f58-67b9-4026-b30f-2abdbb09b87c-openshift-service-ca\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:35:57.894714 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:57.894674    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f83542d0-1d12-4105-b625-54f0764c6f8b" path="/var/lib/kubelet/pods/f83542d0-1d12-4105-b625-54f0764c6f8b/volumes"
Nov 02 12:35:57.944333 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:57.944309    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f080cb5f-6486999cf7-lw2zl"
Nov 02 12:35:58.018308 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:58.018221    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33114556-fa3f-4168-9da6-9ca4a74802f3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33114556-fa3f-4168-9da6-9ca4a74802f3-openshift-service-ca\") pod \"33114556-fa3f-4168-9da6-9ca4a74802f3\" (UID: \"33114556-fa3f-4168-9da6-9ca4a74802f3\") "
Nov 02 12:35:58.018474 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:58.018309    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/33114556-fa3f-4168-9da6-9ca4a74802f3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/33114556-fa3f-4168-9da6-9ca4a74802f3-metrics-certs\") pod \"33114556-fa3f-4168-9da6-9ca4a74802f3\" (UID: \"33114556-fa3f-4168-9da6-9ca4a74802f3\") "
Nov 02 12:35:58.018474 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:58.018403    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33114556-fa3f-4168-9da6-9ca4a74802f3-kube-api-access-fzfwm\" (UniqueName: \"kubernetes.io/projected/33114556-fa3f-4168-9da6-9ca4a74802f3-kube-api-access-fzfwm\") pod \"33114556-fa3f-4168-9da6-9ca4a74802f3\" (UID: \"33114556-fa3f-4168-9da6-9ca4a74802f3\") "
Nov 02 12:35:58.018474 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:58.018441    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/33114556-fa3f-4168-9da6-9ca4a74802f3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33114556-fa3f-4168-9da6-9ca4a74802f3-maas-api-tls\") pod \"33114556-fa3f-4168-9da6-9ca4a74802f3\" (UID: \"33114556-fa3f-4168-9da6-9ca4a74802f3\") "
Nov 02 12:35:58.018704 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:58.018672    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33114556-fa3f-4168-9da6-9ca4a74802f3-openshift-service-ca" pod "33114556-fa3f-4168-9da6-9ca4a74802f3" (UID: "33114556-fa3f-4168-9da6-9ca4a74802f3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:35:58.018789 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:58.018774    4559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33114556-fa3f-4168-9da6-9ca4a74802f3-openshift-service-ca\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:35:58.020782 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:58.020747    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33114556-fa3f-4168-9da6-9ca4a74802f3-maas-api-tls" pod "33114556-fa3f-4168-9da6-9ca4a74802f3" (UID: "33114556-fa3f-4168-9da6-9ca4a74802f3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:35:58.020934 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:58.020913    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33114556-fa3f-4168-9da6-9ca4a74802f3-metrics-certs" pod "33114556-fa3f-4168-9da6-9ca4a74802f3" (UID: "33114556-fa3f-4168-9da6-9ca4a74802f3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:35:58.020993 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:58.020923    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33114556-fa3f-4168-9da6-9ca4a74802f3-kube-api-access-fzfwm" pod "33114556-fa3f-4168-9da6-9ca4a74802f3" (UID: "33114556-fa3f-4168-9da6-9ca4a74802f3"). InnerVolumeSpecName "kube-api-access-fzfwm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:35:58.092368 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:58.092334    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-628cf9bd-openshift-default-75cb8d8c97-g77zh"
Nov 02 12:35:58.093345 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:58.093319    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-628cf9bd-openshift-default-75cb8d8c97-g77zh"
Nov 02 12:35:58.119611 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:58.119581    4559 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33114556-fa3f-4168-9da6-9ca4a74802f3-metrics-certs\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:35:58.119611 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:58.119606    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fzfwm\" (UniqueName: \"kubernetes.io/projected/33114556-fa3f-4168-9da6-9ca4a74802f3-kube-api-access-fzfwm\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:35:58.119818 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:58.119620    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33114556-fa3f-4168-9da6-9ca4a74802f3-maas-api-tls\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:35:58.383165 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:58.383133    4559 generic.go:356] "Generic (PLEG): container finished" podID="33114556-fa3f-4168-9da6-9ca4a74802f3" containerID="1d59509153523e0c0eef638d9038fab237c89a4e64e3c3828511116b4516bc52" exitCode=0
Nov 02 12:35:58.383346 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:58.383194    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f080cb5f-6486999cf7-lw2zl"
Nov 02 12:35:58.383346 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:58.383201    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f080cb5f-6486999cf7-lw2zl" event={"ID":"33114556-fa3f-4168-9da6-9ca4a74802f3","Type":"ContainerDied","Data":"1d59509153523e0c0eef638d9038fab237c89a4e64e3c3828511116b4516bc52"}
Nov 02 12:35:58.383346 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:58.383259    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f080cb5f-6486999cf7-lw2zl" event={"ID":"33114556-fa3f-4168-9da6-9ca4a74802f3","Type":"ContainerDied","Data":"c9787efce920e7e516db12c89a947cc29958efec3326dd002577b090462d6816"}
Nov 02 12:35:58.383346 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:58.383289    4559 scope.go:122] "RemoveContainer" containerID="1d59509153523e0c0eef638d9038fab237c89a4e64e3c3828511116b4516bc52"
Nov 02 12:35:58.384818 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:58.384791    4559 generic.go:356] "Generic (PLEG): container finished" podID="cfa55f58-67b9-4026-b30f-2abdbb09b87c" containerID="0de29d26b955631ff6a120d1c3882c68691d73e3b1ec377effef4530ebb57953" exitCode=0
Nov 02 12:35:58.384940 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:58.384845    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-53b6603d-846469fb44-gc4xc"
Nov 02 12:35:58.384940 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:58.384872    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-53b6603d-846469fb44-gc4xc" event={"ID":"cfa55f58-67b9-4026-b30f-2abdbb09b87c","Type":"ContainerDied","Data":"0de29d26b955631ff6a120d1c3882c68691d73e3b1ec377effef4530ebb57953"}
Nov 02 12:35:58.384940 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:58.384913    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-53b6603d-846469fb44-gc4xc" event={"ID":"cfa55f58-67b9-4026-b30f-2abdbb09b87c","Type":"ContainerDied","Data":"d1855c7411e11817ff15863d8703884780661b63cc888b44cbdc3d67bd52e28c"}
Nov 02 12:35:58.392424 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:58.392407    4559 scope.go:122] "RemoveContainer" containerID="1d59509153523e0c0eef638d9038fab237c89a4e64e3c3828511116b4516bc52"
Nov 02 12:35:58.392692 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:35:58.392668    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d59509153523e0c0eef638d9038fab237c89a4e64e3c3828511116b4516bc52\": container with ID starting with 1d59509153523e0c0eef638d9038fab237c89a4e64e3c3828511116b4516bc52 not found: ID does not exist" containerID="1d59509153523e0c0eef638d9038fab237c89a4e64e3c3828511116b4516bc52"
Nov 02 12:35:58.392782 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:58.392704    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d59509153523e0c0eef638d9038fab237c89a4e64e3c3828511116b4516bc52"} err="failed to get container status \"1d59509153523e0c0eef638d9038fab237c89a4e64e3c3828511116b4516bc52\": rpc error: code = NotFound desc = could not find container \"1d59509153523e0c0eef638d9038fab237c89a4e64e3c3828511116b4516bc52\": container with ID starting with 1d59509153523e0c0eef638d9038fab237c89a4e64e3c3828511116b4516bc52 not found: ID does not exist"
Nov 02 12:35:58.392782 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:58.392730    4559 scope.go:122] "RemoveContainer" containerID="0de29d26b955631ff6a120d1c3882c68691d73e3b1ec377effef4530ebb57953"
Nov 02 12:35:58.400561 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:58.400544    4559 scope.go:122] "RemoveContainer" containerID="0de29d26b955631ff6a120d1c3882c68691d73e3b1ec377effef4530ebb57953"
Nov 02 12:35:58.400840 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:35:58.400820    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0de29d26b955631ff6a120d1c3882c68691d73e3b1ec377effef4530ebb57953\": container with ID starting with 0de29d26b955631ff6a120d1c3882c68691d73e3b1ec377effef4530ebb57953 not found: ID does not exist" containerID="0de29d26b955631ff6a120d1c3882c68691d73e3b1ec377effef4530ebb57953"
Nov 02 12:35:58.400889 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:58.400847    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0de29d26b955631ff6a120d1c3882c68691d73e3b1ec377effef4530ebb57953"} err="failed to get container status \"0de29d26b955631ff6a120d1c3882c68691d73e3b1ec377effef4530ebb57953\": rpc error: code = NotFound desc = could not find container \"0de29d26b955631ff6a120d1c3882c68691d73e3b1ec377effef4530ebb57953\": container with ID starting with 0de29d26b955631ff6a120d1c3882c68691d73e3b1ec377effef4530ebb57953 not found: ID does not exist"
Nov 02 12:35:58.437550 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:58.437486    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f080cb5f-6486999cf7-lw2zl"]
Nov 02 12:35:58.442403 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:58.442373    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f080cb5f-6486999cf7-lw2zl"]
Nov 02 12:35:58.481977 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:58.481949    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-53b6603d-846469fb44-gc4xc"]
Nov 02 12:35:58.486933 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:58.486904    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-53b6603d-846469fb44-gc4xc"]
Nov 02 12:35:59.094834 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:59.094799    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"]
Nov 02 12:35:59.900225 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:59.900188    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33114556-fa3f-4168-9da6-9ca4a74802f3" path="/var/lib/kubelet/pods/33114556-fa3f-4168-9da6-9ca4a74802f3/volumes"
Nov 02 12:35:59.900774 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:59.900754    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cfa55f58-67b9-4026-b30f-2abdbb09b87c" path="/var/lib/kubelet/pods/cfa55f58-67b9-4026-b30f-2abdbb09b87c/volumes"
Nov 02 12:35:59.911870 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:59.911844    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-bh68g"]
Nov 02 12:35:59.912253 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:59.912237    4559 cpu_manager.go:414] "containerMap: removing container" podUID="cfa55f58-67b9-4026-b30f-2abdbb09b87c" containerName="maas-api"
Nov 02 12:35:59.912253 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:59.912252    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cfa55f58-67b9-4026-b30f-2abdbb09b87c" containerName="maas-api"
Nov 02 12:35:59.912253 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:59.912266    4559 cpu_manager.go:414] "containerMap: removing container" podUID="33114556-fa3f-4168-9da6-9ca4a74802f3" containerName="maas-api"
Nov 02 12:35:59.912479 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:59.912274    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33114556-fa3f-4168-9da6-9ca4a74802f3" containerName="maas-api"
Nov 02 12:35:59.912479 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:59.912290    4559 cpu_manager.go:414] "containerMap: removing container" podUID="f83542d0-1d12-4105-b625-54f0764c6f8b" containerName="maas-api"
Nov 02 12:35:59.912479 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:59.912296    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f83542d0-1d12-4105-b625-54f0764c6f8b" containerName="maas-api"
Nov 02 12:35:59.912479 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:59.912351    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="cfa55f58-67b9-4026-b30f-2abdbb09b87c" containerName="maas-api"
Nov 02 12:35:59.912479 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:59.912362    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="33114556-fa3f-4168-9da6-9ca4a74802f3" containerName="maas-api"
Nov 02 12:35:59.912479 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:59.912369    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="f83542d0-1d12-4105-b625-54f0764c6f8b" containerName="maas-api"
Nov 02 12:35:59.917395 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:59.917378    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-bh68g"
Nov 02 12:35:59.937650 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:59.936204    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-95ee26-openshift-default-dockercfg-xjh8d\""
Nov 02 12:35:59.952026 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:35:59.951998    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-bh68g"]
Nov 02 12:36:00.042416 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:00.042373    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76dc13d1-a3a7-4f47-85a2-f596cafbcb94-workload-certs\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-bh68g\" (UID: \"76dc13d1-a3a7-4f47-85a2-f596cafbcb94\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-bh68g"
Nov 02 12:36:00.042592 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:00.042429    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76dc13d1-a3a7-4f47-85a2-f596cafbcb94-istio-envoy\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-bh68g\" (UID: \"76dc13d1-a3a7-4f47-85a2-f596cafbcb94\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-bh68g"
Nov 02 12:36:00.042592 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:00.042460    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76dc13d1-a3a7-4f47-85a2-f596cafbcb94-workload-socket\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-bh68g\" (UID: \"76dc13d1-a3a7-4f47-85a2-f596cafbcb94\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-bh68g"
Nov 02 12:36:00.042592 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:00.042540    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76dc13d1-a3a7-4f47-85a2-f596cafbcb94-istiod-ca-cert\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-bh68g\" (UID: \"76dc13d1-a3a7-4f47-85a2-f596cafbcb94\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-bh68g"
Nov 02 12:36:00.042592 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:00.042582    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76dc13d1-a3a7-4f47-85a2-f596cafbcb94-istio-data\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-bh68g\" (UID: \"76dc13d1-a3a7-4f47-85a2-f596cafbcb94\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-bh68g"
Nov 02 12:36:00.042862 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:00.042616    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76dc13d1-a3a7-4f47-85a2-f596cafbcb94-istio-token\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-bh68g\" (UID: \"76dc13d1-a3a7-4f47-85a2-f596cafbcb94\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-bh68g"
Nov 02 12:36:00.042862 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:00.042677    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76dc13d1-a3a7-4f47-85a2-f596cafbcb94-istio-podinfo\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-bh68g\" (UID: \"76dc13d1-a3a7-4f47-85a2-f596cafbcb94\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-bh68g"
Nov 02 12:36:00.042862 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:00.042709    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfps2\" (UniqueName: \"kubernetes.io/projected/76dc13d1-a3a7-4f47-85a2-f596cafbcb94-kube-api-access-hfps2\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-bh68g\" (UID: \"76dc13d1-a3a7-4f47-85a2-f596cafbcb94\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-bh68g"
Nov 02 12:36:00.042862 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:00.042740    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76dc13d1-a3a7-4f47-85a2-f596cafbcb94-credential-socket\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-bh68g\" (UID: \"76dc13d1-a3a7-4f47-85a2-f596cafbcb94\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-bh68g"
Nov 02 12:36:00.144027 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:00.143995    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76dc13d1-a3a7-4f47-85a2-f596cafbcb94-istio-token\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-bh68g\" (UID: \"76dc13d1-a3a7-4f47-85a2-f596cafbcb94\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-bh68g"
Nov 02 12:36:00.144482 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:00.144039    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76dc13d1-a3a7-4f47-85a2-f596cafbcb94-istio-podinfo\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-bh68g\" (UID: \"76dc13d1-a3a7-4f47-85a2-f596cafbcb94\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-bh68g"
Nov 02 12:36:00.144482 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:00.144058    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfps2\" (UniqueName: \"kubernetes.io/projected/76dc13d1-a3a7-4f47-85a2-f596cafbcb94-kube-api-access-hfps2\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-bh68g\" (UID: \"76dc13d1-a3a7-4f47-85a2-f596cafbcb94\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-bh68g"
Nov 02 12:36:00.144482 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:00.144174    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76dc13d1-a3a7-4f47-85a2-f596cafbcb94-credential-socket\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-bh68g\" (UID: \"76dc13d1-a3a7-4f47-85a2-f596cafbcb94\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-bh68g"
Nov 02 12:36:00.144482 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:00.144266    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76dc13d1-a3a7-4f47-85a2-f596cafbcb94-workload-certs\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-bh68g\" (UID: \"76dc13d1-a3a7-4f47-85a2-f596cafbcb94\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-bh68g"
Nov 02 12:36:00.144482 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:00.144296    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76dc13d1-a3a7-4f47-85a2-f596cafbcb94-istio-envoy\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-bh68g\" (UID: \"76dc13d1-a3a7-4f47-85a2-f596cafbcb94\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-bh68g"
Nov 02 12:36:00.144482 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:00.144326    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76dc13d1-a3a7-4f47-85a2-f596cafbcb94-workload-socket\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-bh68g\" (UID: \"76dc13d1-a3a7-4f47-85a2-f596cafbcb94\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-bh68g"
Nov 02 12:36:00.144482 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:00.144384    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76dc13d1-a3a7-4f47-85a2-f596cafbcb94-istiod-ca-cert\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-bh68g\" (UID: \"76dc13d1-a3a7-4f47-85a2-f596cafbcb94\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-bh68g"
Nov 02 12:36:00.144482 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:00.144421    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76dc13d1-a3a7-4f47-85a2-f596cafbcb94-istio-data\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-bh68g\" (UID: \"76dc13d1-a3a7-4f47-85a2-f596cafbcb94\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-bh68g"
Nov 02 12:36:00.145019 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:00.144635    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76dc13d1-a3a7-4f47-85a2-f596cafbcb94-credential-socket\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-bh68g\" (UID: \"76dc13d1-a3a7-4f47-85a2-f596cafbcb94\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-bh68g"
Nov 02 12:36:00.145019 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:00.144755    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76dc13d1-a3a7-4f47-85a2-f596cafbcb94-workload-certs\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-bh68g\" (UID: \"76dc13d1-a3a7-4f47-85a2-f596cafbcb94\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-bh68g"
Nov 02 12:36:00.145019 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:00.144860    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76dc13d1-a3a7-4f47-85a2-f596cafbcb94-workload-socket\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-bh68g\" (UID: \"76dc13d1-a3a7-4f47-85a2-f596cafbcb94\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-bh68g"
Nov 02 12:36:00.145019 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:00.144934    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76dc13d1-a3a7-4f47-85a2-f596cafbcb94-istio-data\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-bh68g\" (UID: \"76dc13d1-a3a7-4f47-85a2-f596cafbcb94\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-bh68g"
Nov 02 12:36:00.145234 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:00.145150    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76dc13d1-a3a7-4f47-85a2-f596cafbcb94-istiod-ca-cert\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-bh68g\" (UID: \"76dc13d1-a3a7-4f47-85a2-f596cafbcb94\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-bh68g"
Nov 02 12:36:00.146537 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:00.146518    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76dc13d1-a3a7-4f47-85a2-f596cafbcb94-istio-envoy\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-bh68g\" (UID: \"76dc13d1-a3a7-4f47-85a2-f596cafbcb94\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-bh68g"
Nov 02 12:36:00.146696 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:00.146678    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76dc13d1-a3a7-4f47-85a2-f596cafbcb94-istio-podinfo\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-bh68g\" (UID: \"76dc13d1-a3a7-4f47-85a2-f596cafbcb94\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-bh68g"
Nov 02 12:36:00.168298 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:00.168271    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfps2\" (UniqueName: \"kubernetes.io/projected/76dc13d1-a3a7-4f47-85a2-f596cafbcb94-kube-api-access-hfps2\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-bh68g\" (UID: \"76dc13d1-a3a7-4f47-85a2-f596cafbcb94\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-bh68g"
Nov 02 12:36:00.168429 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:00.168412    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76dc13d1-a3a7-4f47-85a2-f596cafbcb94-istio-token\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-bh68g\" (UID: \"76dc13d1-a3a7-4f47-85a2-f596cafbcb94\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-bh68g"
Nov 02 12:36:00.232619 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:00.232588    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-bh68g"
Nov 02 12:36:00.548183 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:00.548159    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-bh68g"]
Nov 02 12:36:00.550417 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:36:00.550386    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76dc13d1_a3a7_4f47_85a2_f596cafbcb94.slice/crio-b4f5e2c2fbf6452a916beb60e9c9295ccef3a361952abd380cb2ff42e0bef271 WatchSource:0}: Error finding container b4f5e2c2fbf6452a916beb60e9c9295ccef3a361952abd380cb2ff42e0bef271: Status 404 returned error can't find the container with id b4f5e2c2fbf6452a916beb60e9c9295ccef3a361952abd380cb2ff42e0bef271
Nov 02 12:36:00.552555 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:00.552521    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:36:00.552623 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:00.552592    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:36:00.552678 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:00.552623    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:36:01.267995 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:01.267967    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-d9dc4ff4-gw-openshift-default-9bfdcc79c-w7gqc"
Nov 02 12:36:01.361118 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:01.361083    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b64510b-aa47-4a38-ac26-7c0df79188b2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b64510b-aa47-4a38-ac26-7c0df79188b2-workload-socket\") pod \"8b64510b-aa47-4a38-ac26-7c0df79188b2\" (UID: \"8b64510b-aa47-4a38-ac26-7c0df79188b2\") "
Nov 02 12:36:01.361283 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:01.361163    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b64510b-aa47-4a38-ac26-7c0df79188b2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b64510b-aa47-4a38-ac26-7c0df79188b2-istio-data\") pod \"8b64510b-aa47-4a38-ac26-7c0df79188b2\" (UID: \"8b64510b-aa47-4a38-ac26-7c0df79188b2\") "
Nov 02 12:36:01.361283 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:01.361223    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b64510b-aa47-4a38-ac26-7c0df79188b2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b64510b-aa47-4a38-ac26-7c0df79188b2-istio-envoy\") pod \"8b64510b-aa47-4a38-ac26-7c0df79188b2\" (UID: \"8b64510b-aa47-4a38-ac26-7c0df79188b2\") "
Nov 02 12:36:01.361283 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:01.361255    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b64510b-aa47-4a38-ac26-7c0df79188b2-kube-api-access-pvgf8\" (UniqueName: \"kubernetes.io/projected/8b64510b-aa47-4a38-ac26-7c0df79188b2-kube-api-access-pvgf8\") pod \"8b64510b-aa47-4a38-ac26-7c0df79188b2\" (UID: \"8b64510b-aa47-4a38-ac26-7c0df79188b2\") "
Nov 02 12:36:01.361408 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:01.361299    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8b64510b-aa47-4a38-ac26-7c0df79188b2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b64510b-aa47-4a38-ac26-7c0df79188b2-istio-podinfo\") pod \"8b64510b-aa47-4a38-ac26-7c0df79188b2\" (UID: \"8b64510b-aa47-4a38-ac26-7c0df79188b2\") "
Nov 02 12:36:01.361408 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:01.361357    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b64510b-aa47-4a38-ac26-7c0df79188b2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b64510b-aa47-4a38-ac26-7c0df79188b2-workload-certs\") pod \"8b64510b-aa47-4a38-ac26-7c0df79188b2\" (UID: \"8b64510b-aa47-4a38-ac26-7c0df79188b2\") "
Nov 02 12:36:01.361479 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:01.361418    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b64510b-aa47-4a38-ac26-7c0df79188b2-istio-token\" (UniqueName: \"kubernetes.io/projected/8b64510b-aa47-4a38-ac26-7c0df79188b2-istio-token\") pod \"8b64510b-aa47-4a38-ac26-7c0df79188b2\" (UID: \"8b64510b-aa47-4a38-ac26-7c0df79188b2\") "
Nov 02 12:36:01.361479 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:01.361443    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b64510b-aa47-4a38-ac26-7c0df79188b2-istio-data" pod "8b64510b-aa47-4a38-ac26-7c0df79188b2" (UID: "8b64510b-aa47-4a38-ac26-7c0df79188b2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:36:01.361479 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:01.361452    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b64510b-aa47-4a38-ac26-7c0df79188b2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b64510b-aa47-4a38-ac26-7c0df79188b2-istiod-ca-cert\") pod \"8b64510b-aa47-4a38-ac26-7c0df79188b2\" (UID: \"8b64510b-aa47-4a38-ac26-7c0df79188b2\") "
Nov 02 12:36:01.361479 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:01.361458    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b64510b-aa47-4a38-ac26-7c0df79188b2-workload-socket" pod "8b64510b-aa47-4a38-ac26-7c0df79188b2" (UID: "8b64510b-aa47-4a38-ac26-7c0df79188b2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:36:01.361703 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:01.361482    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b64510b-aa47-4a38-ac26-7c0df79188b2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b64510b-aa47-4a38-ac26-7c0df79188b2-credential-socket\") pod \"8b64510b-aa47-4a38-ac26-7c0df79188b2\" (UID: \"8b64510b-aa47-4a38-ac26-7c0df79188b2\") "
Nov 02 12:36:01.361893 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:01.361873    4559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b64510b-aa47-4a38-ac26-7c0df79188b2-workload-socket\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:36:01.362021 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:01.361901    4559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b64510b-aa47-4a38-ac26-7c0df79188b2-istio-data\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:36:01.362127 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:01.362087    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b64510b-aa47-4a38-ac26-7c0df79188b2-credential-socket" pod "8b64510b-aa47-4a38-ac26-7c0df79188b2" (UID: "8b64510b-aa47-4a38-ac26-7c0df79188b2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:36:01.362262 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:01.362222    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b64510b-aa47-4a38-ac26-7c0df79188b2-istiod-ca-cert" pod "8b64510b-aa47-4a38-ac26-7c0df79188b2" (UID: "8b64510b-aa47-4a38-ac26-7c0df79188b2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:36:01.362332 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:01.362310    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b64510b-aa47-4a38-ac26-7c0df79188b2-workload-certs" pod "8b64510b-aa47-4a38-ac26-7c0df79188b2" (UID: "8b64510b-aa47-4a38-ac26-7c0df79188b2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:36:01.364453 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:01.364427    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b64510b-aa47-4a38-ac26-7c0df79188b2-kube-api-access-pvgf8" pod "8b64510b-aa47-4a38-ac26-7c0df79188b2" (UID: "8b64510b-aa47-4a38-ac26-7c0df79188b2"). InnerVolumeSpecName "kube-api-access-pvgf8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:36:01.364453 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:01.364432    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8b64510b-aa47-4a38-ac26-7c0df79188b2-istio-podinfo" pod "8b64510b-aa47-4a38-ac26-7c0df79188b2" (UID: "8b64510b-aa47-4a38-ac26-7c0df79188b2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 12:36:01.364573 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:01.364448    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b64510b-aa47-4a38-ac26-7c0df79188b2-istio-envoy" pod "8b64510b-aa47-4a38-ac26-7c0df79188b2" (UID: "8b64510b-aa47-4a38-ac26-7c0df79188b2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:36:01.364573 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:01.364431    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b64510b-aa47-4a38-ac26-7c0df79188b2-istio-token" pod "8b64510b-aa47-4a38-ac26-7c0df79188b2" (UID: "8b64510b-aa47-4a38-ac26-7c0df79188b2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:36:01.402964 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:01.402935    4559 generic.go:356] "Generic (PLEG): container finished" podID="8b64510b-aa47-4a38-ac26-7c0df79188b2" containerID="ef3e4263fb8bfb9a56fb7780ee2a8a52c04f4b659059288983a7fada9ffa8f75" exitCode=0
Nov 02 12:36:01.403129 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:01.402979    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-d9dc4ff4-gw-openshift-default-9bfdcc79c-w7gqc"
Nov 02 12:36:01.403129 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:01.403019    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-d9dc4ff4-gw-openshift-default-9bfdcc79c-w7gqc" event={"ID":"8b64510b-aa47-4a38-ac26-7c0df79188b2","Type":"ContainerDied","Data":"ef3e4263fb8bfb9a56fb7780ee2a8a52c04f4b659059288983a7fada9ffa8f75"}
Nov 02 12:36:01.403129 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:01.403088    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-d9dc4ff4-gw-openshift-default-9bfdcc79c-w7gqc" event={"ID":"8b64510b-aa47-4a38-ac26-7c0df79188b2","Type":"ContainerDied","Data":"4e6eeabe9ac531717c61c56cac1b4463b748c3829d7a3ec3f0066b2c5e916c4d"}
Nov 02 12:36:01.403129 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:01.403122    4559 scope.go:122] "RemoveContainer" containerID="ef3e4263fb8bfb9a56fb7780ee2a8a52c04f4b659059288983a7fada9ffa8f75"
Nov 02 12:36:01.404799 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:01.404773    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-bh68g" event={"ID":"76dc13d1-a3a7-4f47-85a2-f596cafbcb94","Type":"ContainerStarted","Data":"2f30022edc685d1ad8c8dc1e219cd3e77b94ca787c2b9e95f24e108586b01ed7"}
Nov 02 12:36:01.404890 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:01.404813    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-bh68g" event={"ID":"76dc13d1-a3a7-4f47-85a2-f596cafbcb94","Type":"ContainerStarted","Data":"b4f5e2c2fbf6452a916beb60e9c9295ccef3a361952abd380cb2ff42e0bef271"}
Nov 02 12:36:01.411795 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:01.411780    4559 scope.go:122] "RemoveContainer" containerID="7f777244670ed6d28a65b70700029490b9add71a7800a2a20e511bec14316af5"
Nov 02 12:36:01.419837 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:01.419825    4559 scope.go:122] "RemoveContainer" containerID="ef3e4263fb8bfb9a56fb7780ee2a8a52c04f4b659059288983a7fada9ffa8f75"
Nov 02 12:36:01.420107 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:36:01.420088    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef3e4263fb8bfb9a56fb7780ee2a8a52c04f4b659059288983a7fada9ffa8f75\": container with ID starting with ef3e4263fb8bfb9a56fb7780ee2a8a52c04f4b659059288983a7fada9ffa8f75 not found: ID does not exist" containerID="ef3e4263fb8bfb9a56fb7780ee2a8a52c04f4b659059288983a7fada9ffa8f75"
Nov 02 12:36:01.420170 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:01.420115    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef3e4263fb8bfb9a56fb7780ee2a8a52c04f4b659059288983a7fada9ffa8f75"} err="failed to get container status \"ef3e4263fb8bfb9a56fb7780ee2a8a52c04f4b659059288983a7fada9ffa8f75\": rpc error: code = NotFound desc = could not find container \"ef3e4263fb8bfb9a56fb7780ee2a8a52c04f4b659059288983a7fada9ffa8f75\": container with ID starting with ef3e4263fb8bfb9a56fb7780ee2a8a52c04f4b659059288983a7fada9ffa8f75 not found: ID does not exist"
Nov 02 12:36:01.420170 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:01.420134    4559 scope.go:122] "RemoveContainer" containerID="7f777244670ed6d28a65b70700029490b9add71a7800a2a20e511bec14316af5"
Nov 02 12:36:01.420372 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:36:01.420354    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f777244670ed6d28a65b70700029490b9add71a7800a2a20e511bec14316af5\": container with ID starting with 7f777244670ed6d28a65b70700029490b9add71a7800a2a20e511bec14316af5 not found: ID does not exist" containerID="7f777244670ed6d28a65b70700029490b9add71a7800a2a20e511bec14316af5"
Nov 02 12:36:01.420420 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:01.420378    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f777244670ed6d28a65b70700029490b9add71a7800a2a20e511bec14316af5"} err="failed to get container status \"7f777244670ed6d28a65b70700029490b9add71a7800a2a20e511bec14316af5\": rpc error: code = NotFound desc = could not find container \"7f777244670ed6d28a65b70700029490b9add71a7800a2a20e511bec14316af5\": container with ID starting with 7f777244670ed6d28a65b70700029490b9add71a7800a2a20e511bec14316af5 not found: ID does not exist"
Nov 02 12:36:01.463375 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:01.463344    4559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b64510b-aa47-4a38-ac26-7c0df79188b2-istio-envoy\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:36:01.463375 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:01.463372    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pvgf8\" (UniqueName: \"kubernetes.io/projected/8b64510b-aa47-4a38-ac26-7c0df79188b2-kube-api-access-pvgf8\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:36:01.463586 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:01.463385    4559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b64510b-aa47-4a38-ac26-7c0df79188b2-istio-podinfo\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:36:01.463586 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:01.463401    4559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b64510b-aa47-4a38-ac26-7c0df79188b2-workload-certs\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:36:01.463586 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:01.463414    4559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b64510b-aa47-4a38-ac26-7c0df79188b2-istio-token\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:36:01.463586 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:01.463426    4559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b64510b-aa47-4a38-ac26-7c0df79188b2-istiod-ca-cert\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:36:01.463586 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:01.463437    4559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b64510b-aa47-4a38-ac26-7c0df79188b2-credential-socket\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:36:01.467943 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:01.467901    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-bh68g" podStartSLOduration=2.467889073 podStartE2EDuration="2.467889073s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:35:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:36:01.466421704 +0000 UTC m=+557.721820801" watchObservedRunningTime="2026-11-02 12:36:01.467889073 +0000 UTC m=+557.723288167"
Nov 02 12:36:01.505867 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:01.505811    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-d9dc4ff4-gw-openshift-default-9bfdcc79c-w7gqc"]
Nov 02 12:36:01.510340 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:01.510318    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-d9dc4ff4-gw-openshift-default-9bfdcc79c-w7gqc"]
Nov 02 12:36:01.866472 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:01.866388    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-f080cb5f-76f8d88955-vn85z" podUID="07bca5a6-5631-49de-b344-50a8390d1fa6" containerName="payload-pre-processing" containerID="cri-o://b07831d10f4ca8b015e192cc46d451e3d95e481ccdaca902301036f18f11b469" gracePeriod=25
Nov 02 12:36:01.891985 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:01.891958    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b64510b-aa47-4a38-ac26-7c0df79188b2" path="/var/lib/kubelet/pods/8b64510b-aa47-4a38-ac26-7c0df79188b2/volumes"
Nov 02 12:36:02.108348 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:02.108320    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-f080cb5f-76f8d88955-vn85z"
Nov 02 12:36:02.169239 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:02.169215    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07bca5a6-5631-49de-b344-50a8390d1fa6-kube-api-access-4krpl\" (UniqueName: \"kubernetes.io/projected/07bca5a6-5631-49de-b344-50a8390d1fa6-kube-api-access-4krpl\") pod \"07bca5a6-5631-49de-b344-50a8390d1fa6\" (UID: \"07bca5a6-5631-49de-b344-50a8390d1fa6\") "
Nov 02 12:36:02.169386 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:02.169359    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07bca5a6-5631-49de-b344-50a8390d1fa6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07bca5a6-5631-49de-b344-50a8390d1fa6-plugins-config-volume\") pod \"07bca5a6-5631-49de-b344-50a8390d1fa6\" (UID: \"07bca5a6-5631-49de-b344-50a8390d1fa6\") "
Nov 02 12:36:02.169808 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:02.169778    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07bca5a6-5631-49de-b344-50a8390d1fa6-plugins-config-volume" pod "07bca5a6-5631-49de-b344-50a8390d1fa6" (UID: "07bca5a6-5631-49de-b344-50a8390d1fa6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:36:02.171274 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:02.171246    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07bca5a6-5631-49de-b344-50a8390d1fa6-kube-api-access-4krpl" pod "07bca5a6-5631-49de-b344-50a8390d1fa6" (UID: "07bca5a6-5631-49de-b344-50a8390d1fa6"). InnerVolumeSpecName "kube-api-access-4krpl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:36:02.232972 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:02.232940    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-bh68g"
Nov 02 12:36:02.238056 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:02.238031    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-bh68g"
Nov 02 12:36:02.271044 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:02.271014    4559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07bca5a6-5631-49de-b344-50a8390d1fa6-plugins-config-volume\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:36:02.271044 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:02.271041    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4krpl\" (UniqueName: \"kubernetes.io/projected/07bca5a6-5631-49de-b344-50a8390d1fa6-kube-api-access-4krpl\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:36:02.412733 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:02.412628    4559 generic.go:356] "Generic (PLEG): container finished" podID="07bca5a6-5631-49de-b344-50a8390d1fa6" containerID="b07831d10f4ca8b015e192cc46d451e3d95e481ccdaca902301036f18f11b469" exitCode=0
Nov 02 12:36:02.412733 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:02.412688    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-f080cb5f-76f8d88955-vn85z"
Nov 02 12:36:02.412948 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:02.412742    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-f080cb5f-76f8d88955-vn85z" event={"ID":"07bca5a6-5631-49de-b344-50a8390d1fa6","Type":"ContainerDied","Data":"b07831d10f4ca8b015e192cc46d451e3d95e481ccdaca902301036f18f11b469"}
Nov 02 12:36:02.412948 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:02.412786    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-f080cb5f-76f8d88955-vn85z" event={"ID":"07bca5a6-5631-49de-b344-50a8390d1fa6","Type":"ContainerDied","Data":"9169a7eb7c509592c505a4c98bb122bddceb51c96f558cdf48f943799d886cb1"}
Nov 02 12:36:02.412948 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:02.412807    4559 scope.go:122] "RemoveContainer" containerID="b07831d10f4ca8b015e192cc46d451e3d95e481ccdaca902301036f18f11b469"
Nov 02 12:36:02.413111 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:02.413035    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-bh68g"
Nov 02 12:36:02.414166 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:02.413962    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-bh68g"
Nov 02 12:36:02.421880 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:02.421859    4559 scope.go:122] "RemoveContainer" containerID="b07831d10f4ca8b015e192cc46d451e3d95e481ccdaca902301036f18f11b469"
Nov 02 12:36:02.422158 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:36:02.422129    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b07831d10f4ca8b015e192cc46d451e3d95e481ccdaca902301036f18f11b469\": container with ID starting with b07831d10f4ca8b015e192cc46d451e3d95e481ccdaca902301036f18f11b469 not found: ID does not exist" containerID="b07831d10f4ca8b015e192cc46d451e3d95e481ccdaca902301036f18f11b469"
Nov 02 12:36:02.422219 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:02.422161    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b07831d10f4ca8b015e192cc46d451e3d95e481ccdaca902301036f18f11b469"} err="failed to get container status \"b07831d10f4ca8b015e192cc46d451e3d95e481ccdaca902301036f18f11b469\": rpc error: code = NotFound desc = could not find container \"b07831d10f4ca8b015e192cc46d451e3d95e481ccdaca902301036f18f11b469\": container with ID starting with b07831d10f4ca8b015e192cc46d451e3d95e481ccdaca902301036f18f11b469 not found: ID does not exist"
Nov 02 12:36:02.526666 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:02.526600    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f080cb5f-76f8d88955-vn85z"]
Nov 02 12:36:02.532734 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:02.532708    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f080cb5f-76f8d88955-vn85z"]
Nov 02 12:36:03.546458 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:03.546424    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-068756-8565b4cd8c-fpqt6"]
Nov 02 12:36:03.546840 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:03.546815    4559 cpu_manager.go:414] "containerMap: removing container" podUID="8b64510b-aa47-4a38-ac26-7c0df79188b2" containerName="istio-proxy"
Nov 02 12:36:03.546840 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:03.546827    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b64510b-aa47-4a38-ac26-7c0df79188b2" containerName="istio-proxy"
Nov 02 12:36:03.546918 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:03.546845    4559 cpu_manager.go:414] "containerMap: removing container" podUID="07bca5a6-5631-49de-b344-50a8390d1fa6" containerName="payload-pre-processing"
Nov 02 12:36:03.546918 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:03.546850    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07bca5a6-5631-49de-b344-50a8390d1fa6" containerName="payload-pre-processing"
Nov 02 12:36:03.546988 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:03.546933    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b64510b-aa47-4a38-ac26-7c0df79188b2" containerName="istio-proxy"
Nov 02 12:36:03.546988 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:03.546940    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b64510b-aa47-4a38-ac26-7c0df79188b2" containerName="istio-proxy"
Nov 02 12:36:03.546988 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:03.546948    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="07bca5a6-5631-49de-b344-50a8390d1fa6" containerName="payload-pre-processing"
Nov 02 12:36:03.551400 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:03.551384    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-068756-8565b4cd8c-fpqt6"
Nov 02 12:36:03.568923 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:03.568891    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-068756-8565b4cd8c-fpqt6"]
Nov 02 12:36:03.579530 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:03.579507    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-068756\""
Nov 02 12:36:03.579628 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:03.579569    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-068756-dockercfg-jbfd2\""
Nov 02 12:36:03.584005 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:03.583983    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7k87\" (UniqueName: \"kubernetes.io/projected/1972d657-0f0b-40b8-85ba-ac651e09d31f-kube-api-access-j7k87\") pod \"payload-pre-processing-e2e-worker-w3-068756-8565b4cd8c-fpqt6\" (UID: \"1972d657-0f0b-40b8-85ba-ac651e09d31f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-068756-8565b4cd8c-fpqt6"
Nov 02 12:36:03.584119 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:03.584021    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1972d657-0f0b-40b8-85ba-ac651e09d31f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-068756-8565b4cd8c-fpqt6\" (UID: \"1972d657-0f0b-40b8-85ba-ac651e09d31f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-068756-8565b4cd8c-fpqt6"
Nov 02 12:36:03.660380 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:03.660342    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-068756-bdb7bcbbf-nnssk"]
Nov 02 12:36:03.660989 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:03.660968    4559 cpu_manager.go:414] "containerMap: removing container" podUID="8b64510b-aa47-4a38-ac26-7c0df79188b2" containerName="istio-proxy"
Nov 02 12:36:03.661062 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:03.660991    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b64510b-aa47-4a38-ac26-7c0df79188b2" containerName="istio-proxy"
Nov 02 12:36:03.664669 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:03.664456    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-068756-bdb7bcbbf-nnssk"
Nov 02 12:36:03.683817 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:03.683791    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-068756-bdb7bcbbf-nnssk"]
Nov 02 12:36:03.684821 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:03.684796    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7k87\" (UniqueName: \"kubernetes.io/projected/1972d657-0f0b-40b8-85ba-ac651e09d31f-kube-api-access-j7k87\") pod \"payload-pre-processing-e2e-worker-w3-068756-8565b4cd8c-fpqt6\" (UID: \"1972d657-0f0b-40b8-85ba-ac651e09d31f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-068756-8565b4cd8c-fpqt6"
Nov 02 12:36:03.684932 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:03.684838    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1972d657-0f0b-40b8-85ba-ac651e09d31f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-068756-8565b4cd8c-fpqt6\" (UID: \"1972d657-0f0b-40b8-85ba-ac651e09d31f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-068756-8565b4cd8c-fpqt6"
Nov 02 12:36:03.684932 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:03.684881    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4zdr\" (UniqueName: \"kubernetes.io/projected/226d3b62-6d8e-4f46-bf0e-870d7891314f-kube-api-access-f4zdr\") pod \"payload-processing-e2e-worker-w3-068756-bdb7bcbbf-nnssk\" (UID: \"226d3b62-6d8e-4f46-bf0e-870d7891314f\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-068756-bdb7bcbbf-nnssk"
Nov 02 12:36:03.684932 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:03.684911    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/226d3b62-6d8e-4f46-bf0e-870d7891314f-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-068756-bdb7bcbbf-nnssk\" (UID: \"226d3b62-6d8e-4f46-bf0e-870d7891314f\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-068756-bdb7bcbbf-nnssk"
Nov 02 12:36:03.685587 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:03.685565    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1972d657-0f0b-40b8-85ba-ac651e09d31f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-068756-8565b4cd8c-fpqt6\" (UID: \"1972d657-0f0b-40b8-85ba-ac651e09d31f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-068756-8565b4cd8c-fpqt6"
Nov 02 12:36:03.709205 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:03.709180    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7k87\" (UniqueName: \"kubernetes.io/projected/1972d657-0f0b-40b8-85ba-ac651e09d31f-kube-api-access-j7k87\") pod \"payload-pre-processing-e2e-worker-w3-068756-8565b4cd8c-fpqt6\" (UID: \"1972d657-0f0b-40b8-85ba-ac651e09d31f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-068756-8565b4cd8c-fpqt6"
Nov 02 12:36:03.785591 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:03.785553    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4zdr\" (UniqueName: \"kubernetes.io/projected/226d3b62-6d8e-4f46-bf0e-870d7891314f-kube-api-access-f4zdr\") pod \"payload-processing-e2e-worker-w3-068756-bdb7bcbbf-nnssk\" (UID: \"226d3b62-6d8e-4f46-bf0e-870d7891314f\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-068756-bdb7bcbbf-nnssk"
Nov 02 12:36:03.785795 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:03.785599    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/226d3b62-6d8e-4f46-bf0e-870d7891314f-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-068756-bdb7bcbbf-nnssk\" (UID: \"226d3b62-6d8e-4f46-bf0e-870d7891314f\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-068756-bdb7bcbbf-nnssk"
Nov 02 12:36:03.786236 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:03.786216    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/226d3b62-6d8e-4f46-bf0e-870d7891314f-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-068756-bdb7bcbbf-nnssk\" (UID: \"226d3b62-6d8e-4f46-bf0e-870d7891314f\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-068756-bdb7bcbbf-nnssk"
Nov 02 12:36:03.809461 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:03.809400    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4zdr\" (UniqueName: \"kubernetes.io/projected/226d3b62-6d8e-4f46-bf0e-870d7891314f-kube-api-access-f4zdr\") pod \"payload-processing-e2e-worker-w3-068756-bdb7bcbbf-nnssk\" (UID: \"226d3b62-6d8e-4f46-bf0e-870d7891314f\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-068756-bdb7bcbbf-nnssk"
Nov 02 12:36:03.864376 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:03.864354    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-068756-8565b4cd8c-fpqt6"
Nov 02 12:36:03.893515 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:03.893480    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07bca5a6-5631-49de-b344-50a8390d1fa6" path="/var/lib/kubelet/pods/07bca5a6-5631-49de-b344-50a8390d1fa6/volumes"
Nov 02 12:36:03.978814 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:03.978785    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-068756-bdb7bcbbf-nnssk"
Nov 02 12:36:04.184010 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:04.183979    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-068756-8565b4cd8c-fpqt6"]
Nov 02 12:36:04.184164 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:36:04.184080    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1972d657_0f0b_40b8_85ba_ac651e09d31f.slice/crio-518455ef3971849c9c23df2b0ef07076549cc36bfe296d3b346800ea315b3c74 WatchSource:0}: Error finding container 518455ef3971849c9c23df2b0ef07076549cc36bfe296d3b346800ea315b3c74: Status 404 returned error can't find the container with id 518455ef3971849c9c23df2b0ef07076549cc36bfe296d3b346800ea315b3c74
Nov 02 12:36:04.317503 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:04.317474    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-068756-bdb7bcbbf-nnssk"]
Nov 02 12:36:04.317781 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:36:04.317755    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod226d3b62_6d8e_4f46_bf0e_870d7891314f.slice/crio-1370b4784999742a13921b864a158712ce60fc96b7a7675c678cdf2bea1d7526 WatchSource:0}: Error finding container 1370b4784999742a13921b864a158712ce60fc96b7a7675c678cdf2bea1d7526: Status 404 returned error can't find the container with id 1370b4784999742a13921b864a158712ce60fc96b7a7675c678cdf2bea1d7526
Nov 02 12:36:04.424472 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:04.424443    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-068756-bdb7bcbbf-nnssk" event={"ID":"226d3b62-6d8e-4f46-bf0e-870d7891314f","Type":"ContainerStarted","Data":"ff131871a786e5661f10d591961f457330c5df9a3b85751fd502a222c2f76199"}
Nov 02 12:36:04.424597 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:04.424503    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-068756-bdb7bcbbf-nnssk"
Nov 02 12:36:04.424597 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:04.424528    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-068756-bdb7bcbbf-nnssk" event={"ID":"226d3b62-6d8e-4f46-bf0e-870d7891314f","Type":"ContainerStarted","Data":"1370b4784999742a13921b864a158712ce60fc96b7a7675c678cdf2bea1d7526"}
Nov 02 12:36:04.426042 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:04.426015    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-068756-8565b4cd8c-fpqt6" event={"ID":"1972d657-0f0b-40b8-85ba-ac651e09d31f","Type":"ContainerStarted","Data":"81a9d7a5c53155c5c46aff1d22d20825a7c3b50ccb5b34bb4d62778d8a038969"}
Nov 02 12:36:04.426135 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:04.426056    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-068756-8565b4cd8c-fpqt6" event={"ID":"1972d657-0f0b-40b8-85ba-ac651e09d31f","Type":"ContainerStarted","Data":"518455ef3971849c9c23df2b0ef07076549cc36bfe296d3b346800ea315b3c74"}
Nov 02 12:36:04.426135 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:04.426075    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-068756-8565b4cd8c-fpqt6"
Nov 02 12:36:04.477081 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:04.477005    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-068756-bdb7bcbbf-nnssk" podStartSLOduration=1.4769920779999999 podStartE2EDuration="1.476992078s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:36:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:36:04.475014782 +0000 UTC m=+560.730413878" watchObservedRunningTime="2026-11-02 12:36:04.476992078 +0000 UTC m=+560.732391533"
Nov 02 12:36:04.517559 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:04.517516    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-068756-8565b4cd8c-fpqt6" podStartSLOduration=1.5175034680000001 podStartE2EDuration="1.517503468s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:36:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:36:04.516703082 +0000 UTC m=+560.772102178" watchObservedRunningTime="2026-11-02 12:36:04.517503468 +0000 UTC m=+560.772902563"
Nov 02 12:36:06.425380 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:06.425344    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-95ee26-5f96f56f79-2zqdd"]
Nov 02 12:36:06.429537 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:06.429513    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-95ee26-5f96f56f79-2zqdd"
Nov 02 12:36:06.458549 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:06.458515    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-95ee26-5f96f56f79-2zqdd"]
Nov 02 12:36:06.478675 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:06.467015    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-95ee26\""
Nov 02 12:36:06.478675 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:06.467189    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-95ee26-dockercfg-b5b8p\""
Nov 02 12:36:06.521275 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:06.521244    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlnpn\" (UniqueName: \"kubernetes.io/projected/57c75efe-f588-45c2-b5ea-2f948d0e6e13-kube-api-access-zlnpn\") pod \"payload-processing-e2e-worker-w2-95ee26-5f96f56f79-2zqdd\" (UID: \"57c75efe-f588-45c2-b5ea-2f948d0e6e13\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-95ee26-5f96f56f79-2zqdd"
Nov 02 12:36:06.521428 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:06.521407    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57c75efe-f588-45c2-b5ea-2f948d0e6e13-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-95ee26-5f96f56f79-2zqdd\" (UID: \"57c75efe-f588-45c2-b5ea-2f948d0e6e13\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-95ee26-5f96f56f79-2zqdd"
Nov 02 12:36:06.622497 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:06.622460    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57c75efe-f588-45c2-b5ea-2f948d0e6e13-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-95ee26-5f96f56f79-2zqdd\" (UID: \"57c75efe-f588-45c2-b5ea-2f948d0e6e13\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-95ee26-5f96f56f79-2zqdd"
Nov 02 12:36:06.622671 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:06.622508    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlnpn\" (UniqueName: \"kubernetes.io/projected/57c75efe-f588-45c2-b5ea-2f948d0e6e13-kube-api-access-zlnpn\") pod \"payload-processing-e2e-worker-w2-95ee26-5f96f56f79-2zqdd\" (UID: \"57c75efe-f588-45c2-b5ea-2f948d0e6e13\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-95ee26-5f96f56f79-2zqdd"
Nov 02 12:36:06.623226 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:06.623204    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57c75efe-f588-45c2-b5ea-2f948d0e6e13-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-95ee26-5f96f56f79-2zqdd\" (UID: \"57c75efe-f588-45c2-b5ea-2f948d0e6e13\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-95ee26-5f96f56f79-2zqdd"
Nov 02 12:36:06.649656 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:06.649612    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlnpn\" (UniqueName: \"kubernetes.io/projected/57c75efe-f588-45c2-b5ea-2f948d0e6e13-kube-api-access-zlnpn\") pod \"payload-processing-e2e-worker-w2-95ee26-5f96f56f79-2zqdd\" (UID: \"57c75efe-f588-45c2-b5ea-2f948d0e6e13\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-95ee26-5f96f56f79-2zqdd"
Nov 02 12:36:06.759963 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:06.759873    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-95ee26-5f96f56f79-2zqdd"
Nov 02 12:36:07.061506 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:07.061473    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-95ee26-5f96f56f79-2zqdd"]
Nov 02 12:36:07.061753 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:36:07.061727    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57c75efe_f588_45c2_b5ea_2f948d0e6e13.slice/crio-87863d1d01353658282220caa9108ed5df45a0788a29303d91151e46adc13bc8 WatchSource:0}: Error finding container 87863d1d01353658282220caa9108ed5df45a0788a29303d91151e46adc13bc8: Status 404 returned error can't find the container with id 87863d1d01353658282220caa9108ed5df45a0788a29303d91151e46adc13bc8
Nov 02 12:36:07.452519 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:07.452478    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-95ee26-5f96f56f79-2zqdd" event={"ID":"57c75efe-f588-45c2-b5ea-2f948d0e6e13","Type":"ContainerStarted","Data":"c6ee188467a1f51ac30d421c5c1528e70aa6d6757253b757faed5e2768b2e0b6"}
Nov 02 12:36:07.453003 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:07.452549    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-95ee26-5f96f56f79-2zqdd" event={"ID":"57c75efe-f588-45c2-b5ea-2f948d0e6e13","Type":"ContainerStarted","Data":"87863d1d01353658282220caa9108ed5df45a0788a29303d91151e46adc13bc8"}
Nov 02 12:36:07.453003 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:07.452616    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-95ee26-5f96f56f79-2zqdd"
Nov 02 12:36:07.495026 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:07.494978    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w2-95ee26-5f96f56f79-2zqdd" podStartSLOduration=1.494963708 podStartE2EDuration="1.494963708s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:36:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:36:07.493832098 +0000 UTC m=+563.749231198" watchObservedRunningTime="2026-11-02 12:36:07.494963708 +0000 UTC m=+563.750362804"
Nov 02 12:36:09.039704 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:09.039672    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-628cf9bd-f5bbdc6cf-49rc4"]
Nov 02 12:36:09.040187 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:09.039933    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-628cf9bd-f5bbdc6cf-49rc4" podUID="be21bfc9-2d60-4cfd-8ded-41ed733000bb" containerName="maas-api" containerID="cri-o://d9dd371672cad6551c5ea8312529b7b6deebe8b16f875e248afb70338c038bef" gracePeriod=30
Nov 02 12:36:09.221747 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:09.221717    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"]
Nov 02 12:36:09.295237 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:09.295181    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-628cf9bd-f5bbdc6cf-49rc4"
Nov 02 12:36:09.351146 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:09.351121    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/be21bfc9-2d60-4cfd-8ded-41ed733000bb-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be21bfc9-2d60-4cfd-8ded-41ed733000bb-maas-api-tls\") pod \"be21bfc9-2d60-4cfd-8ded-41ed733000bb\" (UID: \"be21bfc9-2d60-4cfd-8ded-41ed733000bb\") "
Nov 02 12:36:09.351314 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:09.351178    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be21bfc9-2d60-4cfd-8ded-41ed733000bb-kube-api-access-jbb76\" (UniqueName: \"kubernetes.io/projected/be21bfc9-2d60-4cfd-8ded-41ed733000bb-kube-api-access-jbb76\") pod \"be21bfc9-2d60-4cfd-8ded-41ed733000bb\" (UID: \"be21bfc9-2d60-4cfd-8ded-41ed733000bb\") "
Nov 02 12:36:09.351314 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:09.351243    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be21bfc9-2d60-4cfd-8ded-41ed733000bb-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be21bfc9-2d60-4cfd-8ded-41ed733000bb-openshift-service-ca\") pod \"be21bfc9-2d60-4cfd-8ded-41ed733000bb\" (UID: \"be21bfc9-2d60-4cfd-8ded-41ed733000bb\") "
Nov 02 12:36:09.351416 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:09.351326    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/be21bfc9-2d60-4cfd-8ded-41ed733000bb-metrics-certs\" (UniqueName: \"kubernetes.io/secret/be21bfc9-2d60-4cfd-8ded-41ed733000bb-metrics-certs\") pod \"be21bfc9-2d60-4cfd-8ded-41ed733000bb\" (UID: \"be21bfc9-2d60-4cfd-8ded-41ed733000bb\") "
Nov 02 12:36:09.351837 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:09.351796    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be21bfc9-2d60-4cfd-8ded-41ed733000bb-openshift-service-ca" pod "be21bfc9-2d60-4cfd-8ded-41ed733000bb" (UID: "be21bfc9-2d60-4cfd-8ded-41ed733000bb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:36:09.353803 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:09.353757    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be21bfc9-2d60-4cfd-8ded-41ed733000bb-maas-api-tls" pod "be21bfc9-2d60-4cfd-8ded-41ed733000bb" (UID: "be21bfc9-2d60-4cfd-8ded-41ed733000bb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:36:09.354007 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:09.353985    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be21bfc9-2d60-4cfd-8ded-41ed733000bb-metrics-certs" pod "be21bfc9-2d60-4cfd-8ded-41ed733000bb" (UID: "be21bfc9-2d60-4cfd-8ded-41ed733000bb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:36:09.354784 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:09.354759    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be21bfc9-2d60-4cfd-8ded-41ed733000bb-kube-api-access-jbb76" pod "be21bfc9-2d60-4cfd-8ded-41ed733000bb" (UID: "be21bfc9-2d60-4cfd-8ded-41ed733000bb"). InnerVolumeSpecName "kube-api-access-jbb76". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:36:09.453149 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:09.453051    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be21bfc9-2d60-4cfd-8ded-41ed733000bb-maas-api-tls\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:36:09.453149 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:09.453090    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jbb76\" (UniqueName: \"kubernetes.io/projected/be21bfc9-2d60-4cfd-8ded-41ed733000bb-kube-api-access-jbb76\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:36:09.453149 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:09.453105    4559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be21bfc9-2d60-4cfd-8ded-41ed733000bb-openshift-service-ca\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:36:09.453149 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:09.453118    4559 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/be21bfc9-2d60-4cfd-8ded-41ed733000bb-metrics-certs\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:36:09.481662 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:09.471072    4559 generic.go:356] "Generic (PLEG): container finished" podID="be21bfc9-2d60-4cfd-8ded-41ed733000bb" containerID="d9dd371672cad6551c5ea8312529b7b6deebe8b16f875e248afb70338c038bef" exitCode=0
Nov 02 12:36:09.481662 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:09.471137    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-628cf9bd-f5bbdc6cf-49rc4"
Nov 02 12:36:09.481662 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:09.471142    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-628cf9bd-f5bbdc6cf-49rc4" event={"ID":"be21bfc9-2d60-4cfd-8ded-41ed733000bb","Type":"ContainerDied","Data":"d9dd371672cad6551c5ea8312529b7b6deebe8b16f875e248afb70338c038bef"}
Nov 02 12:36:09.481662 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:09.471192    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-628cf9bd-f5bbdc6cf-49rc4" event={"ID":"be21bfc9-2d60-4cfd-8ded-41ed733000bb","Type":"ContainerDied","Data":"dacb02c65dfd75eb4883b5c925b10bb4639e489c4c47016c2e8404dc7ad4e8d8"}
Nov 02 12:36:09.481662 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:09.471222    4559 scope.go:122] "RemoveContainer" containerID="d9dd371672cad6551c5ea8312529b7b6deebe8b16f875e248afb70338c038bef"
Nov 02 12:36:09.497816 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:09.497791    4559 scope.go:122] "RemoveContainer" containerID="d9dd371672cad6551c5ea8312529b7b6deebe8b16f875e248afb70338c038bef"
Nov 02 12:36:09.498224 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:36:09.498198    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9dd371672cad6551c5ea8312529b7b6deebe8b16f875e248afb70338c038bef\": container with ID starting with d9dd371672cad6551c5ea8312529b7b6deebe8b16f875e248afb70338c038bef not found: ID does not exist" containerID="d9dd371672cad6551c5ea8312529b7b6deebe8b16f875e248afb70338c038bef"
Nov 02 12:36:09.498330 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:09.498241    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9dd371672cad6551c5ea8312529b7b6deebe8b16f875e248afb70338c038bef"} err="failed to get container status \"d9dd371672cad6551c5ea8312529b7b6deebe8b16f875e248afb70338c038bef\": rpc error: code = NotFound desc = could not find container \"d9dd371672cad6551c5ea8312529b7b6deebe8b16f875e248afb70338c038bef\": container with ID starting with d9dd371672cad6551c5ea8312529b7b6deebe8b16f875e248afb70338c038bef not found: ID does not exist"
Nov 02 12:36:09.534799 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:09.534769    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-628cf9bd-f5bbdc6cf-49rc4"]
Nov 02 12:36:09.541274 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:09.541245    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-628cf9bd-f5bbdc6cf-49rc4"]
Nov 02 12:36:09.619836 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:09.619744    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-95ee26-769ff594c9-wdwnp"]
Nov 02 12:36:09.620754 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:09.620731    4559 cpu_manager.go:414] "containerMap: removing container" podUID="be21bfc9-2d60-4cfd-8ded-41ed733000bb" containerName="maas-api"
Nov 02 12:36:09.620754 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:09.620755    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be21bfc9-2d60-4cfd-8ded-41ed733000bb" containerName="maas-api"
Nov 02 12:36:09.620936 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:09.620925    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="be21bfc9-2d60-4cfd-8ded-41ed733000bb" containerName="maas-api"
Nov 02 12:36:09.626212 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:09.626190    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-95ee26-769ff594c9-wdwnp"
Nov 02 12:36:09.644440 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:09.644373    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-95ee26-769ff594c9-wdwnp"]
Nov 02 12:36:09.644926 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:09.644605    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w2-95ee26-serving-cert\""
Nov 02 12:36:09.755016 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:09.754983    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90adff48-e3d2-445f-9c4c-4133e5820b26-metrics-certs\") pod \"maas-api-e2e-worker-w2-95ee26-769ff594c9-wdwnp\" (UID: \"90adff48-e3d2-445f-9c4c-4133e5820b26\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-95ee26-769ff594c9-wdwnp"
Nov 02 12:36:09.755174 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:09.755036    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tw2fb\" (UniqueName: \"kubernetes.io/projected/90adff48-e3d2-445f-9c4c-4133e5820b26-kube-api-access-tw2fb\") pod \"maas-api-e2e-worker-w2-95ee26-769ff594c9-wdwnp\" (UID: \"90adff48-e3d2-445f-9c4c-4133e5820b26\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-95ee26-769ff594c9-wdwnp"
Nov 02 12:36:09.755174 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:09.755153    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90adff48-e3d2-445f-9c4c-4133e5820b26-maas-api-tls\") pod \"maas-api-e2e-worker-w2-95ee26-769ff594c9-wdwnp\" (UID: \"90adff48-e3d2-445f-9c4c-4133e5820b26\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-95ee26-769ff594c9-wdwnp"
Nov 02 12:36:09.755284 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:09.755209    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/90adff48-e3d2-445f-9c4c-4133e5820b26-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-95ee26-769ff594c9-wdwnp\" (UID: \"90adff48-e3d2-445f-9c4c-4133e5820b26\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-95ee26-769ff594c9-wdwnp"
Nov 02 12:36:09.856654 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:09.856611    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tw2fb\" (UniqueName: \"kubernetes.io/projected/90adff48-e3d2-445f-9c4c-4133e5820b26-kube-api-access-tw2fb\") pod \"maas-api-e2e-worker-w2-95ee26-769ff594c9-wdwnp\" (UID: \"90adff48-e3d2-445f-9c4c-4133e5820b26\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-95ee26-769ff594c9-wdwnp"
Nov 02 12:36:09.856839 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:09.856693    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90adff48-e3d2-445f-9c4c-4133e5820b26-maas-api-tls\") pod \"maas-api-e2e-worker-w2-95ee26-769ff594c9-wdwnp\" (UID: \"90adff48-e3d2-445f-9c4c-4133e5820b26\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-95ee26-769ff594c9-wdwnp"
Nov 02 12:36:09.856839 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:09.856733    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/90adff48-e3d2-445f-9c4c-4133e5820b26-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-95ee26-769ff594c9-wdwnp\" (UID: \"90adff48-e3d2-445f-9c4c-4133e5820b26\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-95ee26-769ff594c9-wdwnp"
Nov 02 12:36:09.856839 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:09.856814    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90adff48-e3d2-445f-9c4c-4133e5820b26-metrics-certs\") pod \"maas-api-e2e-worker-w2-95ee26-769ff594c9-wdwnp\" (UID: \"90adff48-e3d2-445f-9c4c-4133e5820b26\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-95ee26-769ff594c9-wdwnp"
Nov 02 12:36:09.856839 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:36:09.856828    4559 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w2-95ee26-serving-cert: secret "maas-api-e2e-worker-w2-95ee26-serving-cert" not found
Nov 02 12:36:09.857089 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:36:09.856922    4559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90adff48-e3d2-445f-9c4c-4133e5820b26-maas-api-tls podName:90adff48-e3d2-445f-9c4c-4133e5820b26 nodeName:}" failed. No retries permitted until 2026-11-02 12:36:10.356899578 +0000 UTC m=+566.612298670 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/90adff48-e3d2-445f-9c4c-4133e5820b26-maas-api-tls") pod "maas-api-e2e-worker-w2-95ee26-769ff594c9-wdwnp" (UID: "90adff48-e3d2-445f-9c4c-4133e5820b26") : secret "maas-api-e2e-worker-w2-95ee26-serving-cert" not found
Nov 02 12:36:09.857505 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:09.857479    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/90adff48-e3d2-445f-9c4c-4133e5820b26-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-95ee26-769ff594c9-wdwnp\" (UID: \"90adff48-e3d2-445f-9c4c-4133e5820b26\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-95ee26-769ff594c9-wdwnp"
Nov 02 12:36:09.859237 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:09.859221    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90adff48-e3d2-445f-9c4c-4133e5820b26-metrics-certs\") pod \"maas-api-e2e-worker-w2-95ee26-769ff594c9-wdwnp\" (UID: \"90adff48-e3d2-445f-9c4c-4133e5820b26\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-95ee26-769ff594c9-wdwnp"
Nov 02 12:36:09.886286 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:09.886224    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tw2fb\" (UniqueName: \"kubernetes.io/projected/90adff48-e3d2-445f-9c4c-4133e5820b26-kube-api-access-tw2fb\") pod \"maas-api-e2e-worker-w2-95ee26-769ff594c9-wdwnp\" (UID: \"90adff48-e3d2-445f-9c4c-4133e5820b26\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-95ee26-769ff594c9-wdwnp"
Nov 02 12:36:09.892162 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:09.892136    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be21bfc9-2d60-4cfd-8ded-41ed733000bb" path="/var/lib/kubelet/pods/be21bfc9-2d60-4cfd-8ded-41ed733000bb/volumes"
Nov 02 12:36:10.364935 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:10.364900    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90adff48-e3d2-445f-9c4c-4133e5820b26-maas-api-tls\") pod \"maas-api-e2e-worker-w2-95ee26-769ff594c9-wdwnp\" (UID: \"90adff48-e3d2-445f-9c4c-4133e5820b26\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-95ee26-769ff594c9-wdwnp"
Nov 02 12:36:10.387056 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:10.385258    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90adff48-e3d2-445f-9c4c-4133e5820b26-maas-api-tls\") pod \"maas-api-e2e-worker-w2-95ee26-769ff594c9-wdwnp\" (UID: \"90adff48-e3d2-445f-9c4c-4133e5820b26\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-95ee26-769ff594c9-wdwnp"
Nov 02 12:36:10.395197 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:10.395153    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-628cf9bd-openshift-default-75cb8d8c97-g77zh"]
Nov 02 12:36:10.395760 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:10.395729    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-628cf9bd-openshift-default-75cb8d8c97-g77zh" podUID="54f220f8-02ad-4a8b-8c3a-735b0864bdff" containerName="istio-proxy" containerID="cri-o://b09e7499c573a6f4c04b42a3ae75e4892fc57acb825884bb78f5c54606e67046" gracePeriod=30
Nov 02 12:36:10.540653 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:10.540615    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-95ee26-769ff594c9-wdwnp"
Nov 02 12:36:10.885009 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:10.884949    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-95ee26-769ff594c9-wdwnp"]
Nov 02 12:36:10.885291 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:36:10.885262    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90adff48_e3d2_445f_9c4c_4133e5820b26.slice/crio-602723e99fe84622a982c96bd2512e5ba8c094de95f92c5a085c8c9715ddd156 WatchSource:0}: Error finding container 602723e99fe84622a982c96bd2512e5ba8c094de95f92c5a085c8c9715ddd156: Status 404 returned error can't find the container with id 602723e99fe84622a982c96bd2512e5ba8c094de95f92c5a085c8c9715ddd156
Nov 02 12:36:11.484347 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:11.484269    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-95ee26-769ff594c9-wdwnp" event={"ID":"90adff48-e3d2-445f-9c4c-4133e5820b26","Type":"ContainerStarted","Data":"fd10839dd8698d770627382c12113ca42d22a8196b2aa329f9a362fd8853d2b2"}
Nov 02 12:36:11.484347 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:11.484319    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-95ee26-769ff594c9-wdwnp" event={"ID":"90adff48-e3d2-445f-9c4c-4133e5820b26","Type":"ContainerStarted","Data":"602723e99fe84622a982c96bd2512e5ba8c094de95f92c5a085c8c9715ddd156"}
Nov 02 12:36:11.484791 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:11.484399    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-95ee26-769ff594c9-wdwnp"
Nov 02 12:36:11.531399 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:11.531352    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-95ee26-769ff594c9-wdwnp" podStartSLOduration=2.200735663 podStartE2EDuration="2.531338463s" totalImagesPullingTime="330.6028ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:36:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:36:11.529376995 +0000 UTC m=+567.784776103" watchObservedRunningTime="2026-11-02 12:36:11.531338463 +0000 UTC m=+567.786737557"
Nov 02 12:36:11.544028 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:11.543998    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-843978-679c455f86-rj4mr"]
Nov 02 12:36:11.550141 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:11.550119    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-843978-679c455f86-rj4mr"
Nov 02 12:36:11.565675 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:11.565631    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-843978-serving-cert\""
Nov 02 12:36:11.566999 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:11.566978    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-843978-679c455f86-rj4mr"]
Nov 02 12:36:11.683839 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:11.683804    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58hhj\" (UniqueName: \"kubernetes.io/projected/c68bfb23-6abb-4a96-979b-6b00ea4d5f7e-kube-api-access-58hhj\") pod \"maas-api-e2e-worker-w4-843978-679c455f86-rj4mr\" (UID: \"c68bfb23-6abb-4a96-979b-6b00ea4d5f7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-843978-679c455f86-rj4mr"
Nov 02 12:36:11.684039 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:11.683866    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c68bfb23-6abb-4a96-979b-6b00ea4d5f7e-maas-api-tls\") pod \"maas-api-e2e-worker-w4-843978-679c455f86-rj4mr\" (UID: \"c68bfb23-6abb-4a96-979b-6b00ea4d5f7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-843978-679c455f86-rj4mr"
Nov 02 12:36:11.684039 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:11.683973    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c68bfb23-6abb-4a96-979b-6b00ea4d5f7e-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-843978-679c455f86-rj4mr\" (UID: \"c68bfb23-6abb-4a96-979b-6b00ea4d5f7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-843978-679c455f86-rj4mr"
Nov 02 12:36:11.684161 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:11.684055    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c68bfb23-6abb-4a96-979b-6b00ea4d5f7e-metrics-certs\") pod \"maas-api-e2e-worker-w4-843978-679c455f86-rj4mr\" (UID: \"c68bfb23-6abb-4a96-979b-6b00ea4d5f7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-843978-679c455f86-rj4mr"
Nov 02 12:36:11.785552 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:11.785476    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58hhj\" (UniqueName: \"kubernetes.io/projected/c68bfb23-6abb-4a96-979b-6b00ea4d5f7e-kube-api-access-58hhj\") pod \"maas-api-e2e-worker-w4-843978-679c455f86-rj4mr\" (UID: \"c68bfb23-6abb-4a96-979b-6b00ea4d5f7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-843978-679c455f86-rj4mr"
Nov 02 12:36:11.785552 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:11.785515    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c68bfb23-6abb-4a96-979b-6b00ea4d5f7e-maas-api-tls\") pod \"maas-api-e2e-worker-w4-843978-679c455f86-rj4mr\" (UID: \"c68bfb23-6abb-4a96-979b-6b00ea4d5f7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-843978-679c455f86-rj4mr"
Nov 02 12:36:11.785552 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:11.785538    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c68bfb23-6abb-4a96-979b-6b00ea4d5f7e-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-843978-679c455f86-rj4mr\" (UID: \"c68bfb23-6abb-4a96-979b-6b00ea4d5f7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-843978-679c455f86-rj4mr"
Nov 02 12:36:11.785877 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:11.785584    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c68bfb23-6abb-4a96-979b-6b00ea4d5f7e-metrics-certs\") pod \"maas-api-e2e-worker-w4-843978-679c455f86-rj4mr\" (UID: \"c68bfb23-6abb-4a96-979b-6b00ea4d5f7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-843978-679c455f86-rj4mr"
Nov 02 12:36:11.786248 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:11.786224    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c68bfb23-6abb-4a96-979b-6b00ea4d5f7e-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-843978-679c455f86-rj4mr\" (UID: \"c68bfb23-6abb-4a96-979b-6b00ea4d5f7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-843978-679c455f86-rj4mr"
Nov 02 12:36:11.787972 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:11.787947    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c68bfb23-6abb-4a96-979b-6b00ea4d5f7e-metrics-certs\") pod \"maas-api-e2e-worker-w4-843978-679c455f86-rj4mr\" (UID: \"c68bfb23-6abb-4a96-979b-6b00ea4d5f7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-843978-679c455f86-rj4mr"
Nov 02 12:36:11.788064 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:11.787999    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c68bfb23-6abb-4a96-979b-6b00ea4d5f7e-maas-api-tls\") pod \"maas-api-e2e-worker-w4-843978-679c455f86-rj4mr\" (UID: \"c68bfb23-6abb-4a96-979b-6b00ea4d5f7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-843978-679c455f86-rj4mr"
Nov 02 12:36:11.809412 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:11.809389    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-58hhj\" (UniqueName: \"kubernetes.io/projected/c68bfb23-6abb-4a96-979b-6b00ea4d5f7e-kube-api-access-58hhj\") pod \"maas-api-e2e-worker-w4-843978-679c455f86-rj4mr\" (UID: \"c68bfb23-6abb-4a96-979b-6b00ea4d5f7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-843978-679c455f86-rj4mr"
Nov 02 12:36:11.862716 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:11.862690    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-843978-679c455f86-rj4mr"
Nov 02 12:36:12.207886 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:12.207859    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-843978-679c455f86-rj4mr"]
Nov 02 12:36:12.208776 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:36:12.208750    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc68bfb23_6abb_4a96_979b_6b00ea4d5f7e.slice/crio-8275bc7069571f59f93c8cbe87de3340764726e7b7481ae3257dd70d2cdb6c23 WatchSource:0}: Error finding container 8275bc7069571f59f93c8cbe87de3340764726e7b7481ae3257dd70d2cdb6c23: Status 404 returned error can't find the container with id 8275bc7069571f59f93c8cbe87de3340764726e7b7481ae3257dd70d2cdb6c23
Nov 02 12:36:12.496901 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:12.496820    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-843978-679c455f86-rj4mr" event={"ID":"c68bfb23-6abb-4a96-979b-6b00ea4d5f7e","Type":"ContainerStarted","Data":"8275bc7069571f59f93c8cbe87de3340764726e7b7481ae3257dd70d2cdb6c23"}
Nov 02 12:36:13.095665 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:13.095605    4559 patch_prober.go:29] interesting pod/e2e-ait-628cf9bd-openshift-default-75cb8d8c97-g77zh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 12:36:13.095836 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:13.095693    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-628cf9bd-openshift-default-75cb8d8c97-g77zh" podUID="54f220f8-02ad-4a8b-8c3a-735b0864bdff" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 12:36:13.504591 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:13.504551    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-843978-679c455f86-rj4mr" event={"ID":"c68bfb23-6abb-4a96-979b-6b00ea4d5f7e","Type":"ContainerStarted","Data":"37b390df297c04f9616c77c5257b05c3ee9928391c2e5fdde5fbf3a7f7edc728"}
Nov 02 12:36:13.504591 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:13.504674    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-843978-679c455f86-rj4mr"
Nov 02 12:36:13.550452 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:13.550405    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-843978-679c455f86-rj4mr" podStartSLOduration=2.240683356 podStartE2EDuration="2.550391288s" totalImagesPullingTime="309.707932ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:36:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:36:13.549060526 +0000 UTC m=+569.804459622" watchObservedRunningTime="2026-11-02 12:36:13.550391288 +0000 UTC m=+569.805790373"
Nov 02 12:36:15.123957 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:36:15.123919    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqcmj" containerName="extractor"
Nov 02 12:36:15.433575 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:15.433546    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-068756-8565b4cd8c-fpqt6"
Nov 02 12:36:15.433758 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:15.433621    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-068756-bdb7bcbbf-nnssk"
Nov 02 12:36:15.536602 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:15.536577    4559 generic.go:356] "Generic (PLEG): container finished" podID="54f220f8-02ad-4a8b-8c3a-735b0864bdff" containerID="b09e7499c573a6f4c04b42a3ae75e4892fc57acb825884bb78f5c54606e67046" exitCode=0
Nov 02 12:36:15.536738 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:15.536680    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-628cf9bd-openshift-default-75cb8d8c97-g77zh" event={"ID":"54f220f8-02ad-4a8b-8c3a-735b0864bdff","Type":"ContainerDied","Data":"b09e7499c573a6f4c04b42a3ae75e4892fc57acb825884bb78f5c54606e67046"}
Nov 02 12:36:15.536801 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:15.536746    4559 scope.go:122] "RemoveContainer" containerID="e376457369e920ce66d4a6383f4a72ac24b531aeaf6e541fdaf1ffa9e98bd0bb"
Nov 02 12:36:15.736965 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:15.736940    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-628cf9bd-openshift-default-75cb8d8c97-g77zh"
Nov 02 12:36:15.835266 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:15.835226    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54f220f8-02ad-4a8b-8c3a-735b0864bdff-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54f220f8-02ad-4a8b-8c3a-735b0864bdff-istio-data\") pod \"54f220f8-02ad-4a8b-8c3a-735b0864bdff\" (UID: \"54f220f8-02ad-4a8b-8c3a-735b0864bdff\") "
Nov 02 12:36:15.835461 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:15.835282    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54f220f8-02ad-4a8b-8c3a-735b0864bdff-istio-token\" (UniqueName: \"kubernetes.io/projected/54f220f8-02ad-4a8b-8c3a-735b0864bdff-istio-token\") pod \"54f220f8-02ad-4a8b-8c3a-735b0864bdff\" (UID: \"54f220f8-02ad-4a8b-8c3a-735b0864bdff\") "
Nov 02 12:36:15.835461 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:15.835367    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/54f220f8-02ad-4a8b-8c3a-735b0864bdff-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54f220f8-02ad-4a8b-8c3a-735b0864bdff-istio-podinfo\") pod \"54f220f8-02ad-4a8b-8c3a-735b0864bdff\" (UID: \"54f220f8-02ad-4a8b-8c3a-735b0864bdff\") "
Nov 02 12:36:15.835573 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:15.835466    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54f220f8-02ad-4a8b-8c3a-735b0864bdff-istio-data" pod "54f220f8-02ad-4a8b-8c3a-735b0864bdff" (UID: "54f220f8-02ad-4a8b-8c3a-735b0864bdff"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:36:15.835573 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:15.835525    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54f220f8-02ad-4a8b-8c3a-735b0864bdff-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54f220f8-02ad-4a8b-8c3a-735b0864bdff-istio-envoy\") pod \"54f220f8-02ad-4a8b-8c3a-735b0864bdff\" (UID: \"54f220f8-02ad-4a8b-8c3a-735b0864bdff\") "
Nov 02 12:36:15.835737 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:15.835588    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54f220f8-02ad-4a8b-8c3a-735b0864bdff-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54f220f8-02ad-4a8b-8c3a-735b0864bdff-istiod-ca-cert\") pod \"54f220f8-02ad-4a8b-8c3a-735b0864bdff\" (UID: \"54f220f8-02ad-4a8b-8c3a-735b0864bdff\") "
Nov 02 12:36:15.835737 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:15.835622    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54f220f8-02ad-4a8b-8c3a-735b0864bdff-kube-api-access-6465t\" (UniqueName: \"kubernetes.io/projected/54f220f8-02ad-4a8b-8c3a-735b0864bdff-kube-api-access-6465t\") pod \"54f220f8-02ad-4a8b-8c3a-735b0864bdff\" (UID: \"54f220f8-02ad-4a8b-8c3a-735b0864bdff\") "
Nov 02 12:36:15.835737 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:15.835684    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54f220f8-02ad-4a8b-8c3a-735b0864bdff-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54f220f8-02ad-4a8b-8c3a-735b0864bdff-credential-socket\") pod \"54f220f8-02ad-4a8b-8c3a-735b0864bdff\" (UID: \"54f220f8-02ad-4a8b-8c3a-735b0864bdff\") "
Nov 02 12:36:15.835737 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:15.835722    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54f220f8-02ad-4a8b-8c3a-735b0864bdff-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54f220f8-02ad-4a8b-8c3a-735b0864bdff-workload-socket\") pod \"54f220f8-02ad-4a8b-8c3a-735b0864bdff\" (UID: \"54f220f8-02ad-4a8b-8c3a-735b0864bdff\") "
Nov 02 12:36:15.835936 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:15.835782    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54f220f8-02ad-4a8b-8c3a-735b0864bdff-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54f220f8-02ad-4a8b-8c3a-735b0864bdff-workload-certs\") pod \"54f220f8-02ad-4a8b-8c3a-735b0864bdff\" (UID: \"54f220f8-02ad-4a8b-8c3a-735b0864bdff\") "
Nov 02 12:36:15.836038 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:15.836015    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54f220f8-02ad-4a8b-8c3a-735b0864bdff-credential-socket" pod "54f220f8-02ad-4a8b-8c3a-735b0864bdff" (UID: "54f220f8-02ad-4a8b-8c3a-735b0864bdff"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:36:15.836226 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:15.836188    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54f220f8-02ad-4a8b-8c3a-735b0864bdff-workload-socket" pod "54f220f8-02ad-4a8b-8c3a-735b0864bdff" (UID: "54f220f8-02ad-4a8b-8c3a-735b0864bdff"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:36:15.836304 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:15.836208    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54f220f8-02ad-4a8b-8c3a-735b0864bdff-workload-certs" pod "54f220f8-02ad-4a8b-8c3a-735b0864bdff" (UID: "54f220f8-02ad-4a8b-8c3a-735b0864bdff"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:36:15.836304 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:15.836195    4559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54f220f8-02ad-4a8b-8c3a-735b0864bdff-istio-data\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:36:15.836304 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:15.836273    4559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54f220f8-02ad-4a8b-8c3a-735b0864bdff-credential-socket\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:36:15.836467 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:15.836314    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54f220f8-02ad-4a8b-8c3a-735b0864bdff-istiod-ca-cert" pod "54f220f8-02ad-4a8b-8c3a-735b0864bdff" (UID: "54f220f8-02ad-4a8b-8c3a-735b0864bdff"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:36:15.837691 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:15.837663    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54f220f8-02ad-4a8b-8c3a-735b0864bdff-istio-token" pod "54f220f8-02ad-4a8b-8c3a-735b0864bdff" (UID: "54f220f8-02ad-4a8b-8c3a-735b0864bdff"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:36:15.837805 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:15.837700    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/54f220f8-02ad-4a8b-8c3a-735b0864bdff-istio-podinfo" pod "54f220f8-02ad-4a8b-8c3a-735b0864bdff" (UID: "54f220f8-02ad-4a8b-8c3a-735b0864bdff"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 12:36:15.838001 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:15.837980    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54f220f8-02ad-4a8b-8c3a-735b0864bdff-istio-envoy" pod "54f220f8-02ad-4a8b-8c3a-735b0864bdff" (UID: "54f220f8-02ad-4a8b-8c3a-735b0864bdff"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:36:15.838075 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:15.838020    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54f220f8-02ad-4a8b-8c3a-735b0864bdff-kube-api-access-6465t" pod "54f220f8-02ad-4a8b-8c3a-735b0864bdff" (UID: "54f220f8-02ad-4a8b-8c3a-735b0864bdff"). InnerVolumeSpecName "kube-api-access-6465t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:36:15.937455 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:15.937427    4559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54f220f8-02ad-4a8b-8c3a-735b0864bdff-istio-podinfo\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:36:15.937455 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:15.937451    4559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54f220f8-02ad-4a8b-8c3a-735b0864bdff-istio-envoy\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:36:15.937455 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:15.937460    4559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54f220f8-02ad-4a8b-8c3a-735b0864bdff-istiod-ca-cert\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:36:15.937722 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:15.937469    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6465t\" (UniqueName: \"kubernetes.io/projected/54f220f8-02ad-4a8b-8c3a-735b0864bdff-kube-api-access-6465t\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:36:15.937722 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:15.937478    4559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54f220f8-02ad-4a8b-8c3a-735b0864bdff-workload-socket\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:36:15.937722 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:15.937487    4559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54f220f8-02ad-4a8b-8c3a-735b0864bdff-workload-certs\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:36:15.937722 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:15.937496    4559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54f220f8-02ad-4a8b-8c3a-735b0864bdff-istio-token\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:36:16.509870 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:16.508405    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-2f0bef09-68f9f765c8-zngcp"]
Nov 02 12:36:16.509870 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:16.508684    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2f0bef09-68f9f765c8-zngcp" podUID="d393c034-7597-427f-af11-aed585377518" containerName="maas-api" containerID="cri-o://2a3e0d9a1650ae5a2fd3d395c920f9b4b97eedecaf3fec239b6a18c9b5839ca4" gracePeriod=30
Nov 02 12:36:16.543203 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:16.543177    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-628cf9bd-openshift-default-75cb8d8c97-g77zh" event={"ID":"54f220f8-02ad-4a8b-8c3a-735b0864bdff","Type":"ContainerDied","Data":"42be404ad25c4cda44779f1b6c0f5b5106d48779d3d03e8ccb11d762095b2b3e"}
Nov 02 12:36:16.543203 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:16.543200    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-628cf9bd-openshift-default-75cb8d8c97-g77zh"
Nov 02 12:36:16.543361 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:16.543242    4559 scope.go:122] "RemoveContainer" containerID="b09e7499c573a6f4c04b42a3ae75e4892fc57acb825884bb78f5c54606e67046"
Nov 02 12:36:16.620669 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:16.620617    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-628cf9bd-openshift-default-75cb8d8c97-g77zh"]
Nov 02 12:36:16.626474 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:16.626451    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-628cf9bd-openshift-default-75cb8d8c97-g77zh"]
Nov 02 12:36:16.820367 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:16.820339    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2f0bef09-68f9f765c8-zngcp"
Nov 02 12:36:16.849803 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:16.849774    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d393c034-7597-427f-af11-aed585377518-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d393c034-7597-427f-af11-aed585377518-maas-api-tls\") pod \"d393c034-7597-427f-af11-aed585377518\" (UID: \"d393c034-7597-427f-af11-aed585377518\") "
Nov 02 12:36:16.849803 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:16.849807    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d393c034-7597-427f-af11-aed585377518-kube-api-access-92hvr\" (UniqueName: \"kubernetes.io/projected/d393c034-7597-427f-af11-aed585377518-kube-api-access-92hvr\") pod \"d393c034-7597-427f-af11-aed585377518\" (UID: \"d393c034-7597-427f-af11-aed585377518\") "
Nov 02 12:36:16.850009 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:16.849972    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d393c034-7597-427f-af11-aed585377518-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d393c034-7597-427f-af11-aed585377518-openshift-service-ca\") pod \"d393c034-7597-427f-af11-aed585377518\" (UID: \"d393c034-7597-427f-af11-aed585377518\") "
Nov 02 12:36:16.850085 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:16.850059    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d393c034-7597-427f-af11-aed585377518-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d393c034-7597-427f-af11-aed585377518-metrics-certs\") pod \"d393c034-7597-427f-af11-aed585377518\" (UID: \"d393c034-7597-427f-af11-aed585377518\") "
Nov 02 12:36:16.850390 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:16.850365    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d393c034-7597-427f-af11-aed585377518-openshift-service-ca" pod "d393c034-7597-427f-af11-aed585377518" (UID: "d393c034-7597-427f-af11-aed585377518"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:36:16.851048 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:16.851003    4559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d393c034-7597-427f-af11-aed585377518-openshift-service-ca\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:36:16.854679 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:16.852963    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d393c034-7597-427f-af11-aed585377518-kube-api-access-92hvr" pod "d393c034-7597-427f-af11-aed585377518" (UID: "d393c034-7597-427f-af11-aed585377518"). InnerVolumeSpecName "kube-api-access-92hvr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:36:16.858567 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:16.858542    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d393c034-7597-427f-af11-aed585377518-maas-api-tls" pod "d393c034-7597-427f-af11-aed585377518" (UID: "d393c034-7597-427f-af11-aed585377518"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:36:16.859388 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:16.859353    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d393c034-7597-427f-af11-aed585377518-metrics-certs" pod "d393c034-7597-427f-af11-aed585377518" (UID: "d393c034-7597-427f-af11-aed585377518"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:36:16.953748 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:16.952171    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d393c034-7597-427f-af11-aed585377518-maas-api-tls\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:36:16.953748 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:16.952199    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-92hvr\" (UniqueName: \"kubernetes.io/projected/d393c034-7597-427f-af11-aed585377518-kube-api-access-92hvr\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:36:16.953748 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:16.952210    4559 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d393c034-7597-427f-af11-aed585377518-metrics-certs\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:36:17.539821 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:17.539787    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-95ee26-769ff594c9-wdwnp"
Nov 02 12:36:17.554399 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:17.553758    4559 generic.go:356] "Generic (PLEG): container finished" podID="d393c034-7597-427f-af11-aed585377518" containerID="2a3e0d9a1650ae5a2fd3d395c920f9b4b97eedecaf3fec239b6a18c9b5839ca4" exitCode=0
Nov 02 12:36:17.554620 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:17.553834    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2f0bef09-68f9f765c8-zngcp"
Nov 02 12:36:17.554984 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:17.553938    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2f0bef09-68f9f765c8-zngcp" event={"ID":"d393c034-7597-427f-af11-aed585377518","Type":"ContainerDied","Data":"2a3e0d9a1650ae5a2fd3d395c920f9b4b97eedecaf3fec239b6a18c9b5839ca4"}
Nov 02 12:36:17.555063 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:17.555025    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2f0bef09-68f9f765c8-zngcp" event={"ID":"d393c034-7597-427f-af11-aed585377518","Type":"ContainerDied","Data":"e6cd6b2891d2b19e81f3b45ff235c4f867b1ce6cc94d868d0c5818c82113fa53"}
Nov 02 12:36:17.555124 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:17.555066    4559 scope.go:122] "RemoveContainer" containerID="2a3e0d9a1650ae5a2fd3d395c920f9b4b97eedecaf3fec239b6a18c9b5839ca4"
Nov 02 12:36:17.567538 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:17.567513    4559 scope.go:122] "RemoveContainer" containerID="2a3e0d9a1650ae5a2fd3d395c920f9b4b97eedecaf3fec239b6a18c9b5839ca4"
Nov 02 12:36:17.567914 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:36:17.567886    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a3e0d9a1650ae5a2fd3d395c920f9b4b97eedecaf3fec239b6a18c9b5839ca4\": container with ID starting with 2a3e0d9a1650ae5a2fd3d395c920f9b4b97eedecaf3fec239b6a18c9b5839ca4 not found: ID does not exist" containerID="2a3e0d9a1650ae5a2fd3d395c920f9b4b97eedecaf3fec239b6a18c9b5839ca4"
Nov 02 12:36:17.567998 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:17.567929    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a3e0d9a1650ae5a2fd3d395c920f9b4b97eedecaf3fec239b6a18c9b5839ca4"} err="failed to get container status \"2a3e0d9a1650ae5a2fd3d395c920f9b4b97eedecaf3fec239b6a18c9b5839ca4\": rpc error: code = NotFound desc = could not find container \"2a3e0d9a1650ae5a2fd3d395c920f9b4b97eedecaf3fec239b6a18c9b5839ca4\": container with ID starting with 2a3e0d9a1650ae5a2fd3d395c920f9b4b97eedecaf3fec239b6a18c9b5839ca4 not found: ID does not exist"
Nov 02 12:36:17.637816 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:17.637607    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-2f0bef09-68f9f765c8-zngcp"]
Nov 02 12:36:17.641854 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:17.641828    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-2f0bef09-68f9f765c8-zngcp"]
Nov 02 12:36:17.681997 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:17.681964    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-5514e5-cc9847d79-8jprz"]
Nov 02 12:36:17.682372 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:17.682359    4559 cpu_manager.go:414] "containerMap: removing container" podUID="54f220f8-02ad-4a8b-8c3a-735b0864bdff" containerName="istio-proxy"
Nov 02 12:36:17.682415 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:17.682372    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54f220f8-02ad-4a8b-8c3a-735b0864bdff" containerName="istio-proxy"
Nov 02 12:36:17.682415 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:17.682383    4559 cpu_manager.go:414] "containerMap: removing container" podUID="54f220f8-02ad-4a8b-8c3a-735b0864bdff" containerName="istio-proxy"
Nov 02 12:36:17.682415 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:17.682389    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54f220f8-02ad-4a8b-8c3a-735b0864bdff" containerName="istio-proxy"
Nov 02 12:36:17.682502 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:17.682416    4559 cpu_manager.go:414] "containerMap: removing container" podUID="d393c034-7597-427f-af11-aed585377518" containerName="maas-api"
Nov 02 12:36:17.682502 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:17.682422    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d393c034-7597-427f-af11-aed585377518" containerName="maas-api"
Nov 02 12:36:17.682502 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:17.682483    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="d393c034-7597-427f-af11-aed585377518" containerName="maas-api"
Nov 02 12:36:17.682502 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:17.682495    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="54f220f8-02ad-4a8b-8c3a-735b0864bdff" containerName="istio-proxy"
Nov 02 12:36:17.682502 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:17.682502    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="54f220f8-02ad-4a8b-8c3a-735b0864bdff" containerName="istio-proxy"
Nov 02 12:36:17.709475 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:17.709205    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-5514e5-cc9847d79-8jprz"
Nov 02 12:36:17.710688 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:17.709760    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-5514e5-b884d6445-hpkbk"]
Nov 02 12:36:17.724361 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:17.722749    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-5514e5-cc9847d79-8jprz"]
Nov 02 12:36:17.724361 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:17.722849    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-5514e5-b884d6445-hpkbk"
Nov 02 12:36:17.727112 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:17.726863    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-5514e5-dockercfg-ck8bh\""
Nov 02 12:36:17.727112 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:17.727013    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-5514e5\""
Nov 02 12:36:17.729681 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:17.729599    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-5514e5-b884d6445-hpkbk"]
Nov 02 12:36:17.760306 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:17.760280    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/323858c5-a582-47b3-b38f-af6a750580ca-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-5514e5-b884d6445-hpkbk\" (UID: \"323858c5-a582-47b3-b38f-af6a750580ca\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-5514e5-b884d6445-hpkbk"
Nov 02 12:36:17.760582 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:17.760566    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d5d728d-c371-4307-8c83-601a6e63d10c-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-5514e5-cc9847d79-8jprz\" (UID: \"7d5d728d-c371-4307-8c83-601a6e63d10c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-5514e5-cc9847d79-8jprz"
Nov 02 12:36:17.760875 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:17.760846    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlt5t\" (UniqueName: \"kubernetes.io/projected/323858c5-a582-47b3-b38f-af6a750580ca-kube-api-access-zlt5t\") pod \"payload-processing-e2e-worker-w0-5514e5-b884d6445-hpkbk\" (UID: \"323858c5-a582-47b3-b38f-af6a750580ca\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-5514e5-b884d6445-hpkbk"
Nov 02 12:36:17.760981 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:17.760895    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptfvf\" (UniqueName: \"kubernetes.io/projected/7d5d728d-c371-4307-8c83-601a6e63d10c-kube-api-access-ptfvf\") pod \"payload-pre-processing-e2e-worker-w0-5514e5-cc9847d79-8jprz\" (UID: \"7d5d728d-c371-4307-8c83-601a6e63d10c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-5514e5-cc9847d79-8jprz"
Nov 02 12:36:17.862650 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:17.862564    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d5d728d-c371-4307-8c83-601a6e63d10c-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-5514e5-cc9847d79-8jprz\" (UID: \"7d5d728d-c371-4307-8c83-601a6e63d10c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-5514e5-cc9847d79-8jprz"
Nov 02 12:36:17.862886 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:17.862871    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlt5t\" (UniqueName: \"kubernetes.io/projected/323858c5-a582-47b3-b38f-af6a750580ca-kube-api-access-zlt5t\") pod \"payload-processing-e2e-worker-w0-5514e5-b884d6445-hpkbk\" (UID: \"323858c5-a582-47b3-b38f-af6a750580ca\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-5514e5-b884d6445-hpkbk"
Nov 02 12:36:17.862967 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:17.862958    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptfvf\" (UniqueName: \"kubernetes.io/projected/7d5d728d-c371-4307-8c83-601a6e63d10c-kube-api-access-ptfvf\") pod \"payload-pre-processing-e2e-worker-w0-5514e5-cc9847d79-8jprz\" (UID: \"7d5d728d-c371-4307-8c83-601a6e63d10c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-5514e5-cc9847d79-8jprz"
Nov 02 12:36:17.863062 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:17.863053    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/323858c5-a582-47b3-b38f-af6a750580ca-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-5514e5-b884d6445-hpkbk\" (UID: \"323858c5-a582-47b3-b38f-af6a750580ca\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-5514e5-b884d6445-hpkbk"
Nov 02 12:36:17.863299 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:17.863276    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d5d728d-c371-4307-8c83-601a6e63d10c-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-5514e5-cc9847d79-8jprz\" (UID: \"7d5d728d-c371-4307-8c83-601a6e63d10c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-5514e5-cc9847d79-8jprz"
Nov 02 12:36:17.863827 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:17.863810    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/323858c5-a582-47b3-b38f-af6a750580ca-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-5514e5-b884d6445-hpkbk\" (UID: \"323858c5-a582-47b3-b38f-af6a750580ca\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-5514e5-b884d6445-hpkbk"
Nov 02 12:36:17.910675 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:17.910528    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptfvf\" (UniqueName: \"kubernetes.io/projected/7d5d728d-c371-4307-8c83-601a6e63d10c-kube-api-access-ptfvf\") pod \"payload-pre-processing-e2e-worker-w0-5514e5-cc9847d79-8jprz\" (UID: \"7d5d728d-c371-4307-8c83-601a6e63d10c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-5514e5-cc9847d79-8jprz"
Nov 02 12:36:17.912649 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:17.912609    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlt5t\" (UniqueName: \"kubernetes.io/projected/323858c5-a582-47b3-b38f-af6a750580ca-kube-api-access-zlt5t\") pod \"payload-processing-e2e-worker-w0-5514e5-b884d6445-hpkbk\" (UID: \"323858c5-a582-47b3-b38f-af6a750580ca\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-5514e5-b884d6445-hpkbk"
Nov 02 12:36:17.914830 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:17.914806    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54f220f8-02ad-4a8b-8c3a-735b0864bdff" path="/var/lib/kubelet/pods/54f220f8-02ad-4a8b-8c3a-735b0864bdff/volumes"
Nov 02 12:36:17.915227 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:17.915214    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d393c034-7597-427f-af11-aed585377518" path="/var/lib/kubelet/pods/d393c034-7597-427f-af11-aed585377518/volumes"
Nov 02 12:36:18.038316 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:18.038281    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-5514e5-cc9847d79-8jprz"
Nov 02 12:36:18.057674 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:18.057629    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-5514e5-b884d6445-hpkbk"
Nov 02 12:36:18.364544 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:18.364517    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-5514e5-cc9847d79-8jprz"]
Nov 02 12:36:18.365134 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:36:18.365107    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d5d728d_c371_4307_8c83_601a6e63d10c.slice/crio-5bb24b6c27243f6dd7513923f12c0a71c5aa6c3a8a57bff3e7175d0e10939e17 WatchSource:0}: Error finding container 5bb24b6c27243f6dd7513923f12c0a71c5aa6c3a8a57bff3e7175d0e10939e17: Status 404 returned error can't find the container with id 5bb24b6c27243f6dd7513923f12c0a71c5aa6c3a8a57bff3e7175d0e10939e17
Nov 02 12:36:18.410142 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:18.409204    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-5514e5-b884d6445-hpkbk"]
Nov 02 12:36:18.413951 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:36:18.413919    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod323858c5_a582_47b3_b38f_af6a750580ca.slice/crio-810283fd84af18d5942bfda569ee4c04ac3353e8d65a76b2b7041d5d1125135a WatchSource:0}: Error finding container 810283fd84af18d5942bfda569ee4c04ac3353e8d65a76b2b7041d5d1125135a: Status 404 returned error can't find the container with id 810283fd84af18d5942bfda569ee4c04ac3353e8d65a76b2b7041d5d1125135a
Nov 02 12:36:18.460276 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:18.460254    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-95ee26-5f96f56f79-2zqdd"
Nov 02 12:36:18.562867 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:18.562835    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-5514e5-cc9847d79-8jprz" event={"ID":"7d5d728d-c371-4307-8c83-601a6e63d10c","Type":"ContainerStarted","Data":"9e1762642d89763777ed3a88948f10ddbb1af3415403bda5d91a996a88e58931"}
Nov 02 12:36:18.563288 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:18.562886    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-5514e5-cc9847d79-8jprz" event={"ID":"7d5d728d-c371-4307-8c83-601a6e63d10c","Type":"ContainerStarted","Data":"5bb24b6c27243f6dd7513923f12c0a71c5aa6c3a8a57bff3e7175d0e10939e17"}
Nov 02 12:36:18.563288 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:18.563074    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-5514e5-cc9847d79-8jprz"
Nov 02 12:36:18.564632 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:18.564599    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-5514e5-b884d6445-hpkbk" event={"ID":"323858c5-a582-47b3-b38f-af6a750580ca","Type":"ContainerStarted","Data":"f8b322bb97c51daf956a8928056f0df68b3dc8d0d34e7174883bfb0fbac42762"}
Nov 02 12:36:18.564789 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:18.564684    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-5514e5-b884d6445-hpkbk" event={"ID":"323858c5-a582-47b3-b38f-af6a750580ca","Type":"ContainerStarted","Data":"810283fd84af18d5942bfda569ee4c04ac3353e8d65a76b2b7041d5d1125135a"}
Nov 02 12:36:18.564789 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:18.564707    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-5514e5-b884d6445-hpkbk"
Nov 02 12:36:18.653296 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:18.653197    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-5514e5-b884d6445-hpkbk" podStartSLOduration=1.653181658 podStartE2EDuration="1.653181658s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:36:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:36:18.650787962 +0000 UTC m=+574.906187057" watchObservedRunningTime="2026-11-02 12:36:18.653181658 +0000 UTC m=+574.908580784"
Nov 02 12:36:18.653458 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:18.653329    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-5514e5-cc9847d79-8jprz" podStartSLOduration=1.653325874 podStartE2EDuration="1.653325874s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:36:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:36:18.607977641 +0000 UTC m=+574.863376749" watchObservedRunningTime="2026-11-02 12:36:18.653325874 +0000 UTC m=+574.908724980"
Nov 02 12:36:18.837093 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:18.837056    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-ffpz2"]
Nov 02 12:36:18.842509 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:18.842480    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-ffpz2"
Nov 02 12:36:18.862815 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:18.862785    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-61e341-openshift-default-dockercfg-jcvsw\""
Nov 02 12:36:18.866779 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:18.866755    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-ffpz2"]
Nov 02 12:36:18.875789 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:18.875757    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91b0660b-bd2c-4f62-ac8a-e09af747a766-credential-socket\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-ffpz2\" (UID: \"91b0660b-bd2c-4f62-ac8a-e09af747a766\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-ffpz2"
Nov 02 12:36:18.875930 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:18.875813    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tsgbv\" (UniqueName: \"kubernetes.io/projected/91b0660b-bd2c-4f62-ac8a-e09af747a766-kube-api-access-tsgbv\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-ffpz2\" (UID: \"91b0660b-bd2c-4f62-ac8a-e09af747a766\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-ffpz2"
Nov 02 12:36:18.875930 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:18.875844    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91b0660b-bd2c-4f62-ac8a-e09af747a766-istio-token\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-ffpz2\" (UID: \"91b0660b-bd2c-4f62-ac8a-e09af747a766\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-ffpz2"
Nov 02 12:36:18.875930 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:18.875874    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91b0660b-bd2c-4f62-ac8a-e09af747a766-istiod-ca-cert\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-ffpz2\" (UID: \"91b0660b-bd2c-4f62-ac8a-e09af747a766\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-ffpz2"
Nov 02 12:36:18.876100 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:18.875934    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91b0660b-bd2c-4f62-ac8a-e09af747a766-istio-podinfo\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-ffpz2\" (UID: \"91b0660b-bd2c-4f62-ac8a-e09af747a766\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-ffpz2"
Nov 02 12:36:18.876100 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:18.875999    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91b0660b-bd2c-4f62-ac8a-e09af747a766-workload-socket\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-ffpz2\" (UID: \"91b0660b-bd2c-4f62-ac8a-e09af747a766\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-ffpz2"
Nov 02 12:36:18.876220 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:18.876203    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91b0660b-bd2c-4f62-ac8a-e09af747a766-istio-data\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-ffpz2\" (UID: \"91b0660b-bd2c-4f62-ac8a-e09af747a766\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-ffpz2"
Nov 02 12:36:18.876309 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:18.876272    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91b0660b-bd2c-4f62-ac8a-e09af747a766-workload-certs\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-ffpz2\" (UID: \"91b0660b-bd2c-4f62-ac8a-e09af747a766\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-ffpz2"
Nov 02 12:36:18.876368 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:18.876352    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91b0660b-bd2c-4f62-ac8a-e09af747a766-istio-envoy\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-ffpz2\" (UID: \"91b0660b-bd2c-4f62-ac8a-e09af747a766\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-ffpz2"
Nov 02 12:36:18.977546 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:18.977303    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91b0660b-bd2c-4f62-ac8a-e09af747a766-istio-envoy\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-ffpz2\" (UID: \"91b0660b-bd2c-4f62-ac8a-e09af747a766\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-ffpz2"
Nov 02 12:36:18.977792 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:18.977763    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91b0660b-bd2c-4f62-ac8a-e09af747a766-credential-socket\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-ffpz2\" (UID: \"91b0660b-bd2c-4f62-ac8a-e09af747a766\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-ffpz2"
Nov 02 12:36:18.977971 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:18.977952    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tsgbv\" (UniqueName: \"kubernetes.io/projected/91b0660b-bd2c-4f62-ac8a-e09af747a766-kube-api-access-tsgbv\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-ffpz2\" (UID: \"91b0660b-bd2c-4f62-ac8a-e09af747a766\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-ffpz2"
Nov 02 12:36:18.978115 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:18.978098    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91b0660b-bd2c-4f62-ac8a-e09af747a766-istio-token\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-ffpz2\" (UID: \"91b0660b-bd2c-4f62-ac8a-e09af747a766\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-ffpz2"
Nov 02 12:36:18.978343 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:18.978311    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91b0660b-bd2c-4f62-ac8a-e09af747a766-istiod-ca-cert\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-ffpz2\" (UID: \"91b0660b-bd2c-4f62-ac8a-e09af747a766\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-ffpz2"
Nov 02 12:36:18.978542 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:18.978521    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91b0660b-bd2c-4f62-ac8a-e09af747a766-credential-socket\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-ffpz2\" (UID: \"91b0660b-bd2c-4f62-ac8a-e09af747a766\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-ffpz2"
Nov 02 12:36:18.979131 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:18.979105    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91b0660b-bd2c-4f62-ac8a-e09af747a766-istiod-ca-cert\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-ffpz2\" (UID: \"91b0660b-bd2c-4f62-ac8a-e09af747a766\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-ffpz2"
Nov 02 12:36:18.979296 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:18.979273    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91b0660b-bd2c-4f62-ac8a-e09af747a766-istio-podinfo\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-ffpz2\" (UID: \"91b0660b-bd2c-4f62-ac8a-e09af747a766\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-ffpz2"
Nov 02 12:36:18.979364 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:18.979354    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91b0660b-bd2c-4f62-ac8a-e09af747a766-workload-socket\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-ffpz2\" (UID: \"91b0660b-bd2c-4f62-ac8a-e09af747a766\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-ffpz2"
Nov 02 12:36:18.979410 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:18.979393    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91b0660b-bd2c-4f62-ac8a-e09af747a766-istio-data\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-ffpz2\" (UID: \"91b0660b-bd2c-4f62-ac8a-e09af747a766\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-ffpz2"
Nov 02 12:36:18.979460 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:18.979437    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91b0660b-bd2c-4f62-ac8a-e09af747a766-workload-certs\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-ffpz2\" (UID: \"91b0660b-bd2c-4f62-ac8a-e09af747a766\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-ffpz2"
Nov 02 12:36:18.979783 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:18.979734    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91b0660b-bd2c-4f62-ac8a-e09af747a766-workload-socket\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-ffpz2\" (UID: \"91b0660b-bd2c-4f62-ac8a-e09af747a766\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-ffpz2"
Nov 02 12:36:18.979892 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:18.979832    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91b0660b-bd2c-4f62-ac8a-e09af747a766-workload-certs\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-ffpz2\" (UID: \"91b0660b-bd2c-4f62-ac8a-e09af747a766\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-ffpz2"
Nov 02 12:36:18.980107 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:18.980085    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91b0660b-bd2c-4f62-ac8a-e09af747a766-istio-data\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-ffpz2\" (UID: \"91b0660b-bd2c-4f62-ac8a-e09af747a766\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-ffpz2"
Nov 02 12:36:18.980630 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:18.980423    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91b0660b-bd2c-4f62-ac8a-e09af747a766-istio-envoy\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-ffpz2\" (UID: \"91b0660b-bd2c-4f62-ac8a-e09af747a766\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-ffpz2"
Nov 02 12:36:19.026261 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:19.026221    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91b0660b-bd2c-4f62-ac8a-e09af747a766-istio-podinfo\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-ffpz2\" (UID: \"91b0660b-bd2c-4f62-ac8a-e09af747a766\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-ffpz2"
Nov 02 12:36:19.038686 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:19.027529    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tsgbv\" (UniqueName: \"kubernetes.io/projected/91b0660b-bd2c-4f62-ac8a-e09af747a766-kube-api-access-tsgbv\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-ffpz2\" (UID: \"91b0660b-bd2c-4f62-ac8a-e09af747a766\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-ffpz2"
Nov 02 12:36:19.039495 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:19.039464    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91b0660b-bd2c-4f62-ac8a-e09af747a766-istio-token\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-ffpz2\" (UID: \"91b0660b-bd2c-4f62-ac8a-e09af747a766\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-ffpz2"
Nov 02 12:36:19.168221 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:19.168182    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-ffpz2"
Nov 02 12:36:19.512817 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:19.512777    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-ffpz2"]
Nov 02 12:36:19.514018 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:36:19.513981    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91b0660b_bd2c_4f62_ac8a_e09af747a766.slice/crio-a211508dfbe1d54925de5494fe78efdb6c9079096bfc74398787ccd989515b2f WatchSource:0}: Error finding container a211508dfbe1d54925de5494fe78efdb6c9079096bfc74398787ccd989515b2f: Status 404 returned error can't find the container with id a211508dfbe1d54925de5494fe78efdb6c9079096bfc74398787ccd989515b2f
Nov 02 12:36:19.516600 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:19.516564    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:36:19.516703 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:19.516632    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:36:19.516703 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:19.516674    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:36:19.535183 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:19.535165    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-843978-679c455f86-rj4mr"
Nov 02 12:36:19.571591 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:19.571558    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-ffpz2" event={"ID":"91b0660b-bd2c-4f62-ac8a-e09af747a766","Type":"ContainerStarted","Data":"a211508dfbe1d54925de5494fe78efdb6c9079096bfc74398787ccd989515b2f"}
Nov 02 12:36:20.581353 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:20.581313    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-ffpz2" event={"ID":"91b0660b-bd2c-4f62-ac8a-e09af747a766","Type":"ContainerStarted","Data":"f263118547ccecb716514445c970370884eac55bf80594ea71b34d72e20be424"}
Nov 02 12:36:20.652927 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:20.652870    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-ffpz2" podStartSLOduration=2.652854141 podStartE2EDuration="2.652854141s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:36:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:36:20.639537891 +0000 UTC m=+576.894936987" watchObservedRunningTime="2026-11-02 12:36:20.652854141 +0000 UTC m=+576.908253289"
Nov 02 12:36:20.930721 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:20.917120    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"]
Nov 02 12:36:21.168492 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:21.168447    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-ffpz2"
Nov 02 12:36:21.173926 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:21.173897    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-ffpz2"
Nov 02 12:36:21.586582 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:21.586548    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-ffpz2"
Nov 02 12:36:21.587840 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:21.587817    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-ffpz2"
Nov 02 12:36:22.618581 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:22.618543    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-5514e5-75476c695b-8sl6l"]
Nov 02 12:36:22.622260 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:22.622238    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-5514e5-75476c695b-8sl6l"
Nov 02 12:36:22.656901 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:22.656866    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-5514e5-serving-cert\""
Nov 02 12:36:22.659172 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:22.659143    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-5514e5-75476c695b-8sl6l"]
Nov 02 12:36:22.758620 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:22.758576    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75a5e1c8-189e-4005-a8b7-48b007680900-metrics-certs\") pod \"maas-api-e2e-worker-w0-5514e5-75476c695b-8sl6l\" (UID: \"75a5e1c8-189e-4005-a8b7-48b007680900\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-5514e5-75476c695b-8sl6l"
Nov 02 12:36:22.758827 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:22.758705    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/75a5e1c8-189e-4005-a8b7-48b007680900-maas-api-tls\") pod \"maas-api-e2e-worker-w0-5514e5-75476c695b-8sl6l\" (UID: \"75a5e1c8-189e-4005-a8b7-48b007680900\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-5514e5-75476c695b-8sl6l"
Nov 02 12:36:22.758827 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:22.758771    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/75a5e1c8-189e-4005-a8b7-48b007680900-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-5514e5-75476c695b-8sl6l\" (UID: \"75a5e1c8-189e-4005-a8b7-48b007680900\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-5514e5-75476c695b-8sl6l"
Nov 02 12:36:22.758827 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:22.758815    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdfwj\" (UniqueName: \"kubernetes.io/projected/75a5e1c8-189e-4005-a8b7-48b007680900-kube-api-access-xdfwj\") pod \"maas-api-e2e-worker-w0-5514e5-75476c695b-8sl6l\" (UID: \"75a5e1c8-189e-4005-a8b7-48b007680900\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-5514e5-75476c695b-8sl6l"
Nov 02 12:36:22.860478 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:22.860443    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75a5e1c8-189e-4005-a8b7-48b007680900-metrics-certs\") pod \"maas-api-e2e-worker-w0-5514e5-75476c695b-8sl6l\" (UID: \"75a5e1c8-189e-4005-a8b7-48b007680900\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-5514e5-75476c695b-8sl6l"
Nov 02 12:36:22.860689 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:22.860517    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/75a5e1c8-189e-4005-a8b7-48b007680900-maas-api-tls\") pod \"maas-api-e2e-worker-w0-5514e5-75476c695b-8sl6l\" (UID: \"75a5e1c8-189e-4005-a8b7-48b007680900\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-5514e5-75476c695b-8sl6l"
Nov 02 12:36:22.860689 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:22.860572    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/75a5e1c8-189e-4005-a8b7-48b007680900-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-5514e5-75476c695b-8sl6l\" (UID: \"75a5e1c8-189e-4005-a8b7-48b007680900\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-5514e5-75476c695b-8sl6l"
Nov 02 12:36:22.860689 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:22.860612    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdfwj\" (UniqueName: \"kubernetes.io/projected/75a5e1c8-189e-4005-a8b7-48b007680900-kube-api-access-xdfwj\") pod \"maas-api-e2e-worker-w0-5514e5-75476c695b-8sl6l\" (UID: \"75a5e1c8-189e-4005-a8b7-48b007680900\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-5514e5-75476c695b-8sl6l"
Nov 02 12:36:22.861427 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:22.861396    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/75a5e1c8-189e-4005-a8b7-48b007680900-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-5514e5-75476c695b-8sl6l\" (UID: \"75a5e1c8-189e-4005-a8b7-48b007680900\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-5514e5-75476c695b-8sl6l"
Nov 02 12:36:22.862965 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:22.862941    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/75a5e1c8-189e-4005-a8b7-48b007680900-maas-api-tls\") pod \"maas-api-e2e-worker-w0-5514e5-75476c695b-8sl6l\" (UID: \"75a5e1c8-189e-4005-a8b7-48b007680900\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-5514e5-75476c695b-8sl6l"
Nov 02 12:36:22.863065 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:22.863046    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75a5e1c8-189e-4005-a8b7-48b007680900-metrics-certs\") pod \"maas-api-e2e-worker-w0-5514e5-75476c695b-8sl6l\" (UID: \"75a5e1c8-189e-4005-a8b7-48b007680900\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-5514e5-75476c695b-8sl6l"
Nov 02 12:36:22.884573 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:22.884514    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdfwj\" (UniqueName: \"kubernetes.io/projected/75a5e1c8-189e-4005-a8b7-48b007680900-kube-api-access-xdfwj\") pod \"maas-api-e2e-worker-w0-5514e5-75476c695b-8sl6l\" (UID: \"75a5e1c8-189e-4005-a8b7-48b007680900\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-5514e5-75476c695b-8sl6l"
Nov 02 12:36:22.937464 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:22.937431    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-5514e5-75476c695b-8sl6l"
Nov 02 12:36:23.243230 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:23.243199    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-5514e5-75476c695b-8sl6l"]
Nov 02 12:36:23.244227 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:36:23.244203    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75a5e1c8_189e_4005_a8b7_48b007680900.slice/crio-8a6c97821fa2b0ee40f427ea7d4e92022921003fc8fc379c8b3a6d98efae41ca WatchSource:0}: Error finding container 8a6c97821fa2b0ee40f427ea7d4e92022921003fc8fc379c8b3a6d98efae41ca: Status 404 returned error can't find the container with id 8a6c97821fa2b0ee40f427ea7d4e92022921003fc8fc379c8b3a6d98efae41ca
Nov 02 12:36:23.598125 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:23.598043    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-5514e5-75476c695b-8sl6l" event={"ID":"75a5e1c8-189e-4005-a8b7-48b007680900","Type":"ContainerStarted","Data":"8a6c97821fa2b0ee40f427ea7d4e92022921003fc8fc379c8b3a6d98efae41ca"}
Nov 02 12:36:24.604694 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:24.604628    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-5514e5-75476c695b-8sl6l" event={"ID":"75a5e1c8-189e-4005-a8b7-48b007680900","Type":"ContainerStarted","Data":"2b8a153b09aec9dff5645bd1ab9491e455bf99a4ca0d8c0d25cad76eb46a7332"}
Nov 02 12:36:24.605247 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:24.604723    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-5514e5-75476c695b-8sl6l"
Nov 02 12:36:24.606308 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:24.606285    4559 generic.go:356] "Generic (PLEG): container finished" podID="76dc13d1-a3a7-4f47-85a2-f596cafbcb94" containerID="2f30022edc685d1ad8c8dc1e219cd3e77b94ca787c2b9e95f24e108586b01ed7" exitCode=0
Nov 02 12:36:24.606422 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:24.606356    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-bh68g" event={"ID":"76dc13d1-a3a7-4f47-85a2-f596cafbcb94","Type":"ContainerDied","Data":"2f30022edc685d1ad8c8dc1e219cd3e77b94ca787c2b9e95f24e108586b01ed7"}
Nov 02 12:36:24.606725 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:24.606709    4559 scope.go:122] "RemoveContainer" containerID="2f30022edc685d1ad8c8dc1e219cd3e77b94ca787c2b9e95f24e108586b01ed7"
Nov 02 12:36:24.607946 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:24.607914    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:36:24.608081 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:24.608001    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:36:24.608081 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:24.608054    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:36:24.650009 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:24.649931    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-5514e5-75476c695b-8sl6l" podStartSLOduration=2.293278037 podStartE2EDuration="2.649911636s" totalImagesPullingTime="356.633599ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:36:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:36:24.647748221 +0000 UTC m=+580.903147317" watchObservedRunningTime="2026-11-02 12:36:24.649911636 +0000 UTC m=+580.905310733"
Nov 02 12:36:25.626782 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:25.626740    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-bh68g" event={"ID":"76dc13d1-a3a7-4f47-85a2-f596cafbcb94","Type":"ContainerStarted","Data":"50bad0ef48c7d48a03435a979ec8c43e27270fbdc3a8ecf343ef12ba2da24774"}
Nov 02 12:36:26.232922 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:26.232879    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-bh68g"
Nov 02 12:36:26.233122 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:26.233065    4559 patch_prober.go:29] interesting pod/e2e-worker-w2-95ee26-openshift-default-66679d8f57-bh68g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Nov 02 12:36:26.233200 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:26.233131    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-bh68g" podUID="76dc13d1-a3a7-4f47-85a2-f596cafbcb94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Nov 02 12:36:27.233251 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:27.233219    4559 patch_prober.go:29] interesting pod/e2e-worker-w2-95ee26-openshift-default-66679d8f57-bh68g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Nov 02 12:36:27.233707 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:27.233275    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-bh68g" podUID="76dc13d1-a3a7-4f47-85a2-f596cafbcb94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Nov 02 12:36:28.233507 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:28.233462    4559 patch_prober.go:29] interesting pod/e2e-worker-w2-95ee26-openshift-default-66679d8f57-bh68g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Nov 02 12:36:28.233970 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:28.233528    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-bh68g" podUID="76dc13d1-a3a7-4f47-85a2-f596cafbcb94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Nov 02 12:36:28.707270 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:28.707227    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"]
Nov 02 12:36:29.233664 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:29.233548    4559 patch_prober.go:29] interesting pod/e2e-worker-w2-95ee26-openshift-default-66679d8f57-bh68g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Nov 02 12:36:29.233664 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:29.233621    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-bh68g" podUID="76dc13d1-a3a7-4f47-85a2-f596cafbcb94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Nov 02 12:36:29.574613 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:29.574538    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-5514e5-b884d6445-hpkbk"
Nov 02 12:36:29.574613 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:29.574610    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-5514e5-cc9847d79-8jprz"
Nov 02 12:36:30.233555 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:30.233518    4559 patch_prober.go:29] interesting pod/e2e-worker-w2-95ee26-openshift-default-66679d8f57-bh68g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Nov 02 12:36:30.233758 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:30.233587    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-bh68g" podUID="76dc13d1-a3a7-4f47-85a2-f596cafbcb94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Nov 02 12:36:30.633792 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:30.633716    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-5514e5-75476c695b-8sl6l"
Nov 02 12:36:31.237223 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:31.237193    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-bh68g"
Nov 02 12:36:31.237722 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:31.237605    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-bh68g"
Nov 02 12:36:31.238315 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:31.238295    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-bh68g"
Nov 02 12:36:37.538268 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:37.538232    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"]
Nov 02 12:36:45.085730 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:36:45.085688    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqcmj" containerName="extractor"
Nov 02 12:36:46.997866 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:46.997828    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"]
Nov 02 12:36:47.793027 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:47.792998    4559 generic.go:356] "Generic (PLEG): container finished" podID="91b0660b-bd2c-4f62-ac8a-e09af747a766" containerID="f263118547ccecb716514445c970370884eac55bf80594ea71b34d72e20be424" exitCode=0
Nov 02 12:36:47.793319 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:47.793041    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-ffpz2" event={"ID":"91b0660b-bd2c-4f62-ac8a-e09af747a766","Type":"ContainerDied","Data":"f263118547ccecb716514445c970370884eac55bf80594ea71b34d72e20be424"}
Nov 02 12:36:47.793486 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:47.793467    4559 scope.go:122] "RemoveContainer" containerID="f263118547ccecb716514445c970370884eac55bf80594ea71b34d72e20be424"
Nov 02 12:36:47.795187 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:47.795152    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:36:47.795281 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:47.795244    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:36:47.795339 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:47.795323    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:36:48.816432 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:48.816399    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-ffpz2" event={"ID":"91b0660b-bd2c-4f62-ac8a-e09af747a766","Type":"ContainerStarted","Data":"626d04a633ea192250c35ed7413e8b11f2df3f058ee9954bdd020c92e11fc5d2"}
Nov 02 12:36:49.169207 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:49.169176    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-ffpz2"
Nov 02 12:36:49.170351 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:49.170328    4559 patch_prober.go:29] interesting pod/e2e-worker-w1-61e341-openshift-default-59d97476dc-ffpz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Nov 02 12:36:49.170442 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:49.170378    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-ffpz2" podUID="91b0660b-bd2c-4f62-ac8a-e09af747a766" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Nov 02 12:36:50.168539 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:50.168503    4559 patch_prober.go:29] interesting pod/e2e-worker-w1-61e341-openshift-default-59d97476dc-ffpz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Nov 02 12:36:50.168981 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:50.168565    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-ffpz2" podUID="91b0660b-bd2c-4f62-ac8a-e09af747a766" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Nov 02 12:36:51.169019 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:51.168982    4559 patch_prober.go:29] interesting pod/e2e-worker-w1-61e341-openshift-default-59d97476dc-ffpz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Nov 02 12:36:51.169419 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:51.169037    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-ffpz2" podUID="91b0660b-bd2c-4f62-ac8a-e09af747a766" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Nov 02 12:36:51.596721 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:51.591731    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-ffpz2"
Nov 02 12:36:52.168754 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:52.168718    4559 patch_prober.go:29] interesting pod/e2e-worker-w1-61e341-openshift-default-59d97476dc-ffpz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Nov 02 12:36:52.168925 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:52.168777    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-ffpz2" podUID="91b0660b-bd2c-4f62-ac8a-e09af747a766" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Nov 02 12:36:53.169416 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:53.169380    4559 patch_prober.go:29] interesting pod/e2e-worker-w1-61e341-openshift-default-59d97476dc-ffpz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Nov 02 12:36:53.169840 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:53.169440    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-ffpz2" podUID="91b0660b-bd2c-4f62-ac8a-e09af747a766" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Nov 02 12:36:54.173175 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:54.173147    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-ffpz2"
Nov 02 12:36:54.174129 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:54.174108    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-ffpz2"
Nov 02 12:36:57.133119 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:36:57.133082    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"]
Nov 02 12:37:01.206034 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:37:01.206003    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-qk2dp"]
Nov 02 12:37:01.209713 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:37:01.209692    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-qk2dp"
Nov 02 12:37:01.227916 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:37:01.227894    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-843978-openshift-default-dockercfg-nkwwj\""
Nov 02 12:37:01.243152 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:37:01.243130    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-qk2dp"]
Nov 02 12:37:01.304894 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:37:01.304858    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rhm8\" (UniqueName: \"kubernetes.io/projected/ee4f0fd1-9b89-4c29-ba5f-3cf47a18d2be-kube-api-access-6rhm8\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-qk2dp\" (UID: \"ee4f0fd1-9b89-4c29-ba5f-3cf47a18d2be\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-qk2dp"
Nov 02 12:37:01.304894 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:37:01.304894    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee4f0fd1-9b89-4c29-ba5f-3cf47a18d2be-istiod-ca-cert\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-qk2dp\" (UID: \"ee4f0fd1-9b89-4c29-ba5f-3cf47a18d2be\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-qk2dp"
Nov 02 12:37:01.305095 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:37:01.304951    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee4f0fd1-9b89-4c29-ba5f-3cf47a18d2be-istio-podinfo\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-qk2dp\" (UID: \"ee4f0fd1-9b89-4c29-ba5f-3cf47a18d2be\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-qk2dp"
Nov 02 12:37:01.305095 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:37:01.304998    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee4f0fd1-9b89-4c29-ba5f-3cf47a18d2be-workload-socket\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-qk2dp\" (UID: \"ee4f0fd1-9b89-4c29-ba5f-3cf47a18d2be\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-qk2dp"
Nov 02 12:37:01.305161 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:37:01.305105    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee4f0fd1-9b89-4c29-ba5f-3cf47a18d2be-istio-token\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-qk2dp\" (UID: \"ee4f0fd1-9b89-4c29-ba5f-3cf47a18d2be\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-qk2dp"
Nov 02 12:37:01.305161 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:37:01.305132    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee4f0fd1-9b89-4c29-ba5f-3cf47a18d2be-workload-certs\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-qk2dp\" (UID: \"ee4f0fd1-9b89-4c29-ba5f-3cf47a18d2be\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-qk2dp"
Nov 02 12:37:01.305161 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:37:01.305155    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee4f0fd1-9b89-4c29-ba5f-3cf47a18d2be-istio-data\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-qk2dp\" (UID: \"ee4f0fd1-9b89-4c29-ba5f-3cf47a18d2be\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-qk2dp"
Nov 02 12:37:01.305254 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:37:01.305194    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee4f0fd1-9b89-4c29-ba5f-3cf47a18d2be-istio-envoy\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-qk2dp\" (UID: \"ee4f0fd1-9b89-4c29-ba5f-3cf47a18d2be\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-qk2dp"
Nov 02 12:37:01.305288 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:37:01.305272    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee4f0fd1-9b89-4c29-ba5f-3cf47a18d2be-credential-socket\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-qk2dp\" (UID: \"ee4f0fd1-9b89-4c29-ba5f-3cf47a18d2be\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-qk2dp"
Nov 02 12:37:01.406663 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:37:01.406605    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee4f0fd1-9b89-4c29-ba5f-3cf47a18d2be-workload-socket\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-qk2dp\" (UID: \"ee4f0fd1-9b89-4c29-ba5f-3cf47a18d2be\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-qk2dp"
Nov 02 12:37:01.406840 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:37:01.406687    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee4f0fd1-9b89-4c29-ba5f-3cf47a18d2be-istio-token\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-qk2dp\" (UID: \"ee4f0fd1-9b89-4c29-ba5f-3cf47a18d2be\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-qk2dp"
Nov 02 12:37:01.406840 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:37:01.406714    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee4f0fd1-9b89-4c29-ba5f-3cf47a18d2be-workload-certs\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-qk2dp\" (UID: \"ee4f0fd1-9b89-4c29-ba5f-3cf47a18d2be\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-qk2dp"
Nov 02 12:37:01.406840 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:37:01.406735    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee4f0fd1-9b89-4c29-ba5f-3cf47a18d2be-istio-data\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-qk2dp\" (UID: \"ee4f0fd1-9b89-4c29-ba5f-3cf47a18d2be\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-qk2dp"
Nov 02 12:37:01.406840 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:37:01.406751    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee4f0fd1-9b89-4c29-ba5f-3cf47a18d2be-istio-envoy\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-qk2dp\" (UID: \"ee4f0fd1-9b89-4c29-ba5f-3cf47a18d2be\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-qk2dp"
Nov 02 12:37:01.406840 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:37:01.406792    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee4f0fd1-9b89-4c29-ba5f-3cf47a18d2be-credential-socket\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-qk2dp\" (UID: \"ee4f0fd1-9b89-4c29-ba5f-3cf47a18d2be\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-qk2dp"
Nov 02 12:37:01.407067 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:37:01.406870    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rhm8\" (UniqueName: \"kubernetes.io/projected/ee4f0fd1-9b89-4c29-ba5f-3cf47a18d2be-kube-api-access-6rhm8\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-qk2dp\" (UID: \"ee4f0fd1-9b89-4c29-ba5f-3cf47a18d2be\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-qk2dp"
Nov 02 12:37:01.407067 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:37:01.406888    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee4f0fd1-9b89-4c29-ba5f-3cf47a18d2be-istiod-ca-cert\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-qk2dp\" (UID: \"ee4f0fd1-9b89-4c29-ba5f-3cf47a18d2be\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-qk2dp"
Nov 02 12:37:01.407067 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:37:01.406929    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee4f0fd1-9b89-4c29-ba5f-3cf47a18d2be-istio-podinfo\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-qk2dp\" (UID: \"ee4f0fd1-9b89-4c29-ba5f-3cf47a18d2be\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-qk2dp"
Nov 02 12:37:01.407223 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:37:01.407089    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee4f0fd1-9b89-4c29-ba5f-3cf47a18d2be-workload-socket\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-qk2dp\" (UID: \"ee4f0fd1-9b89-4c29-ba5f-3cf47a18d2be\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-qk2dp"
Nov 02 12:37:01.407223 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:37:01.407175    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee4f0fd1-9b89-4c29-ba5f-3cf47a18d2be-workload-certs\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-qk2dp\" (UID: \"ee4f0fd1-9b89-4c29-ba5f-3cf47a18d2be\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-qk2dp"
Nov 02 12:37:01.407340 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:37:01.407321    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee4f0fd1-9b89-4c29-ba5f-3cf47a18d2be-istio-data\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-qk2dp\" (UID: \"ee4f0fd1-9b89-4c29-ba5f-3cf47a18d2be\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-qk2dp"
Nov 02 12:37:01.407381 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:37:01.407318    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee4f0fd1-9b89-4c29-ba5f-3cf47a18d2be-credential-socket\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-qk2dp\" (UID: \"ee4f0fd1-9b89-4c29-ba5f-3cf47a18d2be\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-qk2dp"
Nov 02 12:37:01.407692 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:37:01.407672    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee4f0fd1-9b89-4c29-ba5f-3cf47a18d2be-istiod-ca-cert\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-qk2dp\" (UID: \"ee4f0fd1-9b89-4c29-ba5f-3cf47a18d2be\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-qk2dp"
Nov 02 12:37:01.409000 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:37:01.408973    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee4f0fd1-9b89-4c29-ba5f-3cf47a18d2be-istio-envoy\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-qk2dp\" (UID: \"ee4f0fd1-9b89-4c29-ba5f-3cf47a18d2be\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-qk2dp"
Nov 02 12:37:01.409284 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:37:01.409267    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee4f0fd1-9b89-4c29-ba5f-3cf47a18d2be-istio-podinfo\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-qk2dp\" (UID: \"ee4f0fd1-9b89-4c29-ba5f-3cf47a18d2be\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-qk2dp"
Nov 02 12:37:01.430739 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:37:01.430714    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee4f0fd1-9b89-4c29-ba5f-3cf47a18d2be-istio-token\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-qk2dp\" (UID: \"ee4f0fd1-9b89-4c29-ba5f-3cf47a18d2be\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-qk2dp"
Nov 02 12:37:01.430846 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:37:01.430794    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rhm8\" (UniqueName: \"kubernetes.io/projected/ee4f0fd1-9b89-4c29-ba5f-3cf47a18d2be-kube-api-access-6rhm8\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-qk2dp\" (UID: \"ee4f0fd1-9b89-4c29-ba5f-3cf47a18d2be\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-qk2dp"
Nov 02 12:37:01.522596 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:37:01.522515    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-qk2dp"
Nov 02 12:37:01.836534 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:37:01.836502    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-qk2dp"]
Nov 02 12:37:01.837206 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:37:01.837173    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee4f0fd1_9b89_4c29_ba5f_3cf47a18d2be.slice/crio-31a33445ab51a02222149901ec417661ca4c8e3d8c0a1317d41564c876dd6853 WatchSource:0}: Error finding container 31a33445ab51a02222149901ec417661ca4c8e3d8c0a1317d41564c876dd6853: Status 404 returned error can't find the container with id 31a33445ab51a02222149901ec417661ca4c8e3d8c0a1317d41564c876dd6853
Nov 02 12:37:01.840144 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:37:01.840104    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:37:01.840240 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:37:01.840192    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:37:01.840240 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:37:01.840234    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:37:01.893425 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:37:01.893397    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-qk2dp" event={"ID":"ee4f0fd1-9b89-4c29-ba5f-3cf47a18d2be","Type":"ContainerStarted","Data":"31a33445ab51a02222149901ec417661ca4c8e3d8c0a1317d41564c876dd6853"}
Nov 02 12:37:02.910306 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:37:02.910265    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-qk2dp" event={"ID":"ee4f0fd1-9b89-4c29-ba5f-3cf47a18d2be","Type":"ContainerStarted","Data":"a62e6950cf74c7cf03426b860986d491f12b196efe29b3024eb0966bc2e7d227"}
Nov 02 12:37:02.969835 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:37:02.969784    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-qk2dp" podStartSLOduration=1.969770428 podStartE2EDuration="1.969770428s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:37:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:37:02.968533126 +0000 UTC m=+619.223932233" watchObservedRunningTime="2026-11-02 12:37:02.969770428 +0000 UTC m=+619.225169523"
Nov 02 12:37:03.523476 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:37:03.523443    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-qk2dp"
Nov 02 12:37:03.525213 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:37:03.525186    4559 patch_prober.go:29] interesting pod/e2e-worker-w4-843978-openshift-default-7965f48c7f-qk2dp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Nov 02 12:37:03.525350 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:37:03.525234    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-qk2dp" podUID="ee4f0fd1-9b89-4c29-ba5f-3cf47a18d2be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Nov 02 12:37:04.523113 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:37:04.523075    4559 patch_prober.go:29] interesting pod/e2e-worker-w4-843978-openshift-default-7965f48c7f-qk2dp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Nov 02 12:37:04.523550 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:37:04.523161    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-qk2dp" podUID="ee4f0fd1-9b89-4c29-ba5f-3cf47a18d2be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Nov 02 12:37:05.523917 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:37:05.523879    4559 patch_prober.go:29] interesting pod/e2e-worker-w4-843978-openshift-default-7965f48c7f-qk2dp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Nov 02 12:37:05.524401 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:37:05.523940    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-qk2dp" podUID="ee4f0fd1-9b89-4c29-ba5f-3cf47a18d2be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Nov 02 12:37:06.303662 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:37:06.303189    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"]
Nov 02 12:37:06.523564 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:37:06.523509    4559 patch_prober.go:29] interesting pod/e2e-worker-w4-843978-openshift-default-7965f48c7f-qk2dp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Nov 02 12:37:06.523789 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:37:06.523598    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-qk2dp" podUID="ee4f0fd1-9b89-4c29-ba5f-3cf47a18d2be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Nov 02 12:37:07.523144 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:37:07.523107    4559 patch_prober.go:29] interesting pod/e2e-worker-w4-843978-openshift-default-7965f48c7f-qk2dp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Nov 02 12:37:07.523600 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:37:07.523287    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-qk2dp" podUID="ee4f0fd1-9b89-4c29-ba5f-3cf47a18d2be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Nov 02 12:37:08.527114 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:37:08.527085    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-qk2dp"
Nov 02 12:37:08.527536 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:37:08.527342    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-qk2dp"
Nov 02 12:37:08.528125 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:37:08.528109    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-qk2dp"
Nov 02 12:37:15.085535 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:37:15.085489    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqcmj" containerName="extractor"
Nov 02 12:37:21.490654 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:37:21.490606    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"]
Nov 02 12:37:25.006091 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:37:25.006054    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-61e341/premium-w1-61e341-kserve-746477b99-645g6"]
Nov 02 12:37:25.012037 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:37:25.012016    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-61e341/premium-w1-61e341-kserve-746477b99-645g6"
Nov 02 12:37:25.030273 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:37:25.030249    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-61e341/premium-w1-61e341-kserve-746477b99-645g6"]
Nov 02 12:37:25.040905 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:37:25.040860    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-61e341\"/\"kube-root-ca.crt\""
Nov 02 12:37:25.041039 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:37:25.040918    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-61e341\"/\"openshift-service-ca.crt\""
Nov 02 12:37:25.042986 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:37:25.042966    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-61e341\"/\"default-dockercfg-bhwfk\""
Nov 02 12:37:25.043100 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:37:25.043039    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-61e341\"/\"premium-w1-61e341-kserve-self-signed-certs\""
Nov 02 12:37:25.083539 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:37:25.083507    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/02ff21d3-5a37-493a-ab95-8e4784c29123-home\") pod \"premium-w1-61e341-kserve-746477b99-645g6\" (UID: \"02ff21d3-5a37-493a-ab95-8e4784c29123\") " pod="e2e-models-e2e-worker-w1-61e341/premium-w1-61e341-kserve-746477b99-645g6"
Nov 02 12:37:25.083732 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:37:25.083550    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/02ff21d3-5a37-493a-ab95-8e4784c29123-model-cache\") pod \"premium-w1-61e341-kserve-746477b99-645g6\" (UID: \"02ff21d3-5a37-493a-ab95-8e4784c29123\") " pod="e2e-models-e2e-worker-w1-61e341/premium-w1-61e341-kserve-746477b99-645g6"
Nov 02 12:37:25.083732 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:37:25.083612    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02ff21d3-5a37-493a-ab95-8e4784c29123-tmp-dir\") pod \"premium-w1-61e341-kserve-746477b99-645g6\" (UID: \"02ff21d3-5a37-493a-ab95-8e4784c29123\") " pod="e2e-models-e2e-worker-w1-61e341/premium-w1-61e341-kserve-746477b99-645g6"
Nov 02 12:37:25.083732 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:37:25.083633    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/02ff21d3-5a37-493a-ab95-8e4784c29123-dshm\") pod \"premium-w1-61e341-kserve-746477b99-645g6\" (UID: \"02ff21d3-5a37-493a-ab95-8e4784c29123\") " pod="e2e-models-e2e-worker-w1-61e341/premium-w1-61e341-kserve-746477b99-645g6"
Nov 02 12:37:25.083861 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:37:25.083730    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02ff21d3-5a37-493a-ab95-8e4784c29123-tls-certs\") pod \"premium-w1-61e341-kserve-746477b99-645g6\" (UID: \"02ff21d3-5a37-493a-ab95-8e4784c29123\") " pod="e2e-models-e2e-worker-w1-61e341/premium-w1-61e341-kserve-746477b99-645g6"
Nov 02 12:37:25.083861 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:37:25.083796    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bn9pn\" (UniqueName: \"kubernetes.io/projected/02ff21d3-5a37-493a-ab95-8e4784c29123-kube-api-access-bn9pn\") pod \"premium-w1-61e341-kserve-746477b99-645g6\" (UID: \"02ff21d3-5a37-493a-ab95-8e4784c29123\") " pod="e2e-models-e2e-worker-w1-61e341/premium-w1-61e341-kserve-746477b99-645g6"
Nov 02 12:37:25.186382 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:37:25.186325    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/02ff21d3-5a37-493a-ab95-8e4784c29123-home\") pod \"premium-w1-61e341-kserve-746477b99-645g6\" (UID: \"02ff21d3-5a37-493a-ab95-8e4784c29123\") " pod="e2e-models-e2e-worker-w1-61e341/premium-w1-61e341-kserve-746477b99-645g6"
Nov 02 12:37:25.186904 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:37:25.186854    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/02ff21d3-5a37-493a-ab95-8e4784c29123-model-cache\") pod \"premium-w1-61e341-kserve-746477b99-645g6\" (UID: \"02ff21d3-5a37-493a-ab95-8e4784c29123\") " pod="e2e-models-e2e-worker-w1-61e341/premium-w1-61e341-kserve-746477b99-645g6"
Nov 02 12:37:25.188471 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:37:25.188413    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02ff21d3-5a37-493a-ab95-8e4784c29123-tmp-dir\") pod \"premium-w1-61e341-kserve-746477b99-645g6\" (UID: \"02ff21d3-5a37-493a-ab95-8e4784c29123\") " pod="e2e-models-e2e-worker-w1-61e341/premium-w1-61e341-kserve-746477b99-645g6"
Nov 02 12:37:25.188854 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:37:25.188768    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/02ff21d3-5a37-493a-ab95-8e4784c29123-dshm\") pod \"premium-w1-61e341-kserve-746477b99-645g6\" (UID: \"02ff21d3-5a37-493a-ab95-8e4784c29123\") " pod="e2e-models-e2e-worker-w1-61e341/premium-w1-61e341-kserve-746477b99-645g6"
Nov 02 12:37:25.189038 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:37:25.188859    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02ff21d3-5a37-493a-ab95-8e4784c29123-tls-certs\") pod \"premium-w1-61e341-kserve-746477b99-645g6\" (UID: \"02ff21d3-5a37-493a-ab95-8e4784c29123\") " pod="e2e-models-e2e-worker-w1-61e341/premium-w1-61e341-kserve-746477b99-645g6"
Nov 02 12:37:25.189038 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:37:25.188889    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bn9pn\" (UniqueName: \"kubernetes.io/projected/02ff21d3-5a37-493a-ab95-8e4784c29123-kube-api-access-bn9pn\") pod \"premium-w1-61e341-kserve-746477b99-645g6\" (UID: \"02ff21d3-5a37-493a-ab95-8e4784c29123\") " pod="e2e-models-e2e-worker-w1-61e341/premium-w1-61e341-kserve-746477b99-645g6"
Nov 02 12:37:25.189284 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:37:25.189118    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/02ff21d3-5a37-493a-ab95-8e4784c29123-model-cache\") pod \"premium-w1-61e341-kserve-746477b99-645g6\" (UID: \"02ff21d3-5a37-493a-ab95-8e4784c29123\") " pod="e2e-models-e2e-worker-w1-61e341/premium-w1-61e341-kserve-746477b99-645g6"
Nov 02 12:37:25.194550 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:37:25.194509    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/02ff21d3-5a37-493a-ab95-8e4784c29123-home\") pod \"premium-w1-61e341-kserve-746477b99-645g6\" (UID: \"02ff21d3-5a37-493a-ab95-8e4784c29123\") " pod="e2e-models-e2e-worker-w1-61e341/premium-w1-61e341-kserve-746477b99-645g6"
Nov 02 12:37:25.194717 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:37:25.194624    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02ff21d3-5a37-493a-ab95-8e4784c29123-tmp-dir\") pod \"premium-w1-61e341-kserve-746477b99-645g6\" (UID: \"02ff21d3-5a37-493a-ab95-8e4784c29123\") " pod="e2e-models-e2e-worker-w1-61e341/premium-w1-61e341-kserve-746477b99-645g6"
Nov 02 12:37:25.197578 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:37:25.197532    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02ff21d3-5a37-493a-ab95-8e4784c29123-tls-certs\") pod \"premium-w1-61e341-kserve-746477b99-645g6\" (UID: \"02ff21d3-5a37-493a-ab95-8e4784c29123\") " pod="e2e-models-e2e-worker-w1-61e341/premium-w1-61e341-kserve-746477b99-645g6"
Nov 02 12:37:25.207165 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:37:25.207138    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/02ff21d3-5a37-493a-ab95-8e4784c29123-dshm\") pod \"premium-w1-61e341-kserve-746477b99-645g6\" (UID: \"02ff21d3-5a37-493a-ab95-8e4784c29123\") " pod="e2e-models-e2e-worker-w1-61e341/premium-w1-61e341-kserve-746477b99-645g6"
Nov 02 12:37:25.234362 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:37:25.234326    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bn9pn\" (UniqueName: \"kubernetes.io/projected/02ff21d3-5a37-493a-ab95-8e4784c29123-kube-api-access-bn9pn\") pod \"premium-w1-61e341-kserve-746477b99-645g6\" (UID: \"02ff21d3-5a37-493a-ab95-8e4784c29123\") " pod="e2e-models-e2e-worker-w1-61e341/premium-w1-61e341-kserve-746477b99-645g6"
Nov 02 12:37:25.324966 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:37:25.324886    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-61e341/premium-w1-61e341-kserve-746477b99-645g6"
Nov 02 12:37:25.626888 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:37:25.626857    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-61e341/premium-w1-61e341-kserve-746477b99-645g6"]
Nov 02 12:37:25.627503 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:37:25.627479    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod02ff21d3_5a37_493a_ab95_8e4784c29123.slice/crio-cb2d84e264030d9f5304e00fe71820d17cafbedf5c006ae4feb22391625a5f59 WatchSource:0}: Error finding container cb2d84e264030d9f5304e00fe71820d17cafbedf5c006ae4feb22391625a5f59: Status 404 returned error can't find the container with id cb2d84e264030d9f5304e00fe71820d17cafbedf5c006ae4feb22391625a5f59
Nov 02 12:37:26.063271 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:37:26.063238    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-61e341/premium-w1-61e341-kserve-746477b99-645g6" event={"ID":"02ff21d3-5a37-493a-ab95-8e4784c29123","Type":"ContainerStarted","Data":"39de71ad3ca784367dec8aca9a9aa93ad9d1d9d85bc63dbe5da57936c20ba0af"}
Nov 02 12:37:26.063271 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:37:26.063277    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-61e341/premium-w1-61e341-kserve-746477b99-645g6" event={"ID":"02ff21d3-5a37-493a-ab95-8e4784c29123","Type":"ContainerStarted","Data":"cb2d84e264030d9f5304e00fe71820d17cafbedf5c006ae4feb22391625a5f59"}
Nov 02 12:37:26.105385 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:37:26.105336    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-61e341/premium-w1-61e341-kserve-746477b99-645g6" podStartSLOduration=2.105318145 podStartE2EDuration="2.105318145s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:37:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:37:26.10498709 +0000 UTC m=+642.360386185" watchObservedRunningTime="2026-11-02 12:37:26.105318145 +0000 UTC m=+642.360717230"
Nov 02 12:37:26.325847 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:37:26.325766    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-61e341/premium-w1-61e341-kserve-746477b99-645g6"
Nov 02 12:37:35.325948 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:37:35.325915    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-61e341/premium-w1-61e341-kserve-746477b99-645g6"
Nov 02 12:37:35.338563 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:37:35.338535    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-61e341/premium-w1-61e341-kserve-746477b99-645g6"
Nov 02 12:37:36.150397 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:37:36.150368    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-61e341/premium-w1-61e341-kserve-746477b99-645g6"
Nov 02 12:37:39.311473 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:37:39.311438    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"]
Nov 02 12:37:42.358213 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:37:42.358179    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-dvzvq"]
Nov 02 12:37:42.362055 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:37:42.362034    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-dvzvq"
Nov 02 12:37:42.380123 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:37:42.380103    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-068756-openshift-default-dockercfg-kp7kc\""
Nov 02 12:37:42.396198 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:37:42.396163    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-dvzvq"]
Nov 02 12:37:42.489659 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:37:42.489602    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79ae08e3-f7c3-4e28-aabb-246f42f45b2e-istio-podinfo\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-dvzvq\" (UID: \"79ae08e3-f7c3-4e28-aabb-246f42f45b2e\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-dvzvq"
Nov 02 12:37:42.489832 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:37:42.489679    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79ae08e3-f7c3-4e28-aabb-246f42f45b2e-workload-socket\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-dvzvq\" (UID: \"79ae08e3-f7c3-4e28-aabb-246f42f45b2e\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-dvzvq"
Nov 02 12:37:42.489832 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:37:42.489760    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79ae08e3-f7c3-4e28-aabb-246f42f45b2e-istio-data\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-dvzvq\" (UID: \"79ae08e3-f7c3-4e28-aabb-246f42f45b2e\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-dvzvq"
Nov 02 12:37:42.489832 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:37:42.489797    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79ae08e3-f7c3-4e28-aabb-246f42f45b2e-istiod-ca-cert\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-dvzvq\" (UID: \"79ae08e3-f7c3-4e28-aabb-246f42f45b2e\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-dvzvq"
Nov 02 12:37:42.489960 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:37:42.489888    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79ae08e3-f7c3-4e28-aabb-246f42f45b2e-istio-token\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-dvzvq\" (UID: \"79ae08e3-f7c3-4e28-aabb-246f42f45b2e\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-dvzvq"
Nov 02 12:37:42.489960 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:37:42.489920    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6qw8\" (UniqueName: \"kubernetes.io/projected/79ae08e3-f7c3-4e28-aabb-246f42f45b2e-kube-api-access-l6qw8\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-dvzvq\" (UID: \"79ae08e3-f7c3-4e28-aabb-246f42f45b2e\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-dvzvq"
Nov 02 12:37:42.490030 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:37:42.490002    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79ae08e3-f7c3-4e28-aabb-246f42f45b2e-credential-socket\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-dvzvq\" (UID: \"79ae08e3-f7c3-4e28-aabb-246f42f45b2e\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-dvzvq"
Nov 02 12:37:42.490064 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:37:42.490038    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79ae08e3-f7c3-4e28-aabb-246f42f45b2e-workload-certs\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-dvzvq\" (UID: \"79ae08e3-f7c3-4e28-aabb-246f42f45b2e\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-dvzvq"
Nov 02 12:37:42.490097 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:37:42.490074    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79ae08e3-f7c3-4e28-aabb-246f42f45b2e-istio-envoy\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-dvzvq\" (UID: \"79ae08e3-f7c3-4e28-aabb-246f42f45b2e\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-dvzvq"
Nov 02 12:37:42.590926 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:37:42.590894    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79ae08e3-f7c3-4e28-aabb-246f42f45b2e-istio-token\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-dvzvq\" (UID: \"79ae08e3-f7c3-4e28-aabb-246f42f45b2e\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-dvzvq"
Nov 02 12:37:42.590926 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:37:42.590930    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6qw8\" (UniqueName: \"kubernetes.io/projected/79ae08e3-f7c3-4e28-aabb-246f42f45b2e-kube-api-access-l6qw8\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-dvzvq\" (UID: \"79ae08e3-f7c3-4e28-aabb-246f42f45b2e\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-dvzvq"
Nov 02 12:37:42.591136 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:37:42.591063    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79ae08e3-f7c3-4e28-aabb-246f42f45b2e-credential-socket\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-dvzvq\" (UID: \"79ae08e3-f7c3-4e28-aabb-246f42f45b2e\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-dvzvq"
Nov 02 12:37:42.591174 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:37:42.591140    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79ae08e3-f7c3-4e28-aabb-246f42f45b2e-workload-certs\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-dvzvq\" (UID: \"79ae08e3-f7c3-4e28-aabb-246f42f45b2e\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-dvzvq"
Nov 02 12:37:42.591236 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:37:42.591222    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79ae08e3-f7c3-4e28-aabb-246f42f45b2e-istio-envoy\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-dvzvq\" (UID: \"79ae08e3-f7c3-4e28-aabb-246f42f45b2e\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-dvzvq"
Nov 02 12:37:42.591295 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:37:42.591281    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79ae08e3-f7c3-4e28-aabb-246f42f45b2e-istio-podinfo\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-dvzvq\" (UID: \"79ae08e3-f7c3-4e28-aabb-246f42f45b2e\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-dvzvq"
Nov 02 12:37:42.591346 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:37:42.591332    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79ae08e3-f7c3-4e28-aabb-246f42f45b2e-workload-socket\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-dvzvq\" (UID: \"79ae08e3-f7c3-4e28-aabb-246f42f45b2e\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-dvzvq"
Nov 02 12:37:42.591382 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:37:42.591358    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79ae08e3-f7c3-4e28-aabb-246f42f45b2e-istio-data\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-dvzvq\" (UID: \"79ae08e3-f7c3-4e28-aabb-246f42f45b2e\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-dvzvq"
Nov 02 12:37:42.591432 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:37:42.591396    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79ae08e3-f7c3-4e28-aabb-246f42f45b2e-istiod-ca-cert\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-dvzvq\" (UID: \"79ae08e3-f7c3-4e28-aabb-246f42f45b2e\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-dvzvq"
Nov 02 12:37:42.591537 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:37:42.591514    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79ae08e3-f7c3-4e28-aabb-246f42f45b2e-credential-socket\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-dvzvq\" (UID: \"79ae08e3-f7c3-4e28-aabb-246f42f45b2e\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-dvzvq"
Nov 02 12:37:42.591537 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:37:42.591526    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79ae08e3-f7c3-4e28-aabb-246f42f45b2e-workload-certs\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-dvzvq\" (UID: \"79ae08e3-f7c3-4e28-aabb-246f42f45b2e\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-dvzvq"
Nov 02 12:37:42.591750 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:37:42.591729    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79ae08e3-f7c3-4e28-aabb-246f42f45b2e-workload-socket\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-dvzvq\" (UID: \"79ae08e3-f7c3-4e28-aabb-246f42f45b2e\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-dvzvq"
Nov 02 12:37:42.591809 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:37:42.591748    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79ae08e3-f7c3-4e28-aabb-246f42f45b2e-istio-data\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-dvzvq\" (UID: \"79ae08e3-f7c3-4e28-aabb-246f42f45b2e\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-dvzvq"
Nov 02 12:37:42.592125 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:37:42.592105    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79ae08e3-f7c3-4e28-aabb-246f42f45b2e-istiod-ca-cert\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-dvzvq\" (UID: \"79ae08e3-f7c3-4e28-aabb-246f42f45b2e\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-dvzvq"
Nov 02 12:37:42.594037 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:37:42.594014    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79ae08e3-f7c3-4e28-aabb-246f42f45b2e-istio-envoy\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-dvzvq\" (UID: \"79ae08e3-f7c3-4e28-aabb-246f42f45b2e\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-dvzvq"
Nov 02 12:37:42.594231 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:37:42.594214    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79ae08e3-f7c3-4e28-aabb-246f42f45b2e-istio-podinfo\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-dvzvq\" (UID: \"79ae08e3-f7c3-4e28-aabb-246f42f45b2e\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-dvzvq"
Nov 02 12:37:42.614699 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:37:42.614598    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79ae08e3-f7c3-4e28-aabb-246f42f45b2e-istio-token\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-dvzvq\" (UID: \"79ae08e3-f7c3-4e28-aabb-246f42f45b2e\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-dvzvq"
Nov 02 12:37:42.614859 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:37:42.614839    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6qw8\" (UniqueName: \"kubernetes.io/projected/79ae08e3-f7c3-4e28-aabb-246f42f45b2e-kube-api-access-l6qw8\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-dvzvq\" (UID: \"79ae08e3-f7c3-4e28-aabb-246f42f45b2e\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-dvzvq"
Nov 02 12:37:42.674698 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:37:42.674665    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-dvzvq"
Nov 02 12:37:42.988218 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:37:42.988187    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-dvzvq"]
Nov 02 12:37:42.989204 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:37:42.989180    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79ae08e3_f7c3_4e28_aabb_246f42f45b2e.slice/crio-fac466686e80c7bbe068cd9d438fd552fea7a035932cbdcaf24e8a7be8bcef0b WatchSource:0}: Error finding container fac466686e80c7bbe068cd9d438fd552fea7a035932cbdcaf24e8a7be8bcef0b: Status 404 returned error can't find the container with id fac466686e80c7bbe068cd9d438fd552fea7a035932cbdcaf24e8a7be8bcef0b
Nov 02 12:37:42.991582 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:37:42.991550    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:37:42.991659 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:37:42.991615    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:37:42.991703 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:37:42.991662    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:37:43.185338 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:37:43.185296    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-dvzvq" event={"ID":"79ae08e3-f7c3-4e28-aabb-246f42f45b2e","Type":"ContainerStarted","Data":"ef92179081982d7a3d24dff3c68e9558db8a59fdf9424e883f26899570e86ba6"}
Nov 02 12:37:43.185525 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:37:43.185366    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-dvzvq" event={"ID":"79ae08e3-f7c3-4e28-aabb-246f42f45b2e","Type":"ContainerStarted","Data":"fac466686e80c7bbe068cd9d438fd552fea7a035932cbdcaf24e8a7be8bcef0b"}
Nov 02 12:37:43.243844 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:37:43.243733    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-dvzvq" podStartSLOduration=1.24371309 podStartE2EDuration="1.24371309s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:37:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:37:43.240871032 +0000 UTC m=+659.496270127" watchObservedRunningTime="2026-11-02 12:37:43.24371309 +0000 UTC m=+659.499112189"
Nov 02 12:37:43.675774 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:37:43.675742    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-dvzvq"
Nov 02 12:37:44.677167 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:37:44.677129    4559 patch_prober.go:29] interesting pod/e2e-worker-w3-068756-openshift-default-84f69c9cf5-dvzvq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Nov 02 12:37:44.677675 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:37:44.677190    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-dvzvq" podUID="79ae08e3-f7c3-4e28-aabb-246f42f45b2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Nov 02 12:37:45.115943 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:37:45.115849    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqcmj" containerName="extractor"
Nov 02 12:37:45.675467 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:37:45.675413    4559 patch_prober.go:29] interesting pod/e2e-worker-w3-068756-openshift-default-84f69c9cf5-dvzvq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Nov 02 12:37:45.675831 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:37:45.675797    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-dvzvq" podUID="79ae08e3-f7c3-4e28-aabb-246f42f45b2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Nov 02 12:37:46.675226 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:37:46.675192    4559 patch_prober.go:29] interesting pod/e2e-worker-w3-068756-openshift-default-84f69c9cf5-dvzvq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Nov 02 12:37:46.675670 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:37:46.675257    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-dvzvq" podUID="79ae08e3-f7c3-4e28-aabb-246f42f45b2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Nov 02 12:37:47.675846 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:37:47.675812    4559 patch_prober.go:29] interesting pod/e2e-worker-w3-068756-openshift-default-84f69c9cf5-dvzvq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Nov 02 12:37:47.676220 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:37:47.675867    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-dvzvq" podUID="79ae08e3-f7c3-4e28-aabb-246f42f45b2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Nov 02 12:37:48.675540 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:37:48.675505    4559 patch_prober.go:29] interesting pod/e2e-worker-w3-068756-openshift-default-84f69c9cf5-dvzvq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Nov 02 12:37:48.675788 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:37:48.675571    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-dvzvq" podUID="79ae08e3-f7c3-4e28-aabb-246f42f45b2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Nov 02 12:37:49.766922 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:37:49.766887    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-dvzvq"
Nov 02 12:37:49.767422 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:37:49.767396    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-dvzvq"
Nov 02 12:37:49.768566 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:37:49.768546    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-dvzvq"
Nov 02 12:37:58.511888 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:37:58.511806    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"]
Nov 02 12:38:10.440236 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:10.440199    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-7d66d6cd49-xkqsr"]
Nov 02 12:38:10.444136 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:10.444120    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7d66d6cd49-xkqsr"
Nov 02 12:38:10.467790 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:10.467696    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-1944a056069f\""
Nov 02 12:38:10.467930 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:10.467891    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-1ca6e84d056b\""
Nov 02 12:38:10.480836 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:10.480814    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\""
Nov 02 12:38:10.480984 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:10.480847    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\""
Nov 02 12:38:10.480984 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:10.480837    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\""
Nov 02 12:38:10.480984 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:10.480939    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-242ww\""
Nov 02 12:38:10.480984 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:10.480962    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\""
Nov 02 12:38:10.483236 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:10.483216    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-7d66d6cd49-xkqsr"]
Nov 02 12:38:10.542783 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:10.542754    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/87b8ed31-36bf-421c-8ee0-48fbe61a65e9-provider-credentials\") pod \"payload-processing-external-model-7d66d6cd49-xkqsr\" (UID: \"87b8ed31-36bf-421c-8ee0-48fbe61a65e9\") " pod="models-as-a-service/payload-processing-external-model-7d66d6cd49-xkqsr"
Nov 02 12:38:10.542942 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:10.542793    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/87b8ed31-36bf-421c-8ee0-48fbe61a65e9-routing-overlay-volume\") pod \"payload-processing-external-model-7d66d6cd49-xkqsr\" (UID: \"87b8ed31-36bf-421c-8ee0-48fbe61a65e9\") " pod="models-as-a-service/payload-processing-external-model-7d66d6cd49-xkqsr"
Nov 02 12:38:10.542942 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:10.542828    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7p95c\" (UniqueName: \"kubernetes.io/projected/87b8ed31-36bf-421c-8ee0-48fbe61a65e9-kube-api-access-7p95c\") pod \"payload-processing-external-model-7d66d6cd49-xkqsr\" (UID: \"87b8ed31-36bf-421c-8ee0-48fbe61a65e9\") " pod="models-as-a-service/payload-processing-external-model-7d66d6cd49-xkqsr"
Nov 02 12:38:10.542942 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:10.542909    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87b8ed31-36bf-421c-8ee0-48fbe61a65e9-plugins-config-volume\") pod \"payload-processing-external-model-7d66d6cd49-xkqsr\" (UID: \"87b8ed31-36bf-421c-8ee0-48fbe61a65e9\") " pod="models-as-a-service/payload-processing-external-model-7d66d6cd49-xkqsr"
Nov 02 12:38:10.644415 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:10.644378    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/87b8ed31-36bf-421c-8ee0-48fbe61a65e9-provider-credentials\") pod \"payload-processing-external-model-7d66d6cd49-xkqsr\" (UID: \"87b8ed31-36bf-421c-8ee0-48fbe61a65e9\") " pod="models-as-a-service/payload-processing-external-model-7d66d6cd49-xkqsr"
Nov 02 12:38:10.644595 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:10.644437    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/87b8ed31-36bf-421c-8ee0-48fbe61a65e9-routing-overlay-volume\") pod \"payload-processing-external-model-7d66d6cd49-xkqsr\" (UID: \"87b8ed31-36bf-421c-8ee0-48fbe61a65e9\") " pod="models-as-a-service/payload-processing-external-model-7d66d6cd49-xkqsr"
Nov 02 12:38:10.644595 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:10.644476    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7p95c\" (UniqueName: \"kubernetes.io/projected/87b8ed31-36bf-421c-8ee0-48fbe61a65e9-kube-api-access-7p95c\") pod \"payload-processing-external-model-7d66d6cd49-xkqsr\" (UID: \"87b8ed31-36bf-421c-8ee0-48fbe61a65e9\") " pod="models-as-a-service/payload-processing-external-model-7d66d6cd49-xkqsr"
Nov 02 12:38:10.644595 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:10.644509    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87b8ed31-36bf-421c-8ee0-48fbe61a65e9-plugins-config-volume\") pod \"payload-processing-external-model-7d66d6cd49-xkqsr\" (UID: \"87b8ed31-36bf-421c-8ee0-48fbe61a65e9\") " pod="models-as-a-service/payload-processing-external-model-7d66d6cd49-xkqsr"
Nov 02 12:38:10.645246 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:10.645220    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/87b8ed31-36bf-421c-8ee0-48fbe61a65e9-routing-overlay-volume\") pod \"payload-processing-external-model-7d66d6cd49-xkqsr\" (UID: \"87b8ed31-36bf-421c-8ee0-48fbe61a65e9\") " pod="models-as-a-service/payload-processing-external-model-7d66d6cd49-xkqsr"
Nov 02 12:38:10.645343 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:10.645277    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87b8ed31-36bf-421c-8ee0-48fbe61a65e9-plugins-config-volume\") pod \"payload-processing-external-model-7d66d6cd49-xkqsr\" (UID: \"87b8ed31-36bf-421c-8ee0-48fbe61a65e9\") " pod="models-as-a-service/payload-processing-external-model-7d66d6cd49-xkqsr"
Nov 02 12:38:10.647002 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:10.646983    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/87b8ed31-36bf-421c-8ee0-48fbe61a65e9-provider-credentials\") pod \"payload-processing-external-model-7d66d6cd49-xkqsr\" (UID: \"87b8ed31-36bf-421c-8ee0-48fbe61a65e9\") " pod="models-as-a-service/payload-processing-external-model-7d66d6cd49-xkqsr"
Nov 02 12:38:10.691716 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:10.671703    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7p95c\" (UniqueName: \"kubernetes.io/projected/87b8ed31-36bf-421c-8ee0-48fbe61a65e9-kube-api-access-7p95c\") pod \"payload-processing-external-model-7d66d6cd49-xkqsr\" (UID: \"87b8ed31-36bf-421c-8ee0-48fbe61a65e9\") " pod="models-as-a-service/payload-processing-external-model-7d66d6cd49-xkqsr"
Nov 02 12:38:10.758021 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:10.757990    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7d66d6cd49-xkqsr"
Nov 02 12:38:11.070677 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:11.069661    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-7d66d6cd49-xkqsr"]
Nov 02 12:38:11.071281 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:38:11.071241    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87b8ed31_36bf_421c_8ee0_48fbe61a65e9.slice/crio-ea6ac5b839f52f42f953af05e2ac263a305ab98120ed0b2130b952d8b2c866fe WatchSource:0}: Error finding container ea6ac5b839f52f42f953af05e2ac263a305ab98120ed0b2130b952d8b2c866fe: Status 404 returned error can't find the container with id ea6ac5b839f52f42f953af05e2ac263a305ab98120ed0b2130b952d8b2c866fe
Nov 02 12:38:11.375691 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:11.375572    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7d66d6cd49-xkqsr" event={"ID":"87b8ed31-36bf-421c-8ee0-48fbe61a65e9","Type":"ContainerStarted","Data":"6aa435c0fe858914862edea0bbd087114566dfc9f1ca0d01b6a82f47fbbc4b62"}
Nov 02 12:38:11.375691 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:11.375633    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7d66d6cd49-xkqsr" event={"ID":"87b8ed31-36bf-421c-8ee0-48fbe61a65e9","Type":"ContainerStarted","Data":"ea6ac5b839f52f42f953af05e2ac263a305ab98120ed0b2130b952d8b2c866fe"}
Nov 02 12:38:11.375919 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:11.375819    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-7d66d6cd49-xkqsr"
Nov 02 12:38:11.431681 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:11.430914    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-7d66d6cd49-xkqsr" podStartSLOduration=1.430895757 podStartE2EDuration="1.430895757s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:38:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:38:11.428618165 +0000 UTC m=+687.684017261" watchObservedRunningTime="2026-11-02 12:38:11.430895757 +0000 UTC m=+687.686294853"
Nov 02 12:38:15.086346 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:38:15.086312    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqcmj" containerName="extractor"
Nov 02 12:38:15.574009 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:15.573974    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"]
Nov 02 12:38:22.383388 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:22.383358    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-7d66d6cd49-xkqsr"
Nov 02 12:38:27.069661 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:27.069551    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-4xgzk"]
Nov 02 12:38:27.085676 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:27.083240    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-4xgzk"
Nov 02 12:38:27.105208 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:27.105177    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-4xgzk"]
Nov 02 12:38:27.132374 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:27.132340    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-5514e5-openshift-default-dockercfg-7v87s\""
Nov 02 12:38:27.179547 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:27.179495    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18cc5afa-245b-43f6-b92c-deec76232bd7-workload-certs\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-4xgzk\" (UID: \"18cc5afa-245b-43f6-b92c-deec76232bd7\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-4xgzk"
Nov 02 12:38:27.179547 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:27.179549    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18cc5afa-245b-43f6-b92c-deec76232bd7-istio-podinfo\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-4xgzk\" (UID: \"18cc5afa-245b-43f6-b92c-deec76232bd7\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-4xgzk"
Nov 02 12:38:27.179825 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:27.179597    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18cc5afa-245b-43f6-b92c-deec76232bd7-credential-socket\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-4xgzk\" (UID: \"18cc5afa-245b-43f6-b92c-deec76232bd7\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-4xgzk"
Nov 02 12:38:27.179825 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:27.179634    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18cc5afa-245b-43f6-b92c-deec76232bd7-istio-envoy\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-4xgzk\" (UID: \"18cc5afa-245b-43f6-b92c-deec76232bd7\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-4xgzk"
Nov 02 12:38:27.179825 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:27.179697    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18cc5afa-245b-43f6-b92c-deec76232bd7-istio-data\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-4xgzk\" (UID: \"18cc5afa-245b-43f6-b92c-deec76232bd7\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-4xgzk"
Nov 02 12:38:27.179825 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:27.179746    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18cc5afa-245b-43f6-b92c-deec76232bd7-istio-token\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-4xgzk\" (UID: \"18cc5afa-245b-43f6-b92c-deec76232bd7\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-4xgzk"
Nov 02 12:38:27.180072 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:27.179830    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18cc5afa-245b-43f6-b92c-deec76232bd7-workload-socket\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-4xgzk\" (UID: \"18cc5afa-245b-43f6-b92c-deec76232bd7\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-4xgzk"
Nov 02 12:38:27.180072 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:27.179863    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlghm\" (UniqueName: \"kubernetes.io/projected/18cc5afa-245b-43f6-b92c-deec76232bd7-kube-api-access-wlghm\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-4xgzk\" (UID: \"18cc5afa-245b-43f6-b92c-deec76232bd7\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-4xgzk"
Nov 02 12:38:27.180072 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:27.179895    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18cc5afa-245b-43f6-b92c-deec76232bd7-istiod-ca-cert\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-4xgzk\" (UID: \"18cc5afa-245b-43f6-b92c-deec76232bd7\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-4xgzk"
Nov 02 12:38:27.303752 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:27.284308    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18cc5afa-245b-43f6-b92c-deec76232bd7-istio-data\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-4xgzk\" (UID: \"18cc5afa-245b-43f6-b92c-deec76232bd7\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-4xgzk"
Nov 02 12:38:27.303752 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:27.284436    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18cc5afa-245b-43f6-b92c-deec76232bd7-istio-token\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-4xgzk\" (UID: \"18cc5afa-245b-43f6-b92c-deec76232bd7\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-4xgzk"
Nov 02 12:38:27.303752 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:27.284513    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18cc5afa-245b-43f6-b92c-deec76232bd7-workload-socket\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-4xgzk\" (UID: \"18cc5afa-245b-43f6-b92c-deec76232bd7\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-4xgzk"
Nov 02 12:38:27.303752 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:27.284547    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlghm\" (UniqueName: \"kubernetes.io/projected/18cc5afa-245b-43f6-b92c-deec76232bd7-kube-api-access-wlghm\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-4xgzk\" (UID: \"18cc5afa-245b-43f6-b92c-deec76232bd7\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-4xgzk"
Nov 02 12:38:27.303752 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:27.284583    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18cc5afa-245b-43f6-b92c-deec76232bd7-istiod-ca-cert\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-4xgzk\" (UID: \"18cc5afa-245b-43f6-b92c-deec76232bd7\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-4xgzk"
Nov 02 12:38:27.303752 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:27.284692    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18cc5afa-245b-43f6-b92c-deec76232bd7-workload-certs\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-4xgzk\" (UID: \"18cc5afa-245b-43f6-b92c-deec76232bd7\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-4xgzk"
Nov 02 12:38:27.303752 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:27.284729    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18cc5afa-245b-43f6-b92c-deec76232bd7-istio-podinfo\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-4xgzk\" (UID: \"18cc5afa-245b-43f6-b92c-deec76232bd7\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-4xgzk"
Nov 02 12:38:27.303752 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:27.284781    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18cc5afa-245b-43f6-b92c-deec76232bd7-credential-socket\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-4xgzk\" (UID: \"18cc5afa-245b-43f6-b92c-deec76232bd7\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-4xgzk"
Nov 02 12:38:27.303752 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:27.284817    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18cc5afa-245b-43f6-b92c-deec76232bd7-istio-envoy\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-4xgzk\" (UID: \"18cc5afa-245b-43f6-b92c-deec76232bd7\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-4xgzk"
Nov 02 12:38:27.303752 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:27.285658    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18cc5afa-245b-43f6-b92c-deec76232bd7-istio-data\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-4xgzk\" (UID: \"18cc5afa-245b-43f6-b92c-deec76232bd7\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-4xgzk"
Nov 02 12:38:27.303752 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:27.286244    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18cc5afa-245b-43f6-b92c-deec76232bd7-workload-socket\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-4xgzk\" (UID: \"18cc5afa-245b-43f6-b92c-deec76232bd7\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-4xgzk"
Nov 02 12:38:27.303752 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:27.286796    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18cc5afa-245b-43f6-b92c-deec76232bd7-workload-certs\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-4xgzk\" (UID: \"18cc5afa-245b-43f6-b92c-deec76232bd7\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-4xgzk"
Nov 02 12:38:27.303752 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:27.287394    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18cc5afa-245b-43f6-b92c-deec76232bd7-istiod-ca-cert\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-4xgzk\" (UID: \"18cc5afa-245b-43f6-b92c-deec76232bd7\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-4xgzk"
Nov 02 12:38:27.303752 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:27.294040    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18cc5afa-245b-43f6-b92c-deec76232bd7-istio-envoy\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-4xgzk\" (UID: \"18cc5afa-245b-43f6-b92c-deec76232bd7\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-4xgzk"
Nov 02 12:38:27.303752 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:27.294328    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18cc5afa-245b-43f6-b92c-deec76232bd7-credential-socket\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-4xgzk\" (UID: \"18cc5afa-245b-43f6-b92c-deec76232bd7\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-4xgzk"
Nov 02 12:38:27.313303 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:27.313272    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18cc5afa-245b-43f6-b92c-deec76232bd7-istio-podinfo\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-4xgzk\" (UID: \"18cc5afa-245b-43f6-b92c-deec76232bd7\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-4xgzk"
Nov 02 12:38:27.319586 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:27.319554    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlghm\" (UniqueName: \"kubernetes.io/projected/18cc5afa-245b-43f6-b92c-deec76232bd7-kube-api-access-wlghm\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-4xgzk\" (UID: \"18cc5afa-245b-43f6-b92c-deec76232bd7\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-4xgzk"
Nov 02 12:38:27.324379 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:27.324313    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18cc5afa-245b-43f6-b92c-deec76232bd7-istio-token\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-4xgzk\" (UID: \"18cc5afa-245b-43f6-b92c-deec76232bd7\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-4xgzk"
Nov 02 12:38:27.440887 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:27.440846    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-4xgzk"
Nov 02 12:38:27.799963 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:27.799928    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-4xgzk"]
Nov 02 12:38:27.803982 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:38:27.803950    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18cc5afa_245b_43f6_b92c_deec76232bd7.slice/crio-0dbc46ca2f98e28a6bafcdc807396519c6a46e5612e0108ec1586e09badb2378 WatchSource:0}: Error finding container 0dbc46ca2f98e28a6bafcdc807396519c6a46e5612e0108ec1586e09badb2378: Status 404 returned error can't find the container with id 0dbc46ca2f98e28a6bafcdc807396519c6a46e5612e0108ec1586e09badb2378
Nov 02 12:38:27.806386 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:27.806352    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:38:27.806466 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:27.806419    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:38:27.806466 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:27.806448    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:38:28.482177 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:28.482129    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-4xgzk" event={"ID":"18cc5afa-245b-43f6-b92c-deec76232bd7","Type":"ContainerStarted","Data":"d56acd804fb207eaf32dce09b0f8e3b3579c7a87d8550a4b90b3cc1ee8b4b74c"}
Nov 02 12:38:28.482613 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:28.482206    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-4xgzk" event={"ID":"18cc5afa-245b-43f6-b92c-deec76232bd7","Type":"ContainerStarted","Data":"0dbc46ca2f98e28a6bafcdc807396519c6a46e5612e0108ec1586e09badb2378"}
Nov 02 12:38:28.569659 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:28.566539    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-4xgzk" podStartSLOduration=1.5665172109999999 podStartE2EDuration="1.566517211s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:38:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:38:28.550069118 +0000 UTC m=+704.805468233" watchObservedRunningTime="2026-11-02 12:38:28.566517211 +0000 UTC m=+704.821916307"
Nov 02 12:38:29.447843 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:29.447747    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-4xgzk"
Nov 02 12:38:29.448520 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:29.448164    4559 patch_prober.go:29] interesting pod/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-4xgzk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Nov 02 12:38:29.448520 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:29.448239    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-4xgzk" podUID="18cc5afa-245b-43f6-b92c-deec76232bd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Nov 02 12:38:30.441572 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:30.441528    4559 patch_prober.go:29] interesting pod/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-4xgzk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Nov 02 12:38:30.442205 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:30.442169    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-4xgzk" podUID="18cc5afa-245b-43f6-b92c-deec76232bd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Nov 02 12:38:31.441962 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:31.441930    4559 patch_prober.go:29] interesting pod/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-4xgzk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Nov 02 12:38:31.442423 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:31.441990    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-4xgzk" podUID="18cc5afa-245b-43f6-b92c-deec76232bd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Nov 02 12:38:32.441591 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:32.441552    4559 patch_prober.go:29] interesting pod/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-4xgzk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Nov 02 12:38:32.441796 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:32.441612    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-4xgzk" podUID="18cc5afa-245b-43f6-b92c-deec76232bd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Nov 02 12:38:33.442173 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:33.442135    4559 patch_prober.go:29] interesting pod/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-4xgzk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Nov 02 12:38:33.442565 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:33.442191    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-4xgzk" podUID="18cc5afa-245b-43f6-b92c-deec76232bd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Nov 02 12:38:34.442041 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:34.442002    4559 patch_prober.go:29] interesting pod/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-4xgzk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Nov 02 12:38:34.442226 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:34.442057    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-4xgzk" podUID="18cc5afa-245b-43f6-b92c-deec76232bd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Nov 02 12:38:35.445620 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:35.445589    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-4xgzk"
Nov 02 12:38:35.446009 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:35.445965    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-4xgzk"
Nov 02 12:38:35.446688 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:35.446669    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-4xgzk"
Nov 02 12:38:36.564132 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:36.564094    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-k5zn8"]
Nov 02 12:38:36.568077 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:36.568054    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-k5zn8"
Nov 02 12:38:36.594547 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:36.594506    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-k5zn8"]
Nov 02 12:38:36.638903 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:36.638869    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3b2832d-2aad-406f-86b5-de405575f9b2-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-k5zn8\" (UID: \"d3b2832d-2aad-406f-86b5-de405575f9b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-k5zn8"
Nov 02 12:38:36.639051 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:36.638910    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3b2832d-2aad-406f-86b5-de405575f9b2-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-k5zn8\" (UID: \"d3b2832d-2aad-406f-86b5-de405575f9b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-k5zn8"
Nov 02 12:38:36.639051 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:36.639029    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzzhd\" (UniqueName: \"kubernetes.io/projected/d3b2832d-2aad-406f-86b5-de405575f9b2-kube-api-access-tzzhd\") pod \"maas-default-gateway-openshift-default-5bc884dd87-k5zn8\" (UID: \"d3b2832d-2aad-406f-86b5-de405575f9b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-k5zn8"
Nov 02 12:38:36.639138 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:36.639078    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3b2832d-2aad-406f-86b5-de405575f9b2-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-k5zn8\" (UID: \"d3b2832d-2aad-406f-86b5-de405575f9b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-k5zn8"
Nov 02 12:38:36.639138 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:36.639106    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3b2832d-2aad-406f-86b5-de405575f9b2-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-k5zn8\" (UID: \"d3b2832d-2aad-406f-86b5-de405575f9b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-k5zn8"
Nov 02 12:38:36.639209 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:36.639175    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3b2832d-2aad-406f-86b5-de405575f9b2-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-k5zn8\" (UID: \"d3b2832d-2aad-406f-86b5-de405575f9b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-k5zn8"
Nov 02 12:38:36.639359 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:36.639340    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3b2832d-2aad-406f-86b5-de405575f9b2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-k5zn8\" (UID: \"d3b2832d-2aad-406f-86b5-de405575f9b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-k5zn8"
Nov 02 12:38:36.639398 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:36.639387    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3b2832d-2aad-406f-86b5-de405575f9b2-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-k5zn8\" (UID: \"d3b2832d-2aad-406f-86b5-de405575f9b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-k5zn8"
Nov 02 12:38:36.639528 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:36.639512    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3b2832d-2aad-406f-86b5-de405575f9b2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-k5zn8\" (UID: \"d3b2832d-2aad-406f-86b5-de405575f9b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-k5zn8"
Nov 02 12:38:36.647604 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:36.647580    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"]
Nov 02 12:38:36.740763 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:36.740716    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3b2832d-2aad-406f-86b5-de405575f9b2-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-k5zn8\" (UID: \"d3b2832d-2aad-406f-86b5-de405575f9b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-k5zn8"
Nov 02 12:38:36.740964 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:36.740771    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3b2832d-2aad-406f-86b5-de405575f9b2-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-k5zn8\" (UID: \"d3b2832d-2aad-406f-86b5-de405575f9b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-k5zn8"
Nov 02 12:38:36.740964 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:36.740803    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3b2832d-2aad-406f-86b5-de405575f9b2-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-k5zn8\" (UID: \"d3b2832d-2aad-406f-86b5-de405575f9b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-k5zn8"
Nov 02 12:38:36.740964 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:36.740835    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3b2832d-2aad-406f-86b5-de405575f9b2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-k5zn8\" (UID: \"d3b2832d-2aad-406f-86b5-de405575f9b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-k5zn8"
Nov 02 12:38:36.740964 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:36.740873    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3b2832d-2aad-406f-86b5-de405575f9b2-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-k5zn8\" (UID: \"d3b2832d-2aad-406f-86b5-de405575f9b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-k5zn8"
Nov 02 12:38:36.740964 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:36.740959    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3b2832d-2aad-406f-86b5-de405575f9b2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-k5zn8\" (UID: \"d3b2832d-2aad-406f-86b5-de405575f9b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-k5zn8"
Nov 02 12:38:36.741252 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:36.741048    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3b2832d-2aad-406f-86b5-de405575f9b2-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-k5zn8\" (UID: \"d3b2832d-2aad-406f-86b5-de405575f9b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-k5zn8"
Nov 02 12:38:36.741252 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:36.741084    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3b2832d-2aad-406f-86b5-de405575f9b2-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-k5zn8\" (UID: \"d3b2832d-2aad-406f-86b5-de405575f9b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-k5zn8"
Nov 02 12:38:36.741252 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:36.741166    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzzhd\" (UniqueName: \"kubernetes.io/projected/d3b2832d-2aad-406f-86b5-de405575f9b2-kube-api-access-tzzhd\") pod \"maas-default-gateway-openshift-default-5bc884dd87-k5zn8\" (UID: \"d3b2832d-2aad-406f-86b5-de405575f9b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-k5zn8"
Nov 02 12:38:36.742131 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:36.742101    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3b2832d-2aad-406f-86b5-de405575f9b2-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-k5zn8\" (UID: \"d3b2832d-2aad-406f-86b5-de405575f9b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-k5zn8"
Nov 02 12:38:36.742316 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:36.742195    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3b2832d-2aad-406f-86b5-de405575f9b2-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-k5zn8\" (UID: \"d3b2832d-2aad-406f-86b5-de405575f9b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-k5zn8"
Nov 02 12:38:36.742736 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:36.742715    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3b2832d-2aad-406f-86b5-de405575f9b2-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-k5zn8\" (UID: \"d3b2832d-2aad-406f-86b5-de405575f9b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-k5zn8"
Nov 02 12:38:36.744614 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:36.744484    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3b2832d-2aad-406f-86b5-de405575f9b2-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-k5zn8\" (UID: \"d3b2832d-2aad-406f-86b5-de405575f9b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-k5zn8"
Nov 02 12:38:36.755549 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:36.755496    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3b2832d-2aad-406f-86b5-de405575f9b2-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-k5zn8\" (UID: \"d3b2832d-2aad-406f-86b5-de405575f9b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-k5zn8"
Nov 02 12:38:36.759895 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:36.759856    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3b2832d-2aad-406f-86b5-de405575f9b2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-k5zn8\" (UID: \"d3b2832d-2aad-406f-86b5-de405575f9b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-k5zn8"
Nov 02 12:38:36.760019 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:36.759859    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3b2832d-2aad-406f-86b5-de405575f9b2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-k5zn8\" (UID: \"d3b2832d-2aad-406f-86b5-de405575f9b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-k5zn8"
Nov 02 12:38:36.768968 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:36.768925    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3b2832d-2aad-406f-86b5-de405575f9b2-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-k5zn8\" (UID: \"d3b2832d-2aad-406f-86b5-de405575f9b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-k5zn8"
Nov 02 12:38:36.771833 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:36.771811    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzzhd\" (UniqueName: \"kubernetes.io/projected/d3b2832d-2aad-406f-86b5-de405575f9b2-kube-api-access-tzzhd\") pod \"maas-default-gateway-openshift-default-5bc884dd87-k5zn8\" (UID: \"d3b2832d-2aad-406f-86b5-de405575f9b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-k5zn8"
Nov 02 12:38:36.883263 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:36.883177    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-k5zn8"
Nov 02 12:38:37.216670 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:37.215977    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-k5zn8"]
Nov 02 12:38:37.229107 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:37.222388    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:38:37.229107 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:37.222480    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:38:37.229107 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:37.222524    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:38:37.587729 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:37.583417    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-k5zn8" event={"ID":"d3b2832d-2aad-406f-86b5-de405575f9b2","Type":"ContainerStarted","Data":"e11beb411d5d8cb9ad1ee54ee6367d6245a564a1de071e3b6a623192e83fa165"}
Nov 02 12:38:37.587729 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:37.583487    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-k5zn8" event={"ID":"d3b2832d-2aad-406f-86b5-de405575f9b2","Type":"ContainerStarted","Data":"0ac7d05e8eec691f2f96cd98958f8d22940289c439831f0aa5eed905fd61c8e7"}
Nov 02 12:38:37.641662 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:37.639866    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-k5zn8" podStartSLOduration=1.639849492 podStartE2EDuration="1.639849492s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:38:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:38:37.637178295 +0000 UTC m=+713.892577390" watchObservedRunningTime="2026-11-02 12:38:37.639849492 +0000 UTC m=+713.895248587"
Nov 02 12:38:37.884091 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:37.883884    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-k5zn8"
Nov 02 12:38:38.896665 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:38.888498    4559 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-k5zn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Nov 02 12:38:38.896665 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:38.888577    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-k5zn8" podUID="d3b2832d-2aad-406f-86b5-de405575f9b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Nov 02 12:38:39.888665 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:39.888102    4559 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-k5zn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Nov 02 12:38:39.888665 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:39.888163    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-k5zn8" podUID="d3b2832d-2aad-406f-86b5-de405575f9b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Nov 02 12:38:40.886108 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:40.886067    4559 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-k5zn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Nov 02 12:38:40.886855 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:40.886769    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-k5zn8" podUID="d3b2832d-2aad-406f-86b5-de405575f9b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Nov 02 12:38:41.904781 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:41.904742    4559 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-k5zn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Nov 02 12:38:41.905370 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:41.905339    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-k5zn8" podUID="d3b2832d-2aad-406f-86b5-de405575f9b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Nov 02 12:38:42.896199 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:42.896162    4559 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-k5zn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Nov 02 12:38:42.896481 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:42.896452    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-k5zn8" podUID="d3b2832d-2aad-406f-86b5-de405575f9b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Nov 02 12:38:43.608048 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:43.608006    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-7d66d6cd49-xkqsr"]
Nov 02 12:38:43.884250 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:43.884167    4559 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-k5zn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Nov 02 12:38:43.884250 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:43.884228    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-k5zn8" podUID="d3b2832d-2aad-406f-86b5-de405575f9b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Nov 02 12:38:44.884193 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:44.884154    4559 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-k5zn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Nov 02 12:38:44.884825 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:44.884785    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-k5zn8" podUID="d3b2832d-2aad-406f-86b5-de405575f9b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Nov 02 12:38:45.096256 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:38:45.096220    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqcmj" containerName="extractor"
Nov 02 12:38:46.601664 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:46.580282    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-xs5zn"]
Nov 02 12:38:47.012932 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:47.012893    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-xs5zn"]
Nov 02 12:38:47.013292 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:47.013272    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-xs5zn"
Nov 02 12:38:47.030669 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:47.027286    4559 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-k5zn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 12:38:47.030901 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:47.030866    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-k5zn8" podUID="d3b2832d-2aad-406f-86b5-de405575f9b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:38:47.193367 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:47.193321    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36990652-4474-4f19-94e5-8d540cdc06a1-istio-podinfo\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-xs5zn\" (UID: \"36990652-4474-4f19-94e5-8d540cdc06a1\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-xs5zn"
Nov 02 12:38:47.193711 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:47.193685    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36990652-4474-4f19-94e5-8d540cdc06a1-istiod-ca-cert\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-xs5zn\" (UID: \"36990652-4474-4f19-94e5-8d540cdc06a1\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-xs5zn"
Nov 02 12:38:47.193996 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:47.193972    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8zrh\" (UniqueName: \"kubernetes.io/projected/36990652-4474-4f19-94e5-8d540cdc06a1-kube-api-access-l8zrh\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-xs5zn\" (UID: \"36990652-4474-4f19-94e5-8d540cdc06a1\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-xs5zn"
Nov 02 12:38:47.194146 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:47.194129    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36990652-4474-4f19-94e5-8d540cdc06a1-workload-certs\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-xs5zn\" (UID: \"36990652-4474-4f19-94e5-8d540cdc06a1\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-xs5zn"
Nov 02 12:38:47.194262 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:47.194247    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36990652-4474-4f19-94e5-8d540cdc06a1-credential-socket\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-xs5zn\" (UID: \"36990652-4474-4f19-94e5-8d540cdc06a1\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-xs5zn"
Nov 02 12:38:47.194368 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:47.194354    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36990652-4474-4f19-94e5-8d540cdc06a1-istio-token\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-xs5zn\" (UID: \"36990652-4474-4f19-94e5-8d540cdc06a1\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-xs5zn"
Nov 02 12:38:47.194501 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:47.194486    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36990652-4474-4f19-94e5-8d540cdc06a1-istio-envoy\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-xs5zn\" (UID: \"36990652-4474-4f19-94e5-8d540cdc06a1\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-xs5zn"
Nov 02 12:38:47.194626 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:47.194611    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36990652-4474-4f19-94e5-8d540cdc06a1-istio-data\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-xs5zn\" (UID: \"36990652-4474-4f19-94e5-8d540cdc06a1\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-xs5zn"
Nov 02 12:38:47.194758 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:47.194742    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36990652-4474-4f19-94e5-8d540cdc06a1-workload-socket\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-xs5zn\" (UID: \"36990652-4474-4f19-94e5-8d540cdc06a1\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-xs5zn"
Nov 02 12:38:47.298543 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:47.298410    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36990652-4474-4f19-94e5-8d540cdc06a1-credential-socket\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-xs5zn\" (UID: \"36990652-4474-4f19-94e5-8d540cdc06a1\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-xs5zn"
Nov 02 12:38:47.299417 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:47.299389    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36990652-4474-4f19-94e5-8d540cdc06a1-istio-token\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-xs5zn\" (UID: \"36990652-4474-4f19-94e5-8d540cdc06a1\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-xs5zn"
Nov 02 12:38:47.299612 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:47.299587    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36990652-4474-4f19-94e5-8d540cdc06a1-istio-envoy\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-xs5zn\" (UID: \"36990652-4474-4f19-94e5-8d540cdc06a1\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-xs5zn"
Nov 02 12:38:47.299810 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:47.299774    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36990652-4474-4f19-94e5-8d540cdc06a1-istio-data\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-xs5zn\" (UID: \"36990652-4474-4f19-94e5-8d540cdc06a1\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-xs5zn"
Nov 02 12:38:47.299987 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:47.299952    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36990652-4474-4f19-94e5-8d540cdc06a1-workload-socket\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-xs5zn\" (UID: \"36990652-4474-4f19-94e5-8d540cdc06a1\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-xs5zn"
Nov 02 12:38:47.300221 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:47.300201    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36990652-4474-4f19-94e5-8d540cdc06a1-istio-podinfo\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-xs5zn\" (UID: \"36990652-4474-4f19-94e5-8d540cdc06a1\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-xs5zn"
Nov 02 12:38:47.300356 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:47.299319    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36990652-4474-4f19-94e5-8d540cdc06a1-credential-socket\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-xs5zn\" (UID: \"36990652-4474-4f19-94e5-8d540cdc06a1\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-xs5zn"
Nov 02 12:38:47.301493 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:47.300419    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36990652-4474-4f19-94e5-8d540cdc06a1-istiod-ca-cert\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-xs5zn\" (UID: \"36990652-4474-4f19-94e5-8d540cdc06a1\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-xs5zn"
Nov 02 12:38:47.301752 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:47.301725    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8zrh\" (UniqueName: \"kubernetes.io/projected/36990652-4474-4f19-94e5-8d540cdc06a1-kube-api-access-l8zrh\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-xs5zn\" (UID: \"36990652-4474-4f19-94e5-8d540cdc06a1\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-xs5zn"
Nov 02 12:38:47.301920 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:47.301900    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36990652-4474-4f19-94e5-8d540cdc06a1-workload-certs\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-xs5zn\" (UID: \"36990652-4474-4f19-94e5-8d540cdc06a1\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-xs5zn"
Nov 02 12:38:47.302064 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:47.301438    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36990652-4474-4f19-94e5-8d540cdc06a1-workload-socket\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-xs5zn\" (UID: \"36990652-4474-4f19-94e5-8d540cdc06a1\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-xs5zn"
Nov 02 12:38:47.302186 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:47.301197    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36990652-4474-4f19-94e5-8d540cdc06a1-istio-data\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-xs5zn\" (UID: \"36990652-4474-4f19-94e5-8d540cdc06a1\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-xs5zn"
Nov 02 12:38:47.302539 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:47.301292    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36990652-4474-4f19-94e5-8d540cdc06a1-istiod-ca-cert\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-xs5zn\" (UID: \"36990652-4474-4f19-94e5-8d540cdc06a1\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-xs5zn"
Nov 02 12:38:47.303048 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:47.303027    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36990652-4474-4f19-94e5-8d540cdc06a1-workload-certs\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-xs5zn\" (UID: \"36990652-4474-4f19-94e5-8d540cdc06a1\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-xs5zn"
Nov 02 12:38:47.331099 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:47.331054    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36990652-4474-4f19-94e5-8d540cdc06a1-istio-podinfo\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-xs5zn\" (UID: \"36990652-4474-4f19-94e5-8d540cdc06a1\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-xs5zn"
Nov 02 12:38:47.335295 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:47.335238    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36990652-4474-4f19-94e5-8d540cdc06a1-istio-token\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-xs5zn\" (UID: \"36990652-4474-4f19-94e5-8d540cdc06a1\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-xs5zn"
Nov 02 12:38:47.339412 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:47.338306    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8zrh\" (UniqueName: \"kubernetes.io/projected/36990652-4474-4f19-94e5-8d540cdc06a1-kube-api-access-l8zrh\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-xs5zn\" (UID: \"36990652-4474-4f19-94e5-8d540cdc06a1\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-xs5zn"
Nov 02 12:38:47.348010 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:47.347971    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36990652-4474-4f19-94e5-8d540cdc06a1-istio-envoy\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-xs5zn\" (UID: \"36990652-4474-4f19-94e5-8d540cdc06a1\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-xs5zn"
Nov 02 12:38:47.392185 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:47.391892    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-xs5zn"
Nov 02 12:38:47.416732 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:47.415022    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-k5zn8"
Nov 02 12:38:47.422782 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:47.422694    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-k5zn8"
Nov 02 12:38:47.427944 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:47.427891    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-k5zn8"
Nov 02 12:38:48.064592 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:48.033800    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:38:48.064592 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:48.033895    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:38:48.064592 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:48.033944    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:38:48.064592 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:48.051455    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-xs5zn"]
Nov 02 12:38:48.132205 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:48.127070    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-xs5zn" event={"ID":"36990652-4474-4f19-94e5-8d540cdc06a1","Type":"ContainerStarted","Data":"11acc9bbb88b1b118fb12fd5b6519207f50e4b4996d0a1c887f07aa90c688592"}
Nov 02 12:38:48.608555 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:48.608407    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-7d66d6cd49-xkqsr" podUID="87b8ed31-36bf-421c-8ee0-48fbe61a65e9" containerName="payload-processing" containerID="cri-o://6aa435c0fe858914862edea0bbd087114566dfc9f1ca0d01b6a82f47fbbc4b62" gracePeriod=25
Nov 02 12:38:48.976601 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:48.976578    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7d66d6cd49-xkqsr"
Nov 02 12:38:49.116663 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:49.109185    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87b8ed31-36bf-421c-8ee0-48fbe61a65e9-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/87b8ed31-36bf-421c-8ee0-48fbe61a65e9-routing-overlay-volume\") pod \"87b8ed31-36bf-421c-8ee0-48fbe61a65e9\" (UID: \"87b8ed31-36bf-421c-8ee0-48fbe61a65e9\") "
Nov 02 12:38:49.120856 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:49.120820    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87b8ed31-36bf-421c-8ee0-48fbe61a65e9-routing-overlay-volume" pod "87b8ed31-36bf-421c-8ee0-48fbe61a65e9" (UID: "87b8ed31-36bf-421c-8ee0-48fbe61a65e9"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:38:49.121042 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:49.120868    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87b8ed31-36bf-421c-8ee0-48fbe61a65e9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87b8ed31-36bf-421c-8ee0-48fbe61a65e9-plugins-config-volume\") pod \"87b8ed31-36bf-421c-8ee0-48fbe61a65e9\" (UID: \"87b8ed31-36bf-421c-8ee0-48fbe61a65e9\") "
Nov 02 12:38:49.121451 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:49.121412    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87b8ed31-36bf-421c-8ee0-48fbe61a65e9-provider-credentials\" (UniqueName: \"kubernetes.io/projected/87b8ed31-36bf-421c-8ee0-48fbe61a65e9-provider-credentials\") pod \"87b8ed31-36bf-421c-8ee0-48fbe61a65e9\" (UID: \"87b8ed31-36bf-421c-8ee0-48fbe61a65e9\") "
Nov 02 12:38:49.121704 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:49.121682    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87b8ed31-36bf-421c-8ee0-48fbe61a65e9-kube-api-access-7p95c\" (UniqueName: \"kubernetes.io/projected/87b8ed31-36bf-421c-8ee0-48fbe61a65e9-kube-api-access-7p95c\") pod \"87b8ed31-36bf-421c-8ee0-48fbe61a65e9\" (UID: \"87b8ed31-36bf-421c-8ee0-48fbe61a65e9\") "
Nov 02 12:38:49.122425 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:49.121539    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87b8ed31-36bf-421c-8ee0-48fbe61a65e9-plugins-config-volume" pod "87b8ed31-36bf-421c-8ee0-48fbe61a65e9" (UID: "87b8ed31-36bf-421c-8ee0-48fbe61a65e9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:38:49.123397 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:49.123376    4559 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/87b8ed31-36bf-421c-8ee0-48fbe61a65e9-routing-overlay-volume\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:38:49.123581 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:49.123564    4559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87b8ed31-36bf-421c-8ee0-48fbe61a65e9-plugins-config-volume\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:38:49.124816 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:49.124745    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87b8ed31-36bf-421c-8ee0-48fbe61a65e9-kube-api-access-7p95c" pod "87b8ed31-36bf-421c-8ee0-48fbe61a65e9" (UID: "87b8ed31-36bf-421c-8ee0-48fbe61a65e9"). InnerVolumeSpecName "kube-api-access-7p95c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:38:49.125204 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:49.125166    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87b8ed31-36bf-421c-8ee0-48fbe61a65e9-provider-credentials" pod "87b8ed31-36bf-421c-8ee0-48fbe61a65e9" (UID: "87b8ed31-36bf-421c-8ee0-48fbe61a65e9"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:38:49.144700 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:49.141805    4559 generic.go:356] "Generic (PLEG): container finished" podID="87b8ed31-36bf-421c-8ee0-48fbe61a65e9" containerID="6aa435c0fe858914862edea0bbd087114566dfc9f1ca0d01b6a82f47fbbc4b62" exitCode=0
Nov 02 12:38:49.144700 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:49.141989    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7d66d6cd49-xkqsr" event={"ID":"87b8ed31-36bf-421c-8ee0-48fbe61a65e9","Type":"ContainerDied","Data":"6aa435c0fe858914862edea0bbd087114566dfc9f1ca0d01b6a82f47fbbc4b62"}
Nov 02 12:38:49.144700 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:49.142044    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7d66d6cd49-xkqsr" event={"ID":"87b8ed31-36bf-421c-8ee0-48fbe61a65e9","Type":"ContainerDied","Data":"ea6ac5b839f52f42f953af05e2ac263a305ab98120ed0b2130b952d8b2c866fe"}
Nov 02 12:38:49.144700 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:49.142076    4559 scope.go:122] "RemoveContainer" containerID="6aa435c0fe858914862edea0bbd087114566dfc9f1ca0d01b6a82f47fbbc4b62"
Nov 02 12:38:49.144700 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:49.142237    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7d66d6cd49-xkqsr"
Nov 02 12:38:49.155599 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:49.155236    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-xs5zn" event={"ID":"36990652-4474-4f19-94e5-8d540cdc06a1","Type":"ContainerStarted","Data":"c50dfe46f0cdb8ef0d49145840779c4a42d6ccd66cc4cdcd2707988cfc97a572"}
Nov 02 12:38:49.167617 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:49.167537    4559 scope.go:122] "RemoveContainer" containerID="6aa435c0fe858914862edea0bbd087114566dfc9f1ca0d01b6a82f47fbbc4b62"
Nov 02 12:38:49.168112 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:38:49.168051    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6aa435c0fe858914862edea0bbd087114566dfc9f1ca0d01b6a82f47fbbc4b62\": container with ID starting with 6aa435c0fe858914862edea0bbd087114566dfc9f1ca0d01b6a82f47fbbc4b62 not found: ID does not exist" containerID="6aa435c0fe858914862edea0bbd087114566dfc9f1ca0d01b6a82f47fbbc4b62"
Nov 02 12:38:49.168112 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:49.168084    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6aa435c0fe858914862edea0bbd087114566dfc9f1ca0d01b6a82f47fbbc4b62"} err="failed to get container status \"6aa435c0fe858914862edea0bbd087114566dfc9f1ca0d01b6a82f47fbbc4b62\": rpc error: code = NotFound desc = could not find container \"6aa435c0fe858914862edea0bbd087114566dfc9f1ca0d01b6a82f47fbbc4b62\": container with ID starting with 6aa435c0fe858914862edea0bbd087114566dfc9f1ca0d01b6a82f47fbbc4b62 not found: ID does not exist"
Nov 02 12:38:49.224078 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:49.223937    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-xs5zn" podStartSLOduration=3.2239166 podStartE2EDuration="3.2239166s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:38:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:38:49.220120667 +0000 UTC m=+725.475519764" watchObservedRunningTime="2026-11-02 12:38:49.2239166 +0000 UTC m=+725.479315696"
Nov 02 12:38:49.225594 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:49.225526    4559 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/87b8ed31-36bf-421c-8ee0-48fbe61a65e9-provider-credentials\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:38:49.225594 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:49.225557    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7p95c\" (UniqueName: \"kubernetes.io/projected/87b8ed31-36bf-421c-8ee0-48fbe61a65e9-kube-api-access-7p95c\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:38:49.305947 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:49.305877    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-7d66d6cd49-xkqsr"]
Nov 02 12:38:49.323270 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:49.323221    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-7d66d6cd49-xkqsr"]
Nov 02 12:38:49.395976 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:49.393826    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-xs5zn"
Nov 02 12:38:49.396349 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:49.396310    4559 patch_prober.go:29] interesting pod/e2e-worker-w4-843978-openshift-default-7965f48c7f-xs5zn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Nov 02 12:38:49.396577 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:49.396532    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-xs5zn" podUID="36990652-4474-4f19-94e5-8d540cdc06a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Nov 02 12:38:49.915855 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:49.915816    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87b8ed31-36bf-421c-8ee0-48fbe61a65e9" path="/var/lib/kubelet/pods/87b8ed31-36bf-421c-8ee0-48fbe61a65e9/volumes"
Nov 02 12:38:50.393193 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:50.393156    4559 patch_prober.go:29] interesting pod/e2e-worker-w4-843978-openshift-default-7965f48c7f-xs5zn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Nov 02 12:38:50.393884 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:50.393839    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-xs5zn" podUID="36990652-4474-4f19-94e5-8d540cdc06a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Nov 02 12:38:51.395121 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:51.395072    4559 patch_prober.go:29] interesting pod/e2e-worker-w4-843978-openshift-default-7965f48c7f-xs5zn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Nov 02 12:38:51.395752 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:51.395720    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-xs5zn" podUID="36990652-4474-4f19-94e5-8d540cdc06a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Nov 02 12:38:52.404664 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:52.404089    4559 patch_prober.go:29] interesting pod/e2e-worker-w4-843978-openshift-default-7965f48c7f-xs5zn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Nov 02 12:38:52.404664 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:52.404156    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-xs5zn" podUID="36990652-4474-4f19-94e5-8d540cdc06a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Nov 02 12:38:53.401552 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:53.401516    4559 patch_prober.go:29] interesting pod/e2e-worker-w4-843978-openshift-default-7965f48c7f-xs5zn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Nov 02 12:38:53.401880 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:53.401850    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-xs5zn" podUID="36990652-4474-4f19-94e5-8d540cdc06a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Nov 02 12:38:54.147407 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:54.147377    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-843978/distinct-w4-843978-kserve-6fcf588858-4nrwx"]
Nov 02 12:38:54.148084 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:54.148063    4559 cpu_manager.go:414] "containerMap: removing container" podUID="87b8ed31-36bf-421c-8ee0-48fbe61a65e9" containerName="payload-processing"
Nov 02 12:38:54.148171 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:54.148086    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87b8ed31-36bf-421c-8ee0-48fbe61a65e9" containerName="payload-processing"
Nov 02 12:38:54.148264 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:54.148251    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="87b8ed31-36bf-421c-8ee0-48fbe61a65e9" containerName="payload-processing"
Nov 02 12:38:54.152301 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:54.152280    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-843978/distinct-w4-843978-kserve-6fcf588858-4nrwx"
Nov 02 12:38:54.171662 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:54.171618    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-843978/distinct-w4-843978-kserve-6fcf588858-4nrwx"]
Nov 02 12:38:54.182136 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:54.182112    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-843978\"/\"kube-root-ca.crt\""
Nov 02 12:38:54.197753 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:54.196748    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-843978\"/\"default-dockercfg-h844f\""
Nov 02 12:38:54.197753 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:54.197000    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-843978\"/\"distinct-w4-843978-kserve-self-signed-certs\""
Nov 02 12:38:54.197753 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:54.197055    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-843978\"/\"openshift-service-ca.crt\""
Nov 02 12:38:54.211683 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:54.211266    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82e5d483-e95b-43e6-9135-af960823427a-tmp-dir\") pod \"distinct-w4-843978-kserve-6fcf588858-4nrwx\" (UID: \"82e5d483-e95b-43e6-9135-af960823427a\") " pod="e2e-models-e2e-worker-w4-843978/distinct-w4-843978-kserve-6fcf588858-4nrwx"
Nov 02 12:38:54.211683 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:54.211329    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82e5d483-e95b-43e6-9135-af960823427a-model-cache\") pod \"distinct-w4-843978-kserve-6fcf588858-4nrwx\" (UID: \"82e5d483-e95b-43e6-9135-af960823427a\") " pod="e2e-models-e2e-worker-w4-843978/distinct-w4-843978-kserve-6fcf588858-4nrwx"
Nov 02 12:38:54.211683 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:54.211488    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/82e5d483-e95b-43e6-9135-af960823427a-dshm\") pod \"distinct-w4-843978-kserve-6fcf588858-4nrwx\" (UID: \"82e5d483-e95b-43e6-9135-af960823427a\") " pod="e2e-models-e2e-worker-w4-843978/distinct-w4-843978-kserve-6fcf588858-4nrwx"
Nov 02 12:38:54.211683 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:54.211518    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6f8l\" (UniqueName: \"kubernetes.io/projected/82e5d483-e95b-43e6-9135-af960823427a-kube-api-access-s6f8l\") pod \"distinct-w4-843978-kserve-6fcf588858-4nrwx\" (UID: \"82e5d483-e95b-43e6-9135-af960823427a\") " pod="e2e-models-e2e-worker-w4-843978/distinct-w4-843978-kserve-6fcf588858-4nrwx"
Nov 02 12:38:54.211683 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:54.211548    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82e5d483-e95b-43e6-9135-af960823427a-tls-certs\") pod \"distinct-w4-843978-kserve-6fcf588858-4nrwx\" (UID: \"82e5d483-e95b-43e6-9135-af960823427a\") " pod="e2e-models-e2e-worker-w4-843978/distinct-w4-843978-kserve-6fcf588858-4nrwx"
Nov 02 12:38:54.211683 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:54.211566    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/82e5d483-e95b-43e6-9135-af960823427a-home\") pod \"distinct-w4-843978-kserve-6fcf588858-4nrwx\" (UID: \"82e5d483-e95b-43e6-9135-af960823427a\") " pod="e2e-models-e2e-worker-w4-843978/distinct-w4-843978-kserve-6fcf588858-4nrwx"
Nov 02 12:38:54.333972 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:54.333938    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82e5d483-e95b-43e6-9135-af960823427a-model-cache\") pod \"distinct-w4-843978-kserve-6fcf588858-4nrwx\" (UID: \"82e5d483-e95b-43e6-9135-af960823427a\") " pod="e2e-models-e2e-worker-w4-843978/distinct-w4-843978-kserve-6fcf588858-4nrwx"
Nov 02 12:38:54.334231 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:54.333494    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82e5d483-e95b-43e6-9135-af960823427a-model-cache\") pod \"distinct-w4-843978-kserve-6fcf588858-4nrwx\" (UID: \"82e5d483-e95b-43e6-9135-af960823427a\") " pod="e2e-models-e2e-worker-w4-843978/distinct-w4-843978-kserve-6fcf588858-4nrwx"
Nov 02 12:38:54.341959 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:54.341926    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/82e5d483-e95b-43e6-9135-af960823427a-dshm\") pod \"distinct-w4-843978-kserve-6fcf588858-4nrwx\" (UID: \"82e5d483-e95b-43e6-9135-af960823427a\") " pod="e2e-models-e2e-worker-w4-843978/distinct-w4-843978-kserve-6fcf588858-4nrwx"
Nov 02 12:38:54.353797 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:54.353766    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6f8l\" (UniqueName: \"kubernetes.io/projected/82e5d483-e95b-43e6-9135-af960823427a-kube-api-access-s6f8l\") pod \"distinct-w4-843978-kserve-6fcf588858-4nrwx\" (UID: \"82e5d483-e95b-43e6-9135-af960823427a\") " pod="e2e-models-e2e-worker-w4-843978/distinct-w4-843978-kserve-6fcf588858-4nrwx"
Nov 02 12:38:54.354449 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:54.354425    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82e5d483-e95b-43e6-9135-af960823427a-tls-certs\") pod \"distinct-w4-843978-kserve-6fcf588858-4nrwx\" (UID: \"82e5d483-e95b-43e6-9135-af960823427a\") " pod="e2e-models-e2e-worker-w4-843978/distinct-w4-843978-kserve-6fcf588858-4nrwx"
Nov 02 12:38:54.361207 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:54.361184    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/82e5d483-e95b-43e6-9135-af960823427a-home\") pod \"distinct-w4-843978-kserve-6fcf588858-4nrwx\" (UID: \"82e5d483-e95b-43e6-9135-af960823427a\") " pod="e2e-models-e2e-worker-w4-843978/distinct-w4-843978-kserve-6fcf588858-4nrwx"
Nov 02 12:38:54.372721 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:54.361749    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82e5d483-e95b-43e6-9135-af960823427a-tmp-dir\") pod \"distinct-w4-843978-kserve-6fcf588858-4nrwx\" (UID: \"82e5d483-e95b-43e6-9135-af960823427a\") " pod="e2e-models-e2e-worker-w4-843978/distinct-w4-843978-kserve-6fcf588858-4nrwx"
Nov 02 12:38:54.403707 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:54.367962    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/82e5d483-e95b-43e6-9135-af960823427a-dshm\") pod \"distinct-w4-843978-kserve-6fcf588858-4nrwx\" (UID: \"82e5d483-e95b-43e6-9135-af960823427a\") " pod="e2e-models-e2e-worker-w4-843978/distinct-w4-843978-kserve-6fcf588858-4nrwx"
Nov 02 12:38:54.403707 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:54.361633    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/82e5d483-e95b-43e6-9135-af960823427a-home\") pod \"distinct-w4-843978-kserve-6fcf588858-4nrwx\" (UID: \"82e5d483-e95b-43e6-9135-af960823427a\") " pod="e2e-models-e2e-worker-w4-843978/distinct-w4-843978-kserve-6fcf588858-4nrwx"
Nov 02 12:38:54.403707 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:54.368227    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82e5d483-e95b-43e6-9135-af960823427a-tmp-dir\") pod \"distinct-w4-843978-kserve-6fcf588858-4nrwx\" (UID: \"82e5d483-e95b-43e6-9135-af960823427a\") " pod="e2e-models-e2e-worker-w4-843978/distinct-w4-843978-kserve-6fcf588858-4nrwx"
Nov 02 12:38:54.403707 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:54.390409    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6f8l\" (UniqueName: \"kubernetes.io/projected/82e5d483-e95b-43e6-9135-af960823427a-kube-api-access-s6f8l\") pod \"distinct-w4-843978-kserve-6fcf588858-4nrwx\" (UID: \"82e5d483-e95b-43e6-9135-af960823427a\") " pod="e2e-models-e2e-worker-w4-843978/distinct-w4-843978-kserve-6fcf588858-4nrwx"
Nov 02 12:38:54.403707 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:54.393999    4559 patch_prober.go:29] interesting pod/e2e-worker-w4-843978-openshift-default-7965f48c7f-xs5zn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Nov 02 12:38:54.403707 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:54.394073    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-xs5zn" podUID="36990652-4474-4f19-94e5-8d540cdc06a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Nov 02 12:38:54.417769 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:54.416634    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82e5d483-e95b-43e6-9135-af960823427a-tls-certs\") pod \"distinct-w4-843978-kserve-6fcf588858-4nrwx\" (UID: \"82e5d483-e95b-43e6-9135-af960823427a\") " pod="e2e-models-e2e-worker-w4-843978/distinct-w4-843978-kserve-6fcf588858-4nrwx"
Nov 02 12:38:54.465006 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:54.464967    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-843978/distinct-w4-843978-kserve-6fcf588858-4nrwx"
Nov 02 12:38:54.798705 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:54.798672    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-843978/distinct-w4-843978-kserve-6fcf588858-4nrwx"]
Nov 02 12:38:54.799274 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:38:54.799249    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod82e5d483_e95b_43e6_9135_af960823427a.slice/crio-536c5ed1a3cd5c8bf9dd14f3dbb51dc59dbb5280a187549345e1cec39a8f750c WatchSource:0}: Error finding container 536c5ed1a3cd5c8bf9dd14f3dbb51dc59dbb5280a187549345e1cec39a8f750c: Status 404 returned error can't find the container with id 536c5ed1a3cd5c8bf9dd14f3dbb51dc59dbb5280a187549345e1cec39a8f750c
Nov 02 12:38:55.188964 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:55.188928    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-843978/distinct-w4-843978-kserve-6fcf588858-4nrwx" event={"ID":"82e5d483-e95b-43e6-9135-af960823427a","Type":"ContainerStarted","Data":"d56829801a880a1e94e54324247067e9cf52a5f8e6fd688a4f4c161c95ca6f8e"}
Nov 02 12:38:55.189352 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:55.188971    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-843978/distinct-w4-843978-kserve-6fcf588858-4nrwx" event={"ID":"82e5d483-e95b-43e6-9135-af960823427a","Type":"ContainerStarted","Data":"536c5ed1a3cd5c8bf9dd14f3dbb51dc59dbb5280a187549345e1cec39a8f750c"}
Nov 02 12:38:55.233466 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:55.233419    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-843978/distinct-w4-843978-kserve-6fcf588858-4nrwx" podStartSLOduration=1.233402234 podStartE2EDuration="1.233402234s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:38:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:38:55.230983303 +0000 UTC m=+731.486382409" watchObservedRunningTime="2026-11-02 12:38:55.233402234 +0000 UTC m=+731.488801334"
Nov 02 12:38:55.401169 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:55.401138    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-xs5zn"
Nov 02 12:38:55.401610 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:55.401594    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-xs5zn"
Nov 02 12:38:55.402359 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:55.402337    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-xs5zn"
Nov 02 12:38:55.466076 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:55.466004    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-843978/distinct-w4-843978-kserve-6fcf588858-4nrwx"
Nov 02 12:38:56.446243 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:38:56.446209    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"]
Nov 02 12:39:04.465320 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:39:04.465282    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-843978/distinct-w4-843978-kserve-6fcf588858-4nrwx"
Nov 02 12:39:04.478442 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:39:04.478411    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-843978/distinct-w4-843978-kserve-6fcf588858-4nrwx"
Nov 02 12:39:05.278721 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:39:05.275023    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-843978/distinct-w4-843978-kserve-6fcf588858-4nrwx"
Nov 02 12:39:14.830083 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:39:14.830044    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"]
Nov 02 12:39:15.084680 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:39:15.084573    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqcmj" containerName="extractor"
Nov 02 12:39:30.513851 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:39:30.513771    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"]
Nov 02 12:39:37.678940 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:39:37.678900    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-61e341/distinct-w1-61e341-kserve-787cd566f4-cjj8s"]
Nov 02 12:39:37.682966 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:39:37.682934    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-61e341/distinct-w1-61e341-kserve-787cd566f4-cjj8s"
Nov 02 12:39:37.702478 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:39:37.702448    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-61e341/distinct-w1-61e341-kserve-787cd566f4-cjj8s"]
Nov 02 12:39:37.711921 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:39:37.711885    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-61e341\"/\"distinct-w1-61e341-kserve-self-signed-certs\""
Nov 02 12:39:37.752828 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:39:37.752791    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6886ce28-5800-402d-b893-f92109e289a9-dshm\") pod \"distinct-w1-61e341-kserve-787cd566f4-cjj8s\" (UID: \"6886ce28-5800-402d-b893-f92109e289a9\") " pod="e2e-models-e2e-worker-w1-61e341/distinct-w1-61e341-kserve-787cd566f4-cjj8s"
Nov 02 12:39:37.753018 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:39:37.752853    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6886ce28-5800-402d-b893-f92109e289a9-model-cache\") pod \"distinct-w1-61e341-kserve-787cd566f4-cjj8s\" (UID: \"6886ce28-5800-402d-b893-f92109e289a9\") " pod="e2e-models-e2e-worker-w1-61e341/distinct-w1-61e341-kserve-787cd566f4-cjj8s"
Nov 02 12:39:37.753018 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:39:37.752933    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gj5mj\" (UniqueName: \"kubernetes.io/projected/6886ce28-5800-402d-b893-f92109e289a9-kube-api-access-gj5mj\") pod \"distinct-w1-61e341-kserve-787cd566f4-cjj8s\" (UID: \"6886ce28-5800-402d-b893-f92109e289a9\") " pod="e2e-models-e2e-worker-w1-61e341/distinct-w1-61e341-kserve-787cd566f4-cjj8s"
Nov 02 12:39:37.753018 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:39:37.752965    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6886ce28-5800-402d-b893-f92109e289a9-home\") pod \"distinct-w1-61e341-kserve-787cd566f4-cjj8s\" (UID: \"6886ce28-5800-402d-b893-f92109e289a9\") " pod="e2e-models-e2e-worker-w1-61e341/distinct-w1-61e341-kserve-787cd566f4-cjj8s"
Nov 02 12:39:37.753197 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:39:37.753076    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6886ce28-5800-402d-b893-f92109e289a9-tls-certs\") pod \"distinct-w1-61e341-kserve-787cd566f4-cjj8s\" (UID: \"6886ce28-5800-402d-b893-f92109e289a9\") " pod="e2e-models-e2e-worker-w1-61e341/distinct-w1-61e341-kserve-787cd566f4-cjj8s"
Nov 02 12:39:37.753197 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:39:37.753133    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6886ce28-5800-402d-b893-f92109e289a9-tmp-dir\") pod \"distinct-w1-61e341-kserve-787cd566f4-cjj8s\" (UID: \"6886ce28-5800-402d-b893-f92109e289a9\") " pod="e2e-models-e2e-worker-w1-61e341/distinct-w1-61e341-kserve-787cd566f4-cjj8s"
Nov 02 12:39:37.854268 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:39:37.854235    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6886ce28-5800-402d-b893-f92109e289a9-tls-certs\") pod \"distinct-w1-61e341-kserve-787cd566f4-cjj8s\" (UID: \"6886ce28-5800-402d-b893-f92109e289a9\") " pod="e2e-models-e2e-worker-w1-61e341/distinct-w1-61e341-kserve-787cd566f4-cjj8s"
Nov 02 12:39:37.854268 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:39:37.854274    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6886ce28-5800-402d-b893-f92109e289a9-tmp-dir\") pod \"distinct-w1-61e341-kserve-787cd566f4-cjj8s\" (UID: \"6886ce28-5800-402d-b893-f92109e289a9\") " pod="e2e-models-e2e-worker-w1-61e341/distinct-w1-61e341-kserve-787cd566f4-cjj8s"
Nov 02 12:39:37.854503 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:39:37.854328    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6886ce28-5800-402d-b893-f92109e289a9-dshm\") pod \"distinct-w1-61e341-kserve-787cd566f4-cjj8s\" (UID: \"6886ce28-5800-402d-b893-f92109e289a9\") " pod="e2e-models-e2e-worker-w1-61e341/distinct-w1-61e341-kserve-787cd566f4-cjj8s"
Nov 02 12:39:37.854503 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:39:37.854369    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6886ce28-5800-402d-b893-f92109e289a9-model-cache\") pod \"distinct-w1-61e341-kserve-787cd566f4-cjj8s\" (UID: \"6886ce28-5800-402d-b893-f92109e289a9\") " pod="e2e-models-e2e-worker-w1-61e341/distinct-w1-61e341-kserve-787cd566f4-cjj8s"
Nov 02 12:39:37.854503 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:39:37.854411    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gj5mj\" (UniqueName: \"kubernetes.io/projected/6886ce28-5800-402d-b893-f92109e289a9-kube-api-access-gj5mj\") pod \"distinct-w1-61e341-kserve-787cd566f4-cjj8s\" (UID: \"6886ce28-5800-402d-b893-f92109e289a9\") " pod="e2e-models-e2e-worker-w1-61e341/distinct-w1-61e341-kserve-787cd566f4-cjj8s"
Nov 02 12:39:37.854503 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:39:37.854450    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6886ce28-5800-402d-b893-f92109e289a9-home\") pod \"distinct-w1-61e341-kserve-787cd566f4-cjj8s\" (UID: \"6886ce28-5800-402d-b893-f92109e289a9\") " pod="e2e-models-e2e-worker-w1-61e341/distinct-w1-61e341-kserve-787cd566f4-cjj8s"
Nov 02 12:39:37.854886 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:39:37.854849    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6886ce28-5800-402d-b893-f92109e289a9-model-cache\") pod \"distinct-w1-61e341-kserve-787cd566f4-cjj8s\" (UID: \"6886ce28-5800-402d-b893-f92109e289a9\") " pod="e2e-models-e2e-worker-w1-61e341/distinct-w1-61e341-kserve-787cd566f4-cjj8s"
Nov 02 12:39:37.855021 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:39:37.854844    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6886ce28-5800-402d-b893-f92109e289a9-tmp-dir\") pod \"distinct-w1-61e341-kserve-787cd566f4-cjj8s\" (UID: \"6886ce28-5800-402d-b893-f92109e289a9\") " pod="e2e-models-e2e-worker-w1-61e341/distinct-w1-61e341-kserve-787cd566f4-cjj8s"
Nov 02 12:39:37.855021 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:39:37.854906    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6886ce28-5800-402d-b893-f92109e289a9-home\") pod \"distinct-w1-61e341-kserve-787cd566f4-cjj8s\" (UID: \"6886ce28-5800-402d-b893-f92109e289a9\") " pod="e2e-models-e2e-worker-w1-61e341/distinct-w1-61e341-kserve-787cd566f4-cjj8s"
Nov 02 12:39:37.856729 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:39:37.856664    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6886ce28-5800-402d-b893-f92109e289a9-dshm\") pod \"distinct-w1-61e341-kserve-787cd566f4-cjj8s\" (UID: \"6886ce28-5800-402d-b893-f92109e289a9\") " pod="e2e-models-e2e-worker-w1-61e341/distinct-w1-61e341-kserve-787cd566f4-cjj8s"
Nov 02 12:39:37.857030 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:39:37.857010    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6886ce28-5800-402d-b893-f92109e289a9-tls-certs\") pod \"distinct-w1-61e341-kserve-787cd566f4-cjj8s\" (UID: \"6886ce28-5800-402d-b893-f92109e289a9\") " pod="e2e-models-e2e-worker-w1-61e341/distinct-w1-61e341-kserve-787cd566f4-cjj8s"
Nov 02 12:39:37.898410 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:39:37.898378    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gj5mj\" (UniqueName: \"kubernetes.io/projected/6886ce28-5800-402d-b893-f92109e289a9-kube-api-access-gj5mj\") pod \"distinct-w1-61e341-kserve-787cd566f4-cjj8s\" (UID: \"6886ce28-5800-402d-b893-f92109e289a9\") " pod="e2e-models-e2e-worker-w1-61e341/distinct-w1-61e341-kserve-787cd566f4-cjj8s"
Nov 02 12:39:38.000304 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:39:38.000204    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-61e341/distinct-w1-61e341-kserve-787cd566f4-cjj8s"
Nov 02 12:39:38.298630 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:39:38.298600    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-61e341/distinct-w1-61e341-kserve-787cd566f4-cjj8s"]
Nov 02 12:39:38.298890 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:39:38.298865    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6886ce28_5800_402d_b893_f92109e289a9.slice/crio-bae6e6622784432b8d0a25e4d00141e64ea18e6063492e5c0f4f7e3216467763 WatchSource:0}: Error finding container bae6e6622784432b8d0a25e4d00141e64ea18e6063492e5c0f4f7e3216467763: Status 404 returned error can't find the container with id bae6e6622784432b8d0a25e4d00141e64ea18e6063492e5c0f4f7e3216467763
Nov 02 12:39:38.544450 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:39:38.544413    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-61e341/distinct-w1-61e341-kserve-787cd566f4-cjj8s" event={"ID":"6886ce28-5800-402d-b893-f92109e289a9","Type":"ContainerStarted","Data":"bf5d6ed86bcb2065ce47c40320410f0302766b65c3e6f92ee835402075446aa1"}
Nov 02 12:39:38.544677 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:39:38.544465    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-61e341/distinct-w1-61e341-kserve-787cd566f4-cjj8s" event={"ID":"6886ce28-5800-402d-b893-f92109e289a9","Type":"ContainerStarted","Data":"bae6e6622784432b8d0a25e4d00141e64ea18e6063492e5c0f4f7e3216467763"}
Nov 02 12:39:38.589837 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:39:38.589741    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-61e341/distinct-w1-61e341-kserve-787cd566f4-cjj8s" podStartSLOduration=1.589726218 podStartE2EDuration="1.589726218s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:39:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:39:38.588331276 +0000 UTC m=+774.843730370" watchObservedRunningTime="2026-11-02 12:39:38.589726218 +0000 UTC m=+774.845125313"
Nov 02 12:39:39.000391 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:39:39.000343    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-61e341/distinct-w1-61e341-kserve-787cd566f4-cjj8s"
Nov 02 12:39:42.794751 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:39:42.794718    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-nrntd"]
Nov 02 12:39:42.798819 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:39:42.798802    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-nrntd"
Nov 02 12:39:42.821652 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:39:42.821619    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-nrntd"]
Nov 02 12:39:42.914950 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:39:42.914916    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35f5e006-f020-4166-ab03-aedc1fbf2ce4-workload-socket\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-nrntd\" (UID: \"35f5e006-f020-4166-ab03-aedc1fbf2ce4\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-nrntd"
Nov 02 12:39:42.914950 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:39:42.914955    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35f5e006-f020-4166-ab03-aedc1fbf2ce4-istio-data\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-nrntd\" (UID: \"35f5e006-f020-4166-ab03-aedc1fbf2ce4\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-nrntd"
Nov 02 12:39:42.915184 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:39:42.914988    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35f5e006-f020-4166-ab03-aedc1fbf2ce4-credential-socket\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-nrntd\" (UID: \"35f5e006-f020-4166-ab03-aedc1fbf2ce4\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-nrntd"
Nov 02 12:39:42.915184 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:39:42.915053    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35f5e006-f020-4166-ab03-aedc1fbf2ce4-istio-token\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-nrntd\" (UID: \"35f5e006-f020-4166-ab03-aedc1fbf2ce4\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-nrntd"
Nov 02 12:39:42.915184 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:39:42.915129    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35f5e006-f020-4166-ab03-aedc1fbf2ce4-istiod-ca-cert\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-nrntd\" (UID: \"35f5e006-f020-4166-ab03-aedc1fbf2ce4\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-nrntd"
Nov 02 12:39:42.915184 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:39:42.915173    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35f5e006-f020-4166-ab03-aedc1fbf2ce4-workload-certs\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-nrntd\" (UID: \"35f5e006-f020-4166-ab03-aedc1fbf2ce4\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-nrntd"
Nov 02 12:39:42.915400 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:39:42.915327    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tk6p\" (UniqueName: \"kubernetes.io/projected/35f5e006-f020-4166-ab03-aedc1fbf2ce4-kube-api-access-9tk6p\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-nrntd\" (UID: \"35f5e006-f020-4166-ab03-aedc1fbf2ce4\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-nrntd"
Nov 02 12:39:42.915400 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:39:42.915388    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35f5e006-f020-4166-ab03-aedc1fbf2ce4-istio-podinfo\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-nrntd\" (UID: \"35f5e006-f020-4166-ab03-aedc1fbf2ce4\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-nrntd"
Nov 02 12:39:42.915504 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:39:42.915430    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35f5e006-f020-4166-ab03-aedc1fbf2ce4-istio-envoy\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-nrntd\" (UID: \"35f5e006-f020-4166-ab03-aedc1fbf2ce4\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-nrntd"
Nov 02 12:39:43.016997 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:39:43.016966    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tk6p\" (UniqueName: \"kubernetes.io/projected/35f5e006-f020-4166-ab03-aedc1fbf2ce4-kube-api-access-9tk6p\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-nrntd\" (UID: \"35f5e006-f020-4166-ab03-aedc1fbf2ce4\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-nrntd"
Nov 02 12:39:43.017157 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:39:43.017024    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35f5e006-f020-4166-ab03-aedc1fbf2ce4-istio-podinfo\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-nrntd\" (UID: \"35f5e006-f020-4166-ab03-aedc1fbf2ce4\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-nrntd"
Nov 02 12:39:43.017157 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:39:43.017048    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35f5e006-f020-4166-ab03-aedc1fbf2ce4-istio-envoy\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-nrntd\" (UID: \"35f5e006-f020-4166-ab03-aedc1fbf2ce4\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-nrntd"
Nov 02 12:39:43.017157 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:39:43.017082    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35f5e006-f020-4166-ab03-aedc1fbf2ce4-workload-socket\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-nrntd\" (UID: \"35f5e006-f020-4166-ab03-aedc1fbf2ce4\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-nrntd"
Nov 02 12:39:43.017157 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:39:43.017105    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35f5e006-f020-4166-ab03-aedc1fbf2ce4-istio-data\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-nrntd\" (UID: \"35f5e006-f020-4166-ab03-aedc1fbf2ce4\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-nrntd"
Nov 02 12:39:43.017394 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:39:43.017179    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35f5e006-f020-4166-ab03-aedc1fbf2ce4-credential-socket\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-nrntd\" (UID: \"35f5e006-f020-4166-ab03-aedc1fbf2ce4\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-nrntd"
Nov 02 12:39:43.017394 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:39:43.017218    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35f5e006-f020-4166-ab03-aedc1fbf2ce4-istio-token\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-nrntd\" (UID: \"35f5e006-f020-4166-ab03-aedc1fbf2ce4\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-nrntd"
Nov 02 12:39:43.017394 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:39:43.017284    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35f5e006-f020-4166-ab03-aedc1fbf2ce4-istiod-ca-cert\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-nrntd\" (UID: \"35f5e006-f020-4166-ab03-aedc1fbf2ce4\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-nrntd"
Nov 02 12:39:43.017394 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:39:43.017357    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35f5e006-f020-4166-ab03-aedc1fbf2ce4-workload-certs\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-nrntd\" (UID: \"35f5e006-f020-4166-ab03-aedc1fbf2ce4\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-nrntd"
Nov 02 12:39:43.017617 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:39:43.017596    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35f5e006-f020-4166-ab03-aedc1fbf2ce4-workload-socket\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-nrntd\" (UID: \"35f5e006-f020-4166-ab03-aedc1fbf2ce4\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-nrntd"
Nov 02 12:39:43.017736 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:39:43.017711    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35f5e006-f020-4166-ab03-aedc1fbf2ce4-istio-data\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-nrntd\" (UID: \"35f5e006-f020-4166-ab03-aedc1fbf2ce4\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-nrntd"
Nov 02 12:39:43.017910 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:39:43.017888    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35f5e006-f020-4166-ab03-aedc1fbf2ce4-workload-certs\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-nrntd\" (UID: \"35f5e006-f020-4166-ab03-aedc1fbf2ce4\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-nrntd"
Nov 02 12:39:43.017995 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:39:43.017916    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35f5e006-f020-4166-ab03-aedc1fbf2ce4-credential-socket\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-nrntd\" (UID: \"35f5e006-f020-4166-ab03-aedc1fbf2ce4\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-nrntd"
Nov 02 12:39:43.018378 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:39:43.018353    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35f5e006-f020-4166-ab03-aedc1fbf2ce4-istiod-ca-cert\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-nrntd\" (UID: \"35f5e006-f020-4166-ab03-aedc1fbf2ce4\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-nrntd"
Nov 02 12:39:43.019517 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:39:43.019498    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35f5e006-f020-4166-ab03-aedc1fbf2ce4-istio-envoy\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-nrntd\" (UID: \"35f5e006-f020-4166-ab03-aedc1fbf2ce4\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-nrntd"
Nov 02 12:39:43.019677 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:39:43.019635    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35f5e006-f020-4166-ab03-aedc1fbf2ce4-istio-podinfo\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-nrntd\" (UID: \"35f5e006-f020-4166-ab03-aedc1fbf2ce4\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-nrntd"
Nov 02 12:39:43.040742 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:39:43.040721    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35f5e006-f020-4166-ab03-aedc1fbf2ce4-istio-token\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-nrntd\" (UID: \"35f5e006-f020-4166-ab03-aedc1fbf2ce4\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-nrntd"
Nov 02 12:39:43.040845 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:39:43.040811    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tk6p\" (UniqueName: \"kubernetes.io/projected/35f5e006-f020-4166-ab03-aedc1fbf2ce4-kube-api-access-9tk6p\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-nrntd\" (UID: \"35f5e006-f020-4166-ab03-aedc1fbf2ce4\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-nrntd"
Nov 02 12:39:43.119170 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:39:43.114848    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-nrntd"
Nov 02 12:39:43.439492 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:39:43.439453    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35f5e006_f020_4166_ab03_aedc1fbf2ce4.slice/crio-f7a3cf0cc5613dce317c0eb15e6862604e6c3e23563d98b1f184063e6b80888c WatchSource:0}: Error finding container f7a3cf0cc5613dce317c0eb15e6862604e6c3e23563d98b1f184063e6b80888c: Status 404 returned error can't find the container with id f7a3cf0cc5613dce317c0eb15e6862604e6c3e23563d98b1f184063e6b80888c
Nov 02 12:39:43.439749 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:39:43.439728    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-nrntd"]
Nov 02 12:39:43.441681 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:39:43.441649    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:39:43.441765 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:39:43.441710    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:39:43.441765 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:39:43.441743    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:39:43.612560 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:39:43.612524    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-nrntd" event={"ID":"35f5e006-f020-4166-ab03-aedc1fbf2ce4","Type":"ContainerStarted","Data":"3c630d665d2dbfd303d7ab36682f3d1f8a42ce8c41ac079f7978833d3799d53b"}
Nov 02 12:39:43.612740 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:39:43.612577    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-nrntd" event={"ID":"35f5e006-f020-4166-ab03-aedc1fbf2ce4","Type":"ContainerStarted","Data":"f7a3cf0cc5613dce317c0eb15e6862604e6c3e23563d98b1f184063e6b80888c"}
Nov 02 12:39:43.660673 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:39:43.660600    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-nrntd" podStartSLOduration=1.6605855680000001 podStartE2EDuration="1.660585568s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:39:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:39:43.657610555 +0000 UTC m=+779.913009651" watchObservedRunningTime="2026-11-02 12:39:43.660585568 +0000 UTC m=+779.915984666"
Nov 02 12:39:44.115979 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:39:44.115938    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-nrntd"
Nov 02 12:39:44.117647 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:39:44.117604    4559 patch_prober.go:29] interesting pod/e2e-worker-w3-068756-openshift-default-84f69c9cf5-nrntd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Nov 02 12:39:44.117779 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:39:44.117688    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-nrntd" podUID="35f5e006-f020-4166-ab03-aedc1fbf2ce4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Nov 02 12:39:45.112392 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:39:45.112352    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqcmj" containerName="extractor"
Nov 02 12:39:45.115908 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:39:45.115882    4559 patch_prober.go:29] interesting pod/e2e-worker-w3-068756-openshift-default-84f69c9cf5-nrntd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Nov 02 12:39:45.116040 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:39:45.115937    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-nrntd" podUID="35f5e006-f020-4166-ab03-aedc1fbf2ce4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Nov 02 12:39:46.115706 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:39:46.115666    4559 patch_prober.go:29] interesting pod/e2e-worker-w3-068756-openshift-default-84f69c9cf5-nrntd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Nov 02 12:39:46.115918 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:39:46.115723    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-nrntd" podUID="35f5e006-f020-4166-ab03-aedc1fbf2ce4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Nov 02 12:39:47.115982 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:39:47.115943    4559 patch_prober.go:29] interesting pod/e2e-worker-w3-068756-openshift-default-84f69c9cf5-nrntd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Nov 02 12:39:47.116394 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:39:47.116001    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-nrntd" podUID="35f5e006-f020-4166-ab03-aedc1fbf2ce4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Nov 02 12:39:48.002634 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:39:48.002599    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-61e341/distinct-w1-61e341-kserve-787cd566f4-cjj8s"
Nov 02 12:39:48.022263 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:39:48.022223    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-61e341/distinct-w1-61e341-kserve-787cd566f4-cjj8s"
Nov 02 12:39:48.054294 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:39:48.054263    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"]
Nov 02 12:39:48.115537 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:39:48.115500    4559 patch_prober.go:29] interesting pod/e2e-worker-w3-068756-openshift-default-84f69c9cf5-nrntd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Nov 02 12:39:48.115719 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:39:48.115559    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-nrntd" podUID="35f5e006-f020-4166-ab03-aedc1fbf2ce4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Nov 02 12:39:48.946513 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:39:48.946481    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-61e341/distinct-w1-61e341-kserve-787cd566f4-cjj8s"
Nov 02 12:39:49.115731 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:39:49.115688    4559 patch_prober.go:29] interesting pod/e2e-worker-w3-068756-openshift-default-84f69c9cf5-nrntd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Nov 02 12:39:49.115925 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:39:49.115761    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-nrntd" podUID="35f5e006-f020-4166-ab03-aedc1fbf2ce4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Nov 02 12:39:50.117154 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:39:50.117117    4559 patch_prober.go:29] interesting pod/e2e-worker-w3-068756-openshift-default-84f69c9cf5-nrntd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Nov 02 12:39:50.117828 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:39:50.117778    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-nrntd" podUID="35f5e006-f020-4166-ab03-aedc1fbf2ce4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Nov 02 12:39:51.840727 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:39:51.840696    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-nrntd"
Nov 02 12:39:51.841142 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:39:51.840984    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-nrntd"
Nov 02 12:39:51.842806 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:39:51.842785    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-nrntd"
Nov 02 12:40:01.785682 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:01.782449    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-d7jbc"]
Nov 02 12:40:01.787771 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:01.787749    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-d7jbc"
Nov 02 12:40:01.809969 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:01.809933    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-d7jbc"]
Nov 02 12:40:01.880704 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:01.880625    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39c7e6bf-f8ae-4913-aa8a-5e8325a379d4-workload-socket\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-d7jbc\" (UID: \"39c7e6bf-f8ae-4913-aa8a-5e8325a379d4\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-d7jbc"
Nov 02 12:40:01.880897 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:01.880771    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6bxz\" (UniqueName: \"kubernetes.io/projected/39c7e6bf-f8ae-4913-aa8a-5e8325a379d4-kube-api-access-d6bxz\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-d7jbc\" (UID: \"39c7e6bf-f8ae-4913-aa8a-5e8325a379d4\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-d7jbc"
Nov 02 12:40:01.880897 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:01.880805    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39c7e6bf-f8ae-4913-aa8a-5e8325a379d4-workload-certs\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-d7jbc\" (UID: \"39c7e6bf-f8ae-4913-aa8a-5e8325a379d4\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-d7jbc"
Nov 02 12:40:01.881010 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:01.880895    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39c7e6bf-f8ae-4913-aa8a-5e8325a379d4-istio-podinfo\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-d7jbc\" (UID: \"39c7e6bf-f8ae-4913-aa8a-5e8325a379d4\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-d7jbc"
Nov 02 12:40:01.881010 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:01.880935    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39c7e6bf-f8ae-4913-aa8a-5e8325a379d4-istio-envoy\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-d7jbc\" (UID: \"39c7e6bf-f8ae-4913-aa8a-5e8325a379d4\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-d7jbc"
Nov 02 12:40:01.881010 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:01.880960    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39c7e6bf-f8ae-4913-aa8a-5e8325a379d4-istio-data\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-d7jbc\" (UID: \"39c7e6bf-f8ae-4913-aa8a-5e8325a379d4\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-d7jbc"
Nov 02 12:40:01.881165 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:01.881012    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39c7e6bf-f8ae-4913-aa8a-5e8325a379d4-credential-socket\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-d7jbc\" (UID: \"39c7e6bf-f8ae-4913-aa8a-5e8325a379d4\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-d7jbc"
Nov 02 12:40:01.881165 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:01.881052    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39c7e6bf-f8ae-4913-aa8a-5e8325a379d4-istio-token\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-d7jbc\" (UID: \"39c7e6bf-f8ae-4913-aa8a-5e8325a379d4\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-d7jbc"
Nov 02 12:40:01.881165 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:01.881082    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39c7e6bf-f8ae-4913-aa8a-5e8325a379d4-istiod-ca-cert\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-d7jbc\" (UID: \"39c7e6bf-f8ae-4913-aa8a-5e8325a379d4\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-d7jbc"
Nov 02 12:40:01.981910 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:01.981877    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39c7e6bf-f8ae-4913-aa8a-5e8325a379d4-istio-podinfo\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-d7jbc\" (UID: \"39c7e6bf-f8ae-4913-aa8a-5e8325a379d4\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-d7jbc"
Nov 02 12:40:01.982080 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:01.981929    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39c7e6bf-f8ae-4913-aa8a-5e8325a379d4-istio-envoy\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-d7jbc\" (UID: \"39c7e6bf-f8ae-4913-aa8a-5e8325a379d4\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-d7jbc"
Nov 02 12:40:01.982080 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:01.981965    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39c7e6bf-f8ae-4913-aa8a-5e8325a379d4-istio-data\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-d7jbc\" (UID: \"39c7e6bf-f8ae-4913-aa8a-5e8325a379d4\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-d7jbc"
Nov 02 12:40:01.982080 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:01.981992    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39c7e6bf-f8ae-4913-aa8a-5e8325a379d4-credential-socket\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-d7jbc\" (UID: \"39c7e6bf-f8ae-4913-aa8a-5e8325a379d4\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-d7jbc"
Nov 02 12:40:01.982080 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:01.982048    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39c7e6bf-f8ae-4913-aa8a-5e8325a379d4-istio-token\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-d7jbc\" (UID: \"39c7e6bf-f8ae-4913-aa8a-5e8325a379d4\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-d7jbc"
Nov 02 12:40:01.982283 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:01.982095    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39c7e6bf-f8ae-4913-aa8a-5e8325a379d4-istiod-ca-cert\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-d7jbc\" (UID: \"39c7e6bf-f8ae-4913-aa8a-5e8325a379d4\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-d7jbc"
Nov 02 12:40:01.982283 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:01.982188    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39c7e6bf-f8ae-4913-aa8a-5e8325a379d4-workload-socket\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-d7jbc\" (UID: \"39c7e6bf-f8ae-4913-aa8a-5e8325a379d4\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-d7jbc"
Nov 02 12:40:01.982283 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:01.982262    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6bxz\" (UniqueName: \"kubernetes.io/projected/39c7e6bf-f8ae-4913-aa8a-5e8325a379d4-kube-api-access-d6bxz\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-d7jbc\" (UID: \"39c7e6bf-f8ae-4913-aa8a-5e8325a379d4\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-d7jbc"
Nov 02 12:40:01.982437 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:01.982299    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39c7e6bf-f8ae-4913-aa8a-5e8325a379d4-workload-certs\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-d7jbc\" (UID: \"39c7e6bf-f8ae-4913-aa8a-5e8325a379d4\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-d7jbc"
Nov 02 12:40:01.982495 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:01.982466    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39c7e6bf-f8ae-4913-aa8a-5e8325a379d4-credential-socket\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-d7jbc\" (UID: \"39c7e6bf-f8ae-4913-aa8a-5e8325a379d4\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-d7jbc"
Nov 02 12:40:01.982553 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:01.982488    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39c7e6bf-f8ae-4913-aa8a-5e8325a379d4-istio-data\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-d7jbc\" (UID: \"39c7e6bf-f8ae-4913-aa8a-5e8325a379d4\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-d7jbc"
Nov 02 12:40:01.982724 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:01.982699    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39c7e6bf-f8ae-4913-aa8a-5e8325a379d4-workload-certs\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-d7jbc\" (UID: \"39c7e6bf-f8ae-4913-aa8a-5e8325a379d4\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-d7jbc"
Nov 02 12:40:01.982822 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:01.982765    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39c7e6bf-f8ae-4913-aa8a-5e8325a379d4-workload-socket\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-d7jbc\" (UID: \"39c7e6bf-f8ae-4913-aa8a-5e8325a379d4\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-d7jbc"
Nov 02 12:40:01.982962 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:01.982936    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39c7e6bf-f8ae-4913-aa8a-5e8325a379d4-istiod-ca-cert\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-d7jbc\" (UID: \"39c7e6bf-f8ae-4913-aa8a-5e8325a379d4\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-d7jbc"
Nov 02 12:40:01.984340 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:01.984321    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39c7e6bf-f8ae-4913-aa8a-5e8325a379d4-istio-envoy\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-d7jbc\" (UID: \"39c7e6bf-f8ae-4913-aa8a-5e8325a379d4\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-d7jbc"
Nov 02 12:40:01.984486 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:01.984470    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39c7e6bf-f8ae-4913-aa8a-5e8325a379d4-istio-podinfo\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-d7jbc\" (UID: \"39c7e6bf-f8ae-4913-aa8a-5e8325a379d4\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-d7jbc"
Nov 02 12:40:02.005963 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:02.005938    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39c7e6bf-f8ae-4913-aa8a-5e8325a379d4-istio-token\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-d7jbc\" (UID: \"39c7e6bf-f8ae-4913-aa8a-5e8325a379d4\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-d7jbc"
Nov 02 12:40:02.006088 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:02.006071    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6bxz\" (UniqueName: \"kubernetes.io/projected/39c7e6bf-f8ae-4913-aa8a-5e8325a379d4-kube-api-access-d6bxz\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-d7jbc\" (UID: \"39c7e6bf-f8ae-4913-aa8a-5e8325a379d4\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-d7jbc"
Nov 02 12:40:02.121709 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:02.112011    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-d7jbc"
Nov 02 12:40:02.512225 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:02.509455    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-d7jbc"]
Nov 02 12:40:02.521496 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:02.521449    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:40:02.521758 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:02.521733    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:40:02.521917 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:02.521898    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:40:02.879721 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:02.878777    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-d7jbc" event={"ID":"39c7e6bf-f8ae-4913-aa8a-5e8325a379d4","Type":"ContainerStarted","Data":"81e74f357cf88f7e21ad34989b7f895f1b557fb3fa711c1e63cdf5ac9859b2e3"}
Nov 02 12:40:02.879721 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:02.878854    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-d7jbc" event={"ID":"39c7e6bf-f8ae-4913-aa8a-5e8325a379d4","Type":"ContainerStarted","Data":"b8dbb35b0062baf75b1c50b32b9bfac7a534dee9cf6ed09268fcae974dff5a7f"}
Nov 02 12:40:03.123665 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:03.121633    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-d7jbc"
Nov 02 12:40:03.123665 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:03.121789    4559 patch_prober.go:29] interesting pod/e2e-worker-w4-843978-openshift-default-7965f48c7f-d7jbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Nov 02 12:40:03.123665 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:03.121846    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-d7jbc" podUID="39c7e6bf-f8ae-4913-aa8a-5e8325a379d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Nov 02 12:40:04.113954 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:04.113498    4559 patch_prober.go:29] interesting pod/e2e-worker-w4-843978-openshift-default-7965f48c7f-d7jbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Nov 02 12:40:04.113954 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:04.113578    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-d7jbc" podUID="39c7e6bf-f8ae-4913-aa8a-5e8325a379d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Nov 02 12:40:05.112831 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:05.112794    4559 patch_prober.go:29] interesting pod/e2e-worker-w4-843978-openshift-default-7965f48c7f-d7jbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Nov 02 12:40:05.113013 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:05.112847    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-d7jbc" podUID="39c7e6bf-f8ae-4913-aa8a-5e8325a379d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Nov 02 12:40:06.113409 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:06.113375    4559 patch_prober.go:29] interesting pod/e2e-worker-w4-843978-openshift-default-7965f48c7f-d7jbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Nov 02 12:40:06.113854 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:06.113428    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-d7jbc" podUID="39c7e6bf-f8ae-4913-aa8a-5e8325a379d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Nov 02 12:40:07.112873 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:07.112831    4559 patch_prober.go:29] interesting pod/e2e-worker-w4-843978-openshift-default-7965f48c7f-d7jbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Nov 02 12:40:07.113106 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:07.112892    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-d7jbc" podUID="39c7e6bf-f8ae-4913-aa8a-5e8325a379d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Nov 02 12:40:08.113411 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:08.113371    4559 patch_prober.go:29] interesting pod/e2e-worker-w4-843978-openshift-default-7965f48c7f-d7jbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Nov 02 12:40:08.113863 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:08.113432    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-d7jbc" podUID="39c7e6bf-f8ae-4913-aa8a-5e8325a379d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Nov 02 12:40:09.113745 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:09.113707    4559 patch_prober.go:29] interesting pod/e2e-worker-w4-843978-openshift-default-7965f48c7f-d7jbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Nov 02 12:40:09.114257 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:09.113777    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-d7jbc" podUID="39c7e6bf-f8ae-4913-aa8a-5e8325a379d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Nov 02 12:40:10.456234 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:10.456205    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-d7jbc"
Nov 02 12:40:10.456725 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:10.456620    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-d7jbc"
Nov 02 12:40:10.457714 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:10.457694    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-d7jbc"
Nov 02 12:40:10.518244 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:10.518190    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-d7jbc" podStartSLOduration=9.518176535 podStartE2EDuration="9.518176535s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:40:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:40:02.937162563 +0000 UTC m=+799.192561660" watchObservedRunningTime="2026-11-02 12:40:10.518176535 +0000 UTC m=+806.773575630"
Nov 02 12:40:11.145159 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:11.145124    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-61e341/distinct-2-w1-61e341-kserve-954cf956f-n7jmg"]
Nov 02 12:40:11.150169 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:11.150139    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-61e341/distinct-2-w1-61e341-kserve-954cf956f-n7jmg"
Nov 02 12:40:11.168771 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:11.168741    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-61e341/distinct-2-w1-61e341-kserve-954cf956f-n7jmg"]
Nov 02 12:40:11.178251 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:11.178226    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-61e341\"/\"distinct-2-w1-61e341-kserve-self-signed-certs\""
Nov 02 12:40:11.298056 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:11.298013    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7x74\" (UniqueName: \"kubernetes.io/projected/c5e1a5f8-97b4-469d-be9a-eae7ca995283-kube-api-access-h7x74\") pod \"distinct-2-w1-61e341-kserve-954cf956f-n7jmg\" (UID: \"c5e1a5f8-97b4-469d-be9a-eae7ca995283\") " pod="e2e-models-e2e-worker-w1-61e341/distinct-2-w1-61e341-kserve-954cf956f-n7jmg"
Nov 02 12:40:11.298056 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:11.298058    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c5e1a5f8-97b4-469d-be9a-eae7ca995283-home\") pod \"distinct-2-w1-61e341-kserve-954cf956f-n7jmg\" (UID: \"c5e1a5f8-97b4-469d-be9a-eae7ca995283\") " pod="e2e-models-e2e-worker-w1-61e341/distinct-2-w1-61e341-kserve-954cf956f-n7jmg"
Nov 02 12:40:11.298266 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:11.298221    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c5e1a5f8-97b4-469d-be9a-eae7ca995283-tls-certs\") pod \"distinct-2-w1-61e341-kserve-954cf956f-n7jmg\" (UID: \"c5e1a5f8-97b4-469d-be9a-eae7ca995283\") " pod="e2e-models-e2e-worker-w1-61e341/distinct-2-w1-61e341-kserve-954cf956f-n7jmg"
Nov 02 12:40:11.298362 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:11.298342    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c5e1a5f8-97b4-469d-be9a-eae7ca995283-dshm\") pod \"distinct-2-w1-61e341-kserve-954cf956f-n7jmg\" (UID: \"c5e1a5f8-97b4-469d-be9a-eae7ca995283\") " pod="e2e-models-e2e-worker-w1-61e341/distinct-2-w1-61e341-kserve-954cf956f-n7jmg"
Nov 02 12:40:11.298404 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:11.298369    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c5e1a5f8-97b4-469d-be9a-eae7ca995283-model-cache\") pod \"distinct-2-w1-61e341-kserve-954cf956f-n7jmg\" (UID: \"c5e1a5f8-97b4-469d-be9a-eae7ca995283\") " pod="e2e-models-e2e-worker-w1-61e341/distinct-2-w1-61e341-kserve-954cf956f-n7jmg"
Nov 02 12:40:11.298404 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:11.298399    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c5e1a5f8-97b4-469d-be9a-eae7ca995283-tmp-dir\") pod \"distinct-2-w1-61e341-kserve-954cf956f-n7jmg\" (UID: \"c5e1a5f8-97b4-469d-be9a-eae7ca995283\") " pod="e2e-models-e2e-worker-w1-61e341/distinct-2-w1-61e341-kserve-954cf956f-n7jmg"
Nov 02 12:40:11.400138 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:11.400038    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c5e1a5f8-97b4-469d-be9a-eae7ca995283-tls-certs\") pod \"distinct-2-w1-61e341-kserve-954cf956f-n7jmg\" (UID: \"c5e1a5f8-97b4-469d-be9a-eae7ca995283\") " pod="e2e-models-e2e-worker-w1-61e341/distinct-2-w1-61e341-kserve-954cf956f-n7jmg"
Nov 02 12:40:11.400312 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:11.400168    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c5e1a5f8-97b4-469d-be9a-eae7ca995283-dshm\") pod \"distinct-2-w1-61e341-kserve-954cf956f-n7jmg\" (UID: \"c5e1a5f8-97b4-469d-be9a-eae7ca995283\") " pod="e2e-models-e2e-worker-w1-61e341/distinct-2-w1-61e341-kserve-954cf956f-n7jmg"
Nov 02 12:40:11.400312 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:11.400194    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c5e1a5f8-97b4-469d-be9a-eae7ca995283-model-cache\") pod \"distinct-2-w1-61e341-kserve-954cf956f-n7jmg\" (UID: \"c5e1a5f8-97b4-469d-be9a-eae7ca995283\") " pod="e2e-models-e2e-worker-w1-61e341/distinct-2-w1-61e341-kserve-954cf956f-n7jmg"
Nov 02 12:40:11.400312 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:11.400233    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c5e1a5f8-97b4-469d-be9a-eae7ca995283-tmp-dir\") pod \"distinct-2-w1-61e341-kserve-954cf956f-n7jmg\" (UID: \"c5e1a5f8-97b4-469d-be9a-eae7ca995283\") " pod="e2e-models-e2e-worker-w1-61e341/distinct-2-w1-61e341-kserve-954cf956f-n7jmg"
Nov 02 12:40:11.400312 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:11.400272    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7x74\" (UniqueName: \"kubernetes.io/projected/c5e1a5f8-97b4-469d-be9a-eae7ca995283-kube-api-access-h7x74\") pod \"distinct-2-w1-61e341-kserve-954cf956f-n7jmg\" (UID: \"c5e1a5f8-97b4-469d-be9a-eae7ca995283\") " pod="e2e-models-e2e-worker-w1-61e341/distinct-2-w1-61e341-kserve-954cf956f-n7jmg"
Nov 02 12:40:11.400312 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:11.400297    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c5e1a5f8-97b4-469d-be9a-eae7ca995283-home\") pod \"distinct-2-w1-61e341-kserve-954cf956f-n7jmg\" (UID: \"c5e1a5f8-97b4-469d-be9a-eae7ca995283\") " pod="e2e-models-e2e-worker-w1-61e341/distinct-2-w1-61e341-kserve-954cf956f-n7jmg"
Nov 02 12:40:11.400797 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:11.400762    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c5e1a5f8-97b4-469d-be9a-eae7ca995283-home\") pod \"distinct-2-w1-61e341-kserve-954cf956f-n7jmg\" (UID: \"c5e1a5f8-97b4-469d-be9a-eae7ca995283\") " pod="e2e-models-e2e-worker-w1-61e341/distinct-2-w1-61e341-kserve-954cf956f-n7jmg"
Nov 02 12:40:11.401556 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:11.401532    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c5e1a5f8-97b4-469d-be9a-eae7ca995283-model-cache\") pod \"distinct-2-w1-61e341-kserve-954cf956f-n7jmg\" (UID: \"c5e1a5f8-97b4-469d-be9a-eae7ca995283\") " pod="e2e-models-e2e-worker-w1-61e341/distinct-2-w1-61e341-kserve-954cf956f-n7jmg"
Nov 02 12:40:11.401790 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:11.401735    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c5e1a5f8-97b4-469d-be9a-eae7ca995283-tmp-dir\") pod \"distinct-2-w1-61e341-kserve-954cf956f-n7jmg\" (UID: \"c5e1a5f8-97b4-469d-be9a-eae7ca995283\") " pod="e2e-models-e2e-worker-w1-61e341/distinct-2-w1-61e341-kserve-954cf956f-n7jmg"
Nov 02 12:40:11.422102 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:11.421391    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c5e1a5f8-97b4-469d-be9a-eae7ca995283-tls-certs\") pod \"distinct-2-w1-61e341-kserve-954cf956f-n7jmg\" (UID: \"c5e1a5f8-97b4-469d-be9a-eae7ca995283\") " pod="e2e-models-e2e-worker-w1-61e341/distinct-2-w1-61e341-kserve-954cf956f-n7jmg"
Nov 02 12:40:11.422102 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:11.422008    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c5e1a5f8-97b4-469d-be9a-eae7ca995283-dshm\") pod \"distinct-2-w1-61e341-kserve-954cf956f-n7jmg\" (UID: \"c5e1a5f8-97b4-469d-be9a-eae7ca995283\") " pod="e2e-models-e2e-worker-w1-61e341/distinct-2-w1-61e341-kserve-954cf956f-n7jmg"
Nov 02 12:40:11.428110 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:11.428074    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7x74\" (UniqueName: \"kubernetes.io/projected/c5e1a5f8-97b4-469d-be9a-eae7ca995283-kube-api-access-h7x74\") pod \"distinct-2-w1-61e341-kserve-954cf956f-n7jmg\" (UID: \"c5e1a5f8-97b4-469d-be9a-eae7ca995283\") " pod="e2e-models-e2e-worker-w1-61e341/distinct-2-w1-61e341-kserve-954cf956f-n7jmg"
Nov 02 12:40:11.465944 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:11.465911    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-61e341/distinct-2-w1-61e341-kserve-954cf956f-n7jmg"
Nov 02 12:40:11.766198 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:40:11.766159    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc5e1a5f8_97b4_469d_be9a_eae7ca995283.slice/crio-bd4811df291f82e384b5fe6bf9faeb0703c2ea90657c5060bd957eb955a7f6a4 WatchSource:0}: Error finding container bd4811df291f82e384b5fe6bf9faeb0703c2ea90657c5060bd957eb955a7f6a4: Status 404 returned error can't find the container with id bd4811df291f82e384b5fe6bf9faeb0703c2ea90657c5060bd957eb955a7f6a4
Nov 02 12:40:11.766400 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:11.766380    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-61e341/distinct-2-w1-61e341-kserve-954cf956f-n7jmg"]
Nov 02 12:40:11.948887 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:11.948848    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-61e341/distinct-2-w1-61e341-kserve-954cf956f-n7jmg" event={"ID":"c5e1a5f8-97b4-469d-be9a-eae7ca995283","Type":"ContainerStarted","Data":"b415dc77973de5552aa58690105309e69a20126b11eb57daa9718d17c7d22f87"}
Nov 02 12:40:11.949056 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:11.948899    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-61e341/distinct-2-w1-61e341-kserve-954cf956f-n7jmg" event={"ID":"c5e1a5f8-97b4-469d-be9a-eae7ca995283","Type":"ContainerStarted","Data":"bd4811df291f82e384b5fe6bf9faeb0703c2ea90657c5060bd957eb955a7f6a4"}
Nov 02 12:40:11.992480 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:11.992428    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-61e341/distinct-2-w1-61e341-kserve-954cf956f-n7jmg" podStartSLOduration=0.99241424 podStartE2EDuration="992.41424ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:40:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:40:11.991429525 +0000 UTC m=+808.246828621" watchObservedRunningTime="2026-11-02 12:40:11.99241424 +0000 UTC m=+808.247813335"
Nov 02 12:40:12.364348 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:12.364315    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-nkmdl"]
Nov 02 12:40:12.368472 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:12.368450    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-nkmdl"
Nov 02 12:40:12.399697 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:12.399666    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-nkmdl"]
Nov 02 12:40:12.422837 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:12.422290    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13a4f88f-a38d-4fa4-ad71-8ec24dc429a5-istiod-ca-cert\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-nkmdl\" (UID: \"13a4f88f-a38d-4fa4-ad71-8ec24dc429a5\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-nkmdl"
Nov 02 12:40:12.422837 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:12.422365    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13a4f88f-a38d-4fa4-ad71-8ec24dc429a5-istio-podinfo\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-nkmdl\" (UID: \"13a4f88f-a38d-4fa4-ad71-8ec24dc429a5\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-nkmdl"
Nov 02 12:40:12.422837 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:12.422481    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13a4f88f-a38d-4fa4-ad71-8ec24dc429a5-workload-socket\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-nkmdl\" (UID: \"13a4f88f-a38d-4fa4-ad71-8ec24dc429a5\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-nkmdl"
Nov 02 12:40:12.422837 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:12.422513    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13a4f88f-a38d-4fa4-ad71-8ec24dc429a5-workload-certs\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-nkmdl\" (UID: \"13a4f88f-a38d-4fa4-ad71-8ec24dc429a5\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-nkmdl"
Nov 02 12:40:12.422837 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:12.422565    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13a4f88f-a38d-4fa4-ad71-8ec24dc429a5-istio-token\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-nkmdl\" (UID: \"13a4f88f-a38d-4fa4-ad71-8ec24dc429a5\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-nkmdl"
Nov 02 12:40:12.422837 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:12.422700    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13a4f88f-a38d-4fa4-ad71-8ec24dc429a5-istio-envoy\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-nkmdl\" (UID: \"13a4f88f-a38d-4fa4-ad71-8ec24dc429a5\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-nkmdl"
Nov 02 12:40:12.422837 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:12.422756    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13a4f88f-a38d-4fa4-ad71-8ec24dc429a5-credential-socket\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-nkmdl\" (UID: \"13a4f88f-a38d-4fa4-ad71-8ec24dc429a5\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-nkmdl"
Nov 02 12:40:12.422837 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:12.422813    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfqqx\" (UniqueName: \"kubernetes.io/projected/13a4f88f-a38d-4fa4-ad71-8ec24dc429a5-kube-api-access-tfqqx\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-nkmdl\" (UID: \"13a4f88f-a38d-4fa4-ad71-8ec24dc429a5\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-nkmdl"
Nov 02 12:40:12.423331 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:12.422953    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13a4f88f-a38d-4fa4-ad71-8ec24dc429a5-istio-data\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-nkmdl\" (UID: \"13a4f88f-a38d-4fa4-ad71-8ec24dc429a5\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-nkmdl"
Nov 02 12:40:12.466092 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:12.466060    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-61e341/distinct-2-w1-61e341-kserve-954cf956f-n7jmg"
Nov 02 12:40:12.524409 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:12.524375    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13a4f88f-a38d-4fa4-ad71-8ec24dc429a5-istio-data\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-nkmdl\" (UID: \"13a4f88f-a38d-4fa4-ad71-8ec24dc429a5\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-nkmdl"
Nov 02 12:40:12.524558 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:12.524429    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13a4f88f-a38d-4fa4-ad71-8ec24dc429a5-istiod-ca-cert\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-nkmdl\" (UID: \"13a4f88f-a38d-4fa4-ad71-8ec24dc429a5\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-nkmdl"
Nov 02 12:40:12.524558 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:12.524464    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13a4f88f-a38d-4fa4-ad71-8ec24dc429a5-istio-podinfo\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-nkmdl\" (UID: \"13a4f88f-a38d-4fa4-ad71-8ec24dc429a5\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-nkmdl"
Nov 02 12:40:12.524558 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:12.524484    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13a4f88f-a38d-4fa4-ad71-8ec24dc429a5-workload-socket\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-nkmdl\" (UID: \"13a4f88f-a38d-4fa4-ad71-8ec24dc429a5\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-nkmdl"
Nov 02 12:40:12.524558 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:12.524501    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13a4f88f-a38d-4fa4-ad71-8ec24dc429a5-workload-certs\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-nkmdl\" (UID: \"13a4f88f-a38d-4fa4-ad71-8ec24dc429a5\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-nkmdl"
Nov 02 12:40:12.524558 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:12.524554    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13a4f88f-a38d-4fa4-ad71-8ec24dc429a5-istio-token\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-nkmdl\" (UID: \"13a4f88f-a38d-4fa4-ad71-8ec24dc429a5\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-nkmdl"
Nov 02 12:40:12.524841 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:12.524662    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13a4f88f-a38d-4fa4-ad71-8ec24dc429a5-istio-envoy\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-nkmdl\" (UID: \"13a4f88f-a38d-4fa4-ad71-8ec24dc429a5\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-nkmdl"
Nov 02 12:40:12.524841 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:12.524703    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13a4f88f-a38d-4fa4-ad71-8ec24dc429a5-credential-socket\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-nkmdl\" (UID: \"13a4f88f-a38d-4fa4-ad71-8ec24dc429a5\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-nkmdl"
Nov 02 12:40:12.524841 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:12.524757    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfqqx\" (UniqueName: \"kubernetes.io/projected/13a4f88f-a38d-4fa4-ad71-8ec24dc429a5-kube-api-access-tfqqx\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-nkmdl\" (UID: \"13a4f88f-a38d-4fa4-ad71-8ec24dc429a5\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-nkmdl"
Nov 02 12:40:12.524841 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:12.524813    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13a4f88f-a38d-4fa4-ad71-8ec24dc429a5-istio-data\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-nkmdl\" (UID: \"13a4f88f-a38d-4fa4-ad71-8ec24dc429a5\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-nkmdl"
Nov 02 12:40:12.525067 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:12.524877    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13a4f88f-a38d-4fa4-ad71-8ec24dc429a5-workload-socket\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-nkmdl\" (UID: \"13a4f88f-a38d-4fa4-ad71-8ec24dc429a5\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-nkmdl"
Nov 02 12:40:12.525067 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:12.524943    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13a4f88f-a38d-4fa4-ad71-8ec24dc429a5-workload-certs\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-nkmdl\" (UID: \"13a4f88f-a38d-4fa4-ad71-8ec24dc429a5\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-nkmdl"
Nov 02 12:40:12.525175 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:12.525135    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13a4f88f-a38d-4fa4-ad71-8ec24dc429a5-credential-socket\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-nkmdl\" (UID: \"13a4f88f-a38d-4fa4-ad71-8ec24dc429a5\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-nkmdl"
Nov 02 12:40:12.525303 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:12.525285    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13a4f88f-a38d-4fa4-ad71-8ec24dc429a5-istiod-ca-cert\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-nkmdl\" (UID: \"13a4f88f-a38d-4fa4-ad71-8ec24dc429a5\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-nkmdl"
Nov 02 12:40:12.526968 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:12.526946    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13a4f88f-a38d-4fa4-ad71-8ec24dc429a5-istio-envoy\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-nkmdl\" (UID: \"13a4f88f-a38d-4fa4-ad71-8ec24dc429a5\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-nkmdl"
Nov 02 12:40:12.527220 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:12.527199    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13a4f88f-a38d-4fa4-ad71-8ec24dc429a5-istio-podinfo\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-nkmdl\" (UID: \"13a4f88f-a38d-4fa4-ad71-8ec24dc429a5\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-nkmdl"
Nov 02 12:40:12.548613 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:12.548583    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13a4f88f-a38d-4fa4-ad71-8ec24dc429a5-istio-token\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-nkmdl\" (UID: \"13a4f88f-a38d-4fa4-ad71-8ec24dc429a5\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-nkmdl"
Nov 02 12:40:12.548772 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:12.548754    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfqqx\" (UniqueName: \"kubernetes.io/projected/13a4f88f-a38d-4fa4-ad71-8ec24dc429a5-kube-api-access-tfqqx\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-nkmdl\" (UID: \"13a4f88f-a38d-4fa4-ad71-8ec24dc429a5\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-nkmdl"
Nov 02 12:40:12.681562 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:12.681537    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-nkmdl"
Nov 02 12:40:13.009556 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:40:13.009517    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13a4f88f_a38d_4fa4_ad71_8ec24dc429a5.slice/crio-742ebed4c38a928c0fd4e25a6d3c05388d6475fb293c5030e65c20803237df2b WatchSource:0}: Error finding container 742ebed4c38a928c0fd4e25a6d3c05388d6475fb293c5030e65c20803237df2b: Status 404 returned error can't find the container with id 742ebed4c38a928c0fd4e25a6d3c05388d6475fb293c5030e65c20803237df2b
Nov 02 12:40:13.009841 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:13.009819    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-nkmdl"]
Nov 02 12:40:13.011938 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:13.011906    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:40:13.011998 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:13.011970    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:40:13.012037 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:13.011998    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:40:13.962136 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:13.962095    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-nkmdl" event={"ID":"13a4f88f-a38d-4fa4-ad71-8ec24dc429a5","Type":"ContainerStarted","Data":"9f943f38651bea528490600460e64b2ca8395880a34c6a115a54b6d40f9d8e09"}
Nov 02 12:40:13.962560 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:13.962155    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-nkmdl" event={"ID":"13a4f88f-a38d-4fa4-ad71-8ec24dc429a5","Type":"ContainerStarted","Data":"742ebed4c38a928c0fd4e25a6d3c05388d6475fb293c5030e65c20803237df2b"}
Nov 02 12:40:14.036980 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:14.036927    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-nkmdl" podStartSLOduration=2.036912665 podStartE2EDuration="2.036912665s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:40:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:40:14.035205635 +0000 UTC m=+810.290604749" watchObservedRunningTime="2026-11-02 12:40:14.036912665 +0000 UTC m=+810.292311840"
Nov 02 12:40:14.682700 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:14.682602    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-nkmdl"
Nov 02 12:40:14.703661 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:14.693761    4559 patch_prober.go:29] interesting pod/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-nkmdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Nov 02 12:40:14.703661 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:14.693827    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-nkmdl" podUID="13a4f88f-a38d-4fa4-ad71-8ec24dc429a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Nov 02 12:40:15.171459 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:40:15.171427    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqcmj" containerName="extractor"
Nov 02 12:40:15.682049 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:15.682017    4559 patch_prober.go:29] interesting pod/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-nkmdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Nov 02 12:40:15.682218 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:15.682070    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-nkmdl" podUID="13a4f88f-a38d-4fa4-ad71-8ec24dc429a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Nov 02 12:40:16.686665 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:16.682891    4559 patch_prober.go:29] interesting pod/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-nkmdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Nov 02 12:40:16.686665 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:16.682964    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-nkmdl" podUID="13a4f88f-a38d-4fa4-ad71-8ec24dc429a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Nov 02 12:40:17.682623 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:17.682586    4559 patch_prober.go:29] interesting pod/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-nkmdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Nov 02 12:40:17.682989 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:17.682679    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-nkmdl" podUID="13a4f88f-a38d-4fa4-ad71-8ec24dc429a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Nov 02 12:40:18.682100 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:18.682068    4559 patch_prober.go:29] interesting pod/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-nkmdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Nov 02 12:40:18.682504 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:18.682128    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-nkmdl" podUID="13a4f88f-a38d-4fa4-ad71-8ec24dc429a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Nov 02 12:40:18.853812 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:18.853781    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"]
Nov 02 12:40:19.693663 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:19.684882    4559 patch_prober.go:29] interesting pod/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-nkmdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Nov 02 12:40:19.693663 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:19.684956    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-nkmdl" podUID="13a4f88f-a38d-4fa4-ad71-8ec24dc429a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Nov 02 12:40:21.471772 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:21.471713    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-61e341/distinct-2-w1-61e341-kserve-954cf956f-n7jmg"
Nov 02 12:40:21.735267 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:21.735169    4559 patch_prober.go:29] interesting pod/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-nkmdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 12:40:21.735548 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:21.735516    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-nkmdl" podUID="13a4f88f-a38d-4fa4-ad71-8ec24dc429a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:40:22.143887 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:22.143771    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-61e341/premium-w1-61e341-kserve-746477b99-645g6" podUID="02ff21d3-5a37-493a-ab95-8e4784c29123" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:40:22.172983 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:22.172948    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-61e341/distinct-2-w1-61e341-kserve-954cf956f-n7jmg"
Nov 02 12:40:22.199956 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:22.199922    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-61e341/distinct-2-w1-61e341-kserve-954cf956f-n7jmg"
Nov 02 12:40:22.252768 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:22.252714    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843978/distinct-w4-843978-kserve-6fcf588858-4nrwx" podUID="82e5d483-e95b-43e6-9135-af960823427a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:40:22.366479 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:22.361856    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-nkmdl"
Nov 02 12:40:22.366479 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:22.363212    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-nkmdl"
Nov 02 12:40:22.367086 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:22.367051    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-nkmdl"
Nov 02 12:40:27.925908 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:27.925876    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-cdnsv"]
Nov 02 12:40:27.931011 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:27.930984    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-cdnsv"
Nov 02 12:40:27.951403 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:27.951371    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-cdnsv"]
Nov 02 12:40:28.049148 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:28.049110    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be334717-f78a-4113-bf03-15d5c4365107-istio-token\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-cdnsv\" (UID: \"be334717-f78a-4113-bf03-15d5c4365107\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-cdnsv"
Nov 02 12:40:28.049312 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:28.049156    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be334717-f78a-4113-bf03-15d5c4365107-istiod-ca-cert\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-cdnsv\" (UID: \"be334717-f78a-4113-bf03-15d5c4365107\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-cdnsv"
Nov 02 12:40:28.049312 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:28.049296    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be334717-f78a-4113-bf03-15d5c4365107-credential-socket\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-cdnsv\" (UID: \"be334717-f78a-4113-bf03-15d5c4365107\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-cdnsv"
Nov 02 12:40:28.049393 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:28.049336    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be334717-f78a-4113-bf03-15d5c4365107-istio-envoy\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-cdnsv\" (UID: \"be334717-f78a-4113-bf03-15d5c4365107\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-cdnsv"
Nov 02 12:40:28.049393 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:28.049356    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pt7tk\" (UniqueName: \"kubernetes.io/projected/be334717-f78a-4113-bf03-15d5c4365107-kube-api-access-pt7tk\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-cdnsv\" (UID: \"be334717-f78a-4113-bf03-15d5c4365107\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-cdnsv"
Nov 02 12:40:28.049466 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:28.049451    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be334717-f78a-4113-bf03-15d5c4365107-istio-podinfo\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-cdnsv\" (UID: \"be334717-f78a-4113-bf03-15d5c4365107\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-cdnsv"
Nov 02 12:40:28.049503 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:28.049486    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be334717-f78a-4113-bf03-15d5c4365107-workload-certs\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-cdnsv\" (UID: \"be334717-f78a-4113-bf03-15d5c4365107\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-cdnsv"
Nov 02 12:40:28.049539 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:28.049504    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be334717-f78a-4113-bf03-15d5c4365107-workload-socket\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-cdnsv\" (UID: \"be334717-f78a-4113-bf03-15d5c4365107\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-cdnsv"
Nov 02 12:40:28.049618 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:28.049598    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be334717-f78a-4113-bf03-15d5c4365107-istio-data\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-cdnsv\" (UID: \"be334717-f78a-4113-bf03-15d5c4365107\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-cdnsv"
Nov 02 12:40:28.151382 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:28.151349    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be334717-f78a-4113-bf03-15d5c4365107-credential-socket\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-cdnsv\" (UID: \"be334717-f78a-4113-bf03-15d5c4365107\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-cdnsv"
Nov 02 12:40:28.151549 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:28.151395    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be334717-f78a-4113-bf03-15d5c4365107-istio-envoy\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-cdnsv\" (UID: \"be334717-f78a-4113-bf03-15d5c4365107\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-cdnsv"
Nov 02 12:40:28.151549 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:28.151512    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pt7tk\" (UniqueName: \"kubernetes.io/projected/be334717-f78a-4113-bf03-15d5c4365107-kube-api-access-pt7tk\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-cdnsv\" (UID: \"be334717-f78a-4113-bf03-15d5c4365107\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-cdnsv"
Nov 02 12:40:28.151628 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:28.151606    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be334717-f78a-4113-bf03-15d5c4365107-istio-podinfo\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-cdnsv\" (UID: \"be334717-f78a-4113-bf03-15d5c4365107\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-cdnsv"
Nov 02 12:40:28.151706 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:28.151677    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be334717-f78a-4113-bf03-15d5c4365107-workload-certs\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-cdnsv\" (UID: \"be334717-f78a-4113-bf03-15d5c4365107\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-cdnsv"
Nov 02 12:40:28.151751 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:28.151707    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be334717-f78a-4113-bf03-15d5c4365107-workload-socket\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-cdnsv\" (UID: \"be334717-f78a-4113-bf03-15d5c4365107\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-cdnsv"
Nov 02 12:40:28.151806 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:28.151746    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be334717-f78a-4113-bf03-15d5c4365107-istio-data\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-cdnsv\" (UID: \"be334717-f78a-4113-bf03-15d5c4365107\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-cdnsv"
Nov 02 12:40:28.151870 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:28.151803    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be334717-f78a-4113-bf03-15d5c4365107-credential-socket\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-cdnsv\" (UID: \"be334717-f78a-4113-bf03-15d5c4365107\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-cdnsv"
Nov 02 12:40:28.151870 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:28.151827    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be334717-f78a-4113-bf03-15d5c4365107-istio-token\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-cdnsv\" (UID: \"be334717-f78a-4113-bf03-15d5c4365107\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-cdnsv"
Nov 02 12:40:28.151870 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:28.151864    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be334717-f78a-4113-bf03-15d5c4365107-istiod-ca-cert\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-cdnsv\" (UID: \"be334717-f78a-4113-bf03-15d5c4365107\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-cdnsv"
Nov 02 12:40:28.152090 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:28.152067    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be334717-f78a-4113-bf03-15d5c4365107-workload-certs\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-cdnsv\" (UID: \"be334717-f78a-4113-bf03-15d5c4365107\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-cdnsv"
Nov 02 12:40:28.152160 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:28.152091    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be334717-f78a-4113-bf03-15d5c4365107-workload-socket\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-cdnsv\" (UID: \"be334717-f78a-4113-bf03-15d5c4365107\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-cdnsv"
Nov 02 12:40:28.152208 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:28.152170    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be334717-f78a-4113-bf03-15d5c4365107-istio-data\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-cdnsv\" (UID: \"be334717-f78a-4113-bf03-15d5c4365107\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-cdnsv"
Nov 02 12:40:28.152514 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:28.152497    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be334717-f78a-4113-bf03-15d5c4365107-istiod-ca-cert\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-cdnsv\" (UID: \"be334717-f78a-4113-bf03-15d5c4365107\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-cdnsv"
Nov 02 12:40:28.153737 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:28.153711    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be334717-f78a-4113-bf03-15d5c4365107-istio-envoy\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-cdnsv\" (UID: \"be334717-f78a-4113-bf03-15d5c4365107\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-cdnsv"
Nov 02 12:40:28.154024 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:28.154006    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be334717-f78a-4113-bf03-15d5c4365107-istio-podinfo\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-cdnsv\" (UID: \"be334717-f78a-4113-bf03-15d5c4365107\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-cdnsv"
Nov 02 12:40:28.175820 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:28.175792    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pt7tk\" (UniqueName: \"kubernetes.io/projected/be334717-f78a-4113-bf03-15d5c4365107-kube-api-access-pt7tk\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-cdnsv\" (UID: \"be334717-f78a-4113-bf03-15d5c4365107\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-cdnsv"
Nov 02 12:40:28.175973 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:28.175869    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be334717-f78a-4113-bf03-15d5c4365107-istio-token\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-cdnsv\" (UID: \"be334717-f78a-4113-bf03-15d5c4365107\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-cdnsv"
Nov 02 12:40:28.250714 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:28.250685    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-cdnsv"
Nov 02 12:40:28.566312 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:28.566278    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-cdnsv"]
Nov 02 12:40:28.569332 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:28.569298    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:40:28.569434 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:28.569365    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:40:28.569434 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:28.569395    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:40:29.129544 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:29.129508    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-cdnsv" event={"ID":"be334717-f78a-4113-bf03-15d5c4365107","Type":"ContainerStarted","Data":"32750de39eedcb157478ffa2aca02da0c43e2efc8a63d644f34bbce8d1ef88b2"}
Nov 02 12:40:29.129979 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:29.129568    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-cdnsv" event={"ID":"be334717-f78a-4113-bf03-15d5c4365107","Type":"ContainerStarted","Data":"4062ef3dd834f9ab73f100bd0c11663d045f8293df7c56f6864749d074fd2d1d"}
Nov 02 12:40:29.206664 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:29.206035    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-cdnsv" podStartSLOduration=2.206011523 podStartE2EDuration="2.206011523s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:40:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:40:29.189050846 +0000 UTC m=+825.444449942" watchObservedRunningTime="2026-11-02 12:40:29.206011523 +0000 UTC m=+825.461410620"
Nov 02 12:40:29.252268 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:29.252207    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-cdnsv"
Nov 02 12:40:29.265667 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:29.263545    4559 patch_prober.go:29] interesting pod/e2e-worker-w3-068756-openshift-default-84f69c9cf5-cdnsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Nov 02 12:40:29.265667 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:29.263605    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-cdnsv" podUID="be334717-f78a-4113-bf03-15d5c4365107" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Nov 02 12:40:30.251926 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:30.251888    4559 patch_prober.go:29] interesting pod/e2e-worker-w3-068756-openshift-default-84f69c9cf5-cdnsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Nov 02 12:40:30.252332 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:30.251941    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-cdnsv" podUID="be334717-f78a-4113-bf03-15d5c4365107" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Nov 02 12:40:31.251194 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:31.251158    4559 patch_prober.go:29] interesting pod/e2e-worker-w3-068756-openshift-default-84f69c9cf5-cdnsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Nov 02 12:40:31.251375 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:31.251212    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-cdnsv" podUID="be334717-f78a-4113-bf03-15d5c4365107" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Nov 02 12:40:32.251368 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:32.251334    4559 patch_prober.go:29] interesting pod/e2e-worker-w3-068756-openshift-default-84f69c9cf5-cdnsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Nov 02 12:40:32.251797 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:32.251388    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-cdnsv" podUID="be334717-f78a-4113-bf03-15d5c4365107" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Nov 02 12:40:33.251404 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:33.251367    4559 patch_prober.go:29] interesting pod/e2e-worker-w3-068756-openshift-default-84f69c9cf5-cdnsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Nov 02 12:40:33.251922 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:33.251428    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-cdnsv" podUID="be334717-f78a-4113-bf03-15d5c4365107" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Nov 02 12:40:34.251064 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:34.251032    4559 patch_prober.go:29] interesting pod/e2e-worker-w3-068756-openshift-default-84f69c9cf5-cdnsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Nov 02 12:40:34.251234 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:34.251103    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-cdnsv" podUID="be334717-f78a-4113-bf03-15d5c4365107" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Nov 02 12:40:35.551378 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:35.551336    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-cdnsv"
Nov 02 12:40:35.552198 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:35.552170    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-cdnsv"
Nov 02 12:40:35.553104 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:35.553084    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-cdnsv"
Nov 02 12:40:42.381419 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:42.381386    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-64thh"]
Nov 02 12:40:42.385886 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:42.385865    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-64thh"
Nov 02 12:40:42.420286 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:42.420257    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-64thh"]
Nov 02 12:40:42.503812 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:42.503777    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzzgb\" (UniqueName: \"kubernetes.io/projected/2a48a9d0-407c-4149-abe9-c675e2092854-kube-api-access-mzzgb\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-64thh\" (UID: \"2a48a9d0-407c-4149-abe9-c675e2092854\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-64thh"
Nov 02 12:40:42.503986 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:42.503833    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a48a9d0-407c-4149-abe9-c675e2092854-credential-socket\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-64thh\" (UID: \"2a48a9d0-407c-4149-abe9-c675e2092854\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-64thh"
Nov 02 12:40:42.503986 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:42.503920    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a48a9d0-407c-4149-abe9-c675e2092854-istio-data\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-64thh\" (UID: \"2a48a9d0-407c-4149-abe9-c675e2092854\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-64thh"
Nov 02 12:40:42.503986 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:42.503948    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a48a9d0-407c-4149-abe9-c675e2092854-workload-certs\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-64thh\" (UID: \"2a48a9d0-407c-4149-abe9-c675e2092854\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-64thh"
Nov 02 12:40:42.504134 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:42.504060    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a48a9d0-407c-4149-abe9-c675e2092854-istio-token\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-64thh\" (UID: \"2a48a9d0-407c-4149-abe9-c675e2092854\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-64thh"
Nov 02 12:40:42.504134 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:42.504091    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a48a9d0-407c-4149-abe9-c675e2092854-istiod-ca-cert\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-64thh\" (UID: \"2a48a9d0-407c-4149-abe9-c675e2092854\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-64thh"
Nov 02 12:40:42.504237 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:42.504182    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a48a9d0-407c-4149-abe9-c675e2092854-istio-podinfo\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-64thh\" (UID: \"2a48a9d0-407c-4149-abe9-c675e2092854\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-64thh"
Nov 02 12:40:42.504305 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:42.504291    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a48a9d0-407c-4149-abe9-c675e2092854-istio-envoy\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-64thh\" (UID: \"2a48a9d0-407c-4149-abe9-c675e2092854\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-64thh"
Nov 02 12:40:42.504351 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:42.504324    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a48a9d0-407c-4149-abe9-c675e2092854-workload-socket\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-64thh\" (UID: \"2a48a9d0-407c-4149-abe9-c675e2092854\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-64thh"
Nov 02 12:40:42.605311 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:42.605273    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a48a9d0-407c-4149-abe9-c675e2092854-istio-data\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-64thh\" (UID: \"2a48a9d0-407c-4149-abe9-c675e2092854\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-64thh"
Nov 02 12:40:42.605311 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:42.605314    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a48a9d0-407c-4149-abe9-c675e2092854-workload-certs\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-64thh\" (UID: \"2a48a9d0-407c-4149-abe9-c675e2092854\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-64thh"
Nov 02 12:40:42.605559 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:42.605398    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a48a9d0-407c-4149-abe9-c675e2092854-istio-token\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-64thh\" (UID: \"2a48a9d0-407c-4149-abe9-c675e2092854\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-64thh"
Nov 02 12:40:42.605559 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:42.605427    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a48a9d0-407c-4149-abe9-c675e2092854-istiod-ca-cert\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-64thh\" (UID: \"2a48a9d0-407c-4149-abe9-c675e2092854\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-64thh"
Nov 02 12:40:42.605559 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:42.605468    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a48a9d0-407c-4149-abe9-c675e2092854-istio-podinfo\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-64thh\" (UID: \"2a48a9d0-407c-4149-abe9-c675e2092854\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-64thh"
Nov 02 12:40:42.605559 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:42.605549    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a48a9d0-407c-4149-abe9-c675e2092854-istio-envoy\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-64thh\" (UID: \"2a48a9d0-407c-4149-abe9-c675e2092854\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-64thh"
Nov 02 12:40:42.605832 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:42.605721    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a48a9d0-407c-4149-abe9-c675e2092854-workload-socket\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-64thh\" (UID: \"2a48a9d0-407c-4149-abe9-c675e2092854\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-64thh"
Nov 02 12:40:42.605832 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:42.605757    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a48a9d0-407c-4149-abe9-c675e2092854-istio-data\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-64thh\" (UID: \"2a48a9d0-407c-4149-abe9-c675e2092854\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-64thh"
Nov 02 12:40:42.605832 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:42.605777    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a48a9d0-407c-4149-abe9-c675e2092854-workload-certs\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-64thh\" (UID: \"2a48a9d0-407c-4149-abe9-c675e2092854\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-64thh"
Nov 02 12:40:42.605832 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:42.605798    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzzgb\" (UniqueName: \"kubernetes.io/projected/2a48a9d0-407c-4149-abe9-c675e2092854-kube-api-access-mzzgb\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-64thh\" (UID: \"2a48a9d0-407c-4149-abe9-c675e2092854\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-64thh"
Nov 02 12:40:42.606054 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:42.605852    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a48a9d0-407c-4149-abe9-c675e2092854-credential-socket\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-64thh\" (UID: \"2a48a9d0-407c-4149-abe9-c675e2092854\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-64thh"
Nov 02 12:40:42.606200 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:42.606175    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a48a9d0-407c-4149-abe9-c675e2092854-workload-socket\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-64thh\" (UID: \"2a48a9d0-407c-4149-abe9-c675e2092854\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-64thh"
Nov 02 12:40:42.606256 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:42.606198    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a48a9d0-407c-4149-abe9-c675e2092854-istiod-ca-cert\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-64thh\" (UID: \"2a48a9d0-407c-4149-abe9-c675e2092854\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-64thh"
Nov 02 12:40:42.606256 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:42.606200    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a48a9d0-407c-4149-abe9-c675e2092854-credential-socket\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-64thh\" (UID: \"2a48a9d0-407c-4149-abe9-c675e2092854\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-64thh"
Nov 02 12:40:42.608440 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:42.608412    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a48a9d0-407c-4149-abe9-c675e2092854-istio-envoy\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-64thh\" (UID: \"2a48a9d0-407c-4149-abe9-c675e2092854\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-64thh"
Nov 02 12:40:42.608582 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:42.608565    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a48a9d0-407c-4149-abe9-c675e2092854-istio-podinfo\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-64thh\" (UID: \"2a48a9d0-407c-4149-abe9-c675e2092854\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-64thh"
Nov 02 12:40:42.629286 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:42.629258    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a48a9d0-407c-4149-abe9-c675e2092854-istio-token\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-64thh\" (UID: \"2a48a9d0-407c-4149-abe9-c675e2092854\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-64thh"
Nov 02 12:40:42.629390 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:42.629351    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzzgb\" (UniqueName: \"kubernetes.io/projected/2a48a9d0-407c-4149-abe9-c675e2092854-kube-api-access-mzzgb\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-64thh\" (UID: \"2a48a9d0-407c-4149-abe9-c675e2092854\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-64thh"
Nov 02 12:40:42.701676 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:42.701634    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-64thh"
Nov 02 12:40:43.020740 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:40:43.020708    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a48a9d0_407c_4149_abe9_c675e2092854.slice/crio-74eb292162364d97f724c58b67205f00418353e0610f1c8bcd631377ccba6423 WatchSource:0}: Error finding container 74eb292162364d97f724c58b67205f00418353e0610f1c8bcd631377ccba6423: Status 404 returned error can't find the container with id 74eb292162364d97f724c58b67205f00418353e0610f1c8bcd631377ccba6423
Nov 02 12:40:43.020877 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:43.020831    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-64thh"]
Nov 02 12:40:43.023083 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:43.023048    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:40:43.023171 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:43.023117    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:40:43.023171 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:43.023144    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:40:43.379360 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:43.379273    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-64thh" event={"ID":"2a48a9d0-407c-4149-abe9-c675e2092854","Type":"ContainerStarted","Data":"e20f09afcc45a8e094c97b89927ccd8871b454591c60fb9df4705aa727954bbe"}
Nov 02 12:40:43.379553 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:43.379359    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-64thh" event={"ID":"2a48a9d0-407c-4149-abe9-c675e2092854","Type":"ContainerStarted","Data":"74eb292162364d97f724c58b67205f00418353e0610f1c8bcd631377ccba6423"}
Nov 02 12:40:43.430394 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:43.430328    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-64thh" podStartSLOduration=1.430308631 podStartE2EDuration="1.430308631s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:40:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:40:43.425563131 +0000 UTC m=+839.680962227" watchObservedRunningTime="2026-11-02 12:40:43.430308631 +0000 UTC m=+839.685707729"
Nov 02 12:40:43.705663 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:43.704722    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-64thh"
Nov 02 12:40:44.707791 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:44.707748    4559 patch_prober.go:29] interesting pod/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-64thh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Nov 02 12:40:44.708271 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:44.707807    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-64thh" podUID="2a48a9d0-407c-4149-abe9-c675e2092854" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Nov 02 12:40:45.108722 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:40:45.108463    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqcmj" containerName="extractor"
Nov 02 12:40:45.707827 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:45.707736    4559 patch_prober.go:29] interesting pod/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-64thh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Nov 02 12:40:45.707827 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:45.707815    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-64thh" podUID="2a48a9d0-407c-4149-abe9-c675e2092854" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Nov 02 12:40:46.703664 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:46.702979    4559 patch_prober.go:29] interesting pod/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-64thh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Nov 02 12:40:46.703664 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:46.703045    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-64thh" podUID="2a48a9d0-407c-4149-abe9-c675e2092854" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Nov 02 12:40:47.702021 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:47.701983    4559 patch_prober.go:29] interesting pod/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-64thh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Nov 02 12:40:47.702620 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:47.702573    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-64thh" podUID="2a48a9d0-407c-4149-abe9-c675e2092854" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Nov 02 12:40:48.705663 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:48.705056    4559 patch_prober.go:29] interesting pod/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-64thh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Nov 02 12:40:48.705663 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:48.705142    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-64thh" podUID="2a48a9d0-407c-4149-abe9-c675e2092854" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Nov 02 12:40:49.702276 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:49.702243    4559 patch_prober.go:29] interesting pod/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-64thh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Nov 02 12:40:49.702451 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:49.702297    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-64thh" podUID="2a48a9d0-407c-4149-abe9-c675e2092854" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Nov 02 12:40:49.703765 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:49.703740    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-2hpjf"]
Nov 02 12:40:49.709792 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:49.709760    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-2hpjf"
Nov 02 12:40:49.740424 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:49.740388    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e590f1fe-f50e-4ca1-a7cf-ac07ae240dba-istio-podinfo\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-2hpjf\" (UID: \"e590f1fe-f50e-4ca1-a7cf-ac07ae240dba\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-2hpjf"
Nov 02 12:40:49.740559 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:49.740443    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e590f1fe-f50e-4ca1-a7cf-ac07ae240dba-credential-socket\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-2hpjf\" (UID: \"e590f1fe-f50e-4ca1-a7cf-ac07ae240dba\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-2hpjf"
Nov 02 12:40:49.740605 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:49.740573    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvbh4\" (UniqueName: \"kubernetes.io/projected/e590f1fe-f50e-4ca1-a7cf-ac07ae240dba-kube-api-access-rvbh4\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-2hpjf\" (UID: \"e590f1fe-f50e-4ca1-a7cf-ac07ae240dba\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-2hpjf"
Nov 02 12:40:49.740693 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:49.740612    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e590f1fe-f50e-4ca1-a7cf-ac07ae240dba-istio-data\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-2hpjf\" (UID: \"e590f1fe-f50e-4ca1-a7cf-ac07ae240dba\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-2hpjf"
Nov 02 12:40:49.740693 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:49.740672    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e590f1fe-f50e-4ca1-a7cf-ac07ae240dba-workload-socket\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-2hpjf\" (UID: \"e590f1fe-f50e-4ca1-a7cf-ac07ae240dba\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-2hpjf"
Nov 02 12:40:49.740803 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:49.740735    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e590f1fe-f50e-4ca1-a7cf-ac07ae240dba-istio-envoy\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-2hpjf\" (UID: \"e590f1fe-f50e-4ca1-a7cf-ac07ae240dba\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-2hpjf"
Nov 02 12:40:49.740803 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:49.740776    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e590f1fe-f50e-4ca1-a7cf-ac07ae240dba-workload-certs\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-2hpjf\" (UID: \"e590f1fe-f50e-4ca1-a7cf-ac07ae240dba\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-2hpjf"
Nov 02 12:40:49.740911 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:49.740877    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e590f1fe-f50e-4ca1-a7cf-ac07ae240dba-istiod-ca-cert\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-2hpjf\" (UID: \"e590f1fe-f50e-4ca1-a7cf-ac07ae240dba\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-2hpjf"
Nov 02 12:40:49.740986 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:49.740966    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e590f1fe-f50e-4ca1-a7cf-ac07ae240dba-istio-token\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-2hpjf\" (UID: \"e590f1fe-f50e-4ca1-a7cf-ac07ae240dba\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-2hpjf"
Nov 02 12:40:49.741165 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:49.741135    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-2hpjf"]
Nov 02 12:40:49.842243 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:49.842207    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e590f1fe-f50e-4ca1-a7cf-ac07ae240dba-istiod-ca-cert\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-2hpjf\" (UID: \"e590f1fe-f50e-4ca1-a7cf-ac07ae240dba\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-2hpjf"
Nov 02 12:40:49.842417 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:49.842271    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e590f1fe-f50e-4ca1-a7cf-ac07ae240dba-istio-token\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-2hpjf\" (UID: \"e590f1fe-f50e-4ca1-a7cf-ac07ae240dba\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-2hpjf"
Nov 02 12:40:49.842417 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:49.842323    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e590f1fe-f50e-4ca1-a7cf-ac07ae240dba-istio-podinfo\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-2hpjf\" (UID: \"e590f1fe-f50e-4ca1-a7cf-ac07ae240dba\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-2hpjf"
Nov 02 12:40:49.842417 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:49.842341    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e590f1fe-f50e-4ca1-a7cf-ac07ae240dba-credential-socket\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-2hpjf\" (UID: \"e590f1fe-f50e-4ca1-a7cf-ac07ae240dba\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-2hpjf"
Nov 02 12:40:49.842585 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:49.842429    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvbh4\" (UniqueName: \"kubernetes.io/projected/e590f1fe-f50e-4ca1-a7cf-ac07ae240dba-kube-api-access-rvbh4\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-2hpjf\" (UID: \"e590f1fe-f50e-4ca1-a7cf-ac07ae240dba\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-2hpjf"
Nov 02 12:40:49.842585 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:49.842465    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e590f1fe-f50e-4ca1-a7cf-ac07ae240dba-istio-data\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-2hpjf\" (UID: \"e590f1fe-f50e-4ca1-a7cf-ac07ae240dba\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-2hpjf"
Nov 02 12:40:49.842585 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:49.842491    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e590f1fe-f50e-4ca1-a7cf-ac07ae240dba-workload-socket\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-2hpjf\" (UID: \"e590f1fe-f50e-4ca1-a7cf-ac07ae240dba\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-2hpjf"
Nov 02 12:40:49.842585 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:49.842544    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e590f1fe-f50e-4ca1-a7cf-ac07ae240dba-istio-envoy\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-2hpjf\" (UID: \"e590f1fe-f50e-4ca1-a7cf-ac07ae240dba\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-2hpjf"
Nov 02 12:40:49.842585 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:49.842582    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e590f1fe-f50e-4ca1-a7cf-ac07ae240dba-workload-certs\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-2hpjf\" (UID: \"e590f1fe-f50e-4ca1-a7cf-ac07ae240dba\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-2hpjf"
Nov 02 12:40:49.842949 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:49.842871    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e590f1fe-f50e-4ca1-a7cf-ac07ae240dba-credential-socket\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-2hpjf\" (UID: \"e590f1fe-f50e-4ca1-a7cf-ac07ae240dba\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-2hpjf"
Nov 02 12:40:49.842949 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:49.842920    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e590f1fe-f50e-4ca1-a7cf-ac07ae240dba-workload-certs\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-2hpjf\" (UID: \"e590f1fe-f50e-4ca1-a7cf-ac07ae240dba\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-2hpjf"
Nov 02 12:40:49.843125 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:49.843098    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e590f1fe-f50e-4ca1-a7cf-ac07ae240dba-istiod-ca-cert\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-2hpjf\" (UID: \"e590f1fe-f50e-4ca1-a7cf-ac07ae240dba\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-2hpjf"
Nov 02 12:40:49.843199 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:49.843158    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e590f1fe-f50e-4ca1-a7cf-ac07ae240dba-istio-data\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-2hpjf\" (UID: \"e590f1fe-f50e-4ca1-a7cf-ac07ae240dba\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-2hpjf"
Nov 02 12:40:49.843260 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:49.843234    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e590f1fe-f50e-4ca1-a7cf-ac07ae240dba-workload-socket\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-2hpjf\" (UID: \"e590f1fe-f50e-4ca1-a7cf-ac07ae240dba\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-2hpjf"
Nov 02 12:40:49.844799 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:49.844778    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e590f1fe-f50e-4ca1-a7cf-ac07ae240dba-istio-envoy\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-2hpjf\" (UID: \"e590f1fe-f50e-4ca1-a7cf-ac07ae240dba\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-2hpjf"
Nov 02 12:40:49.844968 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:49.844946    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e590f1fe-f50e-4ca1-a7cf-ac07ae240dba-istio-podinfo\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-2hpjf\" (UID: \"e590f1fe-f50e-4ca1-a7cf-ac07ae240dba\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-2hpjf"
Nov 02 12:40:49.867319 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:49.867273    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e590f1fe-f50e-4ca1-a7cf-ac07ae240dba-istio-token\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-2hpjf\" (UID: \"e590f1fe-f50e-4ca1-a7cf-ac07ae240dba\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-2hpjf"
Nov 02 12:40:49.868363 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:49.868332    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvbh4\" (UniqueName: \"kubernetes.io/projected/e590f1fe-f50e-4ca1-a7cf-ac07ae240dba-kube-api-access-rvbh4\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-2hpjf\" (UID: \"e590f1fe-f50e-4ca1-a7cf-ac07ae240dba\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-2hpjf"
Nov 02 12:40:50.060226 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:50.059890    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-2hpjf"
Nov 02 12:40:50.408513 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:40:50.408473    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode590f1fe_f50e_4ca1_a7cf_ac07ae240dba.slice/crio-e786d443cccac8e794025fa066438b0068c0675a0fb459b8fec8fc7e10cadd1c WatchSource:0}: Error finding container e786d443cccac8e794025fa066438b0068c0675a0fb459b8fec8fc7e10cadd1c: Status 404 returned error can't find the container with id e786d443cccac8e794025fa066438b0068c0675a0fb459b8fec8fc7e10cadd1c
Nov 02 12:40:50.408630 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:50.408552    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-2hpjf"]
Nov 02 12:40:50.410955 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:50.410921    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:40:50.411025 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:50.410990    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:40:50.411025 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:50.411021    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:40:50.447895 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:50.447856    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-2hpjf" event={"ID":"e590f1fe-f50e-4ca1-a7cf-ac07ae240dba","Type":"ContainerStarted","Data":"e786d443cccac8e794025fa066438b0068c0675a0fb459b8fec8fc7e10cadd1c"}
Nov 02 12:40:50.708443 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:50.708414    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-64thh"
Nov 02 12:40:50.708834 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:50.708812    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-64thh"
Nov 02 12:40:50.709668 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:50.709612    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-64thh"
Nov 02 12:40:51.454577 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:51.454540    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-2hpjf" event={"ID":"e590f1fe-f50e-4ca1-a7cf-ac07ae240dba","Type":"ContainerStarted","Data":"49f1142f2c555ccb65a3759cf33dc628efcec7ec6c26d37107392a0e764c636d"}
Nov 02 12:40:51.549244 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:51.549177    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-2hpjf" podStartSLOduration=2.549162067 podStartE2EDuration="2.549162067s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:40:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:40:51.534456779 +0000 UTC m=+847.789855873" watchObservedRunningTime="2026-11-02 12:40:51.549162067 +0000 UTC m=+847.804561214"
Nov 02 12:40:52.061753 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:52.061714    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-2hpjf"
Nov 02 12:40:52.063525 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:52.063502    4559 patch_prober.go:29] interesting pod/e2e-worker-w1-61e341-openshift-default-59d97476dc-2hpjf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Nov 02 12:40:52.063607 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:52.063551    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-2hpjf" podUID="e590f1fe-f50e-4ca1-a7cf-ac07ae240dba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Nov 02 12:40:53.061834 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:53.061799    4559 patch_prober.go:29] interesting pod/e2e-worker-w1-61e341-openshift-default-59d97476dc-2hpjf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Nov 02 12:40:53.062241 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:53.061854    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-2hpjf" podUID="e590f1fe-f50e-4ca1-a7cf-ac07ae240dba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Nov 02 12:40:54.061417 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:54.061384    4559 patch_prober.go:29] interesting pod/e2e-worker-w1-61e341-openshift-default-59d97476dc-2hpjf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Nov 02 12:40:54.061579 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:54.061456    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-2hpjf" podUID="e590f1fe-f50e-4ca1-a7cf-ac07ae240dba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Nov 02 12:40:54.310884 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:54.310833    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"]
Nov 02 12:40:55.064722 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:55.061028    4559 patch_prober.go:29] interesting pod/e2e-worker-w1-61e341-openshift-default-59d97476dc-2hpjf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Nov 02 12:40:55.064722 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:55.061109    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-2hpjf" podUID="e590f1fe-f50e-4ca1-a7cf-ac07ae240dba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Nov 02 12:40:56.067907 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:56.067871    4559 patch_prober.go:29] interesting pod/e2e-worker-w1-61e341-openshift-default-59d97476dc-2hpjf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Nov 02 12:40:56.068528 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:56.068494    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-2hpjf" podUID="e590f1fe-f50e-4ca1-a7cf-ac07ae240dba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Nov 02 12:40:56.795670 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:56.790756    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-61e341/distinct-w1-61e341-kserve-787cd566f4-cjj8s" podUID="6886ce28-5800-402d-b893-f92109e289a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:40:57.064723 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:57.064071    4559 patch_prober.go:29] interesting pod/e2e-worker-w1-61e341-openshift-default-59d97476dc-2hpjf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Nov 02 12:40:57.064723 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:57.064149    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-2hpjf" podUID="e590f1fe-f50e-4ca1-a7cf-ac07ae240dba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Nov 02 12:40:57.181874 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:57.180784    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-61e341/distinct-2-w1-61e341-kserve-954cf956f-n7jmg" podUID="c5e1a5f8-97b4-469d-be9a-eae7ca995283" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:40:57.250449 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:57.249759    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843978/distinct-w4-843978-kserve-6fcf588858-4nrwx" podUID="82e5d483-e95b-43e6-9135-af960823427a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Nov 02 12:40:57.794703 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:57.791413    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-61e341/distinct-w1-61e341-kserve-787cd566f4-cjj8s" podUID="6886ce28-5800-402d-b893-f92109e289a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:40:57.794703 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:57.792117    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-61e341/distinct-w1-61e341-kserve-787cd566f4-cjj8s"
Nov 02 12:40:58.111773 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:58.092685    4559 patch_prober.go:29] interesting pod/e2e-worker-w1-61e341-openshift-default-59d97476dc-2hpjf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Nov 02 12:40:58.111773 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:58.092764    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-2hpjf" podUID="e590f1fe-f50e-4ca1-a7cf-ac07ae240dba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Nov 02 12:40:58.155692 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:58.143336    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-61e341/premium-w1-61e341-kserve-746477b99-645g6" podUID="02ff21d3-5a37-493a-ab95-8e4784c29123" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:40:58.182119 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:58.181252    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-61e341/distinct-2-w1-61e341-kserve-954cf956f-n7jmg" podUID="c5e1a5f8-97b4-469d-be9a-eae7ca995283" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:40:58.212673 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:58.191839    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-61e341/distinct-2-w1-61e341-kserve-954cf956f-n7jmg"
Nov 02 12:40:58.282778 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:58.282734    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843978/distinct-w4-843978-kserve-6fcf588858-4nrwx" podUID="82e5d483-e95b-43e6-9135-af960823427a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Nov 02 12:40:58.296610 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:58.284574    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-843978/distinct-w4-843978-kserve-6fcf588858-4nrwx"
Nov 02 12:40:58.807666 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:58.800092    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-61e341/distinct-w1-61e341-kserve-787cd566f4-cjj8s" podUID="6886ce28-5800-402d-b893-f92109e289a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:40:59.018666 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:59.013971    4559 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-61e341/distinct-w1-61e341-kserve-787cd566f4-cjj8s" podUID="6886ce28-5800-402d-b893-f92109e289a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:40:59.117879 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:59.117789    4559 patch_prober.go:29] interesting pod/e2e-worker-w1-61e341-openshift-default-59d97476dc-2hpjf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Nov 02 12:40:59.118144 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:59.118108    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-2hpjf" podUID="e590f1fe-f50e-4ca1-a7cf-ac07ae240dba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Nov 02 12:40:59.175713 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:59.174023    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-61e341/premium-w1-61e341-kserve-746477b99-645g6" podUID="02ff21d3-5a37-493a-ab95-8e4784c29123" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:40:59.175713 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:59.174706    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-61e341/premium-w1-61e341-kserve-746477b99-645g6"
Nov 02 12:40:59.187797 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:59.187745    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-61e341/distinct-2-w1-61e341-kserve-954cf956f-n7jmg" podUID="c5e1a5f8-97b4-469d-be9a-eae7ca995283" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:40:59.287783 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:59.287732    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843978/distinct-w4-843978-kserve-6fcf588858-4nrwx" podUID="82e5d483-e95b-43e6-9135-af960823427a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:40:59.755301 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:59.728445    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-61e341/distinct-2-w1-61e341-kserve-954cf956f-n7jmg"
Nov 02 12:40:59.822605 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:59.822553    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-61e341/distinct-w1-61e341-kserve-787cd566f4-cjj8s" podUID="6886ce28-5800-402d-b893-f92109e289a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Nov 02 12:40:59.855975 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:59.855945    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-61e341/premium-w1-61e341-kserve-746477b99-645g6"
Nov 02 12:40:59.859807 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:59.859748    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-61e341/distinct-w1-61e341-kserve-787cd566f4-cjj8s"
Nov 02 12:40:59.877189 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:40:59.877160    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-843978/distinct-w4-843978-kserve-6fcf588858-4nrwx"
Nov 02 12:41:00.067150 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:00.067063    4559 patch_prober.go:29] interesting pod/e2e-worker-w1-61e341-openshift-default-59d97476dc-2hpjf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Nov 02 12:41:00.067437 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:00.067397    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-2hpjf" podUID="e590f1fe-f50e-4ca1-a7cf-ac07ae240dba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Nov 02 12:41:02.065993 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:02.065946    4559 patch_prober.go:29] interesting pod/e2e-worker-w1-61e341-openshift-default-59d97476dc-2hpjf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 12:41:02.066514 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:02.066025    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-2hpjf" podUID="e590f1fe-f50e-4ca1-a7cf-ac07ae240dba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:41:02.082250 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:02.082218    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-2hpjf"
Nov 02 12:41:02.082586 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:02.082530    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-2hpjf"
Nov 02 12:41:02.083791 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:02.083768    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-2hpjf"
Nov 02 12:41:12.386981 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:12.386935    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-5b6gt"]
Nov 02 12:41:12.391506 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:12.391480    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-5b6gt"
Nov 02 12:41:12.432492 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:12.432457    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-5b6gt"]
Nov 02 12:41:12.462844 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:12.462795    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63746d03-ec60-494a-96f8-39436d2e6cb8-workload-socket\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-5b6gt\" (UID: \"63746d03-ec60-494a-96f8-39436d2e6cb8\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-5b6gt"
Nov 02 12:41:12.462844 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:12.462845    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63746d03-ec60-494a-96f8-39436d2e6cb8-istio-data\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-5b6gt\" (UID: \"63746d03-ec60-494a-96f8-39436d2e6cb8\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-5b6gt"
Nov 02 12:41:12.463101 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:12.463000    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63746d03-ec60-494a-96f8-39436d2e6cb8-istiod-ca-cert\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-5b6gt\" (UID: \"63746d03-ec60-494a-96f8-39436d2e6cb8\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-5b6gt"
Nov 02 12:41:12.463101 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:12.463043    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63746d03-ec60-494a-96f8-39436d2e6cb8-istio-podinfo\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-5b6gt\" (UID: \"63746d03-ec60-494a-96f8-39436d2e6cb8\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-5b6gt"
Nov 02 12:41:12.463101 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:12.463067    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5lpm\" (UniqueName: \"kubernetes.io/projected/63746d03-ec60-494a-96f8-39436d2e6cb8-kube-api-access-f5lpm\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-5b6gt\" (UID: \"63746d03-ec60-494a-96f8-39436d2e6cb8\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-5b6gt"
Nov 02 12:41:12.463283 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:12.463156    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63746d03-ec60-494a-96f8-39436d2e6cb8-workload-certs\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-5b6gt\" (UID: \"63746d03-ec60-494a-96f8-39436d2e6cb8\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-5b6gt"
Nov 02 12:41:12.463283 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:12.463197    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63746d03-ec60-494a-96f8-39436d2e6cb8-credential-socket\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-5b6gt\" (UID: \"63746d03-ec60-494a-96f8-39436d2e6cb8\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-5b6gt"
Nov 02 12:41:12.463391 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:12.463318    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63746d03-ec60-494a-96f8-39436d2e6cb8-istio-envoy\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-5b6gt\" (UID: \"63746d03-ec60-494a-96f8-39436d2e6cb8\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-5b6gt"
Nov 02 12:41:12.463391 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:12.463348    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63746d03-ec60-494a-96f8-39436d2e6cb8-istio-token\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-5b6gt\" (UID: \"63746d03-ec60-494a-96f8-39436d2e6cb8\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-5b6gt"
Nov 02 12:41:12.571664 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:12.571417    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63746d03-ec60-494a-96f8-39436d2e6cb8-istiod-ca-cert\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-5b6gt\" (UID: \"63746d03-ec60-494a-96f8-39436d2e6cb8\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-5b6gt"
Nov 02 12:41:12.571664 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:12.571493    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63746d03-ec60-494a-96f8-39436d2e6cb8-istio-podinfo\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-5b6gt\" (UID: \"63746d03-ec60-494a-96f8-39436d2e6cb8\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-5b6gt"
Nov 02 12:41:12.571664 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:12.571519    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5lpm\" (UniqueName: \"kubernetes.io/projected/63746d03-ec60-494a-96f8-39436d2e6cb8-kube-api-access-f5lpm\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-5b6gt\" (UID: \"63746d03-ec60-494a-96f8-39436d2e6cb8\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-5b6gt"
Nov 02 12:41:12.571664 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:12.571572    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63746d03-ec60-494a-96f8-39436d2e6cb8-workload-certs\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-5b6gt\" (UID: \"63746d03-ec60-494a-96f8-39436d2e6cb8\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-5b6gt"
Nov 02 12:41:12.571664 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:12.571618    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63746d03-ec60-494a-96f8-39436d2e6cb8-credential-socket\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-5b6gt\" (UID: \"63746d03-ec60-494a-96f8-39436d2e6cb8\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-5b6gt"
Nov 02 12:41:12.572178 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:12.571763    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63746d03-ec60-494a-96f8-39436d2e6cb8-istio-envoy\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-5b6gt\" (UID: \"63746d03-ec60-494a-96f8-39436d2e6cb8\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-5b6gt"
Nov 02 12:41:12.572178 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:12.571800    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63746d03-ec60-494a-96f8-39436d2e6cb8-istio-token\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-5b6gt\" (UID: \"63746d03-ec60-494a-96f8-39436d2e6cb8\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-5b6gt"
Nov 02 12:41:12.572178 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:12.571868    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63746d03-ec60-494a-96f8-39436d2e6cb8-workload-socket\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-5b6gt\" (UID: \"63746d03-ec60-494a-96f8-39436d2e6cb8\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-5b6gt"
Nov 02 12:41:12.572178 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:12.571892    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63746d03-ec60-494a-96f8-39436d2e6cb8-istio-data\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-5b6gt\" (UID: \"63746d03-ec60-494a-96f8-39436d2e6cb8\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-5b6gt"
Nov 02 12:41:12.572398 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:12.572318    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63746d03-ec60-494a-96f8-39436d2e6cb8-istio-data\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-5b6gt\" (UID: \"63746d03-ec60-494a-96f8-39436d2e6cb8\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-5b6gt"
Nov 02 12:41:12.575658 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:12.572899    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63746d03-ec60-494a-96f8-39436d2e6cb8-workload-certs\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-5b6gt\" (UID: \"63746d03-ec60-494a-96f8-39436d2e6cb8\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-5b6gt"
Nov 02 12:41:12.575658 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:12.573136    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63746d03-ec60-494a-96f8-39436d2e6cb8-credential-socket\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-5b6gt\" (UID: \"63746d03-ec60-494a-96f8-39436d2e6cb8\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-5b6gt"
Nov 02 12:41:12.575658 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:12.573299    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63746d03-ec60-494a-96f8-39436d2e6cb8-istiod-ca-cert\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-5b6gt\" (UID: \"63746d03-ec60-494a-96f8-39436d2e6cb8\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-5b6gt"
Nov 02 12:41:12.575658 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:12.573909    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63746d03-ec60-494a-96f8-39436d2e6cb8-workload-socket\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-5b6gt\" (UID: \"63746d03-ec60-494a-96f8-39436d2e6cb8\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-5b6gt"
Nov 02 12:41:12.575658 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:12.574792    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63746d03-ec60-494a-96f8-39436d2e6cb8-istio-podinfo\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-5b6gt\" (UID: \"63746d03-ec60-494a-96f8-39436d2e6cb8\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-5b6gt"
Nov 02 12:41:12.576875 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:12.576852    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63746d03-ec60-494a-96f8-39436d2e6cb8-istio-envoy\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-5b6gt\" (UID: \"63746d03-ec60-494a-96f8-39436d2e6cb8\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-5b6gt"
Nov 02 12:41:12.604429 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:12.604392    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5lpm\" (UniqueName: \"kubernetes.io/projected/63746d03-ec60-494a-96f8-39436d2e6cb8-kube-api-access-f5lpm\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-5b6gt\" (UID: \"63746d03-ec60-494a-96f8-39436d2e6cb8\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-5b6gt"
Nov 02 12:41:12.613191 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:12.613157    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63746d03-ec60-494a-96f8-39436d2e6cb8-istio-token\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-5b6gt\" (UID: \"63746d03-ec60-494a-96f8-39436d2e6cb8\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-5b6gt"
Nov 02 12:41:12.709096 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:12.709044    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-5b6gt"
Nov 02 12:41:13.832128 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:13.829524    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-bb4qf"]
Nov 02 12:41:13.957151 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:41:13.956465    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63746d03_ec60_494a_96f8_39436d2e6cb8.slice/crio-169ce93027b9a3faf245e5d9fc3740c5147e05493ad20598003fe47241c4931a WatchSource:0}: Error finding container 169ce93027b9a3faf245e5d9fc3740c5147e05493ad20598003fe47241c4931a: Status 404 returned error can't find the container with id 169ce93027b9a3faf245e5d9fc3740c5147e05493ad20598003fe47241c4931a
Nov 02 12:41:14.144666 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:14.141354    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:41:14.144666 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:14.141517    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:41:14.144666 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:14.142074    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:41:14.279348 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:14.278520    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dae5c89b-c8f9-46e6-9ca9-5444d1cb9269-istio-envoy\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-bb4qf\" (UID: \"dae5c89b-c8f9-46e6-9ca9-5444d1cb9269\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-bb4qf"
Nov 02 12:41:14.279348 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:14.278684    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dae5c89b-c8f9-46e6-9ca9-5444d1cb9269-istio-podinfo\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-bb4qf\" (UID: \"dae5c89b-c8f9-46e6-9ca9-5444d1cb9269\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-bb4qf"
Nov 02 12:41:14.279348 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:14.278718    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cd8gw\" (UniqueName: \"kubernetes.io/projected/dae5c89b-c8f9-46e6-9ca9-5444d1cb9269-kube-api-access-cd8gw\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-bb4qf\" (UID: \"dae5c89b-c8f9-46e6-9ca9-5444d1cb9269\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-bb4qf"
Nov 02 12:41:14.279348 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:14.278785    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dae5c89b-c8f9-46e6-9ca9-5444d1cb9269-workload-socket\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-bb4qf\" (UID: \"dae5c89b-c8f9-46e6-9ca9-5444d1cb9269\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-bb4qf"
Nov 02 12:41:14.279348 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:14.278811    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dae5c89b-c8f9-46e6-9ca9-5444d1cb9269-credential-socket\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-bb4qf\" (UID: \"dae5c89b-c8f9-46e6-9ca9-5444d1cb9269\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-bb4qf"
Nov 02 12:41:14.279348 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:14.278941    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dae5c89b-c8f9-46e6-9ca9-5444d1cb9269-workload-certs\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-bb4qf\" (UID: \"dae5c89b-c8f9-46e6-9ca9-5444d1cb9269\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-bb4qf"
Nov 02 12:41:14.279348 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:14.278966    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dae5c89b-c8f9-46e6-9ca9-5444d1cb9269-istiod-ca-cert\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-bb4qf\" (UID: \"dae5c89b-c8f9-46e6-9ca9-5444d1cb9269\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-bb4qf"
Nov 02 12:41:14.279348 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:14.278994    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dae5c89b-c8f9-46e6-9ca9-5444d1cb9269-istio-token\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-bb4qf\" (UID: \"dae5c89b-c8f9-46e6-9ca9-5444d1cb9269\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-bb4qf"
Nov 02 12:41:14.279348 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:14.279139    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dae5c89b-c8f9-46e6-9ca9-5444d1cb9269-istio-data\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-bb4qf\" (UID: \"dae5c89b-c8f9-46e6-9ca9-5444d1cb9269\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-bb4qf"
Nov 02 12:41:14.298787 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:14.298746    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-bb4qf"
Nov 02 12:41:14.383664 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:14.383078    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dae5c89b-c8f9-46e6-9ca9-5444d1cb9269-istio-podinfo\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-bb4qf\" (UID: \"dae5c89b-c8f9-46e6-9ca9-5444d1cb9269\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-bb4qf"
Nov 02 12:41:14.383664 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:14.383125    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cd8gw\" (UniqueName: \"kubernetes.io/projected/dae5c89b-c8f9-46e6-9ca9-5444d1cb9269-kube-api-access-cd8gw\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-bb4qf\" (UID: \"dae5c89b-c8f9-46e6-9ca9-5444d1cb9269\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-bb4qf"
Nov 02 12:41:14.383664 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:14.383175    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dae5c89b-c8f9-46e6-9ca9-5444d1cb9269-workload-socket\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-bb4qf\" (UID: \"dae5c89b-c8f9-46e6-9ca9-5444d1cb9269\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-bb4qf"
Nov 02 12:41:14.383664 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:14.383209    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dae5c89b-c8f9-46e6-9ca9-5444d1cb9269-credential-socket\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-bb4qf\" (UID: \"dae5c89b-c8f9-46e6-9ca9-5444d1cb9269\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-bb4qf"
Nov 02 12:41:14.383664 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:14.383306    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dae5c89b-c8f9-46e6-9ca9-5444d1cb9269-workload-certs\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-bb4qf\" (UID: \"dae5c89b-c8f9-46e6-9ca9-5444d1cb9269\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-bb4qf"
Nov 02 12:41:14.383664 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:14.383337    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dae5c89b-c8f9-46e6-9ca9-5444d1cb9269-istiod-ca-cert\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-bb4qf\" (UID: \"dae5c89b-c8f9-46e6-9ca9-5444d1cb9269\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-bb4qf"
Nov 02 12:41:14.383664 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:14.383365    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dae5c89b-c8f9-46e6-9ca9-5444d1cb9269-istio-token\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-bb4qf\" (UID: \"dae5c89b-c8f9-46e6-9ca9-5444d1cb9269\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-bb4qf"
Nov 02 12:41:14.383664 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:14.383424    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dae5c89b-c8f9-46e6-9ca9-5444d1cb9269-istio-data\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-bb4qf\" (UID: \"dae5c89b-c8f9-46e6-9ca9-5444d1cb9269\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-bb4qf"
Nov 02 12:41:14.383664 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:14.383492    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dae5c89b-c8f9-46e6-9ca9-5444d1cb9269-istio-envoy\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-bb4qf\" (UID: \"dae5c89b-c8f9-46e6-9ca9-5444d1cb9269\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-bb4qf"
Nov 02 12:41:14.424665 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:14.421047    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dae5c89b-c8f9-46e6-9ca9-5444d1cb9269-credential-socket\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-bb4qf\" (UID: \"dae5c89b-c8f9-46e6-9ca9-5444d1cb9269\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-bb4qf"
Nov 02 12:41:14.424665 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:14.421387    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dae5c89b-c8f9-46e6-9ca9-5444d1cb9269-workload-certs\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-bb4qf\" (UID: \"dae5c89b-c8f9-46e6-9ca9-5444d1cb9269\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-bb4qf"
Nov 02 12:41:14.424665 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:14.421650    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dae5c89b-c8f9-46e6-9ca9-5444d1cb9269-istio-data\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-bb4qf\" (UID: \"dae5c89b-c8f9-46e6-9ca9-5444d1cb9269\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-bb4qf"
Nov 02 12:41:14.448816 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:14.438170    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dae5c89b-c8f9-46e6-9ca9-5444d1cb9269-istiod-ca-cert\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-bb4qf\" (UID: \"dae5c89b-c8f9-46e6-9ca9-5444d1cb9269\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-bb4qf"
Nov 02 12:41:14.448816 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:14.438409    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dae5c89b-c8f9-46e6-9ca9-5444d1cb9269-workload-socket\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-bb4qf\" (UID: \"dae5c89b-c8f9-46e6-9ca9-5444d1cb9269\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-bb4qf"
Nov 02 12:41:14.517939 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:14.517906    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-bb4qf"]
Nov 02 12:41:14.523798 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:14.523768    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-5b6gt"]
Nov 02 12:41:14.658576 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:14.655677    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dae5c89b-c8f9-46e6-9ca9-5444d1cb9269-istio-token\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-bb4qf\" (UID: \"dae5c89b-c8f9-46e6-9ca9-5444d1cb9269\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-bb4qf"
Nov 02 12:41:14.695681 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:14.695264    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dae5c89b-c8f9-46e6-9ca9-5444d1cb9269-istio-podinfo\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-bb4qf\" (UID: \"dae5c89b-c8f9-46e6-9ca9-5444d1cb9269\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-bb4qf"
Nov 02 12:41:14.699664 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:14.695955    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cd8gw\" (UniqueName: \"kubernetes.io/projected/dae5c89b-c8f9-46e6-9ca9-5444d1cb9269-kube-api-access-cd8gw\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-bb4qf\" (UID: \"dae5c89b-c8f9-46e6-9ca9-5444d1cb9269\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-bb4qf"
Nov 02 12:41:14.775680 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:14.766763    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dae5c89b-c8f9-46e6-9ca9-5444d1cb9269-istio-envoy\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-bb4qf\" (UID: \"dae5c89b-c8f9-46e6-9ca9-5444d1cb9269\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-bb4qf"
Nov 02 12:41:14.845766 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:14.816375    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-bb4qf"
Nov 02 12:41:14.884820 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:14.884763    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-61e341/premium-w1-61e341-kserve-746477b99-645g6" podUID="02ff21d3-5a37-493a-ab95-8e4784c29123" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded"
Nov 02 12:41:14.903532 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:14.889882    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843978/distinct-w4-843978-kserve-6fcf588858-4nrwx" podUID="82e5d483-e95b-43e6-9135-af960823427a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:41:14.918727 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:14.913389    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-61e341/distinct-w1-61e341-kserve-787cd566f4-cjj8s" podUID="6886ce28-5800-402d-b893-f92109e289a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Nov 02 12:41:15.364723 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:15.364222    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-5b6gt" event={"ID":"63746d03-ec60-494a-96f8-39436d2e6cb8","Type":"ContainerStarted","Data":"7e5d2e39bab8890591db07e72702275305d7634ef536bb27272992d1f8026061"}
Nov 02 12:41:15.364723 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:15.364289    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-5b6gt" event={"ID":"63746d03-ec60-494a-96f8-39436d2e6cb8","Type":"ContainerStarted","Data":"169ce93027b9a3faf245e5d9fc3740c5147e05493ad20598003fe47241c4931a"}
Nov 02 12:41:15.464662 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:41:15.463902    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqcmj" containerName="extractor"
Nov 02 12:41:15.464662 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:15.464519    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-5b6gt" podStartSLOduration=3.464501496 podStartE2EDuration="3.464501496s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:41:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:41:15.464106527 +0000 UTC m=+871.719505635" watchObservedRunningTime="2026-11-02 12:41:15.464501496 +0000 UTC m=+871.719900592"
Nov 02 12:41:15.479161 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:15.478723    4559 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-843978/distinct-w4-843978-kserve-6fcf588858-4nrwx" podUID="82e5d483-e95b-43e6-9135-af960823427a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:41:15.794667 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:15.738780    4559 patch_prober.go:29] interesting pod/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-5b6gt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Nov 02 12:41:15.794667 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:15.738876    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-5b6gt" podUID="63746d03-ec60-494a-96f8-39436d2e6cb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Nov 02 12:41:15.794667 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:15.739117    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-5b6gt"
Nov 02 12:41:15.892205 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:15.892131    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843978/distinct-w4-843978-kserve-6fcf588858-4nrwx" podUID="82e5d483-e95b-43e6-9135-af960823427a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:41:15.898421 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:15.898289    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-61e341/premium-w1-61e341-kserve-746477b99-645g6" podUID="02ff21d3-5a37-493a-ab95-8e4784c29123" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:41:15.946664 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:15.918850    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-61e341/distinct-w1-61e341-kserve-787cd566f4-cjj8s" podUID="6886ce28-5800-402d-b893-f92109e289a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:41:15.998392 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:15.998345    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-843978/distinct-w4-843978-kserve-6fcf588858-4nrwx"
Nov 02 12:41:16.075716 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:16.055265    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-61e341/premium-w1-61e341-kserve-746477b99-645g6"
Nov 02 12:41:16.575677 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:16.573035    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-61e341/distinct-w1-61e341-kserve-787cd566f4-cjj8s"
Nov 02 12:41:16.575677 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:16.316094    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-61e341/distinct-2-w1-61e341-kserve-954cf956f-n7jmg" podUID="c5e1a5f8-97b4-469d-be9a-eae7ca995283" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded"
Nov 02 12:41:16.575677 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:16.462512    4559 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-61e341/premium-w1-61e341-kserve-746477b99-645g6" podUID="02ff21d3-5a37-493a-ab95-8e4784c29123" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/health\": context deadline exceeded"
Nov 02 12:41:16.783048 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:16.760297    4559 patch_prober.go:29] interesting pod/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-5b6gt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Nov 02 12:41:16.783048 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:16.760370    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-5b6gt" podUID="63746d03-ec60-494a-96f8-39436d2e6cb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Nov 02 12:41:16.906919 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:16.906818    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843978/distinct-w4-843978-kserve-6fcf588858-4nrwx" podUID="82e5d483-e95b-43e6-9135-af960823427a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:41:16.950776 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:16.950711    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-61e341/distinct-w1-61e341-kserve-787cd566f4-cjj8s" podUID="6886ce28-5800-402d-b893-f92109e289a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Nov 02 12:41:16.993665 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:16.951531    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-61e341/premium-w1-61e341-kserve-746477b99-645g6" podUID="02ff21d3-5a37-493a-ab95-8e4784c29123" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded"
Nov 02 12:41:17.597683 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:17.584019    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-61e341/distinct-2-w1-61e341-kserve-954cf956f-n7jmg" podUID="c5e1a5f8-97b4-469d-be9a-eae7ca995283" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:41:17.627665 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:17.613594    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-61e341/distinct-2-w1-61e341-kserve-954cf956f-n7jmg"
Nov 02 12:41:17.942704 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:17.922602    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:41:18.043672 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:17.928027    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843978/distinct-w4-843978-kserve-6fcf588858-4nrwx" podUID="82e5d483-e95b-43e6-9135-af960823427a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Nov 02 12:41:18.043672 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:17.983042    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:41:18.043672 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:17.983213    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:41:18.043672 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:18.026476    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-61e341/premium-w1-61e341-kserve-746477b99-645g6" podUID="02ff21d3-5a37-493a-ab95-8e4784c29123" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded"
Nov 02 12:41:18.192896 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:18.192776    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-61e341/distinct-w1-61e341-kserve-787cd566f4-cjj8s" podUID="6886ce28-5800-402d-b893-f92109e289a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:41:18.336057 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:18.193742    4559 patch_prober.go:29] interesting pod/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-5b6gt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Nov 02 12:41:18.336925 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:18.336889    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-5b6gt" podUID="63746d03-ec60-494a-96f8-39436d2e6cb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Nov 02 12:41:18.360702 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:18.354755    4559 patch_prober.go:29] interesting pod/e2e-worker-w1-61e341-openshift-default-59d97476dc-2hpjf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 12:41:18.360702 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:18.354823    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-2hpjf" podUID="e590f1fe-f50e-4ca1-a7cf-ac07ae240dba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:41:18.520812 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:18.520715    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-bb4qf"]
Nov 02 12:41:18.614667 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:18.585998    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-61e341/distinct-2-w1-61e341-kserve-954cf956f-n7jmg" podUID="c5e1a5f8-97b4-469d-be9a-eae7ca995283" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:41:18.615274 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:18.586722    4559 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-k5zn8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 12:41:18.615483 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:18.615442    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-k5zn8" podUID="d3b2832d-2aad-406f-86b5-de405575f9b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:41:18.615633 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:18.599207    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-bb4qf" event={"ID":"dae5c89b-c8f9-46e6-9ca9-5444d1cb9269","Type":"ContainerStarted","Data":"518819b8e9c9eefd171e0325b16b1141fc50864b786febe2471ace99739ab374"}
Nov 02 12:41:18.740688 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:18.726803    4559 patch_prober.go:29] interesting pod/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-5b6gt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Nov 02 12:41:18.740688 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:18.726865    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-5b6gt" podUID="63746d03-ec60-494a-96f8-39436d2e6cb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Nov 02 12:41:18.957692 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:18.957249    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-843978/distinct-w4-843978-kserve-6fcf588858-4nrwx"
Nov 02 12:41:19.057770 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:19.030734    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-61e341/premium-w1-61e341-kserve-746477b99-645g6" podUID="02ff21d3-5a37-493a-ab95-8e4784c29123" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:41:19.343703 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:19.336149    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-61e341/distinct-w1-61e341-kserve-787cd566f4-cjj8s" podUID="6886ce28-5800-402d-b893-f92109e289a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:41:19.414665 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:19.412091    4559 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-61e341/distinct-w1-61e341-kserve-787cd566f4-cjj8s" podUID="6886ce28-5800-402d-b893-f92109e289a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": context deadline exceeded"
Nov 02 12:41:19.621882 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:19.621780    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-61e341/distinct-2-w1-61e341-kserve-954cf956f-n7jmg" podUID="c5e1a5f8-97b4-469d-be9a-eae7ca995283" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:41:19.732824 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:19.729297    4559 patch_prober.go:29] interesting pod/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-5b6gt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Nov 02 12:41:19.732824 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:19.729364    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-5b6gt" podUID="63746d03-ec60-494a-96f8-39436d2e6cb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Nov 02 12:41:19.861754 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:19.859480    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-bb4qf" event={"ID":"dae5c89b-c8f9-46e6-9ca9-5444d1cb9269","Type":"ContainerStarted","Data":"4a13ec4d340bcfa618bd48ffa445df3e5a11ce3a54b3e020d57c6e445e7aeb03"}
Nov 02 12:41:19.967724 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:19.959518    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843978/distinct-w4-843978-kserve-6fcf588858-4nrwx" podUID="82e5d483-e95b-43e6-9135-af960823427a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:41:20.035660 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:20.031400    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-61e341/premium-w1-61e341-kserve-746477b99-645g6" podUID="02ff21d3-5a37-493a-ab95-8e4784c29123" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:41:20.347764 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:20.346588    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-61e341/distinct-w1-61e341-kserve-787cd566f4-cjj8s" podUID="6886ce28-5800-402d-b893-f92109e289a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:41:20.629870 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:20.629774    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-61e341/distinct-2-w1-61e341-kserve-954cf956f-n7jmg" podUID="c5e1a5f8-97b4-469d-be9a-eae7ca995283" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:41:20.823664 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:20.818338    4559 patch_prober.go:29] interesting pod/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-5b6gt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Nov 02 12:41:20.823664 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:20.818402    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-5b6gt" podUID="63746d03-ec60-494a-96f8-39436d2e6cb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Nov 02 12:41:20.837744 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:20.837710    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-bb4qf"
Nov 02 12:41:20.873664 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:20.852179    4559 patch_prober.go:29] interesting pod/e2e-worker-w3-068756-openshift-default-84f69c9cf5-bb4qf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Nov 02 12:41:20.873664 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:20.852236    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-bb4qf" podUID="dae5c89b-c8f9-46e6-9ca9-5444d1cb9269" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Nov 02 12:41:20.970675 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:20.960749    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843978/distinct-w4-843978-kserve-6fcf588858-4nrwx" podUID="82e5d483-e95b-43e6-9135-af960823427a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Nov 02 12:41:20.970675 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:20.961603    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-843978/distinct-w4-843978-kserve-6fcf588858-4nrwx"
Nov 02 12:41:21.022285 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:21.022220    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-bb4qf" podStartSLOduration=8.022196938 podStartE2EDuration="8.022196938s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:41:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:41:19.94348329 +0000 UTC m=+876.198882387" watchObservedRunningTime="2026-11-02 12:41:21.022196938 +0000 UTC m=+877.277596035"
Nov 02 12:41:21.099689 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:21.099628    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-61e341/premium-w1-61e341-kserve-746477b99-645g6" podUID="02ff21d3-5a37-493a-ab95-8e4784c29123" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded"
Nov 02 12:41:21.226752 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:21.185904    4559 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-66rd4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 12:41:21.226752 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:21.185976    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-66rd4" podUID="e09c5fe2-c1dd-47ab-8464-facefd253549" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:41:21.360877 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:21.360818    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-61e341/distinct-w1-61e341-kserve-787cd566f4-cjj8s" podUID="6886ce28-5800-402d-b893-f92109e289a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Nov 02 12:41:21.570728 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:21.546072    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-843978/distinct-w4-843978-kserve-6fcf588858-4nrwx"
Nov 02 12:41:21.696162 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:21.646754    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-61e341/distinct-2-w1-61e341-kserve-954cf956f-n7jmg" podUID="c5e1a5f8-97b4-469d-be9a-eae7ca995283" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:41:21.754711 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:21.732740    4559 patch_prober.go:29] interesting pod/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-5b6gt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Nov 02 12:41:21.754711 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:21.732813    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-5b6gt" podUID="63746d03-ec60-494a-96f8-39436d2e6cb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Nov 02 12:41:21.846750 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:21.829233    4559 patch_prober.go:29] interesting pod/e2e-worker-w3-068756-openshift-default-84f69c9cf5-bb4qf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Nov 02 12:41:21.846750 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:21.829292    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-bb4qf" podUID="dae5c89b-c8f9-46e6-9ca9-5444d1cb9269" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Nov 02 12:41:22.129938 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:22.129822    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-61e341/premium-w1-61e341-kserve-746477b99-645g6" podUID="02ff21d3-5a37-493a-ab95-8e4784c29123" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded"
Nov 02 12:41:22.412731 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:22.401368    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-61e341/distinct-w1-61e341-kserve-787cd566f4-cjj8s" podUID="6886ce28-5800-402d-b893-f92109e289a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:41:22.533661 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:22.529082    4559 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-61e341/distinct-2-w1-61e341-kserve-954cf956f-n7jmg" podUID="c5e1a5f8-97b4-469d-be9a-eae7ca995283" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:41:22.581663 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:22.568874    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843978/distinct-w4-843978-kserve-6fcf588858-4nrwx" podUID="82e5d483-e95b-43e6-9135-af960823427a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:41:22.650817 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:22.650758    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-61e341/distinct-2-w1-61e341-kserve-954cf956f-n7jmg" podUID="c5e1a5f8-97b4-469d-be9a-eae7ca995283" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:41:22.744668 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:22.740980    4559 patch_prober.go:29] interesting pod/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-5b6gt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Nov 02 12:41:22.751756 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:22.751710    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-5b6gt" podUID="63746d03-ec60-494a-96f8-39436d2e6cb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Nov 02 12:41:22.827664 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:22.827462    4559 patch_prober.go:29] interesting pod/e2e-worker-w3-068756-openshift-default-84f69c9cf5-bb4qf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Nov 02 12:41:22.827842 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:22.827617    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-bb4qf" podUID="dae5c89b-c8f9-46e6-9ca9-5444d1cb9269" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Nov 02 12:41:23.147718 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:23.147008    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-61e341/premium-w1-61e341-kserve-746477b99-645g6" podUID="02ff21d3-5a37-493a-ab95-8e4784c29123" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:41:23.265224 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:23.265195    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-61e341/premium-w1-61e341-kserve-746477b99-645g6"
Nov 02 12:41:23.428981 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:23.428936    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-61e341/distinct-w1-61e341-kserve-787cd566f4-cjj8s" podUID="6886ce28-5800-402d-b893-f92109e289a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:41:23.465960 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:23.465573    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-61e341/distinct-2-w1-61e341-kserve-954cf956f-n7jmg"
Nov 02 12:41:23.468066 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:23.468040    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-61e341/distinct-w1-61e341-kserve-787cd566f4-cjj8s"
Nov 02 12:41:23.714750 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:23.712018    4559 patch_prober.go:29] interesting pod/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-5b6gt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Nov 02 12:41:23.714750 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:23.712085    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-5b6gt" podUID="63746d03-ec60-494a-96f8-39436d2e6cb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Nov 02 12:41:23.832666 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:23.825173    4559 patch_prober.go:29] interesting pod/e2e-worker-w3-068756-openshift-default-84f69c9cf5-bb4qf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Nov 02 12:41:23.832666 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:23.825233    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-bb4qf" podUID="dae5c89b-c8f9-46e6-9ca9-5444d1cb9269" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Nov 02 12:41:24.711662 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:24.710991    4559 patch_prober.go:29] interesting pod/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-5b6gt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Nov 02 12:41:24.711662 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:24.711052    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-5b6gt" podUID="63746d03-ec60-494a-96f8-39436d2e6cb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Nov 02 12:41:24.828663 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:24.828177    4559 patch_prober.go:29] interesting pod/e2e-worker-w3-068756-openshift-default-84f69c9cf5-bb4qf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Nov 02 12:41:24.828663 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:24.828242    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-bb4qf" podUID="dae5c89b-c8f9-46e6-9ca9-5444d1cb9269" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Nov 02 12:41:25.714670 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:25.710711    4559 patch_prober.go:29] interesting pod/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-5b6gt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Nov 02 12:41:25.714670 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:25.710774    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-5b6gt" podUID="63746d03-ec60-494a-96f8-39436d2e6cb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Nov 02 12:41:25.833660 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:25.831029    4559 patch_prober.go:29] interesting pod/e2e-worker-w3-068756-openshift-default-84f69c9cf5-bb4qf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Nov 02 12:41:25.833660 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:25.831094    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-bb4qf" podUID="dae5c89b-c8f9-46e6-9ca9-5444d1cb9269" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Nov 02 12:41:26.726665 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:26.717665    4559 patch_prober.go:29] interesting pod/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-5b6gt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Nov 02 12:41:26.726665 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:26.717726    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-5b6gt" podUID="63746d03-ec60-494a-96f8-39436d2e6cb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Nov 02 12:41:26.825459 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:26.825417    4559 patch_prober.go:29] interesting pod/e2e-worker-w3-068756-openshift-default-84f69c9cf5-bb4qf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Nov 02 12:41:26.825722 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:26.825488    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-bb4qf" podUID="dae5c89b-c8f9-46e6-9ca9-5444d1cb9269" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Nov 02 12:41:27.714832 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:27.714781    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-5b6gt"
Nov 02 12:41:27.826157 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:27.826121    4559 patch_prober.go:29] interesting pod/e2e-worker-w3-068756-openshift-default-84f69c9cf5-bb4qf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Nov 02 12:41:27.826564 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:27.826207    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-bb4qf" podUID="dae5c89b-c8f9-46e6-9ca9-5444d1cb9269" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Nov 02 12:41:28.332515 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:28.332483    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-5b6gt"
Nov 02 12:41:28.334173 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:28.334113    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-5b6gt"
Nov 02 12:41:28.825947 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:28.825910    4559 patch_prober.go:29] interesting pod/e2e-worker-w3-068756-openshift-default-84f69c9cf5-bb4qf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Nov 02 12:41:28.826109 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:28.826017    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-bb4qf" podUID="dae5c89b-c8f9-46e6-9ca9-5444d1cb9269" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Nov 02 12:41:29.826790 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:29.826499    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-bb4qf"
Nov 02 12:41:29.832577 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:29.832544    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-bb4qf"
Nov 02 12:41:30.152920 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:30.152841    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-bb4qf"
Nov 02 12:41:30.423066 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:30.423020    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"]
Nov 02 12:41:31.392235 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:31.392204    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-cd1386cb-556c7759d7-m98mv"]
Nov 02 12:41:31.396417 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:31.396397    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-cd1386cb-556c7759d7-m98mv"
Nov 02 12:41:31.416833 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:31.416807    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-cd1386cb-556c7759d7-m98mv"]
Nov 02 12:41:31.425821 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:31.425791    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-cd1386cb-dockercfg-klvbg\""
Nov 02 12:41:31.425985 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:31.425889    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-cd1386cb\""
Nov 02 12:41:31.540578 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:31.540530    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7f08638-accd-4b0d-9657-652d7c8df22b-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-cd1386cb-556c7759d7-m98mv\" (UID: \"e7f08638-accd-4b0d-9657-652d7c8df22b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-cd1386cb-556c7759d7-m98mv"
Nov 02 12:41:31.540758 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:31.540691    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4r85\" (UniqueName: \"kubernetes.io/projected/e7f08638-accd-4b0d-9657-652d7c8df22b-kube-api-access-w4r85\") pod \"payload-pre-processing-e2e-mt-cd1386cb-556c7759d7-m98mv\" (UID: \"e7f08638-accd-4b0d-9657-652d7c8df22b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-cd1386cb-556c7759d7-m98mv"
Nov 02 12:41:31.642106 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:31.642066    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7f08638-accd-4b0d-9657-652d7c8df22b-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-cd1386cb-556c7759d7-m98mv\" (UID: \"e7f08638-accd-4b0d-9657-652d7c8df22b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-cd1386cb-556c7759d7-m98mv"
Nov 02 12:41:31.642327 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:31.642164    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4r85\" (UniqueName: \"kubernetes.io/projected/e7f08638-accd-4b0d-9657-652d7c8df22b-kube-api-access-w4r85\") pod \"payload-pre-processing-e2e-mt-cd1386cb-556c7759d7-m98mv\" (UID: \"e7f08638-accd-4b0d-9657-652d7c8df22b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-cd1386cb-556c7759d7-m98mv"
Nov 02 12:41:31.643040 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:31.643011    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7f08638-accd-4b0d-9657-652d7c8df22b-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-cd1386cb-556c7759d7-m98mv\" (UID: \"e7f08638-accd-4b0d-9657-652d7c8df22b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-cd1386cb-556c7759d7-m98mv"
Nov 02 12:41:31.666634 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:31.666603    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4r85\" (UniqueName: \"kubernetes.io/projected/e7f08638-accd-4b0d-9657-652d7c8df22b-kube-api-access-w4r85\") pod \"payload-pre-processing-e2e-mt-cd1386cb-556c7759d7-m98mv\" (UID: \"e7f08638-accd-4b0d-9657-652d7c8df22b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-cd1386cb-556c7759d7-m98mv"
Nov 02 12:41:31.714973 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:31.714589    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-cd1386cb-556c7759d7-m98mv"
Nov 02 12:41:32.294403 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:32.294376    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-cd1386cb-556c7759d7-m98mv"]
Nov 02 12:41:33.375861 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:33.375813    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-cd1386cb-556c7759d7-m98mv" event={"ID":"e7f08638-accd-4b0d-9657-652d7c8df22b","Type":"ContainerStarted","Data":"0986494e30f02d8b39e36ecd1c6b1572a781177b02a3a35df84e4fb1d1b56719"}
Nov 02 12:41:33.376505 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:33.376479    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-cd1386cb-556c7759d7-m98mv" event={"ID":"e7f08638-accd-4b0d-9657-652d7c8df22b","Type":"ContainerStarted","Data":"4ec895e82405aa75aff03839df9f4e0055c21f1af49f18c536cc183b1e7868d6"}
Nov 02 12:41:33.381628 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:33.381604    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-cd1386cb-556c7759d7-m98mv"
Nov 02 12:41:34.395932 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:34.394942    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-61e341/distinct-w1-61e341-kserve-787cd566f4-cjj8s" podUID="6886ce28-5800-402d-b893-f92109e289a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:41:34.936665 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:34.935662    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-cd1386cb-556c7759d7-m98mv" podStartSLOduration=3.935627981 podStartE2EDuration="3.935627981s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:41:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:41:33.514990229 +0000 UTC m=+889.770389337" watchObservedRunningTime="2026-11-02 12:41:34.935627981 +0000 UTC m=+891.191027077"
Nov 02 12:41:34.947665 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:34.944776    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-gr5tx"]
Nov 02 12:41:34.973660 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:34.955895    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-gr5tx"
Nov 02 12:41:34.999662 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:34.975305    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-gr5tx"]
Nov 02 12:41:35.098664 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:35.098193    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d507435-7115-4950-93c5-da6c04d73874-istio-envoy\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-gr5tx\" (UID: \"0d507435-7115-4950-93c5-da6c04d73874\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-gr5tx"
Nov 02 12:41:35.098664 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:35.098254    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zxfw\" (UniqueName: \"kubernetes.io/projected/0d507435-7115-4950-93c5-da6c04d73874-kube-api-access-7zxfw\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-gr5tx\" (UID: \"0d507435-7115-4950-93c5-da6c04d73874\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-gr5tx"
Nov 02 12:41:35.098664 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:35.098294    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d507435-7115-4950-93c5-da6c04d73874-istiod-ca-cert\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-gr5tx\" (UID: \"0d507435-7115-4950-93c5-da6c04d73874\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-gr5tx"
Nov 02 12:41:35.098664 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:35.098344    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d507435-7115-4950-93c5-da6c04d73874-workload-certs\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-gr5tx\" (UID: \"0d507435-7115-4950-93c5-da6c04d73874\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-gr5tx"
Nov 02 12:41:35.098664 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:35.098387    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d507435-7115-4950-93c5-da6c04d73874-workload-socket\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-gr5tx\" (UID: \"0d507435-7115-4950-93c5-da6c04d73874\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-gr5tx"
Nov 02 12:41:35.098664 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:35.098473    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d507435-7115-4950-93c5-da6c04d73874-istio-token\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-gr5tx\" (UID: \"0d507435-7115-4950-93c5-da6c04d73874\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-gr5tx"
Nov 02 12:41:35.098664 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:35.098500    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d507435-7115-4950-93c5-da6c04d73874-istio-data\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-gr5tx\" (UID: \"0d507435-7115-4950-93c5-da6c04d73874\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-gr5tx"
Nov 02 12:41:35.098664 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:35.098595    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d507435-7115-4950-93c5-da6c04d73874-credential-socket\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-gr5tx\" (UID: \"0d507435-7115-4950-93c5-da6c04d73874\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-gr5tx"
Nov 02 12:41:35.099266 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:35.098783    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d507435-7115-4950-93c5-da6c04d73874-istio-podinfo\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-gr5tx\" (UID: \"0d507435-7115-4950-93c5-da6c04d73874\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-gr5tx"
Nov 02 12:41:35.235727 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:35.217496    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d507435-7115-4950-93c5-da6c04d73874-istio-podinfo\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-gr5tx\" (UID: \"0d507435-7115-4950-93c5-da6c04d73874\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-gr5tx"
Nov 02 12:41:35.246854 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:35.246811    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d507435-7115-4950-93c5-da6c04d73874-istio-envoy\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-gr5tx\" (UID: \"0d507435-7115-4950-93c5-da6c04d73874\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-gr5tx"
Nov 02 12:41:35.259801 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:35.259766    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zxfw\" (UniqueName: \"kubernetes.io/projected/0d507435-7115-4950-93c5-da6c04d73874-kube-api-access-7zxfw\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-gr5tx\" (UID: \"0d507435-7115-4950-93c5-da6c04d73874\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-gr5tx"
Nov 02 12:41:35.284664 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:35.260051    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d507435-7115-4950-93c5-da6c04d73874-istiod-ca-cert\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-gr5tx\" (UID: \"0d507435-7115-4950-93c5-da6c04d73874\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-gr5tx"
Nov 02 12:41:35.289661 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:35.286369    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d507435-7115-4950-93c5-da6c04d73874-workload-certs\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-gr5tx\" (UID: \"0d507435-7115-4950-93c5-da6c04d73874\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-gr5tx"
Nov 02 12:41:35.335667 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:35.320818    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d507435-7115-4950-93c5-da6c04d73874-workload-socket\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-gr5tx\" (UID: \"0d507435-7115-4950-93c5-da6c04d73874\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-gr5tx"
Nov 02 12:41:35.397716 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:35.290872    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d507435-7115-4950-93c5-da6c04d73874-workload-certs\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-gr5tx\" (UID: \"0d507435-7115-4950-93c5-da6c04d73874\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-gr5tx"
Nov 02 12:41:35.397716 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:35.273251    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d507435-7115-4950-93c5-da6c04d73874-istiod-ca-cert\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-gr5tx\" (UID: \"0d507435-7115-4950-93c5-da6c04d73874\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-gr5tx"
Nov 02 12:41:35.397716 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:35.337504    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d507435-7115-4950-93c5-da6c04d73874-workload-socket\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-gr5tx\" (UID: \"0d507435-7115-4950-93c5-da6c04d73874\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-gr5tx"
Nov 02 12:41:35.397716 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:35.368289    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zxfw\" (UniqueName: \"kubernetes.io/projected/0d507435-7115-4950-93c5-da6c04d73874-kube-api-access-7zxfw\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-gr5tx\" (UID: \"0d507435-7115-4950-93c5-da6c04d73874\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-gr5tx"
Nov 02 12:41:35.397716 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:35.394264    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d507435-7115-4950-93c5-da6c04d73874-istio-token\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-gr5tx\" (UID: \"0d507435-7115-4950-93c5-da6c04d73874\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-gr5tx"
Nov 02 12:41:35.397716 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:35.394320    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d507435-7115-4950-93c5-da6c04d73874-istio-data\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-gr5tx\" (UID: \"0d507435-7115-4950-93c5-da6c04d73874\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-gr5tx"
Nov 02 12:41:35.397716 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:35.394471    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d507435-7115-4950-93c5-da6c04d73874-credential-socket\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-gr5tx\" (UID: \"0d507435-7115-4950-93c5-da6c04d73874\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-gr5tx"
Nov 02 12:41:35.397716 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:35.395178    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d507435-7115-4950-93c5-da6c04d73874-credential-socket\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-gr5tx\" (UID: \"0d507435-7115-4950-93c5-da6c04d73874\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-gr5tx"
Nov 02 12:41:35.397716 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:35.395928    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d507435-7115-4950-93c5-da6c04d73874-istio-data\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-gr5tx\" (UID: \"0d507435-7115-4950-93c5-da6c04d73874\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-gr5tx"
Nov 02 12:41:35.481173 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:35.481139    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d507435-7115-4950-93c5-da6c04d73874-istio-token\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-gr5tx\" (UID: \"0d507435-7115-4950-93c5-da6c04d73874\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-gr5tx"
Nov 02 12:41:35.545426 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:35.545341    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d507435-7115-4950-93c5-da6c04d73874-istio-podinfo\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-gr5tx\" (UID: \"0d507435-7115-4950-93c5-da6c04d73874\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-gr5tx"
Nov 02 12:41:35.551680 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:35.548182    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d507435-7115-4950-93c5-da6c04d73874-istio-envoy\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-gr5tx\" (UID: \"0d507435-7115-4950-93c5-da6c04d73874\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-gr5tx"
Nov 02 12:41:35.786920 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:35.786881    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-gr5tx"
Nov 02 12:41:36.043874 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:36.043797    4559 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-66rd4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 12:41:36.044261 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:36.044147    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-66rd4" podUID="e09c5fe2-c1dd-47ab-8464-facefd253549" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:41:36.220420 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:36.220120    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-785a7a-openshift-default-57c8cb6f7f-wp4ql"]
Nov 02 12:41:36.445748 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:36.444839    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-785a7a-openshift-default-57c8cb6f7f-wp4ql"]
Nov 02 12:41:36.445748 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:36.445003    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-785a7a-openshift-default-57c8cb6f7f-wp4ql"
Nov 02 12:41:36.537663 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:36.534482    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afbd2f7b-f730-472b-b6ee-f98250881cc3-workload-certs\") pod \"e2e-api-a-785a7a-openshift-default-57c8cb6f7f-wp4ql\" (UID: \"afbd2f7b-f730-472b-b6ee-f98250881cc3\") " pod="openshift-ingress/e2e-api-a-785a7a-openshift-default-57c8cb6f7f-wp4ql"
Nov 02 12:41:36.537663 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:36.534580    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afbd2f7b-f730-472b-b6ee-f98250881cc3-istio-podinfo\") pod \"e2e-api-a-785a7a-openshift-default-57c8cb6f7f-wp4ql\" (UID: \"afbd2f7b-f730-472b-b6ee-f98250881cc3\") " pod="openshift-ingress/e2e-api-a-785a7a-openshift-default-57c8cb6f7f-wp4ql"
Nov 02 12:41:36.537663 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:36.534616    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afbd2f7b-f730-472b-b6ee-f98250881cc3-istio-envoy\") pod \"e2e-api-a-785a7a-openshift-default-57c8cb6f7f-wp4ql\" (UID: \"afbd2f7b-f730-472b-b6ee-f98250881cc3\") " pod="openshift-ingress/e2e-api-a-785a7a-openshift-default-57c8cb6f7f-wp4ql"
Nov 02 12:41:36.537663 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:36.534781    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afbd2f7b-f730-472b-b6ee-f98250881cc3-istio-token\") pod \"e2e-api-a-785a7a-openshift-default-57c8cb6f7f-wp4ql\" (UID: \"afbd2f7b-f730-472b-b6ee-f98250881cc3\") " pod="openshift-ingress/e2e-api-a-785a7a-openshift-default-57c8cb6f7f-wp4ql"
Nov 02 12:41:36.537663 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:36.534814    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9k6dh\" (UniqueName: \"kubernetes.io/projected/afbd2f7b-f730-472b-b6ee-f98250881cc3-kube-api-access-9k6dh\") pod \"e2e-api-a-785a7a-openshift-default-57c8cb6f7f-wp4ql\" (UID: \"afbd2f7b-f730-472b-b6ee-f98250881cc3\") " pod="openshift-ingress/e2e-api-a-785a7a-openshift-default-57c8cb6f7f-wp4ql"
Nov 02 12:41:36.537663 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:36.534847    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afbd2f7b-f730-472b-b6ee-f98250881cc3-credential-socket\") pod \"e2e-api-a-785a7a-openshift-default-57c8cb6f7f-wp4ql\" (UID: \"afbd2f7b-f730-472b-b6ee-f98250881cc3\") " pod="openshift-ingress/e2e-api-a-785a7a-openshift-default-57c8cb6f7f-wp4ql"
Nov 02 12:41:36.537663 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:36.534875    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afbd2f7b-f730-472b-b6ee-f98250881cc3-istio-data\") pod \"e2e-api-a-785a7a-openshift-default-57c8cb6f7f-wp4ql\" (UID: \"afbd2f7b-f730-472b-b6ee-f98250881cc3\") " pod="openshift-ingress/e2e-api-a-785a7a-openshift-default-57c8cb6f7f-wp4ql"
Nov 02 12:41:36.537663 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:36.534933    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afbd2f7b-f730-472b-b6ee-f98250881cc3-istiod-ca-cert\") pod \"e2e-api-a-785a7a-openshift-default-57c8cb6f7f-wp4ql\" (UID: \"afbd2f7b-f730-472b-b6ee-f98250881cc3\") " pod="openshift-ingress/e2e-api-a-785a7a-openshift-default-57c8cb6f7f-wp4ql"
Nov 02 12:41:36.537663 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:36.535091    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afbd2f7b-f730-472b-b6ee-f98250881cc3-workload-socket\") pod \"e2e-api-a-785a7a-openshift-default-57c8cb6f7f-wp4ql\" (UID: \"afbd2f7b-f730-472b-b6ee-f98250881cc3\") " pod="openshift-ingress/e2e-api-a-785a7a-openshift-default-57c8cb6f7f-wp4ql"
Nov 02 12:41:36.630870 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:36.615938    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-785a7a-openshift-default-dockercfg-vv9xk\""
Nov 02 12:41:36.642491 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:36.637426    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afbd2f7b-f730-472b-b6ee-f98250881cc3-istio-envoy\") pod \"e2e-api-a-785a7a-openshift-default-57c8cb6f7f-wp4ql\" (UID: \"afbd2f7b-f730-472b-b6ee-f98250881cc3\") " pod="openshift-ingress/e2e-api-a-785a7a-openshift-default-57c8cb6f7f-wp4ql"
Nov 02 12:41:36.642491 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:36.637542    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afbd2f7b-f730-472b-b6ee-f98250881cc3-istio-token\") pod \"e2e-api-a-785a7a-openshift-default-57c8cb6f7f-wp4ql\" (UID: \"afbd2f7b-f730-472b-b6ee-f98250881cc3\") " pod="openshift-ingress/e2e-api-a-785a7a-openshift-default-57c8cb6f7f-wp4ql"
Nov 02 12:41:36.642491 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:36.637570    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9k6dh\" (UniqueName: \"kubernetes.io/projected/afbd2f7b-f730-472b-b6ee-f98250881cc3-kube-api-access-9k6dh\") pod \"e2e-api-a-785a7a-openshift-default-57c8cb6f7f-wp4ql\" (UID: \"afbd2f7b-f730-472b-b6ee-f98250881cc3\") " pod="openshift-ingress/e2e-api-a-785a7a-openshift-default-57c8cb6f7f-wp4ql"
Nov 02 12:41:36.642491 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:36.637605    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afbd2f7b-f730-472b-b6ee-f98250881cc3-credential-socket\") pod \"e2e-api-a-785a7a-openshift-default-57c8cb6f7f-wp4ql\" (UID: \"afbd2f7b-f730-472b-b6ee-f98250881cc3\") " pod="openshift-ingress/e2e-api-a-785a7a-openshift-default-57c8cb6f7f-wp4ql"
Nov 02 12:41:36.642491 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:36.637628    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afbd2f7b-f730-472b-b6ee-f98250881cc3-istio-data\") pod \"e2e-api-a-785a7a-openshift-default-57c8cb6f7f-wp4ql\" (UID: \"afbd2f7b-f730-472b-b6ee-f98250881cc3\") " pod="openshift-ingress/e2e-api-a-785a7a-openshift-default-57c8cb6f7f-wp4ql"
Nov 02 12:41:36.642491 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:36.637696    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afbd2f7b-f730-472b-b6ee-f98250881cc3-istiod-ca-cert\") pod \"e2e-api-a-785a7a-openshift-default-57c8cb6f7f-wp4ql\" (UID: \"afbd2f7b-f730-472b-b6ee-f98250881cc3\") " pod="openshift-ingress/e2e-api-a-785a7a-openshift-default-57c8cb6f7f-wp4ql"
Nov 02 12:41:36.642491 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:36.637858    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afbd2f7b-f730-472b-b6ee-f98250881cc3-workload-socket\") pod \"e2e-api-a-785a7a-openshift-default-57c8cb6f7f-wp4ql\" (UID: \"afbd2f7b-f730-472b-b6ee-f98250881cc3\") " pod="openshift-ingress/e2e-api-a-785a7a-openshift-default-57c8cb6f7f-wp4ql"
Nov 02 12:41:36.642491 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:36.637921    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afbd2f7b-f730-472b-b6ee-f98250881cc3-workload-certs\") pod \"e2e-api-a-785a7a-openshift-default-57c8cb6f7f-wp4ql\" (UID: \"afbd2f7b-f730-472b-b6ee-f98250881cc3\") " pod="openshift-ingress/e2e-api-a-785a7a-openshift-default-57c8cb6f7f-wp4ql"
Nov 02 12:41:36.642491 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:36.637977    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afbd2f7b-f730-472b-b6ee-f98250881cc3-istio-podinfo\") pod \"e2e-api-a-785a7a-openshift-default-57c8cb6f7f-wp4ql\" (UID: \"afbd2f7b-f730-472b-b6ee-f98250881cc3\") " pod="openshift-ingress/e2e-api-a-785a7a-openshift-default-57c8cb6f7f-wp4ql"
Nov 02 12:41:36.642491 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:36.640335    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afbd2f7b-f730-472b-b6ee-f98250881cc3-credential-socket\") pod \"e2e-api-a-785a7a-openshift-default-57c8cb6f7f-wp4ql\" (UID: \"afbd2f7b-f730-472b-b6ee-f98250881cc3\") " pod="openshift-ingress/e2e-api-a-785a7a-openshift-default-57c8cb6f7f-wp4ql"
Nov 02 12:41:36.642491 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:36.640833    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:41:36.642491 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:36.640924    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:41:36.642491 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:36.640971    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:41:36.642491 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:36.641158    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-gr5tx"]
Nov 02 12:41:36.642491 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:36.641779    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afbd2f7b-f730-472b-b6ee-f98250881cc3-istio-data\") pod \"e2e-api-a-785a7a-openshift-default-57c8cb6f7f-wp4ql\" (UID: \"afbd2f7b-f730-472b-b6ee-f98250881cc3\") " pod="openshift-ingress/e2e-api-a-785a7a-openshift-default-57c8cb6f7f-wp4ql"
Nov 02 12:41:36.642491 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:36.641885    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afbd2f7b-f730-472b-b6ee-f98250881cc3-workload-socket\") pod \"e2e-api-a-785a7a-openshift-default-57c8cb6f7f-wp4ql\" (UID: \"afbd2f7b-f730-472b-b6ee-f98250881cc3\") " pod="openshift-ingress/e2e-api-a-785a7a-openshift-default-57c8cb6f7f-wp4ql"
Nov 02 12:41:36.642491 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:36.641968    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afbd2f7b-f730-472b-b6ee-f98250881cc3-workload-certs\") pod \"e2e-api-a-785a7a-openshift-default-57c8cb6f7f-wp4ql\" (UID: \"afbd2f7b-f730-472b-b6ee-f98250881cc3\") " pod="openshift-ingress/e2e-api-a-785a7a-openshift-default-57c8cb6f7f-wp4ql"
Nov 02 12:41:36.642491 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:36.642190    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afbd2f7b-f730-472b-b6ee-f98250881cc3-istiod-ca-cert\") pod \"e2e-api-a-785a7a-openshift-default-57c8cb6f7f-wp4ql\" (UID: \"afbd2f7b-f730-472b-b6ee-f98250881cc3\") " pod="openshift-ingress/e2e-api-a-785a7a-openshift-default-57c8cb6f7f-wp4ql"
Nov 02 12:41:36.645919 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:36.644557    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afbd2f7b-f730-472b-b6ee-f98250881cc3-istio-envoy\") pod \"e2e-api-a-785a7a-openshift-default-57c8cb6f7f-wp4ql\" (UID: \"afbd2f7b-f730-472b-b6ee-f98250881cc3\") " pod="openshift-ingress/e2e-api-a-785a7a-openshift-default-57c8cb6f7f-wp4ql"
Nov 02 12:41:36.647161 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:36.647131    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afbd2f7b-f730-472b-b6ee-f98250881cc3-istio-podinfo\") pod \"e2e-api-a-785a7a-openshift-default-57c8cb6f7f-wp4ql\" (UID: \"afbd2f7b-f730-472b-b6ee-f98250881cc3\") " pod="openshift-ingress/e2e-api-a-785a7a-openshift-default-57c8cb6f7f-wp4ql"
Nov 02 12:41:36.665103 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:36.665079    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9k6dh\" (UniqueName: \"kubernetes.io/projected/afbd2f7b-f730-472b-b6ee-f98250881cc3-kube-api-access-9k6dh\") pod \"e2e-api-a-785a7a-openshift-default-57c8cb6f7f-wp4ql\" (UID: \"afbd2f7b-f730-472b-b6ee-f98250881cc3\") " pod="openshift-ingress/e2e-api-a-785a7a-openshift-default-57c8cb6f7f-wp4ql"
Nov 02 12:41:36.666235 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:36.666218    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afbd2f7b-f730-472b-b6ee-f98250881cc3-istio-token\") pod \"e2e-api-a-785a7a-openshift-default-57c8cb6f7f-wp4ql\" (UID: \"afbd2f7b-f730-472b-b6ee-f98250881cc3\") " pod="openshift-ingress/e2e-api-a-785a7a-openshift-default-57c8cb6f7f-wp4ql"
Nov 02 12:41:36.802790 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:36.802747    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-785a7a-openshift-default-57c8cb6f7f-wp4ql"
Nov 02 12:41:37.131109 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:41:37.131080    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafbd2f7b_f730_472b_b6ee_f98250881cc3.slice/crio-db93146bc0b0e4633453b94bbe7f3e3275f342c26d0efb334de0792d13f6d66f WatchSource:0}: Error finding container db93146bc0b0e4633453b94bbe7f3e3275f342c26d0efb334de0792d13f6d66f: Status 404 returned error can't find the container with id db93146bc0b0e4633453b94bbe7f3e3275f342c26d0efb334de0792d13f6d66f
Nov 02 12:41:37.131205 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:37.131132    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-785a7a-openshift-default-57c8cb6f7f-wp4ql"]
Nov 02 12:41:37.133600 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:37.133560    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:41:37.133733 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:37.133631    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:41:37.133733 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:37.133701    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:41:37.480136 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:37.480102    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-gr5tx" event={"ID":"0d507435-7115-4950-93c5-da6c04d73874","Type":"ContainerStarted","Data":"cfe4f960ff871c43450fc355da9c07406069824e196b0dd85d5b0af9a7300eb4"}
Nov 02 12:41:37.480609 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:37.480160    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-gr5tx" event={"ID":"0d507435-7115-4950-93c5-da6c04d73874","Type":"ContainerStarted","Data":"c1fbd53aa5b83a82139ffc946e01896498046df3c77d4e136ba2776d14ec7200"}
Nov 02 12:41:37.482299 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:37.482273    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-785a7a-openshift-default-57c8cb6f7f-wp4ql" event={"ID":"afbd2f7b-f730-472b-b6ee-f98250881cc3","Type":"ContainerStarted","Data":"cdf1383f688dc5619eb1923c6e07c0eff091eaaa3f279d62e9b78b17bee0d58f"}
Nov 02 12:41:37.482401 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:37.482323    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-785a7a-openshift-default-57c8cb6f7f-wp4ql" event={"ID":"afbd2f7b-f730-472b-b6ee-f98250881cc3","Type":"ContainerStarted","Data":"db93146bc0b0e4633453b94bbe7f3e3275f342c26d0efb334de0792d13f6d66f"}
Nov 02 12:41:37.544715 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:37.544632    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-gr5tx" podStartSLOduration=3.544617319 podStartE2EDuration="3.544617319s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:41:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:41:37.542000759 +0000 UTC m=+893.797399855" watchObservedRunningTime="2026-11-02 12:41:37.544617319 +0000 UTC m=+893.800016495"
Nov 02 12:41:37.794713 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:37.793972    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-gr5tx"
Nov 02 12:41:37.805675 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:37.802777    4559 patch_prober.go:29] interesting pod/e2e-worker-w1-61e341-openshift-default-59d97476dc-gr5tx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Nov 02 12:41:37.805675 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:37.802845    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-gr5tx" podUID="0d507435-7115-4950-93c5-da6c04d73874" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Nov 02 12:41:37.805675 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:37.803361    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-785a7a-openshift-default-57c8cb6f7f-wp4ql"
Nov 02 12:41:38.789073 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:38.789039    4559 patch_prober.go:29] interesting pod/e2e-worker-w1-61e341-openshift-default-59d97476dc-gr5tx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Nov 02 12:41:38.789730 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:38.789695    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-gr5tx" podUID="0d507435-7115-4950-93c5-da6c04d73874" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Nov 02 12:41:38.817313 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:38.817279    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-785a7a-openshift-default-57c8cb6f7f-wp4ql"
Nov 02 12:41:38.873737 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:38.873685    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-785a7a-openshift-default-57c8cb6f7f-wp4ql" podStartSLOduration=2.873669473 podStartE2EDuration="2.873669473s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:41:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:41:37.609864534 +0000 UTC m=+893.865263655" watchObservedRunningTime="2026-11-02 12:41:38.873669473 +0000 UTC m=+895.129068578"
Nov 02 12:41:39.519018 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:39.518922    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-785a7a-openshift-default-57c8cb6f7f-wp4ql"
Nov 02 12:41:39.519738 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:39.519714    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-785a7a-openshift-default-57c8cb6f7f-wp4ql"
Nov 02 12:41:39.789097 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:39.789006    4559 patch_prober.go:29] interesting pod/e2e-worker-w1-61e341-openshift-default-59d97476dc-gr5tx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Nov 02 12:41:39.800381 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:39.789703    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-gr5tx" podUID="0d507435-7115-4950-93c5-da6c04d73874" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Nov 02 12:41:40.807663 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:40.807035    4559 patch_prober.go:29] interesting pod/e2e-worker-w1-61e341-openshift-default-59d97476dc-gr5tx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Nov 02 12:41:40.807663 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:40.807102    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-gr5tx" podUID="0d507435-7115-4950-93c5-da6c04d73874" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Nov 02 12:41:41.788039 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:41.787998    4559 patch_prober.go:29] interesting pod/e2e-worker-w1-61e341-openshift-default-59d97476dc-gr5tx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Nov 02 12:41:41.800811 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:41.800766    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-gr5tx" podUID="0d507435-7115-4950-93c5-da6c04d73874" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Nov 02 12:41:41.932572 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:41.932542    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-785a7a-7489f6b895-s2swp"]
Nov 02 12:41:41.936898 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:41.936880    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-785a7a-7489f6b895-s2swp"
Nov 02 12:41:41.955148 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:41.955119    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-785a7a-7489f6b895-s2swp"]
Nov 02 12:41:41.964326 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:41.964298    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-785a7a\""
Nov 02 12:41:42.049709 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:42.020046    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-785a7a-dockercfg-hkr9j\""
Nov 02 12:41:42.224717 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:42.194778    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wnc7l\" (UniqueName: \"kubernetes.io/projected/c5a77cc3-ce6a-4a5d-942e-83db434f8efb-kube-api-access-wnc7l\") pod \"payload-pre-processing-e2e-api-a-785a7a-7489f6b895-s2swp\" (UID: \"c5a77cc3-ce6a-4a5d-942e-83db434f8efb\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-785a7a-7489f6b895-s2swp"
Nov 02 12:41:42.224717 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:42.194831    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5a77cc3-ce6a-4a5d-942e-83db434f8efb-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-785a7a-7489f6b895-s2swp\" (UID: \"c5a77cc3-ce6a-4a5d-942e-83db434f8efb\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-785a7a-7489f6b895-s2swp"
Nov 02 12:41:42.224717 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:42.195151    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-785a7a-55bc5f549c-mkv2x"]
Nov 02 12:41:42.255699 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:42.241535    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-785a7a-55bc5f549c-mkv2x"
Nov 02 12:41:42.310717 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:42.297134    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnc7l\" (UniqueName: \"kubernetes.io/projected/c5a77cc3-ce6a-4a5d-942e-83db434f8efb-kube-api-access-wnc7l\") pod \"payload-pre-processing-e2e-api-a-785a7a-7489f6b895-s2swp\" (UID: \"c5a77cc3-ce6a-4a5d-942e-83db434f8efb\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-785a7a-7489f6b895-s2swp"
Nov 02 12:41:42.350412 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:42.350378    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5a77cc3-ce6a-4a5d-942e-83db434f8efb-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-785a7a-7489f6b895-s2swp\" (UID: \"c5a77cc3-ce6a-4a5d-942e-83db434f8efb\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-785a7a-7489f6b895-s2swp"
Nov 02 12:41:42.355179 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:42.351403    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5a77cc3-ce6a-4a5d-942e-83db434f8efb-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-785a7a-7489f6b895-s2swp\" (UID: \"c5a77cc3-ce6a-4a5d-942e-83db434f8efb\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-785a7a-7489f6b895-s2swp"
Nov 02 12:41:42.355179 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:42.351434    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-785a7a-55bc5f549c-mkv2x"]
Nov 02 12:41:42.355868 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:42.355849    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnc7l\" (UniqueName: \"kubernetes.io/projected/c5a77cc3-ce6a-4a5d-942e-83db434f8efb-kube-api-access-wnc7l\") pod \"payload-pre-processing-e2e-api-a-785a7a-7489f6b895-s2swp\" (UID: \"c5a77cc3-ce6a-4a5d-942e-83db434f8efb\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-785a7a-7489f6b895-s2swp"
Nov 02 12:41:42.455854 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:42.455801    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ced16952-82be-49e1-a993-b009b0aa9c64-plugins-config-volume\") pod \"payload-processing-e2e-api-a-785a7a-55bc5f549c-mkv2x\" (UID: \"ced16952-82be-49e1-a993-b009b0aa9c64\") " pod="openshift-ingress/payload-processing-e2e-api-a-785a7a-55bc5f549c-mkv2x"
Nov 02 12:41:42.456044 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:42.456024    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brszw\" (UniqueName: \"kubernetes.io/projected/ced16952-82be-49e1-a993-b009b0aa9c64-kube-api-access-brszw\") pod \"payload-processing-e2e-api-a-785a7a-55bc5f549c-mkv2x\" (UID: \"ced16952-82be-49e1-a993-b009b0aa9c64\") " pod="openshift-ingress/payload-processing-e2e-api-a-785a7a-55bc5f549c-mkv2x"
Nov 02 12:41:42.553981 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:42.553934    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-785a7a-7489f6b895-s2swp"
Nov 02 12:41:42.584699 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:42.584632    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ced16952-82be-49e1-a993-b009b0aa9c64-plugins-config-volume\") pod \"payload-processing-e2e-api-a-785a7a-55bc5f549c-mkv2x\" (UID: \"ced16952-82be-49e1-a993-b009b0aa9c64\") " pod="openshift-ingress/payload-processing-e2e-api-a-785a7a-55bc5f549c-mkv2x"
Nov 02 12:41:42.608008 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:42.607968    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brszw\" (UniqueName: \"kubernetes.io/projected/ced16952-82be-49e1-a993-b009b0aa9c64-kube-api-access-brszw\") pod \"payload-processing-e2e-api-a-785a7a-55bc5f549c-mkv2x\" (UID: \"ced16952-82be-49e1-a993-b009b0aa9c64\") " pod="openshift-ingress/payload-processing-e2e-api-a-785a7a-55bc5f549c-mkv2x"
Nov 02 12:41:42.608790 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:42.586111    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ced16952-82be-49e1-a993-b009b0aa9c64-plugins-config-volume\") pod \"payload-processing-e2e-api-a-785a7a-55bc5f549c-mkv2x\" (UID: \"ced16952-82be-49e1-a993-b009b0aa9c64\") " pod="openshift-ingress/payload-processing-e2e-api-a-785a7a-55bc5f549c-mkv2x"
Nov 02 12:41:42.633373 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:42.633343    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-brszw\" (UniqueName: \"kubernetes.io/projected/ced16952-82be-49e1-a993-b009b0aa9c64-kube-api-access-brszw\") pod \"payload-processing-e2e-api-a-785a7a-55bc5f549c-mkv2x\" (UID: \"ced16952-82be-49e1-a993-b009b0aa9c64\") " pod="openshift-ingress/payload-processing-e2e-api-a-785a7a-55bc5f549c-mkv2x"
Nov 02 12:41:42.679131 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:42.679076    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-785a7a-55bc5f549c-mkv2x"
Nov 02 12:41:42.810740 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:42.810065    4559 patch_prober.go:29] interesting pod/e2e-worker-w1-61e341-openshift-default-59d97476dc-gr5tx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Nov 02 12:41:42.810740 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:42.810129    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-gr5tx" podUID="0d507435-7115-4950-93c5-da6c04d73874" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Nov 02 12:41:43.042515 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:43.042258    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-785a7a-7489f6b895-s2swp"]
Nov 02 12:41:43.075381 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:41:43.074261    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podced16952_82be_49e1_a993_b009b0aa9c64.slice/crio-d883417ba489fee7650e5391224bb6ab1256f9635d11a82759a1a4fa7634ca7a WatchSource:0}: Error finding container d883417ba489fee7650e5391224bb6ab1256f9635d11a82759a1a4fa7634ca7a: Status 404 returned error can't find the container with id d883417ba489fee7650e5391224bb6ab1256f9635d11a82759a1a4fa7634ca7a
Nov 02 12:41:43.075381 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:43.074331    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-785a7a-55bc5f549c-mkv2x"]
Nov 02 12:41:43.570537 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:43.570494    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-785a7a-55bc5f549c-mkv2x" event={"ID":"ced16952-82be-49e1-a993-b009b0aa9c64","Type":"ContainerStarted","Data":"930e1d045753c49edab2c1d23b7962213fe00600f3ce6e89fd5c755005a14b1b"}
Nov 02 12:41:43.570715 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:43.570588    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-785a7a-55bc5f549c-mkv2x" event={"ID":"ced16952-82be-49e1-a993-b009b0aa9c64","Type":"ContainerStarted","Data":"d883417ba489fee7650e5391224bb6ab1256f9635d11a82759a1a4fa7634ca7a"}
Nov 02 12:41:43.570787 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:43.570727    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-785a7a-55bc5f549c-mkv2x"
Nov 02 12:41:43.572350 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:43.572323    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-785a7a-7489f6b895-s2swp" event={"ID":"c5a77cc3-ce6a-4a5d-942e-83db434f8efb","Type":"ContainerStarted","Data":"9724d97e3c400d227a915deeaadc0859a3cc3fef889db0c2ef48487d3d8f30b4"}
Nov 02 12:41:43.572463 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:43.572367    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-785a7a-7489f6b895-s2swp" event={"ID":"c5a77cc3-ce6a-4a5d-942e-83db434f8efb","Type":"ContainerStarted","Data":"7675080ab2186752ddd637fc35261ef7c009ff431079546d2772f057cadfb430"}
Nov 02 12:41:43.572463 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:43.572396    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-785a7a-7489f6b895-s2swp"
Nov 02 12:41:43.613347 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:43.613274    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-785a7a-55bc5f549c-mkv2x" podStartSLOduration=2.613255926 podStartE2EDuration="2.613255926s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:41:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:41:43.611713143 +0000 UTC m=+899.867112240" watchObservedRunningTime="2026-11-02 12:41:43.613255926 +0000 UTC m=+899.868655022"
Nov 02 12:41:43.657805 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:43.657753    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-785a7a-7489f6b895-s2swp" podStartSLOduration=2.657737581 podStartE2EDuration="2.657737581s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:41:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:41:43.655679437 +0000 UTC m=+899.911078532" watchObservedRunningTime="2026-11-02 12:41:43.657737581 +0000 UTC m=+899.913136688"
Nov 02 12:41:43.798298 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:43.795045    4559 patch_prober.go:29] interesting pod/e2e-worker-w1-61e341-openshift-default-59d97476dc-gr5tx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Nov 02 12:41:43.798298 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:43.795110    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-gr5tx" podUID="0d507435-7115-4950-93c5-da6c04d73874" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Nov 02 12:41:44.792892 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:44.792861    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-gr5tx"
Nov 02 12:41:44.793346 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:44.793127    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-gr5tx"
Nov 02 12:41:44.793893 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:44.793872    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-gr5tx"
Nov 02 12:41:45.085699 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:41:45.085567    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqcmj" containerName="extractor"
Nov 02 12:41:45.455472 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:45.455444    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-cd1386cb-556c7759d7-m98mv"
Nov 02 12:41:46.025830 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:46.025799    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-7s2lq"]
Nov 02 12:41:46.031651 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:46.031614    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-7s2lq"
Nov 02 12:41:46.072721 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:46.072684    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-7s2lq"]
Nov 02 12:41:46.172143 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:46.172099    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ac9ccd9-92db-4bc2-8900-772272b4cc80-istio-envoy\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-7s2lq\" (UID: \"0ac9ccd9-92db-4bc2-8900-772272b4cc80\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-7s2lq"
Nov 02 12:41:46.172297 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:46.172158    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ac9ccd9-92db-4bc2-8900-772272b4cc80-istio-token\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-7s2lq\" (UID: \"0ac9ccd9-92db-4bc2-8900-772272b4cc80\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-7s2lq"
Nov 02 12:41:46.172297 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:46.172271    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ac9ccd9-92db-4bc2-8900-772272b4cc80-workload-certs\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-7s2lq\" (UID: \"0ac9ccd9-92db-4bc2-8900-772272b4cc80\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-7s2lq"
Nov 02 12:41:46.172418 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:46.172395    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ac9ccd9-92db-4bc2-8900-772272b4cc80-workload-socket\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-7s2lq\" (UID: \"0ac9ccd9-92db-4bc2-8900-772272b4cc80\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-7s2lq"
Nov 02 12:41:46.172456 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:46.172445    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ac9ccd9-92db-4bc2-8900-772272b4cc80-istio-data\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-7s2lq\" (UID: \"0ac9ccd9-92db-4bc2-8900-772272b4cc80\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-7s2lq"
Nov 02 12:41:46.172506 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:46.172493    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ac9ccd9-92db-4bc2-8900-772272b4cc80-credential-socket\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-7s2lq\" (UID: \"0ac9ccd9-92db-4bc2-8900-772272b4cc80\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-7s2lq"
Nov 02 12:41:46.172549 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:46.172514    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ac9ccd9-92db-4bc2-8900-772272b4cc80-istiod-ca-cert\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-7s2lq\" (UID: \"0ac9ccd9-92db-4bc2-8900-772272b4cc80\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-7s2lq"
Nov 02 12:41:46.172683 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:46.172664    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ac9ccd9-92db-4bc2-8900-772272b4cc80-istio-podinfo\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-7s2lq\" (UID: \"0ac9ccd9-92db-4bc2-8900-772272b4cc80\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-7s2lq"
Nov 02 12:41:46.172739 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:46.172702    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwrqx\" (UniqueName: \"kubernetes.io/projected/0ac9ccd9-92db-4bc2-8900-772272b4cc80-kube-api-access-dwrqx\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-7s2lq\" (UID: \"0ac9ccd9-92db-4bc2-8900-772272b4cc80\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-7s2lq"
Nov 02 12:41:46.273944 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:46.273909    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ac9ccd9-92db-4bc2-8900-772272b4cc80-istio-envoy\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-7s2lq\" (UID: \"0ac9ccd9-92db-4bc2-8900-772272b4cc80\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-7s2lq"
Nov 02 12:41:46.274138 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:46.273951    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ac9ccd9-92db-4bc2-8900-772272b4cc80-istio-token\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-7s2lq\" (UID: \"0ac9ccd9-92db-4bc2-8900-772272b4cc80\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-7s2lq"
Nov 02 12:41:46.274138 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:46.273999    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ac9ccd9-92db-4bc2-8900-772272b4cc80-workload-certs\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-7s2lq\" (UID: \"0ac9ccd9-92db-4bc2-8900-772272b4cc80\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-7s2lq"
Nov 02 12:41:46.274138 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:46.274087    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ac9ccd9-92db-4bc2-8900-772272b4cc80-workload-socket\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-7s2lq\" (UID: \"0ac9ccd9-92db-4bc2-8900-772272b4cc80\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-7s2lq"
Nov 02 12:41:46.274138 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:46.274117    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ac9ccd9-92db-4bc2-8900-772272b4cc80-istio-data\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-7s2lq\" (UID: \"0ac9ccd9-92db-4bc2-8900-772272b4cc80\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-7s2lq"
Nov 02 12:41:46.274351 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:46.274178    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ac9ccd9-92db-4bc2-8900-772272b4cc80-credential-socket\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-7s2lq\" (UID: \"0ac9ccd9-92db-4bc2-8900-772272b4cc80\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-7s2lq"
Nov 02 12:41:46.274351 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:46.274205    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ac9ccd9-92db-4bc2-8900-772272b4cc80-istiod-ca-cert\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-7s2lq\" (UID: \"0ac9ccd9-92db-4bc2-8900-772272b4cc80\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-7s2lq"
Nov 02 12:41:46.274351 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:46.274287    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ac9ccd9-92db-4bc2-8900-772272b4cc80-istio-podinfo\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-7s2lq\" (UID: \"0ac9ccd9-92db-4bc2-8900-772272b4cc80\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-7s2lq"
Nov 02 12:41:46.274351 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:46.274313    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwrqx\" (UniqueName: \"kubernetes.io/projected/0ac9ccd9-92db-4bc2-8900-772272b4cc80-kube-api-access-dwrqx\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-7s2lq\" (UID: \"0ac9ccd9-92db-4bc2-8900-772272b4cc80\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-7s2lq"
Nov 02 12:41:46.274547 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:46.274509    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ac9ccd9-92db-4bc2-8900-772272b4cc80-workload-certs\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-7s2lq\" (UID: \"0ac9ccd9-92db-4bc2-8900-772272b4cc80\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-7s2lq"
Nov 02 12:41:46.274604 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:46.274539    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ac9ccd9-92db-4bc2-8900-772272b4cc80-credential-socket\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-7s2lq\" (UID: \"0ac9ccd9-92db-4bc2-8900-772272b4cc80\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-7s2lq"
Nov 02 12:41:46.274683 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:46.274609    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ac9ccd9-92db-4bc2-8900-772272b4cc80-istio-data\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-7s2lq\" (UID: \"0ac9ccd9-92db-4bc2-8900-772272b4cc80\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-7s2lq"
Nov 02 12:41:46.274683 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:46.274669    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ac9ccd9-92db-4bc2-8900-772272b4cc80-workload-socket\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-7s2lq\" (UID: \"0ac9ccd9-92db-4bc2-8900-772272b4cc80\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-7s2lq"
Nov 02 12:41:46.274935 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:46.274915    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ac9ccd9-92db-4bc2-8900-772272b4cc80-istiod-ca-cert\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-7s2lq\" (UID: \"0ac9ccd9-92db-4bc2-8900-772272b4cc80\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-7s2lq"
Nov 02 12:41:46.276277 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:46.276236    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ac9ccd9-92db-4bc2-8900-772272b4cc80-istio-envoy\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-7s2lq\" (UID: \"0ac9ccd9-92db-4bc2-8900-772272b4cc80\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-7s2lq"
Nov 02 12:41:46.276953 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:46.276934    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ac9ccd9-92db-4bc2-8900-772272b4cc80-istio-podinfo\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-7s2lq\" (UID: \"0ac9ccd9-92db-4bc2-8900-772272b4cc80\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-7s2lq"
Nov 02 12:41:46.298039 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:46.298013    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ac9ccd9-92db-4bc2-8900-772272b4cc80-istio-token\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-7s2lq\" (UID: \"0ac9ccd9-92db-4bc2-8900-772272b4cc80\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-7s2lq"
Nov 02 12:41:46.298039 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:46.298035    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwrqx\" (UniqueName: \"kubernetes.io/projected/0ac9ccd9-92db-4bc2-8900-772272b4cc80-kube-api-access-dwrqx\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-7s2lq\" (UID: \"0ac9ccd9-92db-4bc2-8900-772272b4cc80\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-7s2lq"
Nov 02 12:41:46.346392 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:46.346369    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-7s2lq"
Nov 02 12:41:46.663141 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:41:46.663102    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ac9ccd9_92db_4bc2_8900_772272b4cc80.slice/crio-92a67199357b6c5c526afeb52729c4c2411d137376c2fbc9b03123dceb922aca WatchSource:0}: Error finding container 92a67199357b6c5c526afeb52729c4c2411d137376c2fbc9b03123dceb922aca: Status 404 returned error can't find the container with id 92a67199357b6c5c526afeb52729c4c2411d137376c2fbc9b03123dceb922aca
Nov 02 12:41:46.664350 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:46.664323    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-7s2lq"]
Nov 02 12:41:46.666236 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:46.666199    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:41:46.666341 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:46.666282    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:41:46.666341 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:46.666314    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:41:47.616504 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:47.616462    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-7s2lq" event={"ID":"0ac9ccd9-92db-4bc2-8900-772272b4cc80","Type":"ContainerStarted","Data":"34968672bdf04df79e66a09d897e7c6a67902c459dabd4f3f5870ef9a23542ef"}
Nov 02 12:41:47.619043 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:47.616535    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-7s2lq" event={"ID":"0ac9ccd9-92db-4bc2-8900-772272b4cc80","Type":"ContainerStarted","Data":"92a67199357b6c5c526afeb52729c4c2411d137376c2fbc9b03123dceb922aca"}
Nov 02 12:41:47.676673 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:47.676605    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-7s2lq" podStartSLOduration=2.676587248 podStartE2EDuration="2.676587248s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:41:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:41:47.674564743 +0000 UTC m=+903.929963840" watchObservedRunningTime="2026-11-02 12:41:47.676587248 +0000 UTC m=+903.931986345"
Nov 02 12:41:48.346714 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:48.346680    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-7s2lq"
Nov 02 12:41:48.348424 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:48.348391    4559 patch_prober.go:29] interesting pod/e2e-worker-w2-95ee26-openshift-default-66679d8f57-7s2lq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Nov 02 12:41:48.348582 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:48.348470    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-7s2lq" podUID="0ac9ccd9-92db-4bc2-8900-772272b4cc80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Nov 02 12:41:49.347134 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:49.347101    4559 patch_prober.go:29] interesting pod/e2e-worker-w2-95ee26-openshift-default-66679d8f57-7s2lq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Nov 02 12:41:49.347509 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:49.347167    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-7s2lq" podUID="0ac9ccd9-92db-4bc2-8900-772272b4cc80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Nov 02 12:41:49.958070 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:49.958035    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-tlgjz"]
Nov 02 12:41:49.963607 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:49.963586    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-tlgjz"
Nov 02 12:41:50.026020 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:50.025952    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-tlgjz"]
Nov 02 12:41:50.145935 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:50.145895    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b5ce2e7-bb00-436e-bee3-2d87665168c3-workload-certs\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-tlgjz\" (UID: \"0b5ce2e7-bb00-436e-bee3-2d87665168c3\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-tlgjz"
Nov 02 12:41:50.145935 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:50.145942    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b5ce2e7-bb00-436e-bee3-2d87665168c3-istio-token\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-tlgjz\" (UID: \"0b5ce2e7-bb00-436e-bee3-2d87665168c3\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-tlgjz"
Nov 02 12:41:50.146149 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:50.146054    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b5ce2e7-bb00-436e-bee3-2d87665168c3-istiod-ca-cert\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-tlgjz\" (UID: \"0b5ce2e7-bb00-436e-bee3-2d87665168c3\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-tlgjz"
Nov 02 12:41:50.146149 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:50.146075    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b5ce2e7-bb00-436e-bee3-2d87665168c3-istio-podinfo\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-tlgjz\" (UID: \"0b5ce2e7-bb00-436e-bee3-2d87665168c3\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-tlgjz"
Nov 02 12:41:50.146222 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:50.146188    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b5ce2e7-bb00-436e-bee3-2d87665168c3-workload-socket\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-tlgjz\" (UID: \"0b5ce2e7-bb00-436e-bee3-2d87665168c3\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-tlgjz"
Nov 02 12:41:50.146222 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:50.146217    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b5ce2e7-bb00-436e-bee3-2d87665168c3-istio-data\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-tlgjz\" (UID: \"0b5ce2e7-bb00-436e-bee3-2d87665168c3\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-tlgjz"
Nov 02 12:41:50.146377 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:50.146364    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b5ce2e7-bb00-436e-bee3-2d87665168c3-credential-socket\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-tlgjz\" (UID: \"0b5ce2e7-bb00-436e-bee3-2d87665168c3\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-tlgjz"
Nov 02 12:41:50.146424 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:50.146383    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b5ce2e7-bb00-436e-bee3-2d87665168c3-istio-envoy\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-tlgjz\" (UID: \"0b5ce2e7-bb00-436e-bee3-2d87665168c3\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-tlgjz"
Nov 02 12:41:50.146472 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:50.146426    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnvf5\" (UniqueName: \"kubernetes.io/projected/0b5ce2e7-bb00-436e-bee3-2d87665168c3-kube-api-access-nnvf5\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-tlgjz\" (UID: \"0b5ce2e7-bb00-436e-bee3-2d87665168c3\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-tlgjz"
Nov 02 12:41:50.247330 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:50.247241    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b5ce2e7-bb00-436e-bee3-2d87665168c3-credential-socket\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-tlgjz\" (UID: \"0b5ce2e7-bb00-436e-bee3-2d87665168c3\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-tlgjz"
Nov 02 12:41:50.247330 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:50.247281    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b5ce2e7-bb00-436e-bee3-2d87665168c3-istio-envoy\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-tlgjz\" (UID: \"0b5ce2e7-bb00-436e-bee3-2d87665168c3\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-tlgjz"
Nov 02 12:41:50.247330 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:50.247306    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnvf5\" (UniqueName: \"kubernetes.io/projected/0b5ce2e7-bb00-436e-bee3-2d87665168c3-kube-api-access-nnvf5\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-tlgjz\" (UID: \"0b5ce2e7-bb00-436e-bee3-2d87665168c3\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-tlgjz"
Nov 02 12:41:50.247620 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:50.247345    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b5ce2e7-bb00-436e-bee3-2d87665168c3-workload-certs\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-tlgjz\" (UID: \"0b5ce2e7-bb00-436e-bee3-2d87665168c3\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-tlgjz"
Nov 02 12:41:50.247620 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:50.247377    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b5ce2e7-bb00-436e-bee3-2d87665168c3-istio-token\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-tlgjz\" (UID: \"0b5ce2e7-bb00-436e-bee3-2d87665168c3\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-tlgjz"
Nov 02 12:41:50.247620 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:50.247426    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b5ce2e7-bb00-436e-bee3-2d87665168c3-istiod-ca-cert\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-tlgjz\" (UID: \"0b5ce2e7-bb00-436e-bee3-2d87665168c3\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-tlgjz"
Nov 02 12:41:50.247620 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:50.247456    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b5ce2e7-bb00-436e-bee3-2d87665168c3-istio-podinfo\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-tlgjz\" (UID: \"0b5ce2e7-bb00-436e-bee3-2d87665168c3\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-tlgjz"
Nov 02 12:41:50.247620 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:50.247529    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b5ce2e7-bb00-436e-bee3-2d87665168c3-workload-socket\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-tlgjz\" (UID: \"0b5ce2e7-bb00-436e-bee3-2d87665168c3\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-tlgjz"
Nov 02 12:41:50.247620 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:50.247561    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b5ce2e7-bb00-436e-bee3-2d87665168c3-istio-data\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-tlgjz\" (UID: \"0b5ce2e7-bb00-436e-bee3-2d87665168c3\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-tlgjz"
Nov 02 12:41:50.247867 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:50.247756    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b5ce2e7-bb00-436e-bee3-2d87665168c3-credential-socket\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-tlgjz\" (UID: \"0b5ce2e7-bb00-436e-bee3-2d87665168c3\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-tlgjz"
Nov 02 12:41:50.248117 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:50.248086    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b5ce2e7-bb00-436e-bee3-2d87665168c3-istiod-ca-cert\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-tlgjz\" (UID: \"0b5ce2e7-bb00-436e-bee3-2d87665168c3\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-tlgjz"
Nov 02 12:41:50.248410 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:50.248391    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b5ce2e7-bb00-436e-bee3-2d87665168c3-workload-certs\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-tlgjz\" (UID: \"0b5ce2e7-bb00-436e-bee3-2d87665168c3\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-tlgjz"
Nov 02 12:41:50.248493 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:50.248428    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b5ce2e7-bb00-436e-bee3-2d87665168c3-istio-data\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-tlgjz\" (UID: \"0b5ce2e7-bb00-436e-bee3-2d87665168c3\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-tlgjz"
Nov 02 12:41:50.248540 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:50.248517    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b5ce2e7-bb00-436e-bee3-2d87665168c3-workload-socket\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-tlgjz\" (UID: \"0b5ce2e7-bb00-436e-bee3-2d87665168c3\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-tlgjz"
Nov 02 12:41:50.250126 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:50.250101    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b5ce2e7-bb00-436e-bee3-2d87665168c3-istio-podinfo\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-tlgjz\" (UID: \"0b5ce2e7-bb00-436e-bee3-2d87665168c3\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-tlgjz"
Nov 02 12:41:50.250204 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:50.250129    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b5ce2e7-bb00-436e-bee3-2d87665168c3-istio-envoy\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-tlgjz\" (UID: \"0b5ce2e7-bb00-436e-bee3-2d87665168c3\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-tlgjz"
Nov 02 12:41:50.271291 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:50.271269    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b5ce2e7-bb00-436e-bee3-2d87665168c3-istio-token\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-tlgjz\" (UID: \"0b5ce2e7-bb00-436e-bee3-2d87665168c3\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-tlgjz"
Nov 02 12:41:50.271441 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:50.271351    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnvf5\" (UniqueName: \"kubernetes.io/projected/0b5ce2e7-bb00-436e-bee3-2d87665168c3-kube-api-access-nnvf5\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-tlgjz\" (UID: \"0b5ce2e7-bb00-436e-bee3-2d87665168c3\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-tlgjz"
Nov 02 12:41:50.279255 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:50.279235    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-tlgjz"
Nov 02 12:41:50.347685 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:50.347655    4559 patch_prober.go:29] interesting pod/e2e-worker-w2-95ee26-openshift-default-66679d8f57-7s2lq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Nov 02 12:41:50.348044 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:50.347708    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-7s2lq" podUID="0ac9ccd9-92db-4bc2-8900-772272b4cc80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Nov 02 12:41:50.650630 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:50.650598    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-tlgjz"]
Nov 02 12:41:50.651828 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:41:50.651798    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b5ce2e7_bb00_436e_bee3_2d87665168c3.slice/crio-2544181f85828539114340665b59ee5f0a0c93d7ca77f20324ef1c5ec7b9eb2d WatchSource:0}: Error finding container 2544181f85828539114340665b59ee5f0a0c93d7ca77f20324ef1c5ec7b9eb2d: Status 404 returned error can't find the container with id 2544181f85828539114340665b59ee5f0a0c93d7ca77f20324ef1c5ec7b9eb2d
Nov 02 12:41:50.654795 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:50.654747    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:41:50.654854 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:50.654835    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:41:50.654890 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:50.654868    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:41:51.348682 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:51.347412    4559 patch_prober.go:29] interesting pod/e2e-worker-w2-95ee26-openshift-default-66679d8f57-7s2lq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Nov 02 12:41:51.348682 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:51.347478    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-7s2lq" podUID="0ac9ccd9-92db-4bc2-8900-772272b4cc80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Nov 02 12:41:51.653414 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:51.653318    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-tlgjz" event={"ID":"0b5ce2e7-bb00-436e-bee3-2d87665168c3","Type":"ContainerStarted","Data":"75dc888d7d589386f2c45149c17b2d94d74071e7184e670f201ecd78ab04e560"}
Nov 02 12:41:51.653566 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:51.653450    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-tlgjz" event={"ID":"0b5ce2e7-bb00-436e-bee3-2d87665168c3","Type":"ContainerStarted","Data":"2544181f85828539114340665b59ee5f0a0c93d7ca77f20324ef1c5ec7b9eb2d"}
Nov 02 12:41:51.745542 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:51.744865    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-tlgjz" podStartSLOduration=2.744850675 podStartE2EDuration="2.744850675s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:41:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:41:51.730454689 +0000 UTC m=+907.985853784" watchObservedRunningTime="2026-11-02 12:41:51.744850675 +0000 UTC m=+908.000249770"
Nov 02 12:41:52.279702 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:52.279664    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-tlgjz"
Nov 02 12:41:52.280773 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:52.280738    4559 patch_prober.go:29] interesting pod/e2e-worker-w1-61e341-openshift-default-59d97476dc-tlgjz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Nov 02 12:41:52.280913 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:52.280813    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-tlgjz" podUID="0b5ce2e7-bb00-436e-bee3-2d87665168c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Nov 02 12:41:52.355664 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:52.354854    4559 patch_prober.go:29] interesting pod/e2e-worker-w2-95ee26-openshift-default-66679d8f57-7s2lq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Nov 02 12:41:52.355664 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:52.354918    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-7s2lq" podUID="0ac9ccd9-92db-4bc2-8900-772272b4cc80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Nov 02 12:41:53.280145 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:53.280105    4559 patch_prober.go:29] interesting pod/e2e-worker-w1-61e341-openshift-default-59d97476dc-tlgjz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Nov 02 12:41:53.280356 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:53.280169    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-tlgjz" podUID="0b5ce2e7-bb00-436e-bee3-2d87665168c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Nov 02 12:41:53.353421 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:53.350099    4559 patch_prober.go:29] interesting pod/e2e-worker-w2-95ee26-openshift-default-66679d8f57-7s2lq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Nov 02 12:41:53.353421 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:53.350169    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-7s2lq" podUID="0ac9ccd9-92db-4bc2-8900-772272b4cc80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Nov 02 12:41:54.282667 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:54.280160    4559 patch_prober.go:29] interesting pod/e2e-worker-w1-61e341-openshift-default-59d97476dc-tlgjz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Nov 02 12:41:54.282667 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:54.280253    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-tlgjz" podUID="0b5ce2e7-bb00-436e-bee3-2d87665168c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Nov 02 12:41:54.346908 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:54.346879    4559 patch_prober.go:29] interesting pod/e2e-worker-w2-95ee26-openshift-default-66679d8f57-7s2lq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Nov 02 12:41:54.347133 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:54.347112    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-7s2lq" podUID="0ac9ccd9-92db-4bc2-8900-772272b4cc80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Nov 02 12:41:54.583158 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:54.583063    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-785a7a-7489f6b895-s2swp"
Nov 02 12:41:54.609854 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:54.609811    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-785a7a-55bc5f549c-mkv2x"
Nov 02 12:41:55.317852 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:55.317812    4559 patch_prober.go:29] interesting pod/e2e-worker-w1-61e341-openshift-default-59d97476dc-tlgjz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Nov 02 12:41:55.318419 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:55.318384    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-tlgjz" podUID="0b5ce2e7-bb00-436e-bee3-2d87665168c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Nov 02 12:41:56.279836 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:56.279800    4559 patch_prober.go:29] interesting pod/e2e-worker-w1-61e341-openshift-default-59d97476dc-tlgjz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Nov 02 12:41:56.280010 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:56.279851    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-tlgjz" podUID="0b5ce2e7-bb00-436e-bee3-2d87665168c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Nov 02 12:41:56.351811 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:56.351769    4559 patch_prober.go:29] interesting pod/e2e-worker-w2-95ee26-openshift-default-66679d8f57-7s2lq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 12:41:56.352213 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:56.351835    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-7s2lq" podUID="0ac9ccd9-92db-4bc2-8900-772272b4cc80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:41:57.104666 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:57.103981    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-7s2lq"
Nov 02 12:41:57.114662 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:57.105569    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-7s2lq"
Nov 02 12:41:57.115027 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:57.114990    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-7s2lq"
Nov 02 12:41:57.329951 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:57.327946    4559 patch_prober.go:29] interesting pod/e2e-worker-w1-61e341-openshift-default-59d97476dc-tlgjz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Nov 02 12:41:57.329951 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:57.328030    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-tlgjz" podUID="0b5ce2e7-bb00-436e-bee3-2d87665168c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Nov 02 12:41:58.280260 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:58.280222    4559 patch_prober.go:29] interesting pod/e2e-worker-w1-61e341-openshift-default-59d97476dc-tlgjz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Nov 02 12:41:58.280678 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:58.280294    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-tlgjz" podUID="0b5ce2e7-bb00-436e-bee3-2d87665168c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Nov 02 12:41:59.283932 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:59.283902    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-tlgjz"
Nov 02 12:41:59.284349 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:59.284330    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-tlgjz"
Nov 02 12:41:59.285019 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:41:59.285002    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-tlgjz"
Nov 02 12:42:01.031746 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:01.031712    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-ng469"]
Nov 02 12:42:01.040775 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:01.040749    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-ng469"
Nov 02 12:42:01.068464 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:01.068423    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-ng469"]
Nov 02 12:42:01.160515 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:01.160475    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4991a41e-4031-4dec-b038-7a10b5da7ac5-istio-data\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-ng469\" (UID: \"4991a41e-4031-4dec-b038-7a10b5da7ac5\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-ng469"
Nov 02 12:42:01.160732 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:01.160582    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4991a41e-4031-4dec-b038-7a10b5da7ac5-workload-certs\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-ng469\" (UID: \"4991a41e-4031-4dec-b038-7a10b5da7ac5\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-ng469"
Nov 02 12:42:01.160732 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:01.160606    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4991a41e-4031-4dec-b038-7a10b5da7ac5-istio-token\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-ng469\" (UID: \"4991a41e-4031-4dec-b038-7a10b5da7ac5\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-ng469"
Nov 02 12:42:01.160813 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:01.160789    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4991a41e-4031-4dec-b038-7a10b5da7ac5-istiod-ca-cert\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-ng469\" (UID: \"4991a41e-4031-4dec-b038-7a10b5da7ac5\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-ng469"
Nov 02 12:42:01.160865 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:01.160843    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4991a41e-4031-4dec-b038-7a10b5da7ac5-credential-socket\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-ng469\" (UID: \"4991a41e-4031-4dec-b038-7a10b5da7ac5\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-ng469"
Nov 02 12:42:01.160988 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:01.160970    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ssvvp\" (UniqueName: \"kubernetes.io/projected/4991a41e-4031-4dec-b038-7a10b5da7ac5-kube-api-access-ssvvp\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-ng469\" (UID: \"4991a41e-4031-4dec-b038-7a10b5da7ac5\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-ng469"
Nov 02 12:42:01.161026 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:01.161004    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4991a41e-4031-4dec-b038-7a10b5da7ac5-workload-socket\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-ng469\" (UID: \"4991a41e-4031-4dec-b038-7a10b5da7ac5\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-ng469"
Nov 02 12:42:01.161089 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:01.161076    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4991a41e-4031-4dec-b038-7a10b5da7ac5-istio-podinfo\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-ng469\" (UID: \"4991a41e-4031-4dec-b038-7a10b5da7ac5\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-ng469"
Nov 02 12:42:01.161151 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:01.161140    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4991a41e-4031-4dec-b038-7a10b5da7ac5-istio-envoy\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-ng469\" (UID: \"4991a41e-4031-4dec-b038-7a10b5da7ac5\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-ng469"
Nov 02 12:42:01.262256 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:01.262218    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4991a41e-4031-4dec-b038-7a10b5da7ac5-istio-podinfo\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-ng469\" (UID: \"4991a41e-4031-4dec-b038-7a10b5da7ac5\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-ng469"
Nov 02 12:42:01.262434 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:01.262302    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4991a41e-4031-4dec-b038-7a10b5da7ac5-istio-envoy\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-ng469\" (UID: \"4991a41e-4031-4dec-b038-7a10b5da7ac5\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-ng469"
Nov 02 12:42:01.262434 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:01.262357    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4991a41e-4031-4dec-b038-7a10b5da7ac5-istio-data\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-ng469\" (UID: \"4991a41e-4031-4dec-b038-7a10b5da7ac5\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-ng469"
Nov 02 12:42:01.262571 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:01.262546    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4991a41e-4031-4dec-b038-7a10b5da7ac5-workload-certs\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-ng469\" (UID: \"4991a41e-4031-4dec-b038-7a10b5da7ac5\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-ng469"
Nov 02 12:42:01.262631 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:01.262587    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4991a41e-4031-4dec-b038-7a10b5da7ac5-istio-token\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-ng469\" (UID: \"4991a41e-4031-4dec-b038-7a10b5da7ac5\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-ng469"
Nov 02 12:42:01.262764 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:01.262743    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4991a41e-4031-4dec-b038-7a10b5da7ac5-istiod-ca-cert\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-ng469\" (UID: \"4991a41e-4031-4dec-b038-7a10b5da7ac5\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-ng469"
Nov 02 12:42:01.262910 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:01.262893    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4991a41e-4031-4dec-b038-7a10b5da7ac5-credential-socket\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-ng469\" (UID: \"4991a41e-4031-4dec-b038-7a10b5da7ac5\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-ng469"
Nov 02 12:42:01.263074 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:01.263050    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ssvvp\" (UniqueName: \"kubernetes.io/projected/4991a41e-4031-4dec-b038-7a10b5da7ac5-kube-api-access-ssvvp\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-ng469\" (UID: \"4991a41e-4031-4dec-b038-7a10b5da7ac5\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-ng469"
Nov 02 12:42:01.263159 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:01.262909    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4991a41e-4031-4dec-b038-7a10b5da7ac5-workload-certs\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-ng469\" (UID: \"4991a41e-4031-4dec-b038-7a10b5da7ac5\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-ng469"
Nov 02 12:42:01.263223 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:01.262760    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4991a41e-4031-4dec-b038-7a10b5da7ac5-istio-data\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-ng469\" (UID: \"4991a41e-4031-4dec-b038-7a10b5da7ac5\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-ng469"
Nov 02 12:42:01.263283 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:01.263242    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4991a41e-4031-4dec-b038-7a10b5da7ac5-credential-socket\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-ng469\" (UID: \"4991a41e-4031-4dec-b038-7a10b5da7ac5\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-ng469"
Nov 02 12:42:01.263360 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:01.263342    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4991a41e-4031-4dec-b038-7a10b5da7ac5-workload-socket\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-ng469\" (UID: \"4991a41e-4031-4dec-b038-7a10b5da7ac5\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-ng469"
Nov 02 12:42:01.263586 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:01.263569    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4991a41e-4031-4dec-b038-7a10b5da7ac5-workload-socket\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-ng469\" (UID: \"4991a41e-4031-4dec-b038-7a10b5da7ac5\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-ng469"
Nov 02 12:42:01.263632 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:01.263613    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4991a41e-4031-4dec-b038-7a10b5da7ac5-istiod-ca-cert\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-ng469\" (UID: \"4991a41e-4031-4dec-b038-7a10b5da7ac5\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-ng469"
Nov 02 12:42:01.264730 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:01.264708    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4991a41e-4031-4dec-b038-7a10b5da7ac5-istio-envoy\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-ng469\" (UID: \"4991a41e-4031-4dec-b038-7a10b5da7ac5\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-ng469"
Nov 02 12:42:01.265106 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:01.265083    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4991a41e-4031-4dec-b038-7a10b5da7ac5-istio-podinfo\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-ng469\" (UID: \"4991a41e-4031-4dec-b038-7a10b5da7ac5\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-ng469"
Nov 02 12:42:01.286347 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:01.286286    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4991a41e-4031-4dec-b038-7a10b5da7ac5-istio-token\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-ng469\" (UID: \"4991a41e-4031-4dec-b038-7a10b5da7ac5\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-ng469"
Nov 02 12:42:01.286444 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:01.286421    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ssvvp\" (UniqueName: \"kubernetes.io/projected/4991a41e-4031-4dec-b038-7a10b5da7ac5-kube-api-access-ssvvp\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-ng469\" (UID: \"4991a41e-4031-4dec-b038-7a10b5da7ac5\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-ng469"
Nov 02 12:42:01.366005 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:01.365972    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-ng469"
Nov 02 12:42:01.698479 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:42:01.698450    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4991a41e_4031_4dec_b038_7a10b5da7ac5.slice/crio-074afa5afc393ed865400dfd05bd84e3b1ca0c688e26791746f62deefc953ca3 WatchSource:0}: Error finding container 074afa5afc393ed865400dfd05bd84e3b1ca0c688e26791746f62deefc953ca3: Status 404 returned error can't find the container with id 074afa5afc393ed865400dfd05bd84e3b1ca0c688e26791746f62deefc953ca3
Nov 02 12:42:01.699338 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:01.699278    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-ng469"]
Nov 02 12:42:01.701133 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:01.701102    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:42:01.701253 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:01.701174    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:42:01.701253 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:01.701218    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:42:02.110433 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:02.110321    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-ng469" event={"ID":"4991a41e-4031-4dec-b038-7a10b5da7ac5","Type":"ContainerStarted","Data":"f16fb49d3e7d64d9351e4fb622963b0c0540e3032bb6967ba10c7af8d5ed6ddb"}
Nov 02 12:42:02.110433 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:02.110393    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-ng469" event={"ID":"4991a41e-4031-4dec-b038-7a10b5da7ac5","Type":"ContainerStarted","Data":"074afa5afc393ed865400dfd05bd84e3b1ca0c688e26791746f62deefc953ca3"}
Nov 02 12:42:02.128779 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:02.128753    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-cd1386cb/e2e-lifecycle-model-cd1386cb-kserve-7d9d698845-2579r"]
Nov 02 12:42:02.132791 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:02.132770    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-cd1386cb/e2e-lifecycle-model-cd1386cb-kserve-7d9d698845-2579r"
Nov 02 12:42:02.163054 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:02.163021    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-cd1386cb\"/\"openshift-service-ca.crt\""
Nov 02 12:42:02.163314 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:02.163148    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-cd1386cb\"/\"e2e-lifecycle-model-cd1386cb-kserve-self-signed-certs\""
Nov 02 12:42:02.163314 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:02.163182    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-cd1386cb\"/\"default-dockercfg-rg4nr\""
Nov 02 12:42:02.163314 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:02.163292    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-cd1386cb\"/\"kube-root-ca.crt\""
Nov 02 12:42:02.167432 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:02.167391    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-cd1386cb/e2e-lifecycle-model-cd1386cb-kserve-7d9d698845-2579r"]
Nov 02 12:42:02.176685 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:02.174518    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/572f58c8-cca9-48c0-99d3-0ebde6b82476-dshm\") pod \"e2e-lifecycle-model-cd1386cb-kserve-7d9d698845-2579r\" (UID: \"572f58c8-cca9-48c0-99d3-0ebde6b82476\") " pod="ai-tenant-e2e-mt-cd1386cb/e2e-lifecycle-model-cd1386cb-kserve-7d9d698845-2579r"
Nov 02 12:42:02.176685 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:02.174680    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lt5z5\" (UniqueName: \"kubernetes.io/projected/572f58c8-cca9-48c0-99d3-0ebde6b82476-kube-api-access-lt5z5\") pod \"e2e-lifecycle-model-cd1386cb-kserve-7d9d698845-2579r\" (UID: \"572f58c8-cca9-48c0-99d3-0ebde6b82476\") " pod="ai-tenant-e2e-mt-cd1386cb/e2e-lifecycle-model-cd1386cb-kserve-7d9d698845-2579r"
Nov 02 12:42:02.176685 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:02.174758    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/572f58c8-cca9-48c0-99d3-0ebde6b82476-tls-certs\") pod \"e2e-lifecycle-model-cd1386cb-kserve-7d9d698845-2579r\" (UID: \"572f58c8-cca9-48c0-99d3-0ebde6b82476\") " pod="ai-tenant-e2e-mt-cd1386cb/e2e-lifecycle-model-cd1386cb-kserve-7d9d698845-2579r"
Nov 02 12:42:02.176685 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:02.174866    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/572f58c8-cca9-48c0-99d3-0ebde6b82476-model-cache\") pod \"e2e-lifecycle-model-cd1386cb-kserve-7d9d698845-2579r\" (UID: \"572f58c8-cca9-48c0-99d3-0ebde6b82476\") " pod="ai-tenant-e2e-mt-cd1386cb/e2e-lifecycle-model-cd1386cb-kserve-7d9d698845-2579r"
Nov 02 12:42:02.176685 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:02.174933    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/572f58c8-cca9-48c0-99d3-0ebde6b82476-home\") pod \"e2e-lifecycle-model-cd1386cb-kserve-7d9d698845-2579r\" (UID: \"572f58c8-cca9-48c0-99d3-0ebde6b82476\") " pod="ai-tenant-e2e-mt-cd1386cb/e2e-lifecycle-model-cd1386cb-kserve-7d9d698845-2579r"
Nov 02 12:42:02.176685 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:02.175035    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/572f58c8-cca9-48c0-99d3-0ebde6b82476-tmp-dir\") pod \"e2e-lifecycle-model-cd1386cb-kserve-7d9d698845-2579r\" (UID: \"572f58c8-cca9-48c0-99d3-0ebde6b82476\") " pod="ai-tenant-e2e-mt-cd1386cb/e2e-lifecycle-model-cd1386cb-kserve-7d9d698845-2579r"
Nov 02 12:42:02.179400 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:02.179351    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-ng469" podStartSLOduration=1.1793353739999999 podStartE2EDuration="1.179335374s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:42:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:42:02.17745294 +0000 UTC m=+918.432852035" watchObservedRunningTime="2026-11-02 12:42:02.179335374 +0000 UTC m=+918.434734470"
Nov 02 12:42:02.276846 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:02.276816    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/572f58c8-cca9-48c0-99d3-0ebde6b82476-tmp-dir\") pod \"e2e-lifecycle-model-cd1386cb-kserve-7d9d698845-2579r\" (UID: \"572f58c8-cca9-48c0-99d3-0ebde6b82476\") " pod="ai-tenant-e2e-mt-cd1386cb/e2e-lifecycle-model-cd1386cb-kserve-7d9d698845-2579r"
Nov 02 12:42:02.276983 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:02.276874    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/572f58c8-cca9-48c0-99d3-0ebde6b82476-dshm\") pod \"e2e-lifecycle-model-cd1386cb-kserve-7d9d698845-2579r\" (UID: \"572f58c8-cca9-48c0-99d3-0ebde6b82476\") " pod="ai-tenant-e2e-mt-cd1386cb/e2e-lifecycle-model-cd1386cb-kserve-7d9d698845-2579r"
Nov 02 12:42:02.276983 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:02.276958    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lt5z5\" (UniqueName: \"kubernetes.io/projected/572f58c8-cca9-48c0-99d3-0ebde6b82476-kube-api-access-lt5z5\") pod \"e2e-lifecycle-model-cd1386cb-kserve-7d9d698845-2579r\" (UID: \"572f58c8-cca9-48c0-99d3-0ebde6b82476\") " pod="ai-tenant-e2e-mt-cd1386cb/e2e-lifecycle-model-cd1386cb-kserve-7d9d698845-2579r"
Nov 02 12:42:02.277113 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:02.276992    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/572f58c8-cca9-48c0-99d3-0ebde6b82476-tls-certs\") pod \"e2e-lifecycle-model-cd1386cb-kserve-7d9d698845-2579r\" (UID: \"572f58c8-cca9-48c0-99d3-0ebde6b82476\") " pod="ai-tenant-e2e-mt-cd1386cb/e2e-lifecycle-model-cd1386cb-kserve-7d9d698845-2579r"
Nov 02 12:42:02.277113 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:02.277049    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/572f58c8-cca9-48c0-99d3-0ebde6b82476-model-cache\") pod \"e2e-lifecycle-model-cd1386cb-kserve-7d9d698845-2579r\" (UID: \"572f58c8-cca9-48c0-99d3-0ebde6b82476\") " pod="ai-tenant-e2e-mt-cd1386cb/e2e-lifecycle-model-cd1386cb-kserve-7d9d698845-2579r"
Nov 02 12:42:02.277113 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:02.277081    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/572f58c8-cca9-48c0-99d3-0ebde6b82476-home\") pod \"e2e-lifecycle-model-cd1386cb-kserve-7d9d698845-2579r\" (UID: \"572f58c8-cca9-48c0-99d3-0ebde6b82476\") " pod="ai-tenant-e2e-mt-cd1386cb/e2e-lifecycle-model-cd1386cb-kserve-7d9d698845-2579r"
Nov 02 12:42:02.277341 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:02.277313    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/572f58c8-cca9-48c0-99d3-0ebde6b82476-tmp-dir\") pod \"e2e-lifecycle-model-cd1386cb-kserve-7d9d698845-2579r\" (UID: \"572f58c8-cca9-48c0-99d3-0ebde6b82476\") " pod="ai-tenant-e2e-mt-cd1386cb/e2e-lifecycle-model-cd1386cb-kserve-7d9d698845-2579r"
Nov 02 12:42:02.277469 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:02.277409    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/572f58c8-cca9-48c0-99d3-0ebde6b82476-home\") pod \"e2e-lifecycle-model-cd1386cb-kserve-7d9d698845-2579r\" (UID: \"572f58c8-cca9-48c0-99d3-0ebde6b82476\") " pod="ai-tenant-e2e-mt-cd1386cb/e2e-lifecycle-model-cd1386cb-kserve-7d9d698845-2579r"
Nov 02 12:42:02.277469 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:02.277450    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/572f58c8-cca9-48c0-99d3-0ebde6b82476-model-cache\") pod \"e2e-lifecycle-model-cd1386cb-kserve-7d9d698845-2579r\" (UID: \"572f58c8-cca9-48c0-99d3-0ebde6b82476\") " pod="ai-tenant-e2e-mt-cd1386cb/e2e-lifecycle-model-cd1386cb-kserve-7d9d698845-2579r"
Nov 02 12:42:02.279121 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:02.279089    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/572f58c8-cca9-48c0-99d3-0ebde6b82476-dshm\") pod \"e2e-lifecycle-model-cd1386cb-kserve-7d9d698845-2579r\" (UID: \"572f58c8-cca9-48c0-99d3-0ebde6b82476\") " pod="ai-tenant-e2e-mt-cd1386cb/e2e-lifecycle-model-cd1386cb-kserve-7d9d698845-2579r"
Nov 02 12:42:02.279444 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:02.279424    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/572f58c8-cca9-48c0-99d3-0ebde6b82476-tls-certs\") pod \"e2e-lifecycle-model-cd1386cb-kserve-7d9d698845-2579r\" (UID: \"572f58c8-cca9-48c0-99d3-0ebde6b82476\") " pod="ai-tenant-e2e-mt-cd1386cb/e2e-lifecycle-model-cd1386cb-kserve-7d9d698845-2579r"
Nov 02 12:42:02.316599 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:02.316565    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lt5z5\" (UniqueName: \"kubernetes.io/projected/572f58c8-cca9-48c0-99d3-0ebde6b82476-kube-api-access-lt5z5\") pod \"e2e-lifecycle-model-cd1386cb-kserve-7d9d698845-2579r\" (UID: \"572f58c8-cca9-48c0-99d3-0ebde6b82476\") " pod="ai-tenant-e2e-mt-cd1386cb/e2e-lifecycle-model-cd1386cb-kserve-7d9d698845-2579r"
Nov 02 12:42:02.366358 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:02.366288    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-ng469"
Nov 02 12:42:02.367842 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:02.367811    4559 patch_prober.go:29] interesting pod/e2e-worker-w2-95ee26-openshift-default-66679d8f57-ng469 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Nov 02 12:42:02.367941 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:02.367873    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-ng469" podUID="4991a41e-4031-4dec-b038-7a10b5da7ac5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Nov 02 12:42:02.455665 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:02.450012    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-cd1386cb/e2e-lifecycle-model-cd1386cb-kserve-7d9d698845-2579r"
Nov 02 12:42:03.013282 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:03.013233    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-cd1386cb/e2e-lifecycle-model-cd1386cb-kserve-7d9d698845-2579r"]
Nov 02 12:42:03.122424 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:03.122245    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-cd1386cb/e2e-lifecycle-model-cd1386cb-kserve-7d9d698845-2579r" event={"ID":"572f58c8-cca9-48c0-99d3-0ebde6b82476","Type":"ContainerStarted","Data":"8b744c566dfe165060511d1e4a7599c67c46bbe3ca8eed11c4ef995752448061"}
Nov 02 12:42:03.367117 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:03.367040    4559 patch_prober.go:29] interesting pod/e2e-worker-w2-95ee26-openshift-default-66679d8f57-ng469 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Nov 02 12:42:03.367117 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:03.367092    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-ng469" podUID="4991a41e-4031-4dec-b038-7a10b5da7ac5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Nov 02 12:42:04.161078 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:04.161011    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-cd1386cb/e2e-lifecycle-model-cd1386cb-kserve-7d9d698845-2579r" event={"ID":"572f58c8-cca9-48c0-99d3-0ebde6b82476","Type":"ContainerStarted","Data":"177a2ce7afdb94b482f18b9e727d3c7b26727efc418ffd2328a392e6148b6a68"}
Nov 02 12:42:04.210482 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:04.210427    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-mt-cd1386cb/e2e-lifecycle-model-cd1386cb-kserve-7d9d698845-2579r" podStartSLOduration=2.210413945 podStartE2EDuration="2.210413945s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:42:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:42:04.208166818 +0000 UTC m=+920.463565912" watchObservedRunningTime="2026-11-02 12:42:04.210413945 +0000 UTC m=+920.465813089"
Nov 02 12:42:04.373496 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:04.373418    4559 patch_prober.go:29] interesting pod/e2e-worker-w2-95ee26-openshift-default-66679d8f57-ng469 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Nov 02 12:42:04.373838 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:04.373801    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-ng469" podUID="4991a41e-4031-4dec-b038-7a10b5da7ac5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Nov 02 12:42:04.453104 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:04.452996    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-cd1386cb/e2e-lifecycle-model-cd1386cb-kserve-7d9d698845-2579r"
Nov 02 12:42:05.366779 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:05.366746    4559 patch_prober.go:29] interesting pod/e2e-worker-w2-95ee26-openshift-default-66679d8f57-ng469 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Nov 02 12:42:05.367177 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:05.366803    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-ng469" podUID="4991a41e-4031-4dec-b038-7a10b5da7ac5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Nov 02 12:42:06.367160 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:06.367124    4559 patch_prober.go:29] interesting pod/e2e-worker-w2-95ee26-openshift-default-66679d8f57-ng469 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Nov 02 12:42:06.367576 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:06.367186    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-ng469" podUID="4991a41e-4031-4dec-b038-7a10b5da7ac5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Nov 02 12:42:07.370934 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:07.370893    4559 patch_prober.go:29] interesting pod/e2e-worker-w2-95ee26-openshift-default-66679d8f57-ng469 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Nov 02 12:42:07.371339 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:07.370967    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-ng469" podUID="4991a41e-4031-4dec-b038-7a10b5da7ac5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Nov 02 12:42:08.366874 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:08.366839    4559 patch_prober.go:29] interesting pod/e2e-worker-w2-95ee26-openshift-default-66679d8f57-ng469 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Nov 02 12:42:08.367062 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:08.366908    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-ng469" podUID="4991a41e-4031-4dec-b038-7a10b5da7ac5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Nov 02 12:42:09.371373 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:09.371345    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-ng469"
Nov 02 12:42:09.371847 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:09.371762    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-ng469"
Nov 02 12:42:09.373342 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:09.373317    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-ng469"
Nov 02 12:42:11.300408 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:11.300373    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-96339b-openshift-default-857fd4b79-52nqm"]
Nov 02 12:42:11.306736 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:11.306702    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-96339b-openshift-default-857fd4b79-52nqm"
Nov 02 12:42:11.329447 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:11.329417    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-96339b-openshift-default-dockercfg-hl5t4\""
Nov 02 12:42:11.343242 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:11.343200    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-96339b-openshift-default-857fd4b79-52nqm"]
Nov 02 12:42:11.405610 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:11.405564    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e51ff2f8-753e-4977-a523-90bca8a4ec2a-istiod-ca-cert\") pod \"e2e-api-b-96339b-openshift-default-857fd4b79-52nqm\" (UID: \"e51ff2f8-753e-4977-a523-90bca8a4ec2a\") " pod="openshift-ingress/e2e-api-b-96339b-openshift-default-857fd4b79-52nqm"
Nov 02 12:42:11.405844 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:11.405628    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e51ff2f8-753e-4977-a523-90bca8a4ec2a-istio-token\") pod \"e2e-api-b-96339b-openshift-default-857fd4b79-52nqm\" (UID: \"e51ff2f8-753e-4977-a523-90bca8a4ec2a\") " pod="openshift-ingress/e2e-api-b-96339b-openshift-default-857fd4b79-52nqm"
Nov 02 12:42:11.405844 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:11.405712    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9c8sp\" (UniqueName: \"kubernetes.io/projected/e51ff2f8-753e-4977-a523-90bca8a4ec2a-kube-api-access-9c8sp\") pod \"e2e-api-b-96339b-openshift-default-857fd4b79-52nqm\" (UID: \"e51ff2f8-753e-4977-a523-90bca8a4ec2a\") " pod="openshift-ingress/e2e-api-b-96339b-openshift-default-857fd4b79-52nqm"
Nov 02 12:42:11.405844 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:11.405756    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e51ff2f8-753e-4977-a523-90bca8a4ec2a-istio-envoy\") pod \"e2e-api-b-96339b-openshift-default-857fd4b79-52nqm\" (UID: \"e51ff2f8-753e-4977-a523-90bca8a4ec2a\") " pod="openshift-ingress/e2e-api-b-96339b-openshift-default-857fd4b79-52nqm"
Nov 02 12:42:11.410058 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:11.406201    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e51ff2f8-753e-4977-a523-90bca8a4ec2a-credential-socket\") pod \"e2e-api-b-96339b-openshift-default-857fd4b79-52nqm\" (UID: \"e51ff2f8-753e-4977-a523-90bca8a4ec2a\") " pod="openshift-ingress/e2e-api-b-96339b-openshift-default-857fd4b79-52nqm"
Nov 02 12:42:11.410339 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:11.410310    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e51ff2f8-753e-4977-a523-90bca8a4ec2a-workload-socket\") pod \"e2e-api-b-96339b-openshift-default-857fd4b79-52nqm\" (UID: \"e51ff2f8-753e-4977-a523-90bca8a4ec2a\") " pod="openshift-ingress/e2e-api-b-96339b-openshift-default-857fd4b79-52nqm"
Nov 02 12:42:11.410515 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:11.410376    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e51ff2f8-753e-4977-a523-90bca8a4ec2a-istio-podinfo\") pod \"e2e-api-b-96339b-openshift-default-857fd4b79-52nqm\" (UID: \"e51ff2f8-753e-4977-a523-90bca8a4ec2a\") " pod="openshift-ingress/e2e-api-b-96339b-openshift-default-857fd4b79-52nqm"
Nov 02 12:42:11.410515 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:11.410489    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e51ff2f8-753e-4977-a523-90bca8a4ec2a-workload-certs\") pod \"e2e-api-b-96339b-openshift-default-857fd4b79-52nqm\" (UID: \"e51ff2f8-753e-4977-a523-90bca8a4ec2a\") " pod="openshift-ingress/e2e-api-b-96339b-openshift-default-857fd4b79-52nqm"
Nov 02 12:42:11.410774 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:11.410565    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e51ff2f8-753e-4977-a523-90bca8a4ec2a-istio-data\") pod \"e2e-api-b-96339b-openshift-default-857fd4b79-52nqm\" (UID: \"e51ff2f8-753e-4977-a523-90bca8a4ec2a\") " pod="openshift-ingress/e2e-api-b-96339b-openshift-default-857fd4b79-52nqm"
Nov 02 12:42:11.515671 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:11.514525    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e51ff2f8-753e-4977-a523-90bca8a4ec2a-istio-podinfo\") pod \"e2e-api-b-96339b-openshift-default-857fd4b79-52nqm\" (UID: \"e51ff2f8-753e-4977-a523-90bca8a4ec2a\") " pod="openshift-ingress/e2e-api-b-96339b-openshift-default-857fd4b79-52nqm"
Nov 02 12:42:11.515671 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:11.514683    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e51ff2f8-753e-4977-a523-90bca8a4ec2a-workload-certs\") pod \"e2e-api-b-96339b-openshift-default-857fd4b79-52nqm\" (UID: \"e51ff2f8-753e-4977-a523-90bca8a4ec2a\") " pod="openshift-ingress/e2e-api-b-96339b-openshift-default-857fd4b79-52nqm"
Nov 02 12:42:11.515671 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:11.514770    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e51ff2f8-753e-4977-a523-90bca8a4ec2a-istio-data\") pod \"e2e-api-b-96339b-openshift-default-857fd4b79-52nqm\" (UID: \"e51ff2f8-753e-4977-a523-90bca8a4ec2a\") " pod="openshift-ingress/e2e-api-b-96339b-openshift-default-857fd4b79-52nqm"
Nov 02 12:42:11.515671 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:11.514820    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e51ff2f8-753e-4977-a523-90bca8a4ec2a-istiod-ca-cert\") pod \"e2e-api-b-96339b-openshift-default-857fd4b79-52nqm\" (UID: \"e51ff2f8-753e-4977-a523-90bca8a4ec2a\") " pod="openshift-ingress/e2e-api-b-96339b-openshift-default-857fd4b79-52nqm"
Nov 02 12:42:11.515671 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:11.514849    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e51ff2f8-753e-4977-a523-90bca8a4ec2a-istio-token\") pod \"e2e-api-b-96339b-openshift-default-857fd4b79-52nqm\" (UID: \"e51ff2f8-753e-4977-a523-90bca8a4ec2a\") " pod="openshift-ingress/e2e-api-b-96339b-openshift-default-857fd4b79-52nqm"
Nov 02 12:42:11.515671 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:11.514906    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9c8sp\" (UniqueName: \"kubernetes.io/projected/e51ff2f8-753e-4977-a523-90bca8a4ec2a-kube-api-access-9c8sp\") pod \"e2e-api-b-96339b-openshift-default-857fd4b79-52nqm\" (UID: \"e51ff2f8-753e-4977-a523-90bca8a4ec2a\") " pod="openshift-ingress/e2e-api-b-96339b-openshift-default-857fd4b79-52nqm"
Nov 02 12:42:11.515671 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:11.514936    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e51ff2f8-753e-4977-a523-90bca8a4ec2a-istio-envoy\") pod \"e2e-api-b-96339b-openshift-default-857fd4b79-52nqm\" (UID: \"e51ff2f8-753e-4977-a523-90bca8a4ec2a\") " pod="openshift-ingress/e2e-api-b-96339b-openshift-default-857fd4b79-52nqm"
Nov 02 12:42:11.515671 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:11.515064    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e51ff2f8-753e-4977-a523-90bca8a4ec2a-credential-socket\") pod \"e2e-api-b-96339b-openshift-default-857fd4b79-52nqm\" (UID: \"e51ff2f8-753e-4977-a523-90bca8a4ec2a\") " pod="openshift-ingress/e2e-api-b-96339b-openshift-default-857fd4b79-52nqm"
Nov 02 12:42:11.515671 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:11.515113    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e51ff2f8-753e-4977-a523-90bca8a4ec2a-workload-socket\") pod \"e2e-api-b-96339b-openshift-default-857fd4b79-52nqm\" (UID: \"e51ff2f8-753e-4977-a523-90bca8a4ec2a\") " pod="openshift-ingress/e2e-api-b-96339b-openshift-default-857fd4b79-52nqm"
Nov 02 12:42:11.515671 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:11.515561    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e51ff2f8-753e-4977-a523-90bca8a4ec2a-workload-socket\") pod \"e2e-api-b-96339b-openshift-default-857fd4b79-52nqm\" (UID: \"e51ff2f8-753e-4977-a523-90bca8a4ec2a\") " pod="openshift-ingress/e2e-api-b-96339b-openshift-default-857fd4b79-52nqm"
Nov 02 12:42:11.543756 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:11.528711    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e51ff2f8-753e-4977-a523-90bca8a4ec2a-workload-certs\") pod \"e2e-api-b-96339b-openshift-default-857fd4b79-52nqm\" (UID: \"e51ff2f8-753e-4977-a523-90bca8a4ec2a\") " pod="openshift-ingress/e2e-api-b-96339b-openshift-default-857fd4b79-52nqm"
Nov 02 12:42:11.543756 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:11.530014    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e51ff2f8-753e-4977-a523-90bca8a4ec2a-credential-socket\") pod \"e2e-api-b-96339b-openshift-default-857fd4b79-52nqm\" (UID: \"e51ff2f8-753e-4977-a523-90bca8a4ec2a\") " pod="openshift-ingress/e2e-api-b-96339b-openshift-default-857fd4b79-52nqm"
Nov 02 12:42:11.543756 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:11.530257    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e51ff2f8-753e-4977-a523-90bca8a4ec2a-istio-data\") pod \"e2e-api-b-96339b-openshift-default-857fd4b79-52nqm\" (UID: \"e51ff2f8-753e-4977-a523-90bca8a4ec2a\") " pod="openshift-ingress/e2e-api-b-96339b-openshift-default-857fd4b79-52nqm"
Nov 02 12:42:11.543756 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:11.530890    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e51ff2f8-753e-4977-a523-90bca8a4ec2a-istiod-ca-cert\") pod \"e2e-api-b-96339b-openshift-default-857fd4b79-52nqm\" (UID: \"e51ff2f8-753e-4977-a523-90bca8a4ec2a\") " pod="openshift-ingress/e2e-api-b-96339b-openshift-default-857fd4b79-52nqm"
Nov 02 12:42:11.543756 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:11.539983    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e51ff2f8-753e-4977-a523-90bca8a4ec2a-istio-envoy\") pod \"e2e-api-b-96339b-openshift-default-857fd4b79-52nqm\" (UID: \"e51ff2f8-753e-4977-a523-90bca8a4ec2a\") " pod="openshift-ingress/e2e-api-b-96339b-openshift-default-857fd4b79-52nqm"
Nov 02 12:42:11.543756 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:11.541171    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e51ff2f8-753e-4977-a523-90bca8a4ec2a-istio-podinfo\") pod \"e2e-api-b-96339b-openshift-default-857fd4b79-52nqm\" (UID: \"e51ff2f8-753e-4977-a523-90bca8a4ec2a\") " pod="openshift-ingress/e2e-api-b-96339b-openshift-default-857fd4b79-52nqm"
Nov 02 12:42:11.560717 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:11.558751    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9c8sp\" (UniqueName: \"kubernetes.io/projected/e51ff2f8-753e-4977-a523-90bca8a4ec2a-kube-api-access-9c8sp\") pod \"e2e-api-b-96339b-openshift-default-857fd4b79-52nqm\" (UID: \"e51ff2f8-753e-4977-a523-90bca8a4ec2a\") " pod="openshift-ingress/e2e-api-b-96339b-openshift-default-857fd4b79-52nqm"
Nov 02 12:42:11.572663 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:11.569243    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e51ff2f8-753e-4977-a523-90bca8a4ec2a-istio-token\") pod \"e2e-api-b-96339b-openshift-default-857fd4b79-52nqm\" (UID: \"e51ff2f8-753e-4977-a523-90bca8a4ec2a\") " pod="openshift-ingress/e2e-api-b-96339b-openshift-default-857fd4b79-52nqm"
Nov 02 12:42:11.629663 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:11.624871    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-96339b-openshift-default-857fd4b79-52nqm"
Nov 02 12:42:12.076737 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:42:12.076704    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode51ff2f8_753e_4977_a523_90bca8a4ec2a.slice/crio-6e486983da490d8477d960b384bac287be98b2a69645886f9e9855eed04b6024 WatchSource:0}: Error finding container 6e486983da490d8477d960b384bac287be98b2a69645886f9e9855eed04b6024: Status 404 returned error can't find the container with id 6e486983da490d8477d960b384bac287be98b2a69645886f9e9855eed04b6024
Nov 02 12:42:12.077724 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:12.077695    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-96339b-openshift-default-857fd4b79-52nqm"]
Nov 02 12:42:12.079761 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:12.079567    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:42:12.079761 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:12.079712    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:42:12.079761 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:12.079752    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:42:12.264701 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:12.262729    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-96339b-openshift-default-857fd4b79-52nqm" event={"ID":"e51ff2f8-753e-4977-a523-90bca8a4ec2a","Type":"ContainerStarted","Data":"2d9ba826fa413fa53e83afa46516baa134cdb90eadb757ff07b8e883f8540ca5"}
Nov 02 12:42:12.264701 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:12.262800    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-96339b-openshift-default-857fd4b79-52nqm" event={"ID":"e51ff2f8-753e-4977-a523-90bca8a4ec2a","Type":"ContainerStarted","Data":"6e486983da490d8477d960b384bac287be98b2a69645886f9e9855eed04b6024"}
Nov 02 12:42:12.321421 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:12.321363    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-96339b-openshift-default-857fd4b79-52nqm" podStartSLOduration=1.321347619 podStartE2EDuration="1.321347619s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:42:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:42:12.31934711 +0000 UTC m=+928.574746239" watchObservedRunningTime="2026-11-02 12:42:12.321347619 +0000 UTC m=+928.576746768"
Nov 02 12:42:12.453967 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:12.451502    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-cd1386cb/e2e-lifecycle-model-cd1386cb-kserve-7d9d698845-2579r"
Nov 02 12:42:12.558487 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:12.558342    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-cd1386cb/e2e-lifecycle-model-cd1386cb-kserve-7d9d698845-2579r"
Nov 02 12:42:12.625416 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:12.625383    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-96339b-openshift-default-857fd4b79-52nqm"
Nov 02 12:42:13.280881 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:13.280851    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-cd1386cb/e2e-lifecycle-model-cd1386cb-kserve-7d9d698845-2579r"
Nov 02 12:42:13.631169 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:13.631083    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-96339b-openshift-default-857fd4b79-52nqm"
Nov 02 12:42:14.278384 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:14.278355    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-96339b-openshift-default-857fd4b79-52nqm"
Nov 02 12:42:14.279345 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:14.279319    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-96339b-openshift-default-857fd4b79-52nqm"
Nov 02 12:42:15.087265 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:42:15.087229    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqcmj" containerName="extractor"
Nov 02 12:42:15.461825 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:15.461787    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"]
Nov 02 12:42:20.375775 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:20.375714    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-61e341/distinct-w1-61e341-kserve-787cd566f4-cjj8s" podUID="6886ce28-5800-402d-b893-f92109e289a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:42:21.038829 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:21.038788    4559 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-66rd4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 12:42:21.039158 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:21.039112    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-66rd4" podUID="e09c5fe2-c1dd-47ab-8464-facefd253549" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:42:42.847147 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:42.847109    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-9f7f01-openshift-default-64d8f9568c-pw8tx"]
Nov 02 12:42:42.853482 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:42.853456    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-9f7f01-openshift-default-64d8f9568c-pw8tx"
Nov 02 12:42:42.885013 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:42.884986    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-9f7f01-openshift-default-dockercfg-hkzth\""
Nov 02 12:42:42.892996 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:42.892968    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-9f7f01-openshift-default-64d8f9568c-pw8tx"]
Nov 02 12:42:43.037662 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:43.036994    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26b966b3-2043-474a-95aa-3037982c3fc4-workload-socket\") pod \"e2e-api-a-9f7f01-openshift-default-64d8f9568c-pw8tx\" (UID: \"26b966b3-2043-474a-95aa-3037982c3fc4\") " pod="openshift-ingress/e2e-api-a-9f7f01-openshift-default-64d8f9568c-pw8tx"
Nov 02 12:42:43.037662 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:43.037106    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hmdj\" (UniqueName: \"kubernetes.io/projected/26b966b3-2043-474a-95aa-3037982c3fc4-kube-api-access-6hmdj\") pod \"e2e-api-a-9f7f01-openshift-default-64d8f9568c-pw8tx\" (UID: \"26b966b3-2043-474a-95aa-3037982c3fc4\") " pod="openshift-ingress/e2e-api-a-9f7f01-openshift-default-64d8f9568c-pw8tx"
Nov 02 12:42:43.037662 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:43.037360    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26b966b3-2043-474a-95aa-3037982c3fc4-istio-podinfo\") pod \"e2e-api-a-9f7f01-openshift-default-64d8f9568c-pw8tx\" (UID: \"26b966b3-2043-474a-95aa-3037982c3fc4\") " pod="openshift-ingress/e2e-api-a-9f7f01-openshift-default-64d8f9568c-pw8tx"
Nov 02 12:42:43.037662 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:43.037403    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26b966b3-2043-474a-95aa-3037982c3fc4-credential-socket\") pod \"e2e-api-a-9f7f01-openshift-default-64d8f9568c-pw8tx\" (UID: \"26b966b3-2043-474a-95aa-3037982c3fc4\") " pod="openshift-ingress/e2e-api-a-9f7f01-openshift-default-64d8f9568c-pw8tx"
Nov 02 12:42:43.037662 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:43.037453    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26b966b3-2043-474a-95aa-3037982c3fc4-istio-token\") pod \"e2e-api-a-9f7f01-openshift-default-64d8f9568c-pw8tx\" (UID: \"26b966b3-2043-474a-95aa-3037982c3fc4\") " pod="openshift-ingress/e2e-api-a-9f7f01-openshift-default-64d8f9568c-pw8tx"
Nov 02 12:42:43.037662 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:43.037501    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26b966b3-2043-474a-95aa-3037982c3fc4-istiod-ca-cert\") pod \"e2e-api-a-9f7f01-openshift-default-64d8f9568c-pw8tx\" (UID: \"26b966b3-2043-474a-95aa-3037982c3fc4\") " pod="openshift-ingress/e2e-api-a-9f7f01-openshift-default-64d8f9568c-pw8tx"
Nov 02 12:42:43.037662 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:43.037548    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26b966b3-2043-474a-95aa-3037982c3fc4-workload-certs\") pod \"e2e-api-a-9f7f01-openshift-default-64d8f9568c-pw8tx\" (UID: \"26b966b3-2043-474a-95aa-3037982c3fc4\") " pod="openshift-ingress/e2e-api-a-9f7f01-openshift-default-64d8f9568c-pw8tx"
Nov 02 12:42:43.038189 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:43.037677    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26b966b3-2043-474a-95aa-3037982c3fc4-istio-data\") pod \"e2e-api-a-9f7f01-openshift-default-64d8f9568c-pw8tx\" (UID: \"26b966b3-2043-474a-95aa-3037982c3fc4\") " pod="openshift-ingress/e2e-api-a-9f7f01-openshift-default-64d8f9568c-pw8tx"
Nov 02 12:42:43.038189 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:43.037706    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26b966b3-2043-474a-95aa-3037982c3fc4-istio-envoy\") pod \"e2e-api-a-9f7f01-openshift-default-64d8f9568c-pw8tx\" (UID: \"26b966b3-2043-474a-95aa-3037982c3fc4\") " pod="openshift-ingress/e2e-api-a-9f7f01-openshift-default-64d8f9568c-pw8tx"
Nov 02 12:42:43.143715 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:43.142667    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hmdj\" (UniqueName: \"kubernetes.io/projected/26b966b3-2043-474a-95aa-3037982c3fc4-kube-api-access-6hmdj\") pod \"e2e-api-a-9f7f01-openshift-default-64d8f9568c-pw8tx\" (UID: \"26b966b3-2043-474a-95aa-3037982c3fc4\") " pod="openshift-ingress/e2e-api-a-9f7f01-openshift-default-64d8f9568c-pw8tx"
Nov 02 12:42:43.143715 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:43.142749    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26b966b3-2043-474a-95aa-3037982c3fc4-istio-podinfo\") pod \"e2e-api-a-9f7f01-openshift-default-64d8f9568c-pw8tx\" (UID: \"26b966b3-2043-474a-95aa-3037982c3fc4\") " pod="openshift-ingress/e2e-api-a-9f7f01-openshift-default-64d8f9568c-pw8tx"
Nov 02 12:42:43.143715 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:43.142791    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26b966b3-2043-474a-95aa-3037982c3fc4-credential-socket\") pod \"e2e-api-a-9f7f01-openshift-default-64d8f9568c-pw8tx\" (UID: \"26b966b3-2043-474a-95aa-3037982c3fc4\") " pod="openshift-ingress/e2e-api-a-9f7f01-openshift-default-64d8f9568c-pw8tx"
Nov 02 12:42:43.143715 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:43.142839    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26b966b3-2043-474a-95aa-3037982c3fc4-istio-token\") pod \"e2e-api-a-9f7f01-openshift-default-64d8f9568c-pw8tx\" (UID: \"26b966b3-2043-474a-95aa-3037982c3fc4\") " pod="openshift-ingress/e2e-api-a-9f7f01-openshift-default-64d8f9568c-pw8tx"
Nov 02 12:42:43.143715 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:43.142885    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26b966b3-2043-474a-95aa-3037982c3fc4-istiod-ca-cert\") pod \"e2e-api-a-9f7f01-openshift-default-64d8f9568c-pw8tx\" (UID: \"26b966b3-2043-474a-95aa-3037982c3fc4\") " pod="openshift-ingress/e2e-api-a-9f7f01-openshift-default-64d8f9568c-pw8tx"
Nov 02 12:42:43.143715 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:43.142935    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26b966b3-2043-474a-95aa-3037982c3fc4-workload-certs\") pod \"e2e-api-a-9f7f01-openshift-default-64d8f9568c-pw8tx\" (UID: \"26b966b3-2043-474a-95aa-3037982c3fc4\") " pod="openshift-ingress/e2e-api-a-9f7f01-openshift-default-64d8f9568c-pw8tx"
Nov 02 12:42:43.143715 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:43.143037    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26b966b3-2043-474a-95aa-3037982c3fc4-istio-data\") pod \"e2e-api-a-9f7f01-openshift-default-64d8f9568c-pw8tx\" (UID: \"26b966b3-2043-474a-95aa-3037982c3fc4\") " pod="openshift-ingress/e2e-api-a-9f7f01-openshift-default-64d8f9568c-pw8tx"
Nov 02 12:42:43.143715 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:43.143064    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26b966b3-2043-474a-95aa-3037982c3fc4-istio-envoy\") pod \"e2e-api-a-9f7f01-openshift-default-64d8f9568c-pw8tx\" (UID: \"26b966b3-2043-474a-95aa-3037982c3fc4\") " pod="openshift-ingress/e2e-api-a-9f7f01-openshift-default-64d8f9568c-pw8tx"
Nov 02 12:42:43.143715 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:43.143192    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26b966b3-2043-474a-95aa-3037982c3fc4-workload-socket\") pod \"e2e-api-a-9f7f01-openshift-default-64d8f9568c-pw8tx\" (UID: \"26b966b3-2043-474a-95aa-3037982c3fc4\") " pod="openshift-ingress/e2e-api-a-9f7f01-openshift-default-64d8f9568c-pw8tx"
Nov 02 12:42:43.143715 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:43.143700    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26b966b3-2043-474a-95aa-3037982c3fc4-workload-socket\") pod \"e2e-api-a-9f7f01-openshift-default-64d8f9568c-pw8tx\" (UID: \"26b966b3-2043-474a-95aa-3037982c3fc4\") " pod="openshift-ingress/e2e-api-a-9f7f01-openshift-default-64d8f9568c-pw8tx"
Nov 02 12:42:43.156663 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:43.145087    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26b966b3-2043-474a-95aa-3037982c3fc4-istiod-ca-cert\") pod \"e2e-api-a-9f7f01-openshift-default-64d8f9568c-pw8tx\" (UID: \"26b966b3-2043-474a-95aa-3037982c3fc4\") " pod="openshift-ingress/e2e-api-a-9f7f01-openshift-default-64d8f9568c-pw8tx"
Nov 02 12:42:43.156663 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:43.145338    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26b966b3-2043-474a-95aa-3037982c3fc4-credential-socket\") pod \"e2e-api-a-9f7f01-openshift-default-64d8f9568c-pw8tx\" (UID: \"26b966b3-2043-474a-95aa-3037982c3fc4\") " pod="openshift-ingress/e2e-api-a-9f7f01-openshift-default-64d8f9568c-pw8tx"
Nov 02 12:42:43.156663 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:43.145864    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26b966b3-2043-474a-95aa-3037982c3fc4-istio-data\") pod \"e2e-api-a-9f7f01-openshift-default-64d8f9568c-pw8tx\" (UID: \"26b966b3-2043-474a-95aa-3037982c3fc4\") " pod="openshift-ingress/e2e-api-a-9f7f01-openshift-default-64d8f9568c-pw8tx"
Nov 02 12:42:43.156663 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:43.146091    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26b966b3-2043-474a-95aa-3037982c3fc4-workload-certs\") pod \"e2e-api-a-9f7f01-openshift-default-64d8f9568c-pw8tx\" (UID: \"26b966b3-2043-474a-95aa-3037982c3fc4\") " pod="openshift-ingress/e2e-api-a-9f7f01-openshift-default-64d8f9568c-pw8tx"
Nov 02 12:42:43.178500 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:43.178467    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26b966b3-2043-474a-95aa-3037982c3fc4-istio-envoy\") pod \"e2e-api-a-9f7f01-openshift-default-64d8f9568c-pw8tx\" (UID: \"26b966b3-2043-474a-95aa-3037982c3fc4\") " pod="openshift-ingress/e2e-api-a-9f7f01-openshift-default-64d8f9568c-pw8tx"
Nov 02 12:42:43.179259 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:43.179229    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26b966b3-2043-474a-95aa-3037982c3fc4-istio-podinfo\") pod \"e2e-api-a-9f7f01-openshift-default-64d8f9568c-pw8tx\" (UID: \"26b966b3-2043-474a-95aa-3037982c3fc4\") " pod="openshift-ingress/e2e-api-a-9f7f01-openshift-default-64d8f9568c-pw8tx"
Nov 02 12:42:43.192716 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:43.184445    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hmdj\" (UniqueName: \"kubernetes.io/projected/26b966b3-2043-474a-95aa-3037982c3fc4-kube-api-access-6hmdj\") pod \"e2e-api-a-9f7f01-openshift-default-64d8f9568c-pw8tx\" (UID: \"26b966b3-2043-474a-95aa-3037982c3fc4\") " pod="openshift-ingress/e2e-api-a-9f7f01-openshift-default-64d8f9568c-pw8tx"
Nov 02 12:42:43.192918 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:43.188062    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26b966b3-2043-474a-95aa-3037982c3fc4-istio-token\") pod \"e2e-api-a-9f7f01-openshift-default-64d8f9568c-pw8tx\" (UID: \"26b966b3-2043-474a-95aa-3037982c3fc4\") " pod="openshift-ingress/e2e-api-a-9f7f01-openshift-default-64d8f9568c-pw8tx"
Nov 02 12:42:43.490629 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:43.489113    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-9f7f01-openshift-default-64d8f9568c-pw8tx"
Nov 02 12:42:43.852513 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:42:43.852485    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26b966b3_2043_474a_95aa_3037982c3fc4.slice/crio-2124fa563fc8e9d6e80ec77eacfae49a93490f54ad34b6945a077413d5447cc8 WatchSource:0}: Error finding container 2124fa563fc8e9d6e80ec77eacfae49a93490f54ad34b6945a077413d5447cc8: Status 404 returned error can't find the container with id 2124fa563fc8e9d6e80ec77eacfae49a93490f54ad34b6945a077413d5447cc8
Nov 02 12:42:43.853518 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:43.853491    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-9f7f01-openshift-default-64d8f9568c-pw8tx"]
Nov 02 12:42:43.854945 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:43.854914    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:42:43.855034 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:43.854975    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:42:43.855034 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:43.855008    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:42:44.107186 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:44.107089    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-9f7f01-openshift-default-64d8f9568c-pw8tx" event={"ID":"26b966b3-2043-474a-95aa-3037982c3fc4","Type":"ContainerStarted","Data":"67baa54012d28b2eded48236c9ab1b20b43f10ba74555771f760756c4dba1ec7"}
Nov 02 12:42:44.107186 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:44.107156    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-9f7f01-openshift-default-64d8f9568c-pw8tx" event={"ID":"26b966b3-2043-474a-95aa-3037982c3fc4","Type":"ContainerStarted","Data":"2124fa563fc8e9d6e80ec77eacfae49a93490f54ad34b6945a077413d5447cc8"}
Nov 02 12:42:44.159195 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:44.159133    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-9f7f01-openshift-default-64d8f9568c-pw8tx" podStartSLOduration=2.159113519 podStartE2EDuration="2.159113519s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:42:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:42:44.156957652 +0000 UTC m=+960.412356746" watchObservedRunningTime="2026-11-02 12:42:44.159113519 +0000 UTC m=+960.414512617"
Nov 02 12:42:44.496662 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:44.494536    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-9f7f01-openshift-default-64d8f9568c-pw8tx"
Nov 02 12:42:44.551664 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:44.547774    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-9f7f01-openshift-default-64d8f9568c-pw8tx"
Nov 02 12:42:45.086013 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:42:45.085981    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqcmj" containerName="extractor"
Nov 02 12:42:45.113352 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:45.113329    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-9f7f01-openshift-default-64d8f9568c-pw8tx"
Nov 02 12:42:45.114240 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:45.114220    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-9f7f01-openshift-default-64d8f9568c-pw8tx"
Nov 02 12:42:48.547368 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:48.547338    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-9f7f01-55c4f68b6c-s4sdn"]
Nov 02 12:42:48.551466 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:48.551445    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9f7f01-55c4f68b6c-s4sdn"
Nov 02 12:42:48.563951 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:48.563925    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zv4kj\" (UniqueName: \"kubernetes.io/projected/d2f6353b-9297-4a5b-944f-52afb4380d25-kube-api-access-zv4kj\") pod \"payload-pre-processing-e2e-api-a-9f7f01-55c4f68b6c-s4sdn\" (UID: \"d2f6353b-9297-4a5b-944f-52afb4380d25\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-9f7f01-55c4f68b6c-s4sdn"
Nov 02 12:42:48.564161 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:48.564143    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2f6353b-9297-4a5b-944f-52afb4380d25-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-9f7f01-55c4f68b6c-s4sdn\" (UID: \"d2f6353b-9297-4a5b-944f-52afb4380d25\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-9f7f01-55c4f68b6c-s4sdn"
Nov 02 12:42:48.572142 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:48.572121    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-9f7f01-55c4f68b6c-s4sdn"]
Nov 02 12:42:48.580429 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:48.580400    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-9f7f01\""
Nov 02 12:42:48.580429 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:48.580420    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-9f7f01-dockercfg-s6bnc\""
Nov 02 12:42:48.866745 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:48.808694    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-9f7f01-5548c7f9b5-nq722"]
Nov 02 12:42:48.867988 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:48.867951    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zv4kj\" (UniqueName: \"kubernetes.io/projected/d2f6353b-9297-4a5b-944f-52afb4380d25-kube-api-access-zv4kj\") pod \"payload-pre-processing-e2e-api-a-9f7f01-55c4f68b6c-s4sdn\" (UID: \"d2f6353b-9297-4a5b-944f-52afb4380d25\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-9f7f01-55c4f68b6c-s4sdn"
Nov 02 12:42:48.868626 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:48.868606    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2f6353b-9297-4a5b-944f-52afb4380d25-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-9f7f01-55c4f68b6c-s4sdn\" (UID: \"d2f6353b-9297-4a5b-944f-52afb4380d25\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-9f7f01-55c4f68b6c-s4sdn"
Nov 02 12:42:48.890508 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:48.890472    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2f6353b-9297-4a5b-944f-52afb4380d25-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-9f7f01-55c4f68b6c-s4sdn\" (UID: \"d2f6353b-9297-4a5b-944f-52afb4380d25\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-9f7f01-55c4f68b6c-s4sdn"
Nov 02 12:42:48.912595 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:48.912514    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zv4kj\" (UniqueName: \"kubernetes.io/projected/d2f6353b-9297-4a5b-944f-52afb4380d25-kube-api-access-zv4kj\") pod \"payload-pre-processing-e2e-api-a-9f7f01-55c4f68b6c-s4sdn\" (UID: \"d2f6353b-9297-4a5b-944f-52afb4380d25\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-9f7f01-55c4f68b6c-s4sdn"
Nov 02 12:42:48.936124 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:48.936093    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-9f7f01-5548c7f9b5-nq722"]
Nov 02 12:42:48.936852 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:48.936828    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-9f7f01-5548c7f9b5-nq722"
Nov 02 12:42:48.975582 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:48.974676    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kc4tt\" (UniqueName: \"kubernetes.io/projected/882d2459-4272-4f63-82d6-b8fe06253754-kube-api-access-kc4tt\") pod \"payload-processing-e2e-api-a-9f7f01-5548c7f9b5-nq722\" (UID: \"882d2459-4272-4f63-82d6-b8fe06253754\") " pod="openshift-ingress/payload-processing-e2e-api-a-9f7f01-5548c7f9b5-nq722"
Nov 02 12:42:48.975582 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:48.974746    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/882d2459-4272-4f63-82d6-b8fe06253754-plugins-config-volume\") pod \"payload-processing-e2e-api-a-9f7f01-5548c7f9b5-nq722\" (UID: \"882d2459-4272-4f63-82d6-b8fe06253754\") " pod="openshift-ingress/payload-processing-e2e-api-a-9f7f01-5548c7f9b5-nq722"
Nov 02 12:42:49.075852 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:49.075814    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kc4tt\" (UniqueName: \"kubernetes.io/projected/882d2459-4272-4f63-82d6-b8fe06253754-kube-api-access-kc4tt\") pod \"payload-processing-e2e-api-a-9f7f01-5548c7f9b5-nq722\" (UID: \"882d2459-4272-4f63-82d6-b8fe06253754\") " pod="openshift-ingress/payload-processing-e2e-api-a-9f7f01-5548c7f9b5-nq722"
Nov 02 12:42:49.076018 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:49.075883    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/882d2459-4272-4f63-82d6-b8fe06253754-plugins-config-volume\") pod \"payload-processing-e2e-api-a-9f7f01-5548c7f9b5-nq722\" (UID: \"882d2459-4272-4f63-82d6-b8fe06253754\") " pod="openshift-ingress/payload-processing-e2e-api-a-9f7f01-5548c7f9b5-nq722"
Nov 02 12:42:49.076563 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:49.076542    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/882d2459-4272-4f63-82d6-b8fe06253754-plugins-config-volume\") pod \"payload-processing-e2e-api-a-9f7f01-5548c7f9b5-nq722\" (UID: \"882d2459-4272-4f63-82d6-b8fe06253754\") " pod="openshift-ingress/payload-processing-e2e-api-a-9f7f01-5548c7f9b5-nq722"
Nov 02 12:42:49.099752 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:49.099727    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kc4tt\" (UniqueName: \"kubernetes.io/projected/882d2459-4272-4f63-82d6-b8fe06253754-kube-api-access-kc4tt\") pod \"payload-processing-e2e-api-a-9f7f01-5548c7f9b5-nq722\" (UID: \"882d2459-4272-4f63-82d6-b8fe06253754\") " pod="openshift-ingress/payload-processing-e2e-api-a-9f7f01-5548c7f9b5-nq722"
Nov 02 12:42:49.169007 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:49.168981    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9f7f01-55c4f68b6c-s4sdn"
Nov 02 12:42:49.286316 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:49.286286    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-9f7f01-5548c7f9b5-nq722"
Nov 02 12:42:49.472828 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:42:49.472795    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2f6353b_9297_4a5b_944f_52afb4380d25.slice/crio-0ed3b60232ee1171291b0fa80608077d644f37aff92514b2066aa1735c0f89cb WatchSource:0}: Error finding container 0ed3b60232ee1171291b0fa80608077d644f37aff92514b2066aa1735c0f89cb: Status 404 returned error can't find the container with id 0ed3b60232ee1171291b0fa80608077d644f37aff92514b2066aa1735c0f89cb
Nov 02 12:42:49.473769 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:49.473746    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-9f7f01-55c4f68b6c-s4sdn"]
Nov 02 12:42:49.595479 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:42:49.595453    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod882d2459_4272_4f63_82d6_b8fe06253754.slice/crio-bcc2117bf7d7b3060568e530993bc55679a8b4e7291b6920f2dd0b7547acb0f2 WatchSource:0}: Error finding container bcc2117bf7d7b3060568e530993bc55679a8b4e7291b6920f2dd0b7547acb0f2: Status 404 returned error can't find the container with id bcc2117bf7d7b3060568e530993bc55679a8b4e7291b6920f2dd0b7547acb0f2
Nov 02 12:42:49.598129 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:49.598101    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-9f7f01-5548c7f9b5-nq722"]
Nov 02 12:42:50.150997 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:50.150956    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9f7f01-55c4f68b6c-s4sdn" event={"ID":"d2f6353b-9297-4a5b-944f-52afb4380d25","Type":"ContainerStarted","Data":"443f348dea23bb996e5dddba1ffa83597c7c972fb31c8735516d8ecc89e3d05f"}
Nov 02 12:42:50.151173 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:50.151025    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9f7f01-55c4f68b6c-s4sdn" event={"ID":"d2f6353b-9297-4a5b-944f-52afb4380d25","Type":"ContainerStarted","Data":"0ed3b60232ee1171291b0fa80608077d644f37aff92514b2066aa1735c0f89cb"}
Nov 02 12:42:50.151173 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:50.151057    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9f7f01-55c4f68b6c-s4sdn"
Nov 02 12:42:50.152584 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:50.152550    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-9f7f01-5548c7f9b5-nq722" event={"ID":"882d2459-4272-4f63-82d6-b8fe06253754","Type":"ContainerStarted","Data":"b94287ff0c1a981508b67b95c08001274f7bbea9c4f7f9799bebaa2ce5dfe0fa"}
Nov 02 12:42:50.152710 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:50.152606    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-9f7f01-5548c7f9b5-nq722" event={"ID":"882d2459-4272-4f63-82d6-b8fe06253754","Type":"ContainerStarted","Data":"bcc2117bf7d7b3060568e530993bc55679a8b4e7291b6920f2dd0b7547acb0f2"}
Nov 02 12:42:50.152710 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:50.152657    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-9f7f01-5548c7f9b5-nq722"
Nov 02 12:42:50.194842 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:50.194800    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9f7f01-55c4f68b6c-s4sdn" podStartSLOduration=2.194787567 podStartE2EDuration="2.194787567s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:42:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:42:50.192213637 +0000 UTC m=+966.447612713" watchObservedRunningTime="2026-11-02 12:42:50.194787567 +0000 UTC m=+966.450186662"
Nov 02 12:42:50.237878 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:50.237835    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-9f7f01-5548c7f9b5-nq722" podStartSLOduration=2.237823482 podStartE2EDuration="2.237823482s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:42:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:42:50.235222623 +0000 UTC m=+966.490621717" watchObservedRunningTime="2026-11-02 12:42:50.237823482 +0000 UTC m=+966.493222577"
Nov 02 12:42:54.372660 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:54.367099    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"]
Nov 02 12:42:57.550663 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:57.549702    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-97e08c-69f98f57c9-gb4vp"]
Nov 02 12:42:57.555409 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:57.555385    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-97e08c-69f98f57c9-gb4vp"
Nov 02 12:42:57.573281 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:57.573253    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-97e08c-69f98f57c9-gb4vp"]
Nov 02 12:42:57.581818 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:57.581793    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-97e08c-dockercfg-fm4c7\""
Nov 02 12:42:57.582303 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:57.582237    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-97e08c\""
Nov 02 12:42:57.686663 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:57.685130    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wzgk\" (UniqueName: \"kubernetes.io/projected/9ee37559-86ab-44e9-b50f-e360036fa07c-kube-api-access-7wzgk\") pod \"payload-processing-e2e-api-a-97e08c-69f98f57c9-gb4vp\" (UID: \"9ee37559-86ab-44e9-b50f-e360036fa07c\") " pod="openshift-ingress/payload-processing-e2e-api-a-97e08c-69f98f57c9-gb4vp"
Nov 02 12:42:57.686663 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:57.685479    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ee37559-86ab-44e9-b50f-e360036fa07c-plugins-config-volume\") pod \"payload-processing-e2e-api-a-97e08c-69f98f57c9-gb4vp\" (UID: \"9ee37559-86ab-44e9-b50f-e360036fa07c\") " pod="openshift-ingress/payload-processing-e2e-api-a-97e08c-69f98f57c9-gb4vp"
Nov 02 12:42:57.815677 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:57.815549    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ee37559-86ab-44e9-b50f-e360036fa07c-plugins-config-volume\") pod \"payload-processing-e2e-api-a-97e08c-69f98f57c9-gb4vp\" (UID: \"9ee37559-86ab-44e9-b50f-e360036fa07c\") " pod="openshift-ingress/payload-processing-e2e-api-a-97e08c-69f98f57c9-gb4vp"
Nov 02 12:42:57.816056 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:57.816021    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wzgk\" (UniqueName: \"kubernetes.io/projected/9ee37559-86ab-44e9-b50f-e360036fa07c-kube-api-access-7wzgk\") pod \"payload-processing-e2e-api-a-97e08c-69f98f57c9-gb4vp\" (UID: \"9ee37559-86ab-44e9-b50f-e360036fa07c\") " pod="openshift-ingress/payload-processing-e2e-api-a-97e08c-69f98f57c9-gb4vp"
Nov 02 12:42:57.817757 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:57.817732    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ee37559-86ab-44e9-b50f-e360036fa07c-plugins-config-volume\") pod \"payload-processing-e2e-api-a-97e08c-69f98f57c9-gb4vp\" (UID: \"9ee37559-86ab-44e9-b50f-e360036fa07c\") " pod="openshift-ingress/payload-processing-e2e-api-a-97e08c-69f98f57c9-gb4vp"
Nov 02 12:42:57.907086 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:57.907040    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wzgk\" (UniqueName: \"kubernetes.io/projected/9ee37559-86ab-44e9-b50f-e360036fa07c-kube-api-access-7wzgk\") pod \"payload-processing-e2e-api-a-97e08c-69f98f57c9-gb4vp\" (UID: \"9ee37559-86ab-44e9-b50f-e360036fa07c\") " pod="openshift-ingress/payload-processing-e2e-api-a-97e08c-69f98f57c9-gb4vp"
Nov 02 12:42:58.178352 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:58.178320    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-97e08c-69f98f57c9-gb4vp"
Nov 02 12:42:58.519260 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:42:58.519227    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ee37559_86ab_44e9_b50f_e360036fa07c.slice/crio-a047efaec410b90e764e2fae189da7be01ad3d59837179715b952930136b4d2a WatchSource:0}: Error finding container a047efaec410b90e764e2fae189da7be01ad3d59837179715b952930136b4d2a: Status 404 returned error can't find the container with id a047efaec410b90e764e2fae189da7be01ad3d59837179715b952930136b4d2a
Nov 02 12:42:58.520312 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:58.520285    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-97e08c-69f98f57c9-gb4vp"]
Nov 02 12:42:59.280230 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:59.280191    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-97e08c-69f98f57c9-gb4vp" event={"ID":"9ee37559-86ab-44e9-b50f-e360036fa07c","Type":"ContainerStarted","Data":"fba2eeb8842e0f52d7f056f99cc761089617123806bc8aff1dad1306ee8d180b"}
Nov 02 12:42:59.280624 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:59.280260    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-97e08c-69f98f57c9-gb4vp"
Nov 02 12:42:59.280624 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:59.280288    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-97e08c-69f98f57c9-gb4vp" event={"ID":"9ee37559-86ab-44e9-b50f-e360036fa07c","Type":"ContainerStarted","Data":"a047efaec410b90e764e2fae189da7be01ad3d59837179715b952930136b4d2a"}
Nov 02 12:42:59.325061 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:59.325018    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-97e08c-69f98f57c9-gb4vp" podStartSLOduration=2.32500359 podStartE2EDuration="2.32500359s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:42:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:42:59.321689148 +0000 UTC m=+975.577088243" watchObservedRunningTime="2026-11-02 12:42:59.32500359 +0000 UTC m=+975.580402685"
Nov 02 12:42:59.541150 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:59.539841    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-785a7a-55bc5f549c-mkv2x"]
Nov 02 12:42:59.589476 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:42:59.589443    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-785a7a-7489f6b895-s2swp"]
Nov 02 12:43:01.162216 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:01.162161    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9f7f01-55c4f68b6c-s4sdn"
Nov 02 12:43:01.162825 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:01.162297    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-9f7f01-5548c7f9b5-nq722"
Nov 02 12:43:04.548431 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:04.548358    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-785a7a-55bc5f549c-mkv2x" podUID="ced16952-82be-49e1-a993-b009b0aa9c64" containerName="payload-processing" containerID="cri-o://930e1d045753c49edab2c1d23b7962213fe00600f3ce6e89fd5c755005a14b1b" gracePeriod=25
Nov 02 12:43:04.588345 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:43:04.588316    4559 logging.go:55] [core] [Channel #1052 SubChannel #1053]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.80:50052", ServerName: "10.133.0.80:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.80:50052: connect: connection refused"
Nov 02 12:43:04.591975 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:04.591761    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-785a7a-7489f6b895-s2swp" podUID="c5a77cc3-ce6a-4a5d-942e-83db434f8efb" containerName="payload-pre-processing" containerID="cri-o://9724d97e3c400d227a915deeaadc0859a3cc3fef889db0c2ef48487d3d8f30b4" gracePeriod=25
Nov 02 12:43:05.009660 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:05.002547    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-785a7a-7489f6b895-s2swp"
Nov 02 12:43:05.074599 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:05.074573    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-785a7a-55bc5f549c-mkv2x"
Nov 02 12:43:05.078237 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:05.078210    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ced16952-82be-49e1-a993-b009b0aa9c64-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ced16952-82be-49e1-a993-b009b0aa9c64-plugins-config-volume\") pod \"ced16952-82be-49e1-a993-b009b0aa9c64\" (UID: \"ced16952-82be-49e1-a993-b009b0aa9c64\") "
Nov 02 12:43:05.078394 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:05.078279    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5a77cc3-ce6a-4a5d-942e-83db434f8efb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5a77cc3-ce6a-4a5d-942e-83db434f8efb-plugins-config-volume\") pod \"c5a77cc3-ce6a-4a5d-942e-83db434f8efb\" (UID: \"c5a77cc3-ce6a-4a5d-942e-83db434f8efb\") "
Nov 02 12:43:05.078394 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:05.078349    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5a77cc3-ce6a-4a5d-942e-83db434f8efb-kube-api-access-wnc7l\" (UniqueName: \"kubernetes.io/projected/c5a77cc3-ce6a-4a5d-942e-83db434f8efb-kube-api-access-wnc7l\") pod \"c5a77cc3-ce6a-4a5d-942e-83db434f8efb\" (UID: \"c5a77cc3-ce6a-4a5d-942e-83db434f8efb\") "
Nov 02 12:43:05.078990 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:05.078405    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ced16952-82be-49e1-a993-b009b0aa9c64-kube-api-access-brszw\" (UniqueName: \"kubernetes.io/projected/ced16952-82be-49e1-a993-b009b0aa9c64-kube-api-access-brszw\") pod \"ced16952-82be-49e1-a993-b009b0aa9c64\" (UID: \"ced16952-82be-49e1-a993-b009b0aa9c64\") "
Nov 02 12:43:05.078990 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:05.078687    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5a77cc3-ce6a-4a5d-942e-83db434f8efb-plugins-config-volume" pod "c5a77cc3-ce6a-4a5d-942e-83db434f8efb" (UID: "c5a77cc3-ce6a-4a5d-942e-83db434f8efb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:43:05.079111 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:05.078985    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ced16952-82be-49e1-a993-b009b0aa9c64-plugins-config-volume" pod "ced16952-82be-49e1-a993-b009b0aa9c64" (UID: "ced16952-82be-49e1-a993-b009b0aa9c64"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:43:05.079226 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:05.079207    4559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ced16952-82be-49e1-a993-b009b0aa9c64-plugins-config-volume\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:43:05.079299 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:05.079226    4559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5a77cc3-ce6a-4a5d-942e-83db434f8efb-plugins-config-volume\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:43:05.082082 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:05.082058    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5a77cc3-ce6a-4a5d-942e-83db434f8efb-kube-api-access-wnc7l" pod "c5a77cc3-ce6a-4a5d-942e-83db434f8efb" (UID: "c5a77cc3-ce6a-4a5d-942e-83db434f8efb"). InnerVolumeSpecName "kube-api-access-wnc7l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:43:05.082356 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:05.082340    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ced16952-82be-49e1-a993-b009b0aa9c64-kube-api-access-brszw" pod "ced16952-82be-49e1-a993-b009b0aa9c64" (UID: "ced16952-82be-49e1-a993-b009b0aa9c64"). InnerVolumeSpecName "kube-api-access-brszw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:43:05.181240 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:05.181208    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wnc7l\" (UniqueName: \"kubernetes.io/projected/c5a77cc3-ce6a-4a5d-942e-83db434f8efb-kube-api-access-wnc7l\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:43:05.181240 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:05.181234    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-brszw\" (UniqueName: \"kubernetes.io/projected/ced16952-82be-49e1-a993-b009b0aa9c64-kube-api-access-brszw\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:43:05.406660 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:05.406304    4559 generic.go:356] "Generic (PLEG): container finished" podID="ced16952-82be-49e1-a993-b009b0aa9c64" containerID="930e1d045753c49edab2c1d23b7962213fe00600f3ce6e89fd5c755005a14b1b" exitCode=0
Nov 02 12:43:05.406660 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:05.406435    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-785a7a-55bc5f549c-mkv2x" event={"ID":"ced16952-82be-49e1-a993-b009b0aa9c64","Type":"ContainerDied","Data":"930e1d045753c49edab2c1d23b7962213fe00600f3ce6e89fd5c755005a14b1b"}
Nov 02 12:43:05.406660 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:05.406541    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-785a7a-55bc5f549c-mkv2x" event={"ID":"ced16952-82be-49e1-a993-b009b0aa9c64","Type":"ContainerDied","Data":"d883417ba489fee7650e5391224bb6ab1256f9635d11a82759a1a4fa7634ca7a"}
Nov 02 12:43:05.406660 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:05.406578    4559 scope.go:122] "RemoveContainer" containerID="930e1d045753c49edab2c1d23b7962213fe00600f3ce6e89fd5c755005a14b1b"
Nov 02 12:43:05.430662 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:05.406792    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-785a7a-55bc5f549c-mkv2x"
Nov 02 12:43:05.449787 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:05.443831    4559 generic.go:356] "Generic (PLEG): container finished" podID="c5a77cc3-ce6a-4a5d-942e-83db434f8efb" containerID="9724d97e3c400d227a915deeaadc0859a3cc3fef889db0c2ef48487d3d8f30b4" exitCode=0
Nov 02 12:43:05.449787 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:05.443978    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-785a7a-7489f6b895-s2swp" event={"ID":"c5a77cc3-ce6a-4a5d-942e-83db434f8efb","Type":"ContainerDied","Data":"9724d97e3c400d227a915deeaadc0859a3cc3fef889db0c2ef48487d3d8f30b4"}
Nov 02 12:43:05.449787 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:05.444068    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-785a7a-7489f6b895-s2swp" event={"ID":"c5a77cc3-ce6a-4a5d-942e-83db434f8efb","Type":"ContainerDied","Data":"7675080ab2186752ddd637fc35261ef7c009ff431079546d2772f057cadfb430"}
Nov 02 12:43:05.449787 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:05.444187    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-785a7a-7489f6b895-s2swp"
Nov 02 12:43:05.457480 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:05.451018    4559 scope.go:122] "RemoveContainer" containerID="930e1d045753c49edab2c1d23b7962213fe00600f3ce6e89fd5c755005a14b1b"
Nov 02 12:43:05.457480 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:43:05.451525    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"930e1d045753c49edab2c1d23b7962213fe00600f3ce6e89fd5c755005a14b1b\": container with ID starting with 930e1d045753c49edab2c1d23b7962213fe00600f3ce6e89fd5c755005a14b1b not found: ID does not exist" containerID="930e1d045753c49edab2c1d23b7962213fe00600f3ce6e89fd5c755005a14b1b"
Nov 02 12:43:05.457480 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:05.451567    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"930e1d045753c49edab2c1d23b7962213fe00600f3ce6e89fd5c755005a14b1b"} err="failed to get container status \"930e1d045753c49edab2c1d23b7962213fe00600f3ce6e89fd5c755005a14b1b\": rpc error: code = NotFound desc = could not find container \"930e1d045753c49edab2c1d23b7962213fe00600f3ce6e89fd5c755005a14b1b\": container with ID starting with 930e1d045753c49edab2c1d23b7962213fe00600f3ce6e89fd5c755005a14b1b not found: ID does not exist"
Nov 02 12:43:05.457480 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:05.451593    4559 scope.go:122] "RemoveContainer" containerID="9724d97e3c400d227a915deeaadc0859a3cc3fef889db0c2ef48487d3d8f30b4"
Nov 02 12:43:05.465732 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:05.465621    4559 scope.go:122] "RemoveContainer" containerID="9724d97e3c400d227a915deeaadc0859a3cc3fef889db0c2ef48487d3d8f30b4"
Nov 02 12:43:05.465938 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:43:05.465916    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9724d97e3c400d227a915deeaadc0859a3cc3fef889db0c2ef48487d3d8f30b4\": container with ID starting with 9724d97e3c400d227a915deeaadc0859a3cc3fef889db0c2ef48487d3d8f30b4 not found: ID does not exist" containerID="9724d97e3c400d227a915deeaadc0859a3cc3fef889db0c2ef48487d3d8f30b4"
Nov 02 12:43:05.465996 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:05.465949    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9724d97e3c400d227a915deeaadc0859a3cc3fef889db0c2ef48487d3d8f30b4"} err="failed to get container status \"9724d97e3c400d227a915deeaadc0859a3cc3fef889db0c2ef48487d3d8f30b4\": rpc error: code = NotFound desc = could not find container \"9724d97e3c400d227a915deeaadc0859a3cc3fef889db0c2ef48487d3d8f30b4\": container with ID starting with 9724d97e3c400d227a915deeaadc0859a3cc3fef889db0c2ef48487d3d8f30b4 not found: ID does not exist"
Nov 02 12:43:05.511310 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:05.511281    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-785a7a-7489f6b895-s2swp"]
Nov 02 12:43:05.521194 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:05.521173    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-785a7a-7489f6b895-s2swp"]
Nov 02 12:43:05.547137 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:05.547112    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-785a7a-55bc5f549c-mkv2x"]
Nov 02 12:43:05.551357 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:05.551335    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-785a7a-55bc5f549c-mkv2x"]
Nov 02 12:43:05.591558 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:05.590862    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-api-a-785a7a-55bc5f549c-mkv2x" podUID="ced16952-82be-49e1-a993-b009b0aa9c64" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.80:50052\" within 1s: context deadline exceeded"
Nov 02 12:43:05.914585 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:05.914509    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5a77cc3-ce6a-4a5d-942e-83db434f8efb" path="/var/lib/kubelet/pods/c5a77cc3-ce6a-4a5d-942e-83db434f8efb/volumes"
Nov 02 12:43:05.914951 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:05.914936    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ced16952-82be-49e1-a993-b009b0aa9c64" path="/var/lib/kubelet/pods/ced16952-82be-49e1-a993-b009b0aa9c64/volumes"
Nov 02 12:43:10.308434 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:10.308407    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-97e08c-69f98f57c9-gb4vp"
Nov 02 12:43:10.561871 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:10.561785    4559 patch_prober.go:29] interesting pod/e2e-api-a-785a7a-openshift-default-57c8cb6f7f-wp4ql container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 12:43:10.564402 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:10.564359    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-785a7a-openshift-default-57c8cb6f7f-wp4ql" podUID="afbd2f7b-f730-472b-b6ee-f98250881cc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:43:11.573344 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:11.573311    4559 generic.go:356] "Generic (PLEG): container finished" podID="afbd2f7b-f730-472b-b6ee-f98250881cc3" containerID="cdf1383f688dc5619eb1923c6e07c0eff091eaaa3f279d62e9b78b17bee0d58f" exitCode=0
Nov 02 12:43:11.573818 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:11.573353    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-785a7a-openshift-default-57c8cb6f7f-wp4ql" event={"ID":"afbd2f7b-f730-472b-b6ee-f98250881cc3","Type":"ContainerDied","Data":"cdf1383f688dc5619eb1923c6e07c0eff091eaaa3f279d62e9b78b17bee0d58f"}
Nov 02 12:43:11.573870 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:11.573821    4559 scope.go:122] "RemoveContainer" containerID="cdf1383f688dc5619eb1923c6e07c0eff091eaaa3f279d62e9b78b17bee0d58f"
Nov 02 12:43:11.575378 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:11.575347    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:43:11.575448 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:11.575419    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:43:11.575485 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:11.575456    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:43:12.581119 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:12.581080    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-785a7a-openshift-default-57c8cb6f7f-wp4ql" event={"ID":"afbd2f7b-f730-472b-b6ee-f98250881cc3","Type":"ContainerStarted","Data":"6e3956e5bcf36217b7354b75a1e0890ee6a3272d127e26d9d2cb4cc6b50c10a5"}
Nov 02 12:43:12.804662 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:12.803996    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-785a7a-openshift-default-57c8cb6f7f-wp4ql"
Nov 02 12:43:12.804662 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:12.804158    4559 patch_prober.go:29] interesting pod/e2e-api-a-785a7a-openshift-default-57c8cb6f7f-wp4ql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Nov 02 12:43:12.804662 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:12.804202    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-785a7a-openshift-default-57c8cb6f7f-wp4ql" podUID="afbd2f7b-f730-472b-b6ee-f98250881cc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Nov 02 12:43:13.817072 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:13.817034    4559 patch_prober.go:29] interesting pod/e2e-api-a-785a7a-openshift-default-57c8cb6f7f-wp4ql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Nov 02 12:43:13.817553 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:13.817105    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-785a7a-openshift-default-57c8cb6f7f-wp4ql" podUID="afbd2f7b-f730-472b-b6ee-f98250881cc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Nov 02 12:43:13.819336 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:13.819296    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-785a7a-openshift-default-57c8cb6f7f-wp4ql"]
Nov 02 12:43:14.618938 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:14.618898    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-785a7a-openshift-default-57c8cb6f7f-wp4ql" podUID="afbd2f7b-f730-472b-b6ee-f98250881cc3" containerName="istio-proxy" containerID="cri-o://6e3956e5bcf36217b7354b75a1e0890ee6a3272d127e26d9d2cb4cc6b50c10a5" gracePeriod=30
Nov 02 12:43:15.111676 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:43:15.111620    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqcmj" containerName="extractor"
Nov 02 12:43:20.584443 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:20.584399    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843978/distinct-w4-843978-kserve-6fcf588858-4nrwx" podUID="82e5d483-e95b-43e6-9135-af960823427a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:43:21.031591 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:21.031122    4559 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-66rd4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 12:43:21.031591 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:21.031195    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-66rd4" podUID="e09c5fe2-c1dd-47ab-8464-facefd253549" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:43:21.512840 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:21.512795    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-cd1386cb/e2e-lifecycle-model-cd1386cb-kserve-7d9d698845-2579r" podUID="572f58c8-cca9-48c0-99d3-0ebde6b82476" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded"
Nov 02 12:43:21.580708 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:21.579688    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-61e341/distinct-w1-61e341-kserve-787cd566f4-cjj8s" podUID="6886ce28-5800-402d-b893-f92109e289a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:43:21.589317 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:21.589278    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843978/distinct-w4-843978-kserve-6fcf588858-4nrwx" podUID="82e5d483-e95b-43e6-9135-af960823427a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:43:21.590224 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:21.590201    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-843978/distinct-w4-843978-kserve-6fcf588858-4nrwx"
Nov 02 12:43:21.628102 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:21.628063    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-61e341/distinct-2-w1-61e341-kserve-954cf956f-n7jmg" podUID="c5e1a5f8-97b4-469d-be9a-eae7ca995283" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:43:22.013544 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:22.013486    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-61e341/premium-w1-61e341-kserve-746477b99-645g6" podUID="02ff21d3-5a37-493a-ab95-8e4784c29123" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:43:22.468532 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:22.468487    4559 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-61e341/distinct-2-w1-61e341-kserve-954cf956f-n7jmg" podUID="c5e1a5f8-97b4-469d-be9a-eae7ca995283" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:43:22.523072 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:22.523029    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-cd1386cb/e2e-lifecycle-model-cd1386cb-kserve-7d9d698845-2579r" podUID="572f58c8-cca9-48c0-99d3-0ebde6b82476" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:43:22.523901 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:22.523875    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-cd1386cb/e2e-lifecycle-model-cd1386cb-kserve-7d9d698845-2579r"
Nov 02 12:43:22.593773 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:22.593725    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843978/distinct-w4-843978-kserve-6fcf588858-4nrwx" podUID="82e5d483-e95b-43e6-9135-af960823427a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:43:22.595488 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:22.595457    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-61e341/distinct-w1-61e341-kserve-787cd566f4-cjj8s" podUID="6886ce28-5800-402d-b893-f92109e289a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Nov 02 12:43:22.595993 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:22.595974    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-61e341/distinct-w1-61e341-kserve-787cd566f4-cjj8s"
Nov 02 12:43:22.641735 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:22.629712    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-61e341/distinct-2-w1-61e341-kserve-954cf956f-n7jmg" podUID="c5e1a5f8-97b4-469d-be9a-eae7ca995283" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded"
Nov 02 12:43:22.641735 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:22.630111    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-61e341/distinct-2-w1-61e341-kserve-954cf956f-n7jmg"
Nov 02 12:43:23.467977 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:23.467921    4559 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-mt-cd1386cb/e2e-lifecycle-model-cd1386cb-kserve-7d9d698845-2579r" podUID="572f58c8-cca9-48c0-99d3-0ebde6b82476" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:43:23.523792 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:23.523738    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-cd1386cb/e2e-lifecycle-model-cd1386cb-kserve-7d9d698845-2579r" podUID="572f58c8-cca9-48c0-99d3-0ebde6b82476" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:43:23.596620 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:23.596525    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-61e341/distinct-w1-61e341-kserve-787cd566f4-cjj8s" podUID="6886ce28-5800-402d-b893-f92109e289a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:43:23.597535 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:23.597174    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843978/distinct-w4-843978-kserve-6fcf588858-4nrwx" podUID="82e5d483-e95b-43e6-9135-af960823427a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:43:23.635663 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:23.631719    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-61e341/distinct-2-w1-61e341-kserve-954cf956f-n7jmg" podUID="c5e1a5f8-97b4-469d-be9a-eae7ca995283" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded"
Nov 02 12:43:23.896716 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:23.875819    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-cd1386cb/e2e-lifecycle-model-cd1386cb-kserve-7d9d698845-2579r"
Nov 02 12:43:24.167719 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:24.155300    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-843978/distinct-w4-843978-kserve-6fcf588858-4nrwx"
Nov 02 12:43:24.167995 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:24.167974    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-61e341/distinct-w1-61e341-kserve-787cd566f4-cjj8s"
Nov 02 12:43:24.168125 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:24.168115    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-61e341/distinct-2-w1-61e341-kserve-954cf956f-n7jmg"
Nov 02 12:43:24.579115 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:24.579026    4559 generic.go:356] "Generic (PLEG): container finished" podID="afbd2f7b-f730-472b-b6ee-f98250881cc3" containerID="6e3956e5bcf36217b7354b75a1e0890ee6a3272d127e26d9d2cb4cc6b50c10a5" exitCode=0
Nov 02 12:43:24.579533 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:24.579504    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-785a7a-openshift-default-57c8cb6f7f-wp4ql" event={"ID":"afbd2f7b-f730-472b-b6ee-f98250881cc3","Type":"ContainerDied","Data":"6e3956e5bcf36217b7354b75a1e0890ee6a3272d127e26d9d2cb4cc6b50c10a5"}
Nov 02 12:43:24.579721 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:24.579703    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-785a7a-openshift-default-57c8cb6f7f-wp4ql" event={"ID":"afbd2f7b-f730-472b-b6ee-f98250881cc3","Type":"ContainerDied","Data":"db93146bc0b0e4633453b94bbe7f3e3275f342c26d0efb334de0792d13f6d66f"}
Nov 02 12:43:24.579851 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:24.579837    4559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="db93146bc0b0e4633453b94bbe7f3e3275f342c26d0efb334de0792d13f6d66f"
Nov 02 12:43:24.579948 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:24.579938    4559 scope.go:122] "RemoveContainer" containerID="cdf1383f688dc5619eb1923c6e07c0eff091eaaa3f279d62e9b78b17bee0d58f"
Nov 02 12:43:24.593692 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:24.584234    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-785a7a-openshift-default-57c8cb6f7f-wp4ql"
Nov 02 12:43:24.679285 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:24.679238    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afbd2f7b-f730-472b-b6ee-f98250881cc3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afbd2f7b-f730-472b-b6ee-f98250881cc3-workload-certs\") pod \"afbd2f7b-f730-472b-b6ee-f98250881cc3\" (UID: \"afbd2f7b-f730-472b-b6ee-f98250881cc3\") "
Nov 02 12:43:24.683928 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:24.683893    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afbd2f7b-f730-472b-b6ee-f98250881cc3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afbd2f7b-f730-472b-b6ee-f98250881cc3-workload-socket\") pod \"afbd2f7b-f730-472b-b6ee-f98250881cc3\" (UID: \"afbd2f7b-f730-472b-b6ee-f98250881cc3\") "
Nov 02 12:43:24.684167 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:24.684147    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/afbd2f7b-f730-472b-b6ee-f98250881cc3-istio-token\" (UniqueName: \"kubernetes.io/projected/afbd2f7b-f730-472b-b6ee-f98250881cc3-istio-token\") pod \"afbd2f7b-f730-472b-b6ee-f98250881cc3\" (UID: \"afbd2f7b-f730-472b-b6ee-f98250881cc3\") "
Nov 02 12:43:24.687812 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:24.687788    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afbd2f7b-f730-472b-b6ee-f98250881cc3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afbd2f7b-f730-472b-b6ee-f98250881cc3-istio-envoy\") pod \"afbd2f7b-f730-472b-b6ee-f98250881cc3\" (UID: \"afbd2f7b-f730-472b-b6ee-f98250881cc3\") "
Nov 02 12:43:24.688057 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:24.688040    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/afbd2f7b-f730-472b-b6ee-f98250881cc3-kube-api-access-9k6dh\" (UniqueName: \"kubernetes.io/projected/afbd2f7b-f730-472b-b6ee-f98250881cc3-kube-api-access-9k6dh\") pod \"afbd2f7b-f730-472b-b6ee-f98250881cc3\" (UID: \"afbd2f7b-f730-472b-b6ee-f98250881cc3\") "
Nov 02 12:43:24.688276 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:24.688258    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/afbd2f7b-f730-472b-b6ee-f98250881cc3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afbd2f7b-f730-472b-b6ee-f98250881cc3-istio-podinfo\") pod \"afbd2f7b-f730-472b-b6ee-f98250881cc3\" (UID: \"afbd2f7b-f730-472b-b6ee-f98250881cc3\") "
Nov 02 12:43:24.688471 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:24.688442    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/afbd2f7b-f730-472b-b6ee-f98250881cc3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afbd2f7b-f730-472b-b6ee-f98250881cc3-istiod-ca-cert\") pod \"afbd2f7b-f730-472b-b6ee-f98250881cc3\" (UID: \"afbd2f7b-f730-472b-b6ee-f98250881cc3\") "
Nov 02 12:43:24.724858 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:24.724813    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afbd2f7b-f730-472b-b6ee-f98250881cc3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afbd2f7b-f730-472b-b6ee-f98250881cc3-credential-socket\") pod \"afbd2f7b-f730-472b-b6ee-f98250881cc3\" (UID: \"afbd2f7b-f730-472b-b6ee-f98250881cc3\") "
Nov 02 12:43:24.744675 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:24.743849    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afbd2f7b-f730-472b-b6ee-f98250881cc3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afbd2f7b-f730-472b-b6ee-f98250881cc3-istio-data\") pod \"afbd2f7b-f730-472b-b6ee-f98250881cc3\" (UID: \"afbd2f7b-f730-472b-b6ee-f98250881cc3\") "
Nov 02 12:43:24.758795 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:24.680015    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afbd2f7b-f730-472b-b6ee-f98250881cc3-workload-certs" pod "afbd2f7b-f730-472b-b6ee-f98250881cc3" (UID: "afbd2f7b-f730-472b-b6ee-f98250881cc3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:43:24.758967 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:24.725350    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afbd2f7b-f730-472b-b6ee-f98250881cc3-credential-socket" pod "afbd2f7b-f730-472b-b6ee-f98250881cc3" (UID: "afbd2f7b-f730-472b-b6ee-f98250881cc3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:43:24.759057 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:24.741851    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afbd2f7b-f730-472b-b6ee-f98250881cc3-workload-socket" pod "afbd2f7b-f730-472b-b6ee-f98250881cc3" (UID: "afbd2f7b-f730-472b-b6ee-f98250881cc3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:43:24.759650 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:24.759502    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afbd2f7b-f730-472b-b6ee-f98250881cc3-istio-data" pod "afbd2f7b-f730-472b-b6ee-f98250881cc3" (UID: "afbd2f7b-f730-472b-b6ee-f98250881cc3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:43:24.763445 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:24.762728    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/afbd2f7b-f730-472b-b6ee-f98250881cc3-istio-podinfo" pod "afbd2f7b-f730-472b-b6ee-f98250881cc3" (UID: "afbd2f7b-f730-472b-b6ee-f98250881cc3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 12:43:24.764972 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:24.763699    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afbd2f7b-f730-472b-b6ee-f98250881cc3-istiod-ca-cert" pod "afbd2f7b-f730-472b-b6ee-f98250881cc3" (UID: "afbd2f7b-f730-472b-b6ee-f98250881cc3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:43:24.767126 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:24.767106    4559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afbd2f7b-f730-472b-b6ee-f98250881cc3-credential-socket\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:43:24.767245 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:24.767231    4559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afbd2f7b-f730-472b-b6ee-f98250881cc3-workload-certs\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:43:24.767337 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:24.767326    4559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afbd2f7b-f730-472b-b6ee-f98250881cc3-workload-socket\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:43:24.774149 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:24.774107    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afbd2f7b-f730-472b-b6ee-f98250881cc3-istio-envoy" pod "afbd2f7b-f730-472b-b6ee-f98250881cc3" (UID: "afbd2f7b-f730-472b-b6ee-f98250881cc3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:43:24.838565 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:24.810826    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afbd2f7b-f730-472b-b6ee-f98250881cc3-istio-token" pod "afbd2f7b-f730-472b-b6ee-f98250881cc3" (UID: "afbd2f7b-f730-472b-b6ee-f98250881cc3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:43:24.870142 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:24.868841    4559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afbd2f7b-f730-472b-b6ee-f98250881cc3-istio-envoy\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:43:24.870142 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:24.868879    4559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afbd2f7b-f730-472b-b6ee-f98250881cc3-istio-podinfo\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:43:24.870142 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:24.868893    4559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afbd2f7b-f730-472b-b6ee-f98250881cc3-istiod-ca-cert\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:43:24.870142 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:24.868907    4559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afbd2f7b-f730-472b-b6ee-f98250881cc3-istio-data\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:43:24.870142 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:24.868919    4559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afbd2f7b-f730-472b-b6ee-f98250881cc3-istio-token\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:43:24.872589 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:24.872531    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afbd2f7b-f730-472b-b6ee-f98250881cc3-kube-api-access-9k6dh" pod "afbd2f7b-f730-472b-b6ee-f98250881cc3" (UID: "afbd2f7b-f730-472b-b6ee-f98250881cc3"). InnerVolumeSpecName "kube-api-access-9k6dh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:43:25.033776 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:25.010860    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9k6dh\" (UniqueName: \"kubernetes.io/projected/afbd2f7b-f730-472b-b6ee-f98250881cc3-kube-api-access-9k6dh\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:43:25.644868 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:25.644839    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-785a7a-openshift-default-57c8cb6f7f-wp4ql"
Nov 02 12:43:25.764153 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:25.764114    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-785a7a-openshift-default-57c8cb6f7f-wp4ql"]
Nov 02 12:43:25.769553 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:25.769524    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-785a7a-openshift-default-57c8cb6f7f-wp4ql"]
Nov 02 12:43:25.959245 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:25.959209    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="afbd2f7b-f730-472b-b6ee-f98250881cc3" path="/var/lib/kubelet/pods/afbd2f7b-f730-472b-b6ee-f98250881cc3/volumes"
Nov 02 12:43:25.959976 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:25.959944    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-cd1386cb-openshift-default-5979d457c7-2msnf"]
Nov 02 12:43:25.961312 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:25.961285    4559 cpu_manager.go:414] "containerMap: removing container" podUID="ced16952-82be-49e1-a993-b009b0aa9c64" containerName="payload-processing"
Nov 02 12:43:25.961312 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:25.961313    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ced16952-82be-49e1-a993-b009b0aa9c64" containerName="payload-processing"
Nov 02 12:43:25.961460 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:25.961333    4559 cpu_manager.go:414] "containerMap: removing container" podUID="afbd2f7b-f730-472b-b6ee-f98250881cc3" containerName="istio-proxy"
Nov 02 12:43:25.961460 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:25.961341    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="afbd2f7b-f730-472b-b6ee-f98250881cc3" containerName="istio-proxy"
Nov 02 12:43:25.961460 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:25.961366    4559 cpu_manager.go:414] "containerMap: removing container" podUID="afbd2f7b-f730-472b-b6ee-f98250881cc3" containerName="istio-proxy"
Nov 02 12:43:25.961460 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:25.961373    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="afbd2f7b-f730-472b-b6ee-f98250881cc3" containerName="istio-proxy"
Nov 02 12:43:25.961711 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:25.961686    4559 cpu_manager.go:414] "containerMap: removing container" podUID="c5a77cc3-ce6a-4a5d-942e-83db434f8efb" containerName="payload-pre-processing"
Nov 02 12:43:25.961825 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:25.961717    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5a77cc3-ce6a-4a5d-942e-83db434f8efb" containerName="payload-pre-processing"
Nov 02 12:43:25.961902 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:25.961888    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5a77cc3-ce6a-4a5d-942e-83db434f8efb" containerName="payload-pre-processing"
Nov 02 12:43:25.961935 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:25.961915    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="ced16952-82be-49e1-a993-b009b0aa9c64" containerName="payload-processing"
Nov 02 12:43:25.961935 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:25.961928    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="afbd2f7b-f730-472b-b6ee-f98250881cc3" containerName="istio-proxy"
Nov 02 12:43:25.961998 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:25.961941    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="afbd2f7b-f730-472b-b6ee-f98250881cc3" containerName="istio-proxy"
Nov 02 12:43:25.967514 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:25.967493    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-cd1386cb-openshift-default-5979d457c7-2msnf"]
Nov 02 12:43:25.967633 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:25.967605    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-cd1386cb-openshift-default-5979d457c7-2msnf"
Nov 02 12:43:25.989587 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:25.989561    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-cd1386cb-openshift-default-dockercfg-rshbn\""
Nov 02 12:43:26.052273 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:26.052226    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/154e3aa3-c520-484e-be0a-74ea246b763b-istiod-ca-cert\") pod \"e2e-mt-cd1386cb-openshift-default-5979d457c7-2msnf\" (UID: \"154e3aa3-c520-484e-be0a-74ea246b763b\") " pod="openshift-ingress/e2e-mt-cd1386cb-openshift-default-5979d457c7-2msnf"
Nov 02 12:43:26.052273 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:26.052276    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fjgx\" (UniqueName: \"kubernetes.io/projected/154e3aa3-c520-484e-be0a-74ea246b763b-kube-api-access-4fjgx\") pod \"e2e-mt-cd1386cb-openshift-default-5979d457c7-2msnf\" (UID: \"154e3aa3-c520-484e-be0a-74ea246b763b\") " pod="openshift-ingress/e2e-mt-cd1386cb-openshift-default-5979d457c7-2msnf"
Nov 02 12:43:26.052525 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:26.052421    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/154e3aa3-c520-484e-be0a-74ea246b763b-workload-socket\") pod \"e2e-mt-cd1386cb-openshift-default-5979d457c7-2msnf\" (UID: \"154e3aa3-c520-484e-be0a-74ea246b763b\") " pod="openshift-ingress/e2e-mt-cd1386cb-openshift-default-5979d457c7-2msnf"
Nov 02 12:43:26.052612 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:26.052590    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/154e3aa3-c520-484e-be0a-74ea246b763b-istio-token\") pod \"e2e-mt-cd1386cb-openshift-default-5979d457c7-2msnf\" (UID: \"154e3aa3-c520-484e-be0a-74ea246b763b\") " pod="openshift-ingress/e2e-mt-cd1386cb-openshift-default-5979d457c7-2msnf"
Nov 02 12:43:26.052696 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:26.052653    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/154e3aa3-c520-484e-be0a-74ea246b763b-credential-socket\") pod \"e2e-mt-cd1386cb-openshift-default-5979d457c7-2msnf\" (UID: \"154e3aa3-c520-484e-be0a-74ea246b763b\") " pod="openshift-ingress/e2e-mt-cd1386cb-openshift-default-5979d457c7-2msnf"
Nov 02 12:43:26.052758 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:26.052740    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/154e3aa3-c520-484e-be0a-74ea246b763b-istio-podinfo\") pod \"e2e-mt-cd1386cb-openshift-default-5979d457c7-2msnf\" (UID: \"154e3aa3-c520-484e-be0a-74ea246b763b\") " pod="openshift-ingress/e2e-mt-cd1386cb-openshift-default-5979d457c7-2msnf"
Nov 02 12:43:26.052829 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:26.052801    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/154e3aa3-c520-484e-be0a-74ea246b763b-istio-envoy\") pod \"e2e-mt-cd1386cb-openshift-default-5979d457c7-2msnf\" (UID: \"154e3aa3-c520-484e-be0a-74ea246b763b\") " pod="openshift-ingress/e2e-mt-cd1386cb-openshift-default-5979d457c7-2msnf"
Nov 02 12:43:26.052918 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:26.052894    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/154e3aa3-c520-484e-be0a-74ea246b763b-istio-data\") pod \"e2e-mt-cd1386cb-openshift-default-5979d457c7-2msnf\" (UID: \"154e3aa3-c520-484e-be0a-74ea246b763b\") " pod="openshift-ingress/e2e-mt-cd1386cb-openshift-default-5979d457c7-2msnf"
Nov 02 12:43:26.052974 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:26.052962    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/154e3aa3-c520-484e-be0a-74ea246b763b-workload-certs\") pod \"e2e-mt-cd1386cb-openshift-default-5979d457c7-2msnf\" (UID: \"154e3aa3-c520-484e-be0a-74ea246b763b\") " pod="openshift-ingress/e2e-mt-cd1386cb-openshift-default-5979d457c7-2msnf"
Nov 02 12:43:26.154734 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:26.154690    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/154e3aa3-c520-484e-be0a-74ea246b763b-workload-certs\") pod \"e2e-mt-cd1386cb-openshift-default-5979d457c7-2msnf\" (UID: \"154e3aa3-c520-484e-be0a-74ea246b763b\") " pod="openshift-ingress/e2e-mt-cd1386cb-openshift-default-5979d457c7-2msnf"
Nov 02 12:43:26.154944 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:26.154786    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/154e3aa3-c520-484e-be0a-74ea246b763b-istiod-ca-cert\") pod \"e2e-mt-cd1386cb-openshift-default-5979d457c7-2msnf\" (UID: \"154e3aa3-c520-484e-be0a-74ea246b763b\") " pod="openshift-ingress/e2e-mt-cd1386cb-openshift-default-5979d457c7-2msnf"
Nov 02 12:43:26.154944 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:26.154812    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fjgx\" (UniqueName: \"kubernetes.io/projected/154e3aa3-c520-484e-be0a-74ea246b763b-kube-api-access-4fjgx\") pod \"e2e-mt-cd1386cb-openshift-default-5979d457c7-2msnf\" (UID: \"154e3aa3-c520-484e-be0a-74ea246b763b\") " pod="openshift-ingress/e2e-mt-cd1386cb-openshift-default-5979d457c7-2msnf"
Nov 02 12:43:26.154944 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:26.154866    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/154e3aa3-c520-484e-be0a-74ea246b763b-workload-socket\") pod \"e2e-mt-cd1386cb-openshift-default-5979d457c7-2msnf\" (UID: \"154e3aa3-c520-484e-be0a-74ea246b763b\") " pod="openshift-ingress/e2e-mt-cd1386cb-openshift-default-5979d457c7-2msnf"
Nov 02 12:43:26.155116 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:26.154958    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/154e3aa3-c520-484e-be0a-74ea246b763b-istio-token\") pod \"e2e-mt-cd1386cb-openshift-default-5979d457c7-2msnf\" (UID: \"154e3aa3-c520-484e-be0a-74ea246b763b\") " pod="openshift-ingress/e2e-mt-cd1386cb-openshift-default-5979d457c7-2msnf"
Nov 02 12:43:26.155116 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:26.154995    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/154e3aa3-c520-484e-be0a-74ea246b763b-credential-socket\") pod \"e2e-mt-cd1386cb-openshift-default-5979d457c7-2msnf\" (UID: \"154e3aa3-c520-484e-be0a-74ea246b763b\") " pod="openshift-ingress/e2e-mt-cd1386cb-openshift-default-5979d457c7-2msnf"
Nov 02 12:43:26.155116 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:26.155034    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/154e3aa3-c520-484e-be0a-74ea246b763b-istio-podinfo\") pod \"e2e-mt-cd1386cb-openshift-default-5979d457c7-2msnf\" (UID: \"154e3aa3-c520-484e-be0a-74ea246b763b\") " pod="openshift-ingress/e2e-mt-cd1386cb-openshift-default-5979d457c7-2msnf"
Nov 02 12:43:26.155116 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:26.155088    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/154e3aa3-c520-484e-be0a-74ea246b763b-workload-certs\") pod \"e2e-mt-cd1386cb-openshift-default-5979d457c7-2msnf\" (UID: \"154e3aa3-c520-484e-be0a-74ea246b763b\") " pod="openshift-ingress/e2e-mt-cd1386cb-openshift-default-5979d457c7-2msnf"
Nov 02 12:43:26.155116 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:26.155101    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/154e3aa3-c520-484e-be0a-74ea246b763b-istio-envoy\") pod \"e2e-mt-cd1386cb-openshift-default-5979d457c7-2msnf\" (UID: \"154e3aa3-c520-484e-be0a-74ea246b763b\") " pod="openshift-ingress/e2e-mt-cd1386cb-openshift-default-5979d457c7-2msnf"
Nov 02 12:43:26.155379 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:26.155168    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/154e3aa3-c520-484e-be0a-74ea246b763b-istio-data\") pod \"e2e-mt-cd1386cb-openshift-default-5979d457c7-2msnf\" (UID: \"154e3aa3-c520-484e-be0a-74ea246b763b\") " pod="openshift-ingress/e2e-mt-cd1386cb-openshift-default-5979d457c7-2msnf"
Nov 02 12:43:26.155379 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:26.155270    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/154e3aa3-c520-484e-be0a-74ea246b763b-workload-socket\") pod \"e2e-mt-cd1386cb-openshift-default-5979d457c7-2msnf\" (UID: \"154e3aa3-c520-484e-be0a-74ea246b763b\") " pod="openshift-ingress/e2e-mt-cd1386cb-openshift-default-5979d457c7-2msnf"
Nov 02 12:43:26.155480 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:26.155388    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/154e3aa3-c520-484e-be0a-74ea246b763b-istio-data\") pod \"e2e-mt-cd1386cb-openshift-default-5979d457c7-2msnf\" (UID: \"154e3aa3-c520-484e-be0a-74ea246b763b\") " pod="openshift-ingress/e2e-mt-cd1386cb-openshift-default-5979d457c7-2msnf"
Nov 02 12:43:26.155530 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:26.155505    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/154e3aa3-c520-484e-be0a-74ea246b763b-credential-socket\") pod \"e2e-mt-cd1386cb-openshift-default-5979d457c7-2msnf\" (UID: \"154e3aa3-c520-484e-be0a-74ea246b763b\") " pod="openshift-ingress/e2e-mt-cd1386cb-openshift-default-5979d457c7-2msnf"
Nov 02 12:43:26.155568 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:26.155553    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/154e3aa3-c520-484e-be0a-74ea246b763b-istiod-ca-cert\") pod \"e2e-mt-cd1386cb-openshift-default-5979d457c7-2msnf\" (UID: \"154e3aa3-c520-484e-be0a-74ea246b763b\") " pod="openshift-ingress/e2e-mt-cd1386cb-openshift-default-5979d457c7-2msnf"
Nov 02 12:43:26.157221 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:26.157197    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/154e3aa3-c520-484e-be0a-74ea246b763b-istio-envoy\") pod \"e2e-mt-cd1386cb-openshift-default-5979d457c7-2msnf\" (UID: \"154e3aa3-c520-484e-be0a-74ea246b763b\") " pod="openshift-ingress/e2e-mt-cd1386cb-openshift-default-5979d457c7-2msnf"
Nov 02 12:43:26.157506 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:26.157486    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/154e3aa3-c520-484e-be0a-74ea246b763b-istio-podinfo\") pod \"e2e-mt-cd1386cb-openshift-default-5979d457c7-2msnf\" (UID: \"154e3aa3-c520-484e-be0a-74ea246b763b\") " pod="openshift-ingress/e2e-mt-cd1386cb-openshift-default-5979d457c7-2msnf"
Nov 02 12:43:26.178934 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:26.178905    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/154e3aa3-c520-484e-be0a-74ea246b763b-istio-token\") pod \"e2e-mt-cd1386cb-openshift-default-5979d457c7-2msnf\" (UID: \"154e3aa3-c520-484e-be0a-74ea246b763b\") " pod="openshift-ingress/e2e-mt-cd1386cb-openshift-default-5979d457c7-2msnf"
Nov 02 12:43:26.179033 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:26.179004    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fjgx\" (UniqueName: \"kubernetes.io/projected/154e3aa3-c520-484e-be0a-74ea246b763b-kube-api-access-4fjgx\") pod \"e2e-mt-cd1386cb-openshift-default-5979d457c7-2msnf\" (UID: \"154e3aa3-c520-484e-be0a-74ea246b763b\") " pod="openshift-ingress/e2e-mt-cd1386cb-openshift-default-5979d457c7-2msnf"
Nov 02 12:43:26.282668 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:26.282576    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-cd1386cb-openshift-default-5979d457c7-2msnf"
Nov 02 12:43:26.677634 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:43:26.677590    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod154e3aa3_c520_484e_be0a_74ea246b763b.slice/crio-ea7df6d548f1da643bce894a5d550aa62179997b66352f99e7408c643c700c42 WatchSource:0}: Error finding container ea7df6d548f1da643bce894a5d550aa62179997b66352f99e7408c643c700c42: Status 404 returned error can't find the container with id ea7df6d548f1da643bce894a5d550aa62179997b66352f99e7408c643c700c42
Nov 02 12:43:26.678927 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:26.678905    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-cd1386cb-openshift-default-5979d457c7-2msnf"]
Nov 02 12:43:26.680258 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:26.680227    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:43:26.680329 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:26.680290    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:43:26.680329 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:26.680321    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:43:27.664830 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:27.664790    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-cd1386cb-openshift-default-5979d457c7-2msnf" event={"ID":"154e3aa3-c520-484e-be0a-74ea246b763b","Type":"ContainerStarted","Data":"4c3090e0a62f5efd133b12cd690636736ae43a8dc6e3a0ffd00a5b92159629b9"}
Nov 02 12:43:27.665347 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:27.664855    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-cd1386cb-openshift-default-5979d457c7-2msnf" event={"ID":"154e3aa3-c520-484e-be0a-74ea246b763b","Type":"ContainerStarted","Data":"ea7df6d548f1da643bce894a5d550aa62179997b66352f99e7408c643c700c42"}
Nov 02 12:43:27.764963 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:27.764904    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-cd1386cb-openshift-default-5979d457c7-2msnf" podStartSLOduration=2.764889571 podStartE2EDuration="2.764889571s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:43:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:43:27.761000811 +0000 UTC m=+1004.016399907" watchObservedRunningTime="2026-11-02 12:43:27.764889571 +0000 UTC m=+1004.020288666"
Nov 02 12:43:28.031664 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:28.031143    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7004-0d46b6"]
Nov 02 12:43:28.042829 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:28.042793    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7004-0d46b6"
Nov 02 12:43:28.053460 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:28.053353    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7004-0d46b6"]
Nov 02 12:43:28.064804 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:28.064775    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-hfcns\""
Nov 02 12:43:28.154603 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:28.154554    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9wqm\" (UniqueName: \"kubernetes.io/projected/50ca0973-9e27-4042-9300-4ad78edae135-kube-api-access-g9wqm\") pod \"test-curl-7004-0d46b6\" (UID: \"50ca0973-9e27-4042-9300-4ad78edae135\") " pod="opendatahub/test-curl-7004-0d46b6"
Nov 02 12:43:28.256045 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:28.255987    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9wqm\" (UniqueName: \"kubernetes.io/projected/50ca0973-9e27-4042-9300-4ad78edae135-kube-api-access-g9wqm\") pod \"test-curl-7004-0d46b6\" (UID: \"50ca0973-9e27-4042-9300-4ad78edae135\") " pod="opendatahub/test-curl-7004-0d46b6"
Nov 02 12:43:28.280154 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:28.280124    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9wqm\" (UniqueName: \"kubernetes.io/projected/50ca0973-9e27-4042-9300-4ad78edae135-kube-api-access-g9wqm\") pod \"test-curl-7004-0d46b6\" (UID: \"50ca0973-9e27-4042-9300-4ad78edae135\") " pod="opendatahub/test-curl-7004-0d46b6"
Nov 02 12:43:28.308945 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:28.308855    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-cd1386cb-openshift-default-5979d457c7-2msnf"
Nov 02 12:43:28.314192 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:28.314159    4559 patch_prober.go:29] interesting pod/e2e-mt-cd1386cb-openshift-default-5979d457c7-2msnf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Nov 02 12:43:28.314463 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:28.314418    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-cd1386cb-openshift-default-5979d457c7-2msnf" podUID="154e3aa3-c520-484e-be0a-74ea246b763b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Nov 02 12:43:28.366464 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:28.366431    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7004-0d46b6"
Nov 02 12:43:29.135811 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:29.135732    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-c79cde-openshift-default-54dbf475f7-g56xt"]
Nov 02 12:43:29.143909 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:29.142324    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-c79cde-openshift-default-54dbf475f7-g56xt"
Nov 02 12:43:29.165789 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:29.162881    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-c79cde-openshift-default-54dbf475f7-g56xt"]
Nov 02 12:43:29.167553 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:29.167508    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-c79cde-openshift-default-dockercfg-6vx4k\""
Nov 02 12:43:29.186127 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:29.185083    4559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 12:43:29.218666 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:29.217958    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7004-0d46b6"]
Nov 02 12:43:29.220662 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:29.219655    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea9a3b60-d2ac-4837-b979-2ea31f79cee0-istio-envoy\") pod \"e2e-api-b-c79cde-openshift-default-54dbf475f7-g56xt\" (UID: \"ea9a3b60-d2ac-4837-b979-2ea31f79cee0\") " pod="openshift-ingress/e2e-api-b-c79cde-openshift-default-54dbf475f7-g56xt"
Nov 02 12:43:29.220662 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:29.219712    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea9a3b60-d2ac-4837-b979-2ea31f79cee0-istio-token\") pod \"e2e-api-b-c79cde-openshift-default-54dbf475f7-g56xt\" (UID: \"ea9a3b60-d2ac-4837-b979-2ea31f79cee0\") " pod="openshift-ingress/e2e-api-b-c79cde-openshift-default-54dbf475f7-g56xt"
Nov 02 12:43:29.220662 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:29.219762    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea9a3b60-d2ac-4837-b979-2ea31f79cee0-workload-socket\") pod \"e2e-api-b-c79cde-openshift-default-54dbf475f7-g56xt\" (UID: \"ea9a3b60-d2ac-4837-b979-2ea31f79cee0\") " pod="openshift-ingress/e2e-api-b-c79cde-openshift-default-54dbf475f7-g56xt"
Nov 02 12:43:29.221047 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:29.221026    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea9a3b60-d2ac-4837-b979-2ea31f79cee0-workload-certs\") pod \"e2e-api-b-c79cde-openshift-default-54dbf475f7-g56xt\" (UID: \"ea9a3b60-d2ac-4837-b979-2ea31f79cee0\") " pod="openshift-ingress/e2e-api-b-c79cde-openshift-default-54dbf475f7-g56xt"
Nov 02 12:43:29.221226 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:29.221213    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea9a3b60-d2ac-4837-b979-2ea31f79cee0-istiod-ca-cert\") pod \"e2e-api-b-c79cde-openshift-default-54dbf475f7-g56xt\" (UID: \"ea9a3b60-d2ac-4837-b979-2ea31f79cee0\") " pod="openshift-ingress/e2e-api-b-c79cde-openshift-default-54dbf475f7-g56xt"
Nov 02 12:43:29.221322 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:29.221310    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea9a3b60-d2ac-4837-b979-2ea31f79cee0-credential-socket\") pod \"e2e-api-b-c79cde-openshift-default-54dbf475f7-g56xt\" (UID: \"ea9a3b60-d2ac-4837-b979-2ea31f79cee0\") " pod="openshift-ingress/e2e-api-b-c79cde-openshift-default-54dbf475f7-g56xt"
Nov 02 12:43:29.221401 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:29.221391    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea9a3b60-d2ac-4837-b979-2ea31f79cee0-istio-data\") pod \"e2e-api-b-c79cde-openshift-default-54dbf475f7-g56xt\" (UID: \"ea9a3b60-d2ac-4837-b979-2ea31f79cee0\") " pod="openshift-ingress/e2e-api-b-c79cde-openshift-default-54dbf475f7-g56xt"
Nov 02 12:43:29.221468 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:29.221460    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2s6lw\" (UniqueName: \"kubernetes.io/projected/ea9a3b60-d2ac-4837-b979-2ea31f79cee0-kube-api-access-2s6lw\") pod \"e2e-api-b-c79cde-openshift-default-54dbf475f7-g56xt\" (UID: \"ea9a3b60-d2ac-4837-b979-2ea31f79cee0\") " pod="openshift-ingress/e2e-api-b-c79cde-openshift-default-54dbf475f7-g56xt"
Nov 02 12:43:29.221601 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:29.221584    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea9a3b60-d2ac-4837-b979-2ea31f79cee0-istio-podinfo\") pod \"e2e-api-b-c79cde-openshift-default-54dbf475f7-g56xt\" (UID: \"ea9a3b60-d2ac-4837-b979-2ea31f79cee0\") " pod="openshift-ingress/e2e-api-b-c79cde-openshift-default-54dbf475f7-g56xt"
Nov 02 12:43:29.355385 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:29.355351    4559 patch_prober.go:29] interesting pod/e2e-mt-cd1386cb-openshift-default-5979d457c7-2msnf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Nov 02 12:43:29.362612 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:29.355613    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-cd1386cb-openshift-default-5979d457c7-2msnf" podUID="154e3aa3-c520-484e-be0a-74ea246b763b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Nov 02 12:43:29.362869 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:29.361896    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea9a3b60-d2ac-4837-b979-2ea31f79cee0-istio-podinfo\") pod \"e2e-api-b-c79cde-openshift-default-54dbf475f7-g56xt\" (UID: \"ea9a3b60-d2ac-4837-b979-2ea31f79cee0\") " pod="openshift-ingress/e2e-api-b-c79cde-openshift-default-54dbf475f7-g56xt"
Nov 02 12:43:29.363065 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:29.363047    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea9a3b60-d2ac-4837-b979-2ea31f79cee0-istio-envoy\") pod \"e2e-api-b-c79cde-openshift-default-54dbf475f7-g56xt\" (UID: \"ea9a3b60-d2ac-4837-b979-2ea31f79cee0\") " pod="openshift-ingress/e2e-api-b-c79cde-openshift-default-54dbf475f7-g56xt"
Nov 02 12:43:29.363211 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:29.363194    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea9a3b60-d2ac-4837-b979-2ea31f79cee0-istio-token\") pod \"e2e-api-b-c79cde-openshift-default-54dbf475f7-g56xt\" (UID: \"ea9a3b60-d2ac-4837-b979-2ea31f79cee0\") " pod="openshift-ingress/e2e-api-b-c79cde-openshift-default-54dbf475f7-g56xt"
Nov 02 12:43:29.366036 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:29.366013    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea9a3b60-d2ac-4837-b979-2ea31f79cee0-workload-socket\") pod \"e2e-api-b-c79cde-openshift-default-54dbf475f7-g56xt\" (UID: \"ea9a3b60-d2ac-4837-b979-2ea31f79cee0\") " pod="openshift-ingress/e2e-api-b-c79cde-openshift-default-54dbf475f7-g56xt"
Nov 02 12:43:29.366222 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:29.366205    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea9a3b60-d2ac-4837-b979-2ea31f79cee0-workload-certs\") pod \"e2e-api-b-c79cde-openshift-default-54dbf475f7-g56xt\" (UID: \"ea9a3b60-d2ac-4837-b979-2ea31f79cee0\") " pod="openshift-ingress/e2e-api-b-c79cde-openshift-default-54dbf475f7-g56xt"
Nov 02 12:43:29.366457 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:29.366417    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea9a3b60-d2ac-4837-b979-2ea31f79cee0-istiod-ca-cert\") pod \"e2e-api-b-c79cde-openshift-default-54dbf475f7-g56xt\" (UID: \"ea9a3b60-d2ac-4837-b979-2ea31f79cee0\") " pod="openshift-ingress/e2e-api-b-c79cde-openshift-default-54dbf475f7-g56xt"
Nov 02 12:43:29.366626 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:29.366608    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea9a3b60-d2ac-4837-b979-2ea31f79cee0-credential-socket\") pod \"e2e-api-b-c79cde-openshift-default-54dbf475f7-g56xt\" (UID: \"ea9a3b60-d2ac-4837-b979-2ea31f79cee0\") " pod="openshift-ingress/e2e-api-b-c79cde-openshift-default-54dbf475f7-g56xt"
Nov 02 12:43:29.366793 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:29.366776    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea9a3b60-d2ac-4837-b979-2ea31f79cee0-istio-data\") pod \"e2e-api-b-c79cde-openshift-default-54dbf475f7-g56xt\" (UID: \"ea9a3b60-d2ac-4837-b979-2ea31f79cee0\") " pod="openshift-ingress/e2e-api-b-c79cde-openshift-default-54dbf475f7-g56xt"
Nov 02 12:43:29.366934 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:29.366905    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2s6lw\" (UniqueName: \"kubernetes.io/projected/ea9a3b60-d2ac-4837-b979-2ea31f79cee0-kube-api-access-2s6lw\") pod \"e2e-api-b-c79cde-openshift-default-54dbf475f7-g56xt\" (UID: \"ea9a3b60-d2ac-4837-b979-2ea31f79cee0\") " pod="openshift-ingress/e2e-api-b-c79cde-openshift-default-54dbf475f7-g56xt"
Nov 02 12:43:29.367679 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:29.367632    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea9a3b60-d2ac-4837-b979-2ea31f79cee0-workload-certs\") pod \"e2e-api-b-c79cde-openshift-default-54dbf475f7-g56xt\" (UID: \"ea9a3b60-d2ac-4837-b979-2ea31f79cee0\") " pod="openshift-ingress/e2e-api-b-c79cde-openshift-default-54dbf475f7-g56xt"
Nov 02 12:43:29.367805 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:29.367782    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea9a3b60-d2ac-4837-b979-2ea31f79cee0-credential-socket\") pod \"e2e-api-b-c79cde-openshift-default-54dbf475f7-g56xt\" (UID: \"ea9a3b60-d2ac-4837-b979-2ea31f79cee0\") " pod="openshift-ingress/e2e-api-b-c79cde-openshift-default-54dbf475f7-g56xt"
Nov 02 12:43:29.367875 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:29.366446    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea9a3b60-d2ac-4837-b979-2ea31f79cee0-workload-socket\") pod \"e2e-api-b-c79cde-openshift-default-54dbf475f7-g56xt\" (UID: \"ea9a3b60-d2ac-4837-b979-2ea31f79cee0\") " pod="openshift-ingress/e2e-api-b-c79cde-openshift-default-54dbf475f7-g56xt"
Nov 02 12:43:29.368108 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:29.368083    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea9a3b60-d2ac-4837-b979-2ea31f79cee0-istio-data\") pod \"e2e-api-b-c79cde-openshift-default-54dbf475f7-g56xt\" (UID: \"ea9a3b60-d2ac-4837-b979-2ea31f79cee0\") " pod="openshift-ingress/e2e-api-b-c79cde-openshift-default-54dbf475f7-g56xt"
Nov 02 12:43:29.369311 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:29.369116    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea9a3b60-d2ac-4837-b979-2ea31f79cee0-istio-podinfo\") pod \"e2e-api-b-c79cde-openshift-default-54dbf475f7-g56xt\" (UID: \"ea9a3b60-d2ac-4837-b979-2ea31f79cee0\") " pod="openshift-ingress/e2e-api-b-c79cde-openshift-default-54dbf475f7-g56xt"
Nov 02 12:43:29.431503 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:29.371403    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea9a3b60-d2ac-4837-b979-2ea31f79cee0-istiod-ca-cert\") pod \"e2e-api-b-c79cde-openshift-default-54dbf475f7-g56xt\" (UID: \"ea9a3b60-d2ac-4837-b979-2ea31f79cee0\") " pod="openshift-ingress/e2e-api-b-c79cde-openshift-default-54dbf475f7-g56xt"
Nov 02 12:43:29.455667 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:29.436384    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea9a3b60-d2ac-4837-b979-2ea31f79cee0-istio-envoy\") pod \"e2e-api-b-c79cde-openshift-default-54dbf475f7-g56xt\" (UID: \"ea9a3b60-d2ac-4837-b979-2ea31f79cee0\") " pod="openshift-ingress/e2e-api-b-c79cde-openshift-default-54dbf475f7-g56xt"
Nov 02 12:43:29.474327 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:29.474273    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea9a3b60-d2ac-4837-b979-2ea31f79cee0-istio-token\") pod \"e2e-api-b-c79cde-openshift-default-54dbf475f7-g56xt\" (UID: \"ea9a3b60-d2ac-4837-b979-2ea31f79cee0\") " pod="openshift-ingress/e2e-api-b-c79cde-openshift-default-54dbf475f7-g56xt"
Nov 02 12:43:29.488344 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:29.488305    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2s6lw\" (UniqueName: \"kubernetes.io/projected/ea9a3b60-d2ac-4837-b979-2ea31f79cee0-kube-api-access-2s6lw\") pod \"e2e-api-b-c79cde-openshift-default-54dbf475f7-g56xt\" (UID: \"ea9a3b60-d2ac-4837-b979-2ea31f79cee0\") " pod="openshift-ingress/e2e-api-b-c79cde-openshift-default-54dbf475f7-g56xt"
Nov 02 12:43:29.488620 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:29.488599    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-c79cde-openshift-default-54dbf475f7-g56xt"
Nov 02 12:43:29.921540 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:29.921508    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7004-0d46b6" event={"ID":"50ca0973-9e27-4042-9300-4ad78edae135","Type":"ContainerStarted","Data":"a050f45b619229f71017bb816094214fcff7ac095a2384a3bbc1d881b0c70cba"}
Nov 02 12:43:30.015066 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:30.014832    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-c79cde-openshift-default-54dbf475f7-g56xt"]
Nov 02 12:43:30.015066 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:43:30.015037    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea9a3b60_d2ac_4837_b979_2ea31f79cee0.slice/crio-d2d6c052637aac0649d8d12cddbe1f17bb9ea297fdf961faa0fd9f10e582ebd4 WatchSource:0}: Error finding container d2d6c052637aac0649d8d12cddbe1f17bb9ea297fdf961faa0fd9f10e582ebd4: Status 404 returned error can't find the container with id d2d6c052637aac0649d8d12cddbe1f17bb9ea297fdf961faa0fd9f10e582ebd4
Nov 02 12:43:30.018318 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:30.018275    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:43:30.018428 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:30.018361    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:43:30.018428 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:30.018409    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:43:30.295714 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:30.291029    4559 patch_prober.go:29] interesting pod/e2e-mt-cd1386cb-openshift-default-5979d457c7-2msnf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Nov 02 12:43:30.295714 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:30.291087    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-cd1386cb-openshift-default-5979d457c7-2msnf" podUID="154e3aa3-c520-484e-be0a-74ea246b763b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Nov 02 12:43:30.426351 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:30.424772    4559 patch_prober.go:29] interesting pod/e2e-api-b-96339b-openshift-default-857fd4b79-52nqm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 12:43:30.426351 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:30.424836    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-96339b-openshift-default-857fd4b79-52nqm" podUID="e51ff2f8-753e-4977-a523-90bca8a4ec2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:43:30.933170 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:30.933130    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c79cde-openshift-default-54dbf475f7-g56xt" event={"ID":"ea9a3b60-d2ac-4837-b979-2ea31f79cee0","Type":"ContainerStarted","Data":"6618ad71a8db78e22aa9f5dfcbf9c761ac8e0f287f7d1924fa12329ce2ffb07f"}
Nov 02 12:43:30.933388 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:30.933183    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c79cde-openshift-default-54dbf475f7-g56xt" event={"ID":"ea9a3b60-d2ac-4837-b979-2ea31f79cee0","Type":"ContainerStarted","Data":"d2d6c052637aac0649d8d12cddbe1f17bb9ea297fdf961faa0fd9f10e582ebd4"}
Nov 02 12:43:31.043367 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:31.043315    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-c79cde-openshift-default-54dbf475f7-g56xt" podStartSLOduration=2.043299178 podStartE2EDuration="2.043299178s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:43:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:43:31.008175757 +0000 UTC m=+1007.263574853" watchObservedRunningTime="2026-11-02 12:43:31.043299178 +0000 UTC m=+1007.298698275"
Nov 02 12:43:31.170787 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:31.170742    4559 patch_prober.go:29] interesting pod/e2e-api-a-9f7f01-openshift-default-64d8f9568c-pw8tx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 12:43:31.171386 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:31.171339    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-9f7f01-openshift-default-64d8f9568c-pw8tx" podUID="26b966b3-2043-474a-95aa-3037982c3fc4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:43:31.299714 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:31.295057    4559 patch_prober.go:29] interesting pod/e2e-mt-cd1386cb-openshift-default-5979d457c7-2msnf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Nov 02 12:43:31.299714 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:31.295125    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-cd1386cb-openshift-default-5979d457c7-2msnf" podUID="154e3aa3-c520-484e-be0a-74ea246b763b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Nov 02 12:43:31.491754 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:31.489949    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-c79cde-openshift-default-54dbf475f7-g56xt"
Nov 02 12:43:31.507016 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:31.506671    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-c79cde-openshift-default-54dbf475f7-g56xt"
Nov 02 12:43:32.006689 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:31.999778    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-c79cde-openshift-default-54dbf475f7-g56xt"
Nov 02 12:43:32.028203 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:32.027898    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-c79cde-openshift-default-54dbf475f7-g56xt"
Nov 02 12:43:32.308987 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:32.308412    4559 patch_prober.go:29] interesting pod/e2e-mt-cd1386cb-openshift-default-5979d457c7-2msnf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Nov 02 12:43:32.308987 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:32.308477    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-cd1386cb-openshift-default-5979d457c7-2msnf" podUID="154e3aa3-c520-484e-be0a-74ea246b763b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Nov 02 12:43:33.307073 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:33.307033    4559 patch_prober.go:29] interesting pod/e2e-mt-cd1386cb-openshift-default-5979d457c7-2msnf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Nov 02 12:43:33.307257 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:33.307097    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-cd1386cb-openshift-default-5979d457c7-2msnf" podUID="154e3aa3-c520-484e-be0a-74ea246b763b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Nov 02 12:43:34.288656 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:34.288614    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-cd1386cb-openshift-default-5979d457c7-2msnf"
Nov 02 12:43:34.289107 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:34.289017    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-cd1386cb-openshift-default-5979d457c7-2msnf"
Nov 02 12:43:34.289940 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:34.289922    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-cd1386cb-openshift-default-5979d457c7-2msnf"
Nov 02 12:43:35.045081 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:35.045040    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7004-0d46b6" event={"ID":"50ca0973-9e27-4042-9300-4ad78edae135","Type":"ContainerStarted","Data":"0a131e581c036d1dad8c689ea80b32fa696f06cb2cc240dabb6bfb775e8dc727"}
Nov 02 12:43:35.087528 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:35.087482    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7004-0d46b6" podStartSLOduration=3.454486803 podStartE2EDuration="8.087468286s" totalImagesPullingTime="4.632981483s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:43:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:43:35.084312657 +0000 UTC m=+1011.339711750" watchObservedRunningTime="2026-11-02 12:43:35.087468286 +0000 UTC m=+1011.342867381"
Nov 02 12:43:35.320483 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:35.320396    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-c79cde-76d5b754d7-2vhqc"]
Nov 02 12:43:35.343149 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:35.343113    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-c79cde-789d76c79-cnjxb"]
Nov 02 12:43:35.343444 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:35.343419    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c79cde-76d5b754d7-2vhqc"
Nov 02 12:43:35.353689 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:35.353005    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-c79cde-789d76c79-cnjxb"
Nov 02 12:43:35.353689 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:35.353528    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-c79cde-76d5b754d7-2vhqc"]
Nov 02 12:43:35.363170 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:35.361546    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-c79cde\""
Nov 02 12:43:35.363170 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:35.362626    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bq2ff\" (UniqueName: \"kubernetes.io/projected/9077a191-58b9-4dbd-aa96-b87eaa5398bf-kube-api-access-bq2ff\") pod \"payload-pre-processing-e2e-api-b-c79cde-76d5b754d7-2vhqc\" (UID: \"9077a191-58b9-4dbd-aa96-b87eaa5398bf\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-c79cde-76d5b754d7-2vhqc"
Nov 02 12:43:35.363170 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:35.362687    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/730c9cc3-ba43-4ed4-8769-13999c58cf13-plugins-config-volume\") pod \"payload-processing-e2e-api-b-c79cde-789d76c79-cnjxb\" (UID: \"730c9cc3-ba43-4ed4-8769-13999c58cf13\") " pod="openshift-ingress/payload-processing-e2e-api-b-c79cde-789d76c79-cnjxb"
Nov 02 12:43:35.363170 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:35.362797    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9077a191-58b9-4dbd-aa96-b87eaa5398bf-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-c79cde-76d5b754d7-2vhqc\" (UID: \"9077a191-58b9-4dbd-aa96-b87eaa5398bf\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-c79cde-76d5b754d7-2vhqc"
Nov 02 12:43:35.363170 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:35.362856    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4zrc\" (UniqueName: \"kubernetes.io/projected/730c9cc3-ba43-4ed4-8769-13999c58cf13-kube-api-access-l4zrc\") pod \"payload-processing-e2e-api-b-c79cde-789d76c79-cnjxb\" (UID: \"730c9cc3-ba43-4ed4-8769-13999c58cf13\") " pod="openshift-ingress/payload-processing-e2e-api-b-c79cde-789d76c79-cnjxb"
Nov 02 12:43:35.376795 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:35.376770    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-c79cde-dockercfg-cb4gf\""
Nov 02 12:43:35.381185 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:35.381159    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-c79cde-789d76c79-cnjxb"]
Nov 02 12:43:35.463847 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:35.463817    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bq2ff\" (UniqueName: \"kubernetes.io/projected/9077a191-58b9-4dbd-aa96-b87eaa5398bf-kube-api-access-bq2ff\") pod \"payload-pre-processing-e2e-api-b-c79cde-76d5b754d7-2vhqc\" (UID: \"9077a191-58b9-4dbd-aa96-b87eaa5398bf\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-c79cde-76d5b754d7-2vhqc"
Nov 02 12:43:35.464013 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:35.463850    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/730c9cc3-ba43-4ed4-8769-13999c58cf13-plugins-config-volume\") pod \"payload-processing-e2e-api-b-c79cde-789d76c79-cnjxb\" (UID: \"730c9cc3-ba43-4ed4-8769-13999c58cf13\") " pod="openshift-ingress/payload-processing-e2e-api-b-c79cde-789d76c79-cnjxb"
Nov 02 12:43:35.464013 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:35.463917    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9077a191-58b9-4dbd-aa96-b87eaa5398bf-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-c79cde-76d5b754d7-2vhqc\" (UID: \"9077a191-58b9-4dbd-aa96-b87eaa5398bf\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-c79cde-76d5b754d7-2vhqc"
Nov 02 12:43:35.464013 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:35.463951    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4zrc\" (UniqueName: \"kubernetes.io/projected/730c9cc3-ba43-4ed4-8769-13999c58cf13-kube-api-access-l4zrc\") pod \"payload-processing-e2e-api-b-c79cde-789d76c79-cnjxb\" (UID: \"730c9cc3-ba43-4ed4-8769-13999c58cf13\") " pod="openshift-ingress/payload-processing-e2e-api-b-c79cde-789d76c79-cnjxb"
Nov 02 12:43:35.464624 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:35.464595    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/730c9cc3-ba43-4ed4-8769-13999c58cf13-plugins-config-volume\") pod \"payload-processing-e2e-api-b-c79cde-789d76c79-cnjxb\" (UID: \"730c9cc3-ba43-4ed4-8769-13999c58cf13\") " pod="openshift-ingress/payload-processing-e2e-api-b-c79cde-789d76c79-cnjxb"
Nov 02 12:43:35.464806 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:35.464784    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9077a191-58b9-4dbd-aa96-b87eaa5398bf-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-c79cde-76d5b754d7-2vhqc\" (UID: \"9077a191-58b9-4dbd-aa96-b87eaa5398bf\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-c79cde-76d5b754d7-2vhqc"
Nov 02 12:43:35.565961 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:35.565177    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4zrc\" (UniqueName: \"kubernetes.io/projected/730c9cc3-ba43-4ed4-8769-13999c58cf13-kube-api-access-l4zrc\") pod \"payload-processing-e2e-api-b-c79cde-789d76c79-cnjxb\" (UID: \"730c9cc3-ba43-4ed4-8769-13999c58cf13\") " pod="openshift-ingress/payload-processing-e2e-api-b-c79cde-789d76c79-cnjxb"
Nov 02 12:43:35.580615 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:35.580323    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bq2ff\" (UniqueName: \"kubernetes.io/projected/9077a191-58b9-4dbd-aa96-b87eaa5398bf-kube-api-access-bq2ff\") pod \"payload-pre-processing-e2e-api-b-c79cde-76d5b754d7-2vhqc\" (UID: \"9077a191-58b9-4dbd-aa96-b87eaa5398bf\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-c79cde-76d5b754d7-2vhqc"
Nov 02 12:43:35.689607 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:35.689242    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c79cde-76d5b754d7-2vhqc"
Nov 02 12:43:35.736380 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:35.730985    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-c79cde-789d76c79-cnjxb"
Nov 02 12:43:36.518575 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:36.518486    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7004-0d46b6"]
Nov 02 12:43:36.522295 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:36.522268    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7004-0d46b6"]
Nov 02 12:43:36.586045 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:43:36.586018    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9077a191_58b9_4dbd_aa96_b87eaa5398bf.slice/crio-5f3207dd88374acce49f9aee0787d843ee147d6b10cf8bbe17792ca869e90864 WatchSource:0}: Error finding container 5f3207dd88374acce49f9aee0787d843ee147d6b10cf8bbe17792ca869e90864: Status 404 returned error can't find the container with id 5f3207dd88374acce49f9aee0787d843ee147d6b10cf8bbe17792ca869e90864
Nov 02 12:43:36.587306 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:36.587283    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-c79cde-76d5b754d7-2vhqc"]
Nov 02 12:43:36.612217 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:36.612189    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-c79cde-789d76c79-cnjxb"]
Nov 02 12:43:37.164162 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:37.164132    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-6a3af1-openshift-default-666bcf8bbc-2hjz9"]
Nov 02 12:43:37.164826 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:37.164811    4559 cpu_manager.go:414] "containerMap: removing container" podUID="50ca0973-9e27-4042-9300-4ad78edae135" containerName="test-curl-7004-0d46b6"
Nov 02 12:43:37.164890 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:37.164827    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50ca0973-9e27-4042-9300-4ad78edae135" containerName="test-curl-7004-0d46b6"
Nov 02 12:43:37.164942 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:37.164933    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="50ca0973-9e27-4042-9300-4ad78edae135" containerName="test-curl-7004-0d46b6"
Nov 02 12:43:37.168296 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:37.168276    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-6a3af1-openshift-default-666bcf8bbc-2hjz9"
Nov 02 12:43:37.181025 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:37.180993    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c79cde-76d5b754d7-2vhqc" event={"ID":"9077a191-58b9-4dbd-aa96-b87eaa5398bf","Type":"ContainerStarted","Data":"33300948ccd06393ac439495af4550576623e716490d238d50fc5df3dc78af42"}
Nov 02 12:43:37.181150 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:37.181049    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c79cde-76d5b754d7-2vhqc" event={"ID":"9077a191-58b9-4dbd-aa96-b87eaa5398bf","Type":"ContainerStarted","Data":"5f3207dd88374acce49f9aee0787d843ee147d6b10cf8bbe17792ca869e90864"}
Nov 02 12:43:37.181778 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:37.181504    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c79cde-76d5b754d7-2vhqc"
Nov 02 12:43:37.183749 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:37.183724    4559 generic.go:356] "Generic (PLEG): container finished" podID="26b966b3-2043-474a-95aa-3037982c3fc4" containerID="67baa54012d28b2eded48236c9ab1b20b43f10ba74555771f760756c4dba1ec7" exitCode=0
Nov 02 12:43:37.183850 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:37.183756    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-9f7f01-openshift-default-64d8f9568c-pw8tx" event={"ID":"26b966b3-2043-474a-95aa-3037982c3fc4","Type":"ContainerDied","Data":"67baa54012d28b2eded48236c9ab1b20b43f10ba74555771f760756c4dba1ec7"}
Nov 02 12:43:37.184202 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:37.184178    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-6a3af1-openshift-default-dockercfg-2lw5r\""
Nov 02 12:43:37.184331 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:37.184239    4559 scope.go:122] "RemoveContainer" containerID="67baa54012d28b2eded48236c9ab1b20b43f10ba74555771f760756c4dba1ec7"
Nov 02 12:43:37.186177 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:37.186142    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:43:37.186280 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:37.186237    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:43:37.186348 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:37.186292    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:43:37.188825 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:37.188798    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-c79cde-789d76c79-cnjxb" event={"ID":"730c9cc3-ba43-4ed4-8769-13999c58cf13","Type":"ContainerStarted","Data":"3f6b2633003d3ce771ffc57324358907b056216f6282579d876e71b663dedcda"}
Nov 02 12:43:37.188934 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:37.188853    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-c79cde-789d76c79-cnjxb" event={"ID":"730c9cc3-ba43-4ed4-8769-13999c58cf13","Type":"ContainerStarted","Data":"0ba944fc5562971da3ac0cf0ded0f334fa31d5b8511e1d6695b74446e8c6acfd"}
Nov 02 12:43:37.189534 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:37.189519    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-c79cde-789d76c79-cnjxb"
Nov 02 12:43:37.195901 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:37.191013    4559 generic.go:356] "Generic (PLEG): container finished" podID="e51ff2f8-753e-4977-a523-90bca8a4ec2a" containerID="2d9ba826fa413fa53e83afa46516baa134cdb90eadb757ff07b8e883f8540ca5" exitCode=0
Nov 02 12:43:37.195901 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:37.191217    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7004-0d46b6" podUID="50ca0973-9e27-4042-9300-4ad78edae135" containerName="test-curl-7004-0d46b6" containerID="cri-o://0a131e581c036d1dad8c689ea80b32fa696f06cb2cc240dabb6bfb775e8dc727" gracePeriod=2
Nov 02 12:43:37.195901 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:37.191296    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-96339b-openshift-default-857fd4b79-52nqm" event={"ID":"e51ff2f8-753e-4977-a523-90bca8a4ec2a","Type":"ContainerDied","Data":"2d9ba826fa413fa53e83afa46516baa134cdb90eadb757ff07b8e883f8540ca5"}
Nov 02 12:43:37.195901 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:37.191569    4559 scope.go:122] "RemoveContainer" containerID="2d9ba826fa413fa53e83afa46516baa134cdb90eadb757ff07b8e883f8540ca5"
Nov 02 12:43:37.199333 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:37.199300    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:43:37.199442 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:37.199368    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:43:37.199442 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:37.199404    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:43:37.207154 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:37.207128    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-6a3af1-openshift-default-666bcf8bbc-2hjz9"]
Nov 02 12:43:37.295933 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:37.295900    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0992908f-5ad3-4600-b8d4-f26d07958ee2-istiod-ca-cert\") pod \"e2e-api-b-6a3af1-openshift-default-666bcf8bbc-2hjz9\" (UID: \"0992908f-5ad3-4600-b8d4-f26d07958ee2\") " pod="openshift-ingress/e2e-api-b-6a3af1-openshift-default-666bcf8bbc-2hjz9"
Nov 02 12:43:37.296106 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:37.295949    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0992908f-5ad3-4600-b8d4-f26d07958ee2-istio-envoy\") pod \"e2e-api-b-6a3af1-openshift-default-666bcf8bbc-2hjz9\" (UID: \"0992908f-5ad3-4600-b8d4-f26d07958ee2\") " pod="openshift-ingress/e2e-api-b-6a3af1-openshift-default-666bcf8bbc-2hjz9"
Nov 02 12:43:37.296184 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:37.296152    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0992908f-5ad3-4600-b8d4-f26d07958ee2-istio-data\") pod \"e2e-api-b-6a3af1-openshift-default-666bcf8bbc-2hjz9\" (UID: \"0992908f-5ad3-4600-b8d4-f26d07958ee2\") " pod="openshift-ingress/e2e-api-b-6a3af1-openshift-default-666bcf8bbc-2hjz9"
Nov 02 12:43:37.296580 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:37.296549    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0992908f-5ad3-4600-b8d4-f26d07958ee2-workload-certs\") pod \"e2e-api-b-6a3af1-openshift-default-666bcf8bbc-2hjz9\" (UID: \"0992908f-5ad3-4600-b8d4-f26d07958ee2\") " pod="openshift-ingress/e2e-api-b-6a3af1-openshift-default-666bcf8bbc-2hjz9"
Nov 02 12:43:37.296726 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:37.296608    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0992908f-5ad3-4600-b8d4-f26d07958ee2-workload-socket\") pod \"e2e-api-b-6a3af1-openshift-default-666bcf8bbc-2hjz9\" (UID: \"0992908f-5ad3-4600-b8d4-f26d07958ee2\") " pod="openshift-ingress/e2e-api-b-6a3af1-openshift-default-666bcf8bbc-2hjz9"
Nov 02 12:43:37.296781 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:37.296731    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjdcj\" (UniqueName: \"kubernetes.io/projected/0992908f-5ad3-4600-b8d4-f26d07958ee2-kube-api-access-jjdcj\") pod \"e2e-api-b-6a3af1-openshift-default-666bcf8bbc-2hjz9\" (UID: \"0992908f-5ad3-4600-b8d4-f26d07958ee2\") " pod="openshift-ingress/e2e-api-b-6a3af1-openshift-default-666bcf8bbc-2hjz9"
Nov 02 12:43:37.296781 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:37.296765    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0992908f-5ad3-4600-b8d4-f26d07958ee2-istio-token\") pod \"e2e-api-b-6a3af1-openshift-default-666bcf8bbc-2hjz9\" (UID: \"0992908f-5ad3-4600-b8d4-f26d07958ee2\") " pod="openshift-ingress/e2e-api-b-6a3af1-openshift-default-666bcf8bbc-2hjz9"
Nov 02 12:43:37.296885 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:37.296832    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0992908f-5ad3-4600-b8d4-f26d07958ee2-credential-socket\") pod \"e2e-api-b-6a3af1-openshift-default-666bcf8bbc-2hjz9\" (UID: \"0992908f-5ad3-4600-b8d4-f26d07958ee2\") " pod="openshift-ingress/e2e-api-b-6a3af1-openshift-default-666bcf8bbc-2hjz9"
Nov 02 12:43:37.296960 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:37.296942    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0992908f-5ad3-4600-b8d4-f26d07958ee2-istio-podinfo\") pod \"e2e-api-b-6a3af1-openshift-default-666bcf8bbc-2hjz9\" (UID: \"0992908f-5ad3-4600-b8d4-f26d07958ee2\") " pod="openshift-ingress/e2e-api-b-6a3af1-openshift-default-666bcf8bbc-2hjz9"
Nov 02 12:43:37.346217 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:37.345749    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-c79cde-789d76c79-cnjxb" podStartSLOduration=2.345728891 podStartE2EDuration="2.345728891s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:43:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:43:37.34377522 +0000 UTC m=+1013.599174326" watchObservedRunningTime="2026-11-02 12:43:37.345728891 +0000 UTC m=+1013.601127986"
Nov 02 12:43:37.398435 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:37.398388    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0992908f-5ad3-4600-b8d4-f26d07958ee2-istiod-ca-cert\") pod \"e2e-api-b-6a3af1-openshift-default-666bcf8bbc-2hjz9\" (UID: \"0992908f-5ad3-4600-b8d4-f26d07958ee2\") " pod="openshift-ingress/e2e-api-b-6a3af1-openshift-default-666bcf8bbc-2hjz9"
Nov 02 12:43:37.398618 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:37.398442    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0992908f-5ad3-4600-b8d4-f26d07958ee2-istio-envoy\") pod \"e2e-api-b-6a3af1-openshift-default-666bcf8bbc-2hjz9\" (UID: \"0992908f-5ad3-4600-b8d4-f26d07958ee2\") " pod="openshift-ingress/e2e-api-b-6a3af1-openshift-default-666bcf8bbc-2hjz9"
Nov 02 12:43:37.398618 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:37.398552    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0992908f-5ad3-4600-b8d4-f26d07958ee2-istio-data\") pod \"e2e-api-b-6a3af1-openshift-default-666bcf8bbc-2hjz9\" (UID: \"0992908f-5ad3-4600-b8d4-f26d07958ee2\") " pod="openshift-ingress/e2e-api-b-6a3af1-openshift-default-666bcf8bbc-2hjz9"
Nov 02 12:43:37.398776 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:37.398687    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0992908f-5ad3-4600-b8d4-f26d07958ee2-workload-certs\") pod \"e2e-api-b-6a3af1-openshift-default-666bcf8bbc-2hjz9\" (UID: \"0992908f-5ad3-4600-b8d4-f26d07958ee2\") " pod="openshift-ingress/e2e-api-b-6a3af1-openshift-default-666bcf8bbc-2hjz9"
Nov 02 12:43:37.398776 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:37.398718    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0992908f-5ad3-4600-b8d4-f26d07958ee2-workload-socket\") pod \"e2e-api-b-6a3af1-openshift-default-666bcf8bbc-2hjz9\" (UID: \"0992908f-5ad3-4600-b8d4-f26d07958ee2\") " pod="openshift-ingress/e2e-api-b-6a3af1-openshift-default-666bcf8bbc-2hjz9"
Nov 02 12:43:37.399293 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:37.399268    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0992908f-5ad3-4600-b8d4-f26d07958ee2-istiod-ca-cert\") pod \"e2e-api-b-6a3af1-openshift-default-666bcf8bbc-2hjz9\" (UID: \"0992908f-5ad3-4600-b8d4-f26d07958ee2\") " pod="openshift-ingress/e2e-api-b-6a3af1-openshift-default-666bcf8bbc-2hjz9"
Nov 02 12:43:37.399414 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:37.398965    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0992908f-5ad3-4600-b8d4-f26d07958ee2-istio-data\") pod \"e2e-api-b-6a3af1-openshift-default-666bcf8bbc-2hjz9\" (UID: \"0992908f-5ad3-4600-b8d4-f26d07958ee2\") " pod="openshift-ingress/e2e-api-b-6a3af1-openshift-default-666bcf8bbc-2hjz9"
Nov 02 12:43:37.399414 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:37.399061    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0992908f-5ad3-4600-b8d4-f26d07958ee2-workload-certs\") pod \"e2e-api-b-6a3af1-openshift-default-666bcf8bbc-2hjz9\" (UID: \"0992908f-5ad3-4600-b8d4-f26d07958ee2\") " pod="openshift-ingress/e2e-api-b-6a3af1-openshift-default-666bcf8bbc-2hjz9"
Nov 02 12:43:37.399414 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:37.399252    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0992908f-5ad3-4600-b8d4-f26d07958ee2-workload-socket\") pod \"e2e-api-b-6a3af1-openshift-default-666bcf8bbc-2hjz9\" (UID: \"0992908f-5ad3-4600-b8d4-f26d07958ee2\") " pod="openshift-ingress/e2e-api-b-6a3af1-openshift-default-666bcf8bbc-2hjz9"
Nov 02 12:43:37.399414 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:37.399379    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjdcj\" (UniqueName: \"kubernetes.io/projected/0992908f-5ad3-4600-b8d4-f26d07958ee2-kube-api-access-jjdcj\") pod \"e2e-api-b-6a3af1-openshift-default-666bcf8bbc-2hjz9\" (UID: \"0992908f-5ad3-4600-b8d4-f26d07958ee2\") " pod="openshift-ingress/e2e-api-b-6a3af1-openshift-default-666bcf8bbc-2hjz9"
Nov 02 12:43:37.399414 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:37.399410    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0992908f-5ad3-4600-b8d4-f26d07958ee2-istio-token\") pod \"e2e-api-b-6a3af1-openshift-default-666bcf8bbc-2hjz9\" (UID: \"0992908f-5ad3-4600-b8d4-f26d07958ee2\") " pod="openshift-ingress/e2e-api-b-6a3af1-openshift-default-666bcf8bbc-2hjz9"
Nov 02 12:43:37.399698 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:37.399453    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0992908f-5ad3-4600-b8d4-f26d07958ee2-credential-socket\") pod \"e2e-api-b-6a3af1-openshift-default-666bcf8bbc-2hjz9\" (UID: \"0992908f-5ad3-4600-b8d4-f26d07958ee2\") " pod="openshift-ingress/e2e-api-b-6a3af1-openshift-default-666bcf8bbc-2hjz9"
Nov 02 12:43:37.399698 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:37.399526    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0992908f-5ad3-4600-b8d4-f26d07958ee2-istio-podinfo\") pod \"e2e-api-b-6a3af1-openshift-default-666bcf8bbc-2hjz9\" (UID: \"0992908f-5ad3-4600-b8d4-f26d07958ee2\") " pod="openshift-ingress/e2e-api-b-6a3af1-openshift-default-666bcf8bbc-2hjz9"
Nov 02 12:43:37.399818 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:37.399800    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0992908f-5ad3-4600-b8d4-f26d07958ee2-credential-socket\") pod \"e2e-api-b-6a3af1-openshift-default-666bcf8bbc-2hjz9\" (UID: \"0992908f-5ad3-4600-b8d4-f26d07958ee2\") " pod="openshift-ingress/e2e-api-b-6a3af1-openshift-default-666bcf8bbc-2hjz9"
Nov 02 12:43:37.401779 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:37.401757    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0992908f-5ad3-4600-b8d4-f26d07958ee2-istio-envoy\") pod \"e2e-api-b-6a3af1-openshift-default-666bcf8bbc-2hjz9\" (UID: \"0992908f-5ad3-4600-b8d4-f26d07958ee2\") " pod="openshift-ingress/e2e-api-b-6a3af1-openshift-default-666bcf8bbc-2hjz9"
Nov 02 12:43:37.403776 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:37.403736    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c79cde-76d5b754d7-2vhqc" podStartSLOduration=2.403724095 podStartE2EDuration="2.403724095s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:43:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:43:37.40153991 +0000 UTC m=+1013.656939005" watchObservedRunningTime="2026-11-02 12:43:37.403724095 +0000 UTC m=+1013.659123190"
Nov 02 12:43:37.404233 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:37.404100    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0992908f-5ad3-4600-b8d4-f26d07958ee2-istio-podinfo\") pod \"e2e-api-b-6a3af1-openshift-default-666bcf8bbc-2hjz9\" (UID: \"0992908f-5ad3-4600-b8d4-f26d07958ee2\") " pod="openshift-ingress/e2e-api-b-6a3af1-openshift-default-666bcf8bbc-2hjz9"
Nov 02 12:43:37.472478 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:37.472403    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0992908f-5ad3-4600-b8d4-f26d07958ee2-istio-token\") pod \"e2e-api-b-6a3af1-openshift-default-666bcf8bbc-2hjz9\" (UID: \"0992908f-5ad3-4600-b8d4-f26d07958ee2\") " pod="openshift-ingress/e2e-api-b-6a3af1-openshift-default-666bcf8bbc-2hjz9"
Nov 02 12:43:37.475439 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:37.475413    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjdcj\" (UniqueName: \"kubernetes.io/projected/0992908f-5ad3-4600-b8d4-f26d07958ee2-kube-api-access-jjdcj\") pod \"e2e-api-b-6a3af1-openshift-default-666bcf8bbc-2hjz9\" (UID: \"0992908f-5ad3-4600-b8d4-f26d07958ee2\") " pod="openshift-ingress/e2e-api-b-6a3af1-openshift-default-666bcf8bbc-2hjz9"
Nov 02 12:43:37.489397 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:37.489375    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-6a3af1-openshift-default-666bcf8bbc-2hjz9"
Nov 02 12:43:37.514872 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:43:37.514816    4559 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7004-0d46b6\" is forbidden: User \"system:node:ip-10-0-128-132.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-132.ec2.internal' and this object" podUID="50ca0973-9e27-4042-9300-4ad78edae135" pod="opendatahub/test-curl-7004-0d46b6"
Nov 02 12:43:37.952456 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:37.952424    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-6a3af1-openshift-default-666bcf8bbc-2hjz9"]
Nov 02 12:43:38.046991 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:38.046851    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:43:38.047333 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:38.047288    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:43:38.047547 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:38.047512    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:43:38.200077 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:38.200046    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-9f7f01-openshift-default-64d8f9568c-pw8tx" event={"ID":"26b966b3-2043-474a-95aa-3037982c3fc4","Type":"ContainerStarted","Data":"2e2bbe96217d9f6965c75c614cb63495b7c0e945cf5dc32aca4c7557f46e8f5f"}
Nov 02 12:43:38.202400 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:38.202377    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-96339b-openshift-default-857fd4b79-52nqm" event={"ID":"e51ff2f8-753e-4977-a523-90bca8a4ec2a","Type":"ContainerStarted","Data":"dff1816eb1b190886c76fbe891761a02d676a0ea1101e11f1ff6a73e97f9a32c"}
Nov 02 12:43:38.204061 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:38.204006    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-6a3af1-openshift-default-666bcf8bbc-2hjz9" event={"ID":"0992908f-5ad3-4600-b8d4-f26d07958ee2","Type":"ContainerStarted","Data":"7bf0822da13088100666569a3d151001bc54e5fb2866e48502e17c8510bac656"}
Nov 02 12:43:38.491251 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:38.491164    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-9f7f01-openshift-default-64d8f9568c-pw8tx"
Nov 02 12:43:38.491746 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:38.491725    4559 patch_prober.go:29] interesting pod/e2e-api-a-9f7f01-openshift-default-64d8f9568c-pw8tx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Nov 02 12:43:38.491802 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:38.491774    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-9f7f01-openshift-default-64d8f9568c-pw8tx" podUID="26b966b3-2043-474a-95aa-3037982c3fc4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Nov 02 12:43:38.629633 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:38.629599    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-96339b-openshift-default-857fd4b79-52nqm"
Nov 02 12:43:38.630039 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:38.630016    4559 patch_prober.go:29] interesting pod/e2e-api-b-96339b-openshift-default-857fd4b79-52nqm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Nov 02 12:43:38.630977 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:38.630949    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-96339b-openshift-default-857fd4b79-52nqm" podUID="e51ff2f8-753e-4977-a523-90bca8a4ec2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Nov 02 12:43:39.243762 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:39.243718    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-6a3af1-openshift-default-666bcf8bbc-2hjz9" event={"ID":"0992908f-5ad3-4600-b8d4-f26d07958ee2","Type":"ContainerStarted","Data":"3c680aa9299d50dbc18dc2288a5ba61232752ea9c393f42d1f8ab47b681c7714"}
Nov 02 12:43:39.339400 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:39.339327    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-6a3af1-openshift-default-666bcf8bbc-2hjz9" podStartSLOduration=2.339302143 podStartE2EDuration="2.339302143s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:43:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:43:39.291159726 +0000 UTC m=+1015.546558844" watchObservedRunningTime="2026-11-02 12:43:39.339302143 +0000 UTC m=+1015.594701239"
Nov 02 12:43:39.503825 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:39.503723    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-6a3af1-openshift-default-666bcf8bbc-2hjz9"
Nov 02 12:43:39.507676 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:39.504171    4559 patch_prober.go:29] interesting pod/e2e-api-a-9f7f01-openshift-default-64d8f9568c-pw8tx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Nov 02 12:43:39.549870 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:39.504307    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-9f7f01-openshift-default-64d8f9568c-pw8tx" podUID="26b966b3-2043-474a-95aa-3037982c3fc4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Nov 02 12:43:39.575157 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:39.573445    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-6a3af1-openshift-default-666bcf8bbc-2hjz9"
Nov 02 12:43:39.641017 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:39.638173    4559 patch_prober.go:29] interesting pod/e2e-api-b-96339b-openshift-default-857fd4b79-52nqm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Nov 02 12:43:39.641017 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:39.638247    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-96339b-openshift-default-857fd4b79-52nqm" podUID="e51ff2f8-753e-4977-a523-90bca8a4ec2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Nov 02 12:43:39.708675 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:39.705837    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7004-0d46b6"
Nov 02 12:43:39.778844 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:39.778748    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50ca0973-9e27-4042-9300-4ad78edae135-kube-api-access-g9wqm\" (UniqueName: \"kubernetes.io/projected/50ca0973-9e27-4042-9300-4ad78edae135-kube-api-access-g9wqm\") pod \"50ca0973-9e27-4042-9300-4ad78edae135\" (UID: \"50ca0973-9e27-4042-9300-4ad78edae135\") "
Nov 02 12:43:39.781284 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:39.781260    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50ca0973-9e27-4042-9300-4ad78edae135-kube-api-access-g9wqm" pod "50ca0973-9e27-4042-9300-4ad78edae135" (UID: "50ca0973-9e27-4042-9300-4ad78edae135"). InnerVolumeSpecName "kube-api-access-g9wqm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:43:39.881141 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:39.881106    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g9wqm\" (UniqueName: \"kubernetes.io/projected/50ca0973-9e27-4042-9300-4ad78edae135-kube-api-access-g9wqm\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:43:39.950921 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:39.950868    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50ca0973-9e27-4042-9300-4ad78edae135" path="/var/lib/kubelet/pods/50ca0973-9e27-4042-9300-4ad78edae135/volumes"
Nov 02 12:43:40.251528 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:40.251491    4559 generic.go:356] "Generic (PLEG): container finished" podID="50ca0973-9e27-4042-9300-4ad78edae135" containerID="0a131e581c036d1dad8c689ea80b32fa696f06cb2cc240dabb6bfb775e8dc727" exitCode=137
Nov 02 12:43:40.252097 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:40.251534    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7004-0d46b6"
Nov 02 12:43:40.252097 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:40.251586    4559 scope.go:122] "RemoveContainer" containerID="0a131e581c036d1dad8c689ea80b32fa696f06cb2cc240dabb6bfb775e8dc727"
Nov 02 12:43:40.252097 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:40.251881    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-6a3af1-openshift-default-666bcf8bbc-2hjz9"
Nov 02 12:43:40.253517 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:40.253500    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-6a3af1-openshift-default-666bcf8bbc-2hjz9"
Nov 02 12:43:40.261261 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:40.261247    4559 scope.go:122] "RemoveContainer" containerID="0a131e581c036d1dad8c689ea80b32fa696f06cb2cc240dabb6bfb775e8dc727"
Nov 02 12:43:40.261547 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:43:40.261528    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a131e581c036d1dad8c689ea80b32fa696f06cb2cc240dabb6bfb775e8dc727\": container with ID starting with 0a131e581c036d1dad8c689ea80b32fa696f06cb2cc240dabb6bfb775e8dc727 not found: ID does not exist" containerID="0a131e581c036d1dad8c689ea80b32fa696f06cb2cc240dabb6bfb775e8dc727"
Nov 02 12:43:40.261604 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:40.261560    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a131e581c036d1dad8c689ea80b32fa696f06cb2cc240dabb6bfb775e8dc727"} err="failed to get container status \"0a131e581c036d1dad8c689ea80b32fa696f06cb2cc240dabb6bfb775e8dc727\": rpc error: code = NotFound desc = could not find container \"0a131e581c036d1dad8c689ea80b32fa696f06cb2cc240dabb6bfb775e8dc727\": container with ID starting with 0a131e581c036d1dad8c689ea80b32fa696f06cb2cc240dabb6bfb775e8dc727 not found: ID does not exist"
Nov 02 12:43:40.495668 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:40.495000    4559 patch_prober.go:29] interesting pod/e2e-api-a-9f7f01-openshift-default-64d8f9568c-pw8tx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Nov 02 12:43:40.495668 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:40.495092    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-9f7f01-openshift-default-64d8f9568c-pw8tx" podUID="26b966b3-2043-474a-95aa-3037982c3fc4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Nov 02 12:43:40.630665 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:40.630187    4559 patch_prober.go:29] interesting pod/e2e-api-b-96339b-openshift-default-857fd4b79-52nqm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Nov 02 12:43:40.630665 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:40.630284    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-96339b-openshift-default-857fd4b79-52nqm" podUID="e51ff2f8-753e-4977-a523-90bca8a4ec2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Nov 02 12:43:41.503664 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:41.497984    4559 patch_prober.go:29] interesting pod/e2e-api-a-9f7f01-openshift-default-64d8f9568c-pw8tx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Nov 02 12:43:41.503664 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:41.498045    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-9f7f01-openshift-default-64d8f9568c-pw8tx" podUID="26b966b3-2043-474a-95aa-3037982c3fc4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Nov 02 12:43:41.632027 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:41.631991    4559 patch_prober.go:29] interesting pod/e2e-api-b-96339b-openshift-default-857fd4b79-52nqm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Nov 02 12:43:41.633009 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:41.632978    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-96339b-openshift-default-857fd4b79-52nqm" podUID="e51ff2f8-753e-4977-a523-90bca8a4ec2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Nov 02 12:43:42.417661 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:42.416043    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"]
Nov 02 12:43:42.501773 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:42.501736    4559 patch_prober.go:29] interesting pod/e2e-api-a-9f7f01-openshift-default-64d8f9568c-pw8tx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Nov 02 12:43:42.502065 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:42.502037    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-9f7f01-openshift-default-64d8f9568c-pw8tx" podUID="26b966b3-2043-474a-95aa-3037982c3fc4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Nov 02 12:43:42.626145 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:42.626106    4559 patch_prober.go:29] interesting pod/e2e-api-b-96339b-openshift-default-857fd4b79-52nqm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Nov 02 12:43:42.626792 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:42.626751    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-96339b-openshift-default-857fd4b79-52nqm" podUID="e51ff2f8-753e-4977-a523-90bca8a4ec2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Nov 02 12:43:43.492489 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:43.492453    4559 patch_prober.go:29] interesting pod/e2e-api-a-9f7f01-openshift-default-64d8f9568c-pw8tx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Nov 02 12:43:43.492792 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:43.492762    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-9f7f01-openshift-default-64d8f9568c-pw8tx" podUID="26b966b3-2043-474a-95aa-3037982c3fc4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Nov 02 12:43:43.632076 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:43.632039    4559 patch_prober.go:29] interesting pod/e2e-api-b-96339b-openshift-default-857fd4b79-52nqm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Nov 02 12:43:43.633426 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:43.633391    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-96339b-openshift-default-857fd4b79-52nqm" podUID="e51ff2f8-753e-4977-a523-90bca8a4ec2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Nov 02 12:43:44.279843 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:44.279808    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-96339b-openshift-default-857fd4b79-52nqm"
Nov 02 12:43:44.643857 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:44.643824    4559 patch_prober.go:29] interesting pod/e2e-api-b-96339b-openshift-default-857fd4b79-52nqm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Nov 02 12:43:44.644429 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:44.644396    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-96339b-openshift-default-857fd4b79-52nqm" podUID="e51ff2f8-753e-4977-a523-90bca8a4ec2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Nov 02 12:43:44.656482 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:44.656297    4559 patch_prober.go:29] interesting pod/e2e-api-a-9f7f01-openshift-default-64d8f9568c-pw8tx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Nov 02 12:43:44.656482 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:44.656358    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-9f7f01-openshift-default-64d8f9568c-pw8tx" podUID="26b966b3-2043-474a-95aa-3037982c3fc4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Nov 02 12:43:45.130663 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:45.117798    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-9f7f01-openshift-default-64d8f9568c-pw8tx"
Nov 02 12:43:45.144347 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:43:45.141533    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqcmj" containerName="extractor"
Nov 02 12:43:45.610739 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:45.609692    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-9f7f01-openshift-default-64d8f9568c-pw8tx"
Nov 02 12:43:45.701666 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:45.698212    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-9f7f01-openshift-default-64d8f9568c-pw8tx"
Nov 02 12:43:45.964888 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:45.964859    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-96339b-openshift-default-857fd4b79-52nqm"
Nov 02 12:43:45.966221 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:45.966197    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-96339b-openshift-default-857fd4b79-52nqm"
Nov 02 12:43:46.933729 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:46.933681    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-96339b-openshift-default-857fd4b79-52nqm"]
Nov 02 12:43:47.703489 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:47.703438    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-96339b-openshift-default-857fd4b79-52nqm" podUID="e51ff2f8-753e-4977-a523-90bca8a4ec2a" containerName="istio-proxy" containerID="cri-o://dff1816eb1b190886c76fbe891761a02d676a0ea1101e11f1ff6a73e97f9a32c" gracePeriod=30
Nov 02 12:43:47.843722 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:47.843681    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-cd1386cb-556c7759d7-m98mv"]
Nov 02 12:43:48.207349 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:48.207322    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-c79cde-789d76c79-cnjxb"
Nov 02 12:43:48.207865 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:48.207805    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c79cde-76d5b754d7-2vhqc"
Nov 02 12:43:52.844890 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:52.844849    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-cd1386cb-556c7759d7-m98mv" podUID="e7f08638-accd-4b0d-9657-652d7c8df22b" containerName="payload-pre-processing" containerID="cri-o://0986494e30f02d8b39e36ecd1c6b1572a781177b02a3a35df84e4fb1d1b56719" gracePeriod=25
Nov 02 12:43:52.867187 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:52.867161    4559 generic.go:356] "Generic (PLEG): container finished" podID="e51ff2f8-753e-4977-a523-90bca8a4ec2a" containerID="dff1816eb1b190886c76fbe891761a02d676a0ea1101e11f1ff6a73e97f9a32c" exitCode=0
Nov 02 12:43:52.867324 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:52.867233    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-96339b-openshift-default-857fd4b79-52nqm" event={"ID":"e51ff2f8-753e-4977-a523-90bca8a4ec2a","Type":"ContainerDied","Data":"dff1816eb1b190886c76fbe891761a02d676a0ea1101e11f1ff6a73e97f9a32c"}
Nov 02 12:43:52.867324 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:52.867294    4559 scope.go:122] "RemoveContainer" containerID="2d9ba826fa413fa53e83afa46516baa134cdb90eadb757ff07b8e883f8540ca5"
Nov 02 12:43:52.988303 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:52.988249    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-96339b-openshift-default-857fd4b79-52nqm"
Nov 02 12:43:53.012848 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:53.012826    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e51ff2f8-753e-4977-a523-90bca8a4ec2a-kube-api-access-9c8sp\" (UniqueName: \"kubernetes.io/projected/e51ff2f8-753e-4977-a523-90bca8a4ec2a-kube-api-access-9c8sp\") pod \"e51ff2f8-753e-4977-a523-90bca8a4ec2a\" (UID: \"e51ff2f8-753e-4977-a523-90bca8a4ec2a\") "
Nov 02 12:43:53.012968 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:53.012864    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e51ff2f8-753e-4977-a523-90bca8a4ec2a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e51ff2f8-753e-4977-a523-90bca8a4ec2a-istiod-ca-cert\") pod \"e51ff2f8-753e-4977-a523-90bca8a4ec2a\" (UID: \"e51ff2f8-753e-4977-a523-90bca8a4ec2a\") "
Nov 02 12:43:53.012968 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:53.012954    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e51ff2f8-753e-4977-a523-90bca8a4ec2a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e51ff2f8-753e-4977-a523-90bca8a4ec2a-credential-socket\") pod \"e51ff2f8-753e-4977-a523-90bca8a4ec2a\" (UID: \"e51ff2f8-753e-4977-a523-90bca8a4ec2a\") "
Nov 02 12:43:53.013080 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:53.013069    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e51ff2f8-753e-4977-a523-90bca8a4ec2a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e51ff2f8-753e-4977-a523-90bca8a4ec2a-istio-envoy\") pod \"e51ff2f8-753e-4977-a523-90bca8a4ec2a\" (UID: \"e51ff2f8-753e-4977-a523-90bca8a4ec2a\") "
Nov 02 12:43:53.013119 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:53.013093    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e51ff2f8-753e-4977-a523-90bca8a4ec2a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e51ff2f8-753e-4977-a523-90bca8a4ec2a-istio-podinfo\") pod \"e51ff2f8-753e-4977-a523-90bca8a4ec2a\" (UID: \"e51ff2f8-753e-4977-a523-90bca8a4ec2a\") "
Nov 02 12:43:53.013119 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:53.013112    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e51ff2f8-753e-4977-a523-90bca8a4ec2a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e51ff2f8-753e-4977-a523-90bca8a4ec2a-istio-data\") pod \"e51ff2f8-753e-4977-a523-90bca8a4ec2a\" (UID: \"e51ff2f8-753e-4977-a523-90bca8a4ec2a\") "
Nov 02 12:43:53.013223 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:53.013168    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e51ff2f8-753e-4977-a523-90bca8a4ec2a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e51ff2f8-753e-4977-a523-90bca8a4ec2a-workload-socket\") pod \"e51ff2f8-753e-4977-a523-90bca8a4ec2a\" (UID: \"e51ff2f8-753e-4977-a523-90bca8a4ec2a\") "
Nov 02 12:43:53.013288 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:53.013248    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e51ff2f8-753e-4977-a523-90bca8a4ec2a-credential-socket" pod "e51ff2f8-753e-4977-a523-90bca8a4ec2a" (UID: "e51ff2f8-753e-4977-a523-90bca8a4ec2a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:43:53.013346 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:53.013284    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e51ff2f8-753e-4977-a523-90bca8a4ec2a-istiod-ca-cert" pod "e51ff2f8-753e-4977-a523-90bca8a4ec2a" (UID: "e51ff2f8-753e-4977-a523-90bca8a4ec2a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:43:53.013346 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:53.013299    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e51ff2f8-753e-4977-a523-90bca8a4ec2a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e51ff2f8-753e-4977-a523-90bca8a4ec2a-workload-certs\") pod \"e51ff2f8-753e-4977-a523-90bca8a4ec2a\" (UID: \"e51ff2f8-753e-4977-a523-90bca8a4ec2a\") "
Nov 02 12:43:53.013346 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:53.013340    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e51ff2f8-753e-4977-a523-90bca8a4ec2a-istio-token\" (UniqueName: \"kubernetes.io/projected/e51ff2f8-753e-4977-a523-90bca8a4ec2a-istio-token\") pod \"e51ff2f8-753e-4977-a523-90bca8a4ec2a\" (UID: \"e51ff2f8-753e-4977-a523-90bca8a4ec2a\") "
Nov 02 12:43:53.013535 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:53.013492    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e51ff2f8-753e-4977-a523-90bca8a4ec2a-istio-data" pod "e51ff2f8-753e-4977-a523-90bca8a4ec2a" (UID: "e51ff2f8-753e-4977-a523-90bca8a4ec2a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:43:53.013610 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:53.013550    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e51ff2f8-753e-4977-a523-90bca8a4ec2a-workload-socket" pod "e51ff2f8-753e-4977-a523-90bca8a4ec2a" (UID: "e51ff2f8-753e-4977-a523-90bca8a4ec2a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:43:53.013610 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:53.013562    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e51ff2f8-753e-4977-a523-90bca8a4ec2a-workload-certs" pod "e51ff2f8-753e-4977-a523-90bca8a4ec2a" (UID: "e51ff2f8-753e-4977-a523-90bca8a4ec2a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:43:53.014265 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:53.014243    4559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e51ff2f8-753e-4977-a523-90bca8a4ec2a-workload-certs\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:43:53.014265 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:53.014265    4559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e51ff2f8-753e-4977-a523-90bca8a4ec2a-istiod-ca-cert\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:43:53.014389 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:53.014278    4559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e51ff2f8-753e-4977-a523-90bca8a4ec2a-credential-socket\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:43:53.014389 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:53.014290    4559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e51ff2f8-753e-4977-a523-90bca8a4ec2a-istio-data\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:43:53.014389 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:53.014303    4559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e51ff2f8-753e-4977-a523-90bca8a4ec2a-workload-socket\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:43:53.015318 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:53.015289    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e51ff2f8-753e-4977-a523-90bca8a4ec2a-istio-envoy" pod "e51ff2f8-753e-4977-a523-90bca8a4ec2a" (UID: "e51ff2f8-753e-4977-a523-90bca8a4ec2a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:43:53.015318 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:53.015307    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e51ff2f8-753e-4977-a523-90bca8a4ec2a-istio-podinfo" pod "e51ff2f8-753e-4977-a523-90bca8a4ec2a" (UID: "e51ff2f8-753e-4977-a523-90bca8a4ec2a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 12:43:53.015517 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:53.015499    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e51ff2f8-753e-4977-a523-90bca8a4ec2a-kube-api-access-9c8sp" pod "e51ff2f8-753e-4977-a523-90bca8a4ec2a" (UID: "e51ff2f8-753e-4977-a523-90bca8a4ec2a"). InnerVolumeSpecName "kube-api-access-9c8sp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:43:53.015703 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:53.015686    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e51ff2f8-753e-4977-a523-90bca8a4ec2a-istio-token" pod "e51ff2f8-753e-4977-a523-90bca8a4ec2a" (UID: "e51ff2f8-753e-4977-a523-90bca8a4ec2a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:43:53.115368 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:53.115338    4559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e51ff2f8-753e-4977-a523-90bca8a4ec2a-istio-token\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:43:53.115368 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:53.115365    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9c8sp\" (UniqueName: \"kubernetes.io/projected/e51ff2f8-753e-4977-a523-90bca8a4ec2a-kube-api-access-9c8sp\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:43:53.115547 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:53.115375    4559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e51ff2f8-753e-4977-a523-90bca8a4ec2a-istio-envoy\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:43:53.115547 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:53.115386    4559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e51ff2f8-753e-4977-a523-90bca8a4ec2a-istio-podinfo\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:43:53.874959 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:53.874926    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-96339b-openshift-default-857fd4b79-52nqm" event={"ID":"e51ff2f8-753e-4977-a523-90bca8a4ec2a","Type":"ContainerDied","Data":"6e486983da490d8477d960b384bac287be98b2a69645886f9e9855eed04b6024"}
Nov 02 12:43:53.874959 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:53.874955    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-96339b-openshift-default-857fd4b79-52nqm"
Nov 02 12:43:53.875512 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:53.875017    4559 scope.go:122] "RemoveContainer" containerID="dff1816eb1b190886c76fbe891761a02d676a0ea1101e11f1ff6a73e97f9a32c"
Nov 02 12:43:53.935751 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:53.935722    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-96339b-openshift-default-857fd4b79-52nqm"]
Nov 02 12:43:53.948611 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:53.948589    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-96339b-openshift-default-857fd4b79-52nqm"]
Nov 02 12:43:55.454591 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:55.454514    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-cd1386cb-556c7759d7-m98mv" podUID="e7f08638-accd-4b0d-9657-652d7c8df22b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 12:43:55.962088 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:55.962052    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e51ff2f8-753e-4977-a523-90bca8a4ec2a" path="/var/lib/kubelet/pods/e51ff2f8-753e-4977-a523-90bca8a4ec2a/volumes"
Nov 02 12:43:58.770984 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:58.770951    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-6a3af1-84b6d48b4b-rb9gx"]
Nov 02 12:43:58.771820 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:58.771799    4559 cpu_manager.go:414] "containerMap: removing container" podUID="e51ff2f8-753e-4977-a523-90bca8a4ec2a" containerName="istio-proxy"
Nov 02 12:43:58.771820 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:58.771820    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e51ff2f8-753e-4977-a523-90bca8a4ec2a" containerName="istio-proxy"
Nov 02 12:43:58.772000 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:58.771977    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="e51ff2f8-753e-4977-a523-90bca8a4ec2a" containerName="istio-proxy"
Nov 02 12:43:58.772000 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:58.771990    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="e51ff2f8-753e-4977-a523-90bca8a4ec2a" containerName="istio-proxy"
Nov 02 12:43:58.777058 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:58.777030    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6a3af1-84b6d48b4b-rb9gx"
Nov 02 12:43:58.804464 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:58.802482    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-6a3af1-serving-cert\""
Nov 02 12:43:58.808990 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:58.808957    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mm54s\" (UniqueName: \"kubernetes.io/projected/227d2e4d-cc59-4cc6-b705-f257e3470f16-kube-api-access-mm54s\") pod \"maas-api-e2e-api-b-6a3af1-84b6d48b4b-rb9gx\" (UID: \"227d2e4d-cc59-4cc6-b705-f257e3470f16\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6a3af1-84b6d48b4b-rb9gx"
Nov 02 12:43:58.809119 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:58.809064    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/227d2e4d-cc59-4cc6-b705-f257e3470f16-maas-api-tls\") pod \"maas-api-e2e-api-b-6a3af1-84b6d48b4b-rb9gx\" (UID: \"227d2e4d-cc59-4cc6-b705-f257e3470f16\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6a3af1-84b6d48b4b-rb9gx"
Nov 02 12:43:58.809119 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:58.809108    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/227d2e4d-cc59-4cc6-b705-f257e3470f16-metrics-certs\") pod \"maas-api-e2e-api-b-6a3af1-84b6d48b4b-rb9gx\" (UID: \"227d2e4d-cc59-4cc6-b705-f257e3470f16\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6a3af1-84b6d48b4b-rb9gx"
Nov 02 12:43:58.809404 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:58.809385    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/227d2e4d-cc59-4cc6-b705-f257e3470f16-openshift-service-ca\") pod \"maas-api-e2e-api-b-6a3af1-84b6d48b4b-rb9gx\" (UID: \"227d2e4d-cc59-4cc6-b705-f257e3470f16\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6a3af1-84b6d48b4b-rb9gx"
Nov 02 12:43:58.811010 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:58.810988    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-6a3af1-84b6d48b4b-rb9gx"]
Nov 02 12:43:58.911026 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:58.910995    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/227d2e4d-cc59-4cc6-b705-f257e3470f16-openshift-service-ca\") pod \"maas-api-e2e-api-b-6a3af1-84b6d48b4b-rb9gx\" (UID: \"227d2e4d-cc59-4cc6-b705-f257e3470f16\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6a3af1-84b6d48b4b-rb9gx"
Nov 02 12:43:58.911211 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:58.911066    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mm54s\" (UniqueName: \"kubernetes.io/projected/227d2e4d-cc59-4cc6-b705-f257e3470f16-kube-api-access-mm54s\") pod \"maas-api-e2e-api-b-6a3af1-84b6d48b4b-rb9gx\" (UID: \"227d2e4d-cc59-4cc6-b705-f257e3470f16\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6a3af1-84b6d48b4b-rb9gx"
Nov 02 12:43:58.911211 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:58.911134    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/227d2e4d-cc59-4cc6-b705-f257e3470f16-maas-api-tls\") pod \"maas-api-e2e-api-b-6a3af1-84b6d48b4b-rb9gx\" (UID: \"227d2e4d-cc59-4cc6-b705-f257e3470f16\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6a3af1-84b6d48b4b-rb9gx"
Nov 02 12:43:58.911211 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:58.911175    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/227d2e4d-cc59-4cc6-b705-f257e3470f16-metrics-certs\") pod \"maas-api-e2e-api-b-6a3af1-84b6d48b4b-rb9gx\" (UID: \"227d2e4d-cc59-4cc6-b705-f257e3470f16\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6a3af1-84b6d48b4b-rb9gx"
Nov 02 12:43:58.911752 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:58.911726    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/227d2e4d-cc59-4cc6-b705-f257e3470f16-openshift-service-ca\") pod \"maas-api-e2e-api-b-6a3af1-84b6d48b4b-rb9gx\" (UID: \"227d2e4d-cc59-4cc6-b705-f257e3470f16\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6a3af1-84b6d48b4b-rb9gx"
Nov 02 12:43:58.913435 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:58.913407    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/227d2e4d-cc59-4cc6-b705-f257e3470f16-metrics-certs\") pod \"maas-api-e2e-api-b-6a3af1-84b6d48b4b-rb9gx\" (UID: \"227d2e4d-cc59-4cc6-b705-f257e3470f16\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6a3af1-84b6d48b4b-rb9gx"
Nov 02 12:43:58.913692 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:58.913671    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/227d2e4d-cc59-4cc6-b705-f257e3470f16-maas-api-tls\") pod \"maas-api-e2e-api-b-6a3af1-84b6d48b4b-rb9gx\" (UID: \"227d2e4d-cc59-4cc6-b705-f257e3470f16\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6a3af1-84b6d48b4b-rb9gx"
Nov 02 12:43:58.937974 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:58.937945    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mm54s\" (UniqueName: \"kubernetes.io/projected/227d2e4d-cc59-4cc6-b705-f257e3470f16-kube-api-access-mm54s\") pod \"maas-api-e2e-api-b-6a3af1-84b6d48b4b-rb9gx\" (UID: \"227d2e4d-cc59-4cc6-b705-f257e3470f16\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6a3af1-84b6d48b4b-rb9gx"
Nov 02 12:43:59.090012 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:59.089922    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6a3af1-84b6d48b4b-rb9gx"
Nov 02 12:43:59.495528 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:43:59.495498    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod227d2e4d_cc59_4cc6_b705_f257e3470f16.slice/crio-76c4e83e9ab2495e42895e086f0cf9454028d566ea0b3ff60f9e7153e1070afc WatchSource:0}: Error finding container 76c4e83e9ab2495e42895e086f0cf9454028d566ea0b3ff60f9e7153e1070afc: Status 404 returned error can't find the container with id 76c4e83e9ab2495e42895e086f0cf9454028d566ea0b3ff60f9e7153e1070afc
Nov 02 12:43:59.495924 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:43:59.495900    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-6a3af1-84b6d48b4b-rb9gx"]
Nov 02 12:44:00.180671 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:00.174198    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6a3af1-84b6d48b4b-rb9gx" event={"ID":"227d2e4d-cc59-4cc6-b705-f257e3470f16","Type":"ContainerStarted","Data":"7c9aa713c0f86c6ada47f985a5972f1d4ff210053e1e477aab191a167f2b02ae"}
Nov 02 12:44:00.180671 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:00.174320    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6a3af1-84b6d48b4b-rb9gx" event={"ID":"227d2e4d-cc59-4cc6-b705-f257e3470f16","Type":"ContainerStarted","Data":"76c4e83e9ab2495e42895e086f0cf9454028d566ea0b3ff60f9e7153e1070afc"}
Nov 02 12:44:00.180671 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:00.175337    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6a3af1-84b6d48b4b-rb9gx"
Nov 02 12:44:00.227395 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:00.227343    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6a3af1-84b6d48b4b-rb9gx" podStartSLOduration=1.913512815 podStartE2EDuration="2.227325165s" totalImagesPullingTime="313.81235ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:43:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:44:00.226258274 +0000 UTC m=+1036.481657368" watchObservedRunningTime="2026-11-02 12:44:00.227325165 +0000 UTC m=+1036.482724259"
Nov 02 12:44:05.454159 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:05.454116    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-cd1386cb-556c7759d7-m98mv" podUID="e7f08638-accd-4b0d-9657-652d7c8df22b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 12:44:06.233681 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:06.220924    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6a3af1-84b6d48b4b-rb9gx"
Nov 02 12:44:13.482892 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:13.482719    4559 generic.go:356] "Generic (PLEG): container finished" podID="e7f08638-accd-4b0d-9657-652d7c8df22b" containerID="0986494e30f02d8b39e36ecd1c6b1572a781177b02a3a35df84e4fb1d1b56719" exitCode=0
Nov 02 12:44:13.482892 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:13.482786    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-cd1386cb-556c7759d7-m98mv" event={"ID":"e7f08638-accd-4b0d-9657-652d7c8df22b","Type":"ContainerDied","Data":"0986494e30f02d8b39e36ecd1c6b1572a781177b02a3a35df84e4fb1d1b56719"}
Nov 02 12:44:13.598338 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:13.598314    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-cd1386cb-556c7759d7-m98mv"
Nov 02 12:44:13.693247 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:13.693209    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7f08638-accd-4b0d-9657-652d7c8df22b-kube-api-access-w4r85\" (UniqueName: \"kubernetes.io/projected/e7f08638-accd-4b0d-9657-652d7c8df22b-kube-api-access-w4r85\") pod \"e7f08638-accd-4b0d-9657-652d7c8df22b\" (UID: \"e7f08638-accd-4b0d-9657-652d7c8df22b\") "
Nov 02 12:44:13.693429 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:13.693383    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7f08638-accd-4b0d-9657-652d7c8df22b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7f08638-accd-4b0d-9657-652d7c8df22b-plugins-config-volume\") pod \"e7f08638-accd-4b0d-9657-652d7c8df22b\" (UID: \"e7f08638-accd-4b0d-9657-652d7c8df22b\") "
Nov 02 12:44:13.694218 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:13.693839    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7f08638-accd-4b0d-9657-652d7c8df22b-plugins-config-volume" pod "e7f08638-accd-4b0d-9657-652d7c8df22b" (UID: "e7f08638-accd-4b0d-9657-652d7c8df22b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:44:13.694218 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:13.694181    4559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7f08638-accd-4b0d-9657-652d7c8df22b-plugins-config-volume\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:44:13.696140 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:13.696099    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7f08638-accd-4b0d-9657-652d7c8df22b-kube-api-access-w4r85" pod "e7f08638-accd-4b0d-9657-652d7c8df22b" (UID: "e7f08638-accd-4b0d-9657-652d7c8df22b"). InnerVolumeSpecName "kube-api-access-w4r85". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:44:13.795838 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:13.795803    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w4r85\" (UniqueName: \"kubernetes.io/projected/e7f08638-accd-4b0d-9657-652d7c8df22b-kube-api-access-w4r85\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:44:14.508917 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:14.505689    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-cd1386cb-556c7759d7-m98mv"
Nov 02 12:44:14.508917 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:14.505891    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-cd1386cb-556c7759d7-m98mv" event={"ID":"e7f08638-accd-4b0d-9657-652d7c8df22b","Type":"ContainerDied","Data":"4ec895e82405aa75aff03839df9f4e0055c21f1af49f18c536cc183b1e7868d6"}
Nov 02 12:44:14.508917 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:14.505961    4559 scope.go:122] "RemoveContainer" containerID="0986494e30f02d8b39e36ecd1c6b1572a781177b02a3a35df84e4fb1d1b56719"
Nov 02 12:44:14.561550 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:14.561521    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-cd1386cb-556c7759d7-m98mv"]
Nov 02 12:44:14.574600 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:14.574573    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-cd1386cb-556c7759d7-m98mv"]
Nov 02 12:44:15.090236 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:44:15.090180    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqcmj" containerName="extractor"
Nov 02 12:44:15.895783 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:15.895751    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7f08638-accd-4b0d-9657-652d7c8df22b" path="/var/lib/kubelet/pods/e7f08638-accd-4b0d-9657-652d7c8df22b/volumes"
Nov 02 12:44:17.294669 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:17.294307    4559 patch_prober.go:29] interesting pod/e2e-api-b-c79cde-openshift-default-54dbf475f7-g56xt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": read tcp 10.133.0.2:35866->10.133.0.92:15021: read: connection reset by peer" start-of-body=
Nov 02 12:44:17.294669 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:17.294398    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-c79cde-openshift-default-54dbf475f7-g56xt" podUID="ea9a3b60-d2ac-4837-b979-2ea31f79cee0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": read tcp 10.133.0.2:35866->10.133.0.92:15021: read: connection reset by peer"
Nov 02 12:44:17.554447 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:17.554349    4559 generic.go:356] "Generic (PLEG): container finished" podID="ea9a3b60-d2ac-4837-b979-2ea31f79cee0" containerID="6618ad71a8db78e22aa9f5dfcbf9c761ac8e0f287f7d1924fa12329ce2ffb07f" exitCode=0
Nov 02 12:44:17.563712 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:17.554691    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c79cde-openshift-default-54dbf475f7-g56xt" event={"ID":"ea9a3b60-d2ac-4837-b979-2ea31f79cee0","Type":"ContainerDied","Data":"6618ad71a8db78e22aa9f5dfcbf9c761ac8e0f287f7d1924fa12329ce2ffb07f"}
Nov 02 12:44:17.563712 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:17.555435    4559 scope.go:122] "RemoveContainer" containerID="6618ad71a8db78e22aa9f5dfcbf9c761ac8e0f287f7d1924fa12329ce2ffb07f"
Nov 02 12:44:17.568719 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:17.567828    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:44:17.568719 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:17.567931    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:44:17.568719 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:17.567981    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:44:18.621378 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:18.621334    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c79cde-openshift-default-54dbf475f7-g56xt" event={"ID":"ea9a3b60-d2ac-4837-b979-2ea31f79cee0","Type":"ContainerStarted","Data":"a67e9c2c37d610837c24c172d688bf1a2a32f784482327da22bcbdef0c4a7f4c"}
Nov 02 12:44:19.492664 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:19.492131    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-c79cde-openshift-default-54dbf475f7-g56xt"
Nov 02 12:44:19.492664 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:19.492259    4559 patch_prober.go:29] interesting pod/e2e-api-b-c79cde-openshift-default-54dbf475f7-g56xt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body=
Nov 02 12:44:19.492664 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:19.492303    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c79cde-openshift-default-54dbf475f7-g56xt" podUID="ea9a3b60-d2ac-4837-b979-2ea31f79cee0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused"
Nov 02 12:44:19.703660 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:19.703014    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"]
Nov 02 12:44:20.492665 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:20.489989    4559 patch_prober.go:29] interesting pod/e2e-api-b-c79cde-openshift-default-54dbf475f7-g56xt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body=
Nov 02 12:44:20.492665 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:20.490049    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c79cde-openshift-default-54dbf475f7-g56xt" podUID="ea9a3b60-d2ac-4837-b979-2ea31f79cee0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused"
Nov 02 12:44:21.489664 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:21.489213    4559 patch_prober.go:29] interesting pod/e2e-api-b-c79cde-openshift-default-54dbf475f7-g56xt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body=
Nov 02 12:44:21.489664 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:21.489274    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c79cde-openshift-default-54dbf475f7-g56xt" podUID="ea9a3b60-d2ac-4837-b979-2ea31f79cee0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused"
Nov 02 12:44:22.020591 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:22.020553    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-9f7f01-5548c7f9b5-nq722"]
Nov 02 12:44:22.048650 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:22.048038    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-9f7f01-55c4f68b6c-s4sdn"]
Nov 02 12:44:22.515665 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:22.508754    4559 patch_prober.go:29] interesting pod/e2e-api-b-c79cde-openshift-default-54dbf475f7-g56xt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body=
Nov 02 12:44:22.515665 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:22.508854    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c79cde-openshift-default-54dbf475f7-g56xt" podUID="ea9a3b60-d2ac-4837-b979-2ea31f79cee0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused"
Nov 02 12:44:23.457666 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:23.456720    4559 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-mt-cd1386cb/e2e-lifecycle-model-cd1386cb-kserve-7d9d698845-2579r" podUID="572f58c8-cca9-48c0-99d3-0ebde6b82476" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:44:23.490074 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:23.490034    4559 patch_prober.go:29] interesting pod/e2e-api-b-c79cde-openshift-default-54dbf475f7-g56xt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body=
Nov 02 12:44:23.490353 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:23.490321    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c79cde-openshift-default-54dbf475f7-g56xt" podUID="ea9a3b60-d2ac-4837-b979-2ea31f79cee0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused"
Nov 02 12:44:24.016766 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:24.016720    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-61e341/premium-w1-61e341-kserve-746477b99-645g6" podUID="02ff21d3-5a37-493a-ab95-8e4784c29123" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:44:24.500062 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:24.500020    4559 patch_prober.go:29] interesting pod/e2e-api-b-c79cde-openshift-default-54dbf475f7-g56xt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body=
Nov 02 12:44:24.500433 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:24.500399    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c79cde-openshift-default-54dbf475f7-g56xt" podUID="ea9a3b60-d2ac-4837-b979-2ea31f79cee0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused"
Nov 02 12:44:24.598232 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:24.598172    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-61e341/distinct-w1-61e341-kserve-787cd566f4-cjj8s" podUID="6886ce28-5800-402d-b893-f92109e289a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:44:24.620686 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:24.598822    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843978/distinct-w4-843978-kserve-6fcf588858-4nrwx" podUID="82e5d483-e95b-43e6-9135-af960823427a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Nov 02 12:44:24.638331 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:24.638252    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-61e341/distinct-2-w1-61e341-kserve-954cf956f-n7jmg" podUID="c5e1a5f8-97b4-469d-be9a-eae7ca995283" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded"
Nov 02 12:44:24.901811 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:24.901710    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-cd1386cb/e2e-lifecycle-model-cd1386cb-kserve-7d9d698845-2579r" podUID="572f58c8-cca9-48c0-99d3-0ebde6b82476" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded"
Nov 02 12:44:25.022760 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:25.022715    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-61e341/premium-w1-61e341-kserve-746477b99-645g6" podUID="02ff21d3-5a37-493a-ab95-8e4784c29123" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:44:25.045662 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:25.024022    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-61e341/premium-w1-61e341-kserve-746477b99-645g6"
Nov 02 12:44:25.506666 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:25.492896    4559 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-843978/distinct-w4-843978-kserve-6fcf588858-4nrwx" podUID="82e5d483-e95b-43e6-9135-af960823427a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:44:25.518342 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:25.518116    4559 patch_prober.go:29] interesting pod/e2e-api-b-c79cde-openshift-default-54dbf475f7-g56xt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body=
Nov 02 12:44:25.518601 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:25.518569    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c79cde-openshift-default-54dbf475f7-g56xt" podUID="ea9a3b60-d2ac-4837-b979-2ea31f79cee0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused"
Nov 02 12:44:25.612557 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:25.599509    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843978/distinct-w4-843978-kserve-6fcf588858-4nrwx" podUID="82e5d483-e95b-43e6-9135-af960823427a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:44:25.612557 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:25.600183    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-843978/distinct-w4-843978-kserve-6fcf588858-4nrwx"
Nov 02 12:44:26.032834 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:26.032780    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-61e341/premium-w1-61e341-kserve-746477b99-645g6" podUID="02ff21d3-5a37-493a-ab95-8e4784c29123" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:44:26.096451 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:26.095265    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-cd1386cb/e2e-lifecycle-model-cd1386cb-kserve-7d9d698845-2579r" podUID="572f58c8-cca9-48c0-99d3-0ebde6b82476" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:44:26.142560 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:26.142526    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-843978/distinct-w4-843978-kserve-6fcf588858-4nrwx"
Nov 02 12:44:26.142885 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:26.142866    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-cd1386cb/e2e-lifecycle-model-cd1386cb-kserve-7d9d698845-2579r"
Nov 02 12:44:26.155572 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:26.155546    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-cd1386cb/e2e-lifecycle-model-cd1386cb-kserve-7d9d698845-2579r"
Nov 02 12:44:26.195300 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:26.195268    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-61e341/premium-w1-61e341-kserve-746477b99-645g6"
Nov 02 12:44:26.494036 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:26.494007    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-c79cde-openshift-default-54dbf475f7-g56xt"
Nov 02 12:44:26.494518 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:26.494496    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-c79cde-openshift-default-54dbf475f7-g56xt"
Nov 02 12:44:26.495172 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:26.495157    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-c79cde-openshift-default-54dbf475f7-g56xt"
Nov 02 12:44:27.027118 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:27.026705    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-9f7f01-5548c7f9b5-nq722" podUID="882d2459-4272-4f63-82d6-b8fe06253754" containerName="payload-processing" containerID="cri-o://b94287ff0c1a981508b67b95c08001274f7bbea9c4f7f9799bebaa2ce5dfe0fa" gracePeriod=25
Nov 02 12:44:27.054205 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:27.054113    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9f7f01-55c4f68b6c-s4sdn" podUID="d2f6353b-9297-4a5b-944f-52afb4380d25" containerName="payload-pre-processing" containerID="cri-o://443f348dea23bb996e5dddba1ffa83597c7c972fb31c8735516d8ecc89e3d05f" gracePeriod=25
Nov 02 12:44:27.748852 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:27.748823    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9f7f01-55c4f68b6c-s4sdn"
Nov 02 12:44:27.802729 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:27.802699    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-9f7f01-5548c7f9b5-nq722"
Nov 02 12:44:27.831583 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:27.831552    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/882d2459-4272-4f63-82d6-b8fe06253754-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/882d2459-4272-4f63-82d6-b8fe06253754-plugins-config-volume\") pod \"882d2459-4272-4f63-82d6-b8fe06253754\" (UID: \"882d2459-4272-4f63-82d6-b8fe06253754\") "
Nov 02 12:44:27.831809 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:27.831716    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/882d2459-4272-4f63-82d6-b8fe06253754-kube-api-access-kc4tt\" (UniqueName: \"kubernetes.io/projected/882d2459-4272-4f63-82d6-b8fe06253754-kube-api-access-kc4tt\") pod \"882d2459-4272-4f63-82d6-b8fe06253754\" (UID: \"882d2459-4272-4f63-82d6-b8fe06253754\") "
Nov 02 12:44:27.831918 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:27.831826    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2f6353b-9297-4a5b-944f-52afb4380d25-kube-api-access-zv4kj\" (UniqueName: \"kubernetes.io/projected/d2f6353b-9297-4a5b-944f-52afb4380d25-kube-api-access-zv4kj\") pod \"d2f6353b-9297-4a5b-944f-52afb4380d25\" (UID: \"d2f6353b-9297-4a5b-944f-52afb4380d25\") "
Nov 02 12:44:27.831990 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:27.831932    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2f6353b-9297-4a5b-944f-52afb4380d25-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2f6353b-9297-4a5b-944f-52afb4380d25-plugins-config-volume\") pod \"d2f6353b-9297-4a5b-944f-52afb4380d25\" (UID: \"d2f6353b-9297-4a5b-944f-52afb4380d25\") "
Nov 02 12:44:27.832049 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:27.832006    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/882d2459-4272-4f63-82d6-b8fe06253754-plugins-config-volume" pod "882d2459-4272-4f63-82d6-b8fe06253754" (UID: "882d2459-4272-4f63-82d6-b8fe06253754"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:44:27.832360 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:27.832334    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2f6353b-9297-4a5b-944f-52afb4380d25-plugins-config-volume" pod "d2f6353b-9297-4a5b-944f-52afb4380d25" (UID: "d2f6353b-9297-4a5b-944f-52afb4380d25"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:44:27.833128 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:27.833105    4559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2f6353b-9297-4a5b-944f-52afb4380d25-plugins-config-volume\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:44:27.833226 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:27.833134    4559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/882d2459-4272-4f63-82d6-b8fe06253754-plugins-config-volume\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:44:27.834025 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:27.834006    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2f6353b-9297-4a5b-944f-52afb4380d25-kube-api-access-zv4kj" pod "d2f6353b-9297-4a5b-944f-52afb4380d25" (UID: "d2f6353b-9297-4a5b-944f-52afb4380d25"). InnerVolumeSpecName "kube-api-access-zv4kj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:44:27.834338 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:27.834319    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/882d2459-4272-4f63-82d6-b8fe06253754-kube-api-access-kc4tt" pod "882d2459-4272-4f63-82d6-b8fe06253754" (UID: "882d2459-4272-4f63-82d6-b8fe06253754"). InnerVolumeSpecName "kube-api-access-kc4tt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:44:27.934775 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:27.934715    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kc4tt\" (UniqueName: \"kubernetes.io/projected/882d2459-4272-4f63-82d6-b8fe06253754-kube-api-access-kc4tt\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:44:27.934775 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:27.934746    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zv4kj\" (UniqueName: \"kubernetes.io/projected/d2f6353b-9297-4a5b-944f-52afb4380d25-kube-api-access-zv4kj\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:44:28.162849 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:28.162811    4559 generic.go:356] "Generic (PLEG): container finished" podID="d2f6353b-9297-4a5b-944f-52afb4380d25" containerID="443f348dea23bb996e5dddba1ffa83597c7c972fb31c8735516d8ecc89e3d05f" exitCode=0
Nov 02 12:44:28.163234 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:28.162851    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9f7f01-55c4f68b6c-s4sdn"
Nov 02 12:44:28.163234 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:28.162886    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9f7f01-55c4f68b6c-s4sdn" event={"ID":"d2f6353b-9297-4a5b-944f-52afb4380d25","Type":"ContainerDied","Data":"443f348dea23bb996e5dddba1ffa83597c7c972fb31c8735516d8ecc89e3d05f"}
Nov 02 12:44:28.163234 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:28.162940    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9f7f01-55c4f68b6c-s4sdn" event={"ID":"d2f6353b-9297-4a5b-944f-52afb4380d25","Type":"ContainerDied","Data":"0ed3b60232ee1171291b0fa80608077d644f37aff92514b2066aa1735c0f89cb"}
Nov 02 12:44:28.163234 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:28.162967    4559 scope.go:122] "RemoveContainer" containerID="443f348dea23bb996e5dddba1ffa83597c7c972fb31c8735516d8ecc89e3d05f"
Nov 02 12:44:28.164440 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:28.164420    4559 generic.go:356] "Generic (PLEG): container finished" podID="882d2459-4272-4f63-82d6-b8fe06253754" containerID="b94287ff0c1a981508b67b95c08001274f7bbea9c4f7f9799bebaa2ce5dfe0fa" exitCode=0
Nov 02 12:44:28.164524 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:28.164465    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-9f7f01-5548c7f9b5-nq722"
Nov 02 12:44:28.164524 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:28.164501    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-9f7f01-5548c7f9b5-nq722" event={"ID":"882d2459-4272-4f63-82d6-b8fe06253754","Type":"ContainerDied","Data":"b94287ff0c1a981508b67b95c08001274f7bbea9c4f7f9799bebaa2ce5dfe0fa"}
Nov 02 12:44:28.164600 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:28.164548    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-9f7f01-5548c7f9b5-nq722" event={"ID":"882d2459-4272-4f63-82d6-b8fe06253754","Type":"ContainerDied","Data":"bcc2117bf7d7b3060568e530993bc55679a8b4e7291b6920f2dd0b7547acb0f2"}
Nov 02 12:44:28.173999 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:28.173981    4559 scope.go:122] "RemoveContainer" containerID="443f348dea23bb996e5dddba1ffa83597c7c972fb31c8735516d8ecc89e3d05f"
Nov 02 12:44:28.174260 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:44:28.174242    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"443f348dea23bb996e5dddba1ffa83597c7c972fb31c8735516d8ecc89e3d05f\": container with ID starting with 443f348dea23bb996e5dddba1ffa83597c7c972fb31c8735516d8ecc89e3d05f not found: ID does not exist" containerID="443f348dea23bb996e5dddba1ffa83597c7c972fb31c8735516d8ecc89e3d05f"
Nov 02 12:44:28.174341 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:28.174273    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"443f348dea23bb996e5dddba1ffa83597c7c972fb31c8735516d8ecc89e3d05f"} err="failed to get container status \"443f348dea23bb996e5dddba1ffa83597c7c972fb31c8735516d8ecc89e3d05f\": rpc error: code = NotFound desc = could not find container \"443f348dea23bb996e5dddba1ffa83597c7c972fb31c8735516d8ecc89e3d05f\": container with ID starting with 443f348dea23bb996e5dddba1ffa83597c7c972fb31c8735516d8ecc89e3d05f not found: ID does not exist"
Nov 02 12:44:28.174341 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:28.174297    4559 scope.go:122] "RemoveContainer" containerID="b94287ff0c1a981508b67b95c08001274f7bbea9c4f7f9799bebaa2ce5dfe0fa"
Nov 02 12:44:28.183085 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:28.183069    4559 scope.go:122] "RemoveContainer" containerID="b94287ff0c1a981508b67b95c08001274f7bbea9c4f7f9799bebaa2ce5dfe0fa"
Nov 02 12:44:28.183359 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:44:28.183336    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b94287ff0c1a981508b67b95c08001274f7bbea9c4f7f9799bebaa2ce5dfe0fa\": container with ID starting with b94287ff0c1a981508b67b95c08001274f7bbea9c4f7f9799bebaa2ce5dfe0fa not found: ID does not exist" containerID="b94287ff0c1a981508b67b95c08001274f7bbea9c4f7f9799bebaa2ce5dfe0fa"
Nov 02 12:44:28.183409 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:28.183367    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b94287ff0c1a981508b67b95c08001274f7bbea9c4f7f9799bebaa2ce5dfe0fa"} err="failed to get container status \"b94287ff0c1a981508b67b95c08001274f7bbea9c4f7f9799bebaa2ce5dfe0fa\": rpc error: code = NotFound desc = could not find container \"b94287ff0c1a981508b67b95c08001274f7bbea9c4f7f9799bebaa2ce5dfe0fa\": container with ID starting with b94287ff0c1a981508b67b95c08001274f7bbea9c4f7f9799bebaa2ce5dfe0fa not found: ID does not exist"
Nov 02 12:44:28.224428 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:28.224400    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-9f7f01-5548c7f9b5-nq722"]
Nov 02 12:44:28.228171 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:28.228151    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-9f7f01-5548c7f9b5-nq722"]
Nov 02 12:44:28.255965 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:28.255944    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-9f7f01-55c4f68b6c-s4sdn"]
Nov 02 12:44:28.259122 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:28.259105    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-9f7f01-55c4f68b6c-s4sdn"]
Nov 02 12:44:29.894314 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:29.894283    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="882d2459-4272-4f63-82d6-b8fe06253754" path="/var/lib/kubelet/pods/882d2459-4272-4f63-82d6-b8fe06253754/volumes"
Nov 02 12:44:29.894857 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:29.894717    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2f6353b-9297-4a5b-944f-52afb4380d25" path="/var/lib/kubelet/pods/d2f6353b-9297-4a5b-944f-52afb4380d25/volumes"
Nov 02 12:44:30.123270 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:30.123234    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-97e08c-69f98f57c9-gb4vp"]
Nov 02 12:44:30.867338 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:30.867305    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-rxdxm"]
Nov 02 12:44:30.868308 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:30.868289    4559 cpu_manager.go:414] "containerMap: removing container" podUID="e7f08638-accd-4b0d-9657-652d7c8df22b" containerName="payload-pre-processing"
Nov 02 12:44:30.868364 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:30.868312    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7f08638-accd-4b0d-9657-652d7c8df22b" containerName="payload-pre-processing"
Nov 02 12:44:30.868364 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:30.868338    4559 cpu_manager.go:414] "containerMap: removing container" podUID="e51ff2f8-753e-4977-a523-90bca8a4ec2a" containerName="istio-proxy"
Nov 02 12:44:30.868364 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:30.868345    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e51ff2f8-753e-4977-a523-90bca8a4ec2a" containerName="istio-proxy"
Nov 02 12:44:30.868364 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:30.868359    4559 cpu_manager.go:414] "containerMap: removing container" podUID="d2f6353b-9297-4a5b-944f-52afb4380d25" containerName="payload-pre-processing"
Nov 02 12:44:30.868496 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:30.868367    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2f6353b-9297-4a5b-944f-52afb4380d25" containerName="payload-pre-processing"
Nov 02 12:44:30.868496 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:30.868401    4559 cpu_manager.go:414] "containerMap: removing container" podUID="882d2459-4272-4f63-82d6-b8fe06253754" containerName="payload-processing"
Nov 02 12:44:30.868496 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:30.868409    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="882d2459-4272-4f63-82d6-b8fe06253754" containerName="payload-processing"
Nov 02 12:44:30.868605 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:30.868594    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2f6353b-9297-4a5b-944f-52afb4380d25" containerName="payload-pre-processing"
Nov 02 12:44:30.868679 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:30.868612    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="882d2459-4272-4f63-82d6-b8fe06253754" containerName="payload-processing"
Nov 02 12:44:30.868679 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:30.868628    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7f08638-accd-4b0d-9657-652d7c8df22b" containerName="payload-pre-processing"
Nov 02 12:44:30.872730 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:30.872708    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-rxdxm"
Nov 02 12:44:30.901014 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:30.900984    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-f0cf7e-openshift-default-dockercfg-s488n\""
Nov 02 12:44:30.909189 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:30.909161    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-rxdxm"]
Nov 02 12:44:31.025665 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:31.010213    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2989390-a19e-408b-ac9e-ec01df76bf12-credential-socket\") pod \"e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-rxdxm\" (UID: \"c2989390-a19e-408b-ac9e-ec01df76bf12\") " pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-rxdxm"
Nov 02 12:44:31.025665 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:31.010313    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gvgx\" (UniqueName: \"kubernetes.io/projected/c2989390-a19e-408b-ac9e-ec01df76bf12-kube-api-access-9gvgx\") pod \"e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-rxdxm\" (UID: \"c2989390-a19e-408b-ac9e-ec01df76bf12\") " pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-rxdxm"
Nov 02 12:44:31.025665 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:31.010364    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2989390-a19e-408b-ac9e-ec01df76bf12-istio-podinfo\") pod \"e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-rxdxm\" (UID: \"c2989390-a19e-408b-ac9e-ec01df76bf12\") " pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-rxdxm"
Nov 02 12:44:31.025665 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:31.010431    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2989390-a19e-408b-ac9e-ec01df76bf12-istiod-ca-cert\") pod \"e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-rxdxm\" (UID: \"c2989390-a19e-408b-ac9e-ec01df76bf12\") " pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-rxdxm"
Nov 02 12:44:31.025665 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:31.010611    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2989390-a19e-408b-ac9e-ec01df76bf12-istio-token\") pod \"e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-rxdxm\" (UID: \"c2989390-a19e-408b-ac9e-ec01df76bf12\") " pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-rxdxm"
Nov 02 12:44:31.025665 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:31.010719    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2989390-a19e-408b-ac9e-ec01df76bf12-workload-certs\") pod \"e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-rxdxm\" (UID: \"c2989390-a19e-408b-ac9e-ec01df76bf12\") " pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-rxdxm"
Nov 02 12:44:31.025665 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:31.010815    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2989390-a19e-408b-ac9e-ec01df76bf12-workload-socket\") pod \"e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-rxdxm\" (UID: \"c2989390-a19e-408b-ac9e-ec01df76bf12\") " pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-rxdxm"
Nov 02 12:44:31.025665 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:31.010867    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2989390-a19e-408b-ac9e-ec01df76bf12-istio-envoy\") pod \"e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-rxdxm\" (UID: \"c2989390-a19e-408b-ac9e-ec01df76bf12\") " pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-rxdxm"
Nov 02 12:44:31.025665 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:31.010917    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2989390-a19e-408b-ac9e-ec01df76bf12-istio-data\") pod \"e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-rxdxm\" (UID: \"c2989390-a19e-408b-ac9e-ec01df76bf12\") " pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-rxdxm"
Nov 02 12:44:31.118732 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:31.118144    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2989390-a19e-408b-ac9e-ec01df76bf12-credential-socket\") pod \"e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-rxdxm\" (UID: \"c2989390-a19e-408b-ac9e-ec01df76bf12\") " pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-rxdxm"
Nov 02 12:44:31.118732 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:31.118203    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gvgx\" (UniqueName: \"kubernetes.io/projected/c2989390-a19e-408b-ac9e-ec01df76bf12-kube-api-access-9gvgx\") pod \"e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-rxdxm\" (UID: \"c2989390-a19e-408b-ac9e-ec01df76bf12\") " pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-rxdxm"
Nov 02 12:44:31.118732 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:31.118245    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2989390-a19e-408b-ac9e-ec01df76bf12-istio-podinfo\") pod \"e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-rxdxm\" (UID: \"c2989390-a19e-408b-ac9e-ec01df76bf12\") " pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-rxdxm"
Nov 02 12:44:31.118732 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:31.118301    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2989390-a19e-408b-ac9e-ec01df76bf12-istiod-ca-cert\") pod \"e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-rxdxm\" (UID: \"c2989390-a19e-408b-ac9e-ec01df76bf12\") " pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-rxdxm"
Nov 02 12:44:31.118732 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:31.118405    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2989390-a19e-408b-ac9e-ec01df76bf12-istio-token\") pod \"e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-rxdxm\" (UID: \"c2989390-a19e-408b-ac9e-ec01df76bf12\") " pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-rxdxm"
Nov 02 12:44:31.118732 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:31.118470    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2989390-a19e-408b-ac9e-ec01df76bf12-workload-certs\") pod \"e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-rxdxm\" (UID: \"c2989390-a19e-408b-ac9e-ec01df76bf12\") " pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-rxdxm"
Nov 02 12:44:31.118732 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:31.118534    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2989390-a19e-408b-ac9e-ec01df76bf12-workload-socket\") pod \"e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-rxdxm\" (UID: \"c2989390-a19e-408b-ac9e-ec01df76bf12\") " pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-rxdxm"
Nov 02 12:44:31.118732 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:31.118575    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2989390-a19e-408b-ac9e-ec01df76bf12-istio-envoy\") pod \"e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-rxdxm\" (UID: \"c2989390-a19e-408b-ac9e-ec01df76bf12\") " pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-rxdxm"
Nov 02 12:44:31.118732 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:31.118618    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2989390-a19e-408b-ac9e-ec01df76bf12-istio-data\") pod \"e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-rxdxm\" (UID: \"c2989390-a19e-408b-ac9e-ec01df76bf12\") " pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-rxdxm"
Nov 02 12:44:31.119315 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:31.119062    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2989390-a19e-408b-ac9e-ec01df76bf12-istio-data\") pod \"e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-rxdxm\" (UID: \"c2989390-a19e-408b-ac9e-ec01df76bf12\") " pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-rxdxm"
Nov 02 12:44:31.119315 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:31.119283    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2989390-a19e-408b-ac9e-ec01df76bf12-credential-socket\") pod \"e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-rxdxm\" (UID: \"c2989390-a19e-408b-ac9e-ec01df76bf12\") " pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-rxdxm"
Nov 02 12:44:31.122660 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:31.120167    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2989390-a19e-408b-ac9e-ec01df76bf12-workload-certs\") pod \"e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-rxdxm\" (UID: \"c2989390-a19e-408b-ac9e-ec01df76bf12\") " pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-rxdxm"
Nov 02 12:44:31.122660 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:31.120882    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2989390-a19e-408b-ac9e-ec01df76bf12-workload-socket\") pod \"e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-rxdxm\" (UID: \"c2989390-a19e-408b-ac9e-ec01df76bf12\") " pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-rxdxm"
Nov 02 12:44:31.135760 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:31.134328    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2989390-a19e-408b-ac9e-ec01df76bf12-istio-podinfo\") pod \"e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-rxdxm\" (UID: \"c2989390-a19e-408b-ac9e-ec01df76bf12\") " pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-rxdxm"
Nov 02 12:44:31.135760 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:31.134532    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2989390-a19e-408b-ac9e-ec01df76bf12-istiod-ca-cert\") pod \"e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-rxdxm\" (UID: \"c2989390-a19e-408b-ac9e-ec01df76bf12\") " pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-rxdxm"
Nov 02 12:44:31.168799 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:31.168211    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2989390-a19e-408b-ac9e-ec01df76bf12-istio-envoy\") pod \"e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-rxdxm\" (UID: \"c2989390-a19e-408b-ac9e-ec01df76bf12\") " pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-rxdxm"
Nov 02 12:44:31.183441 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:31.183401    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gvgx\" (UniqueName: \"kubernetes.io/projected/c2989390-a19e-408b-ac9e-ec01df76bf12-kube-api-access-9gvgx\") pod \"e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-rxdxm\" (UID: \"c2989390-a19e-408b-ac9e-ec01df76bf12\") " pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-rxdxm"
Nov 02 12:44:31.186481 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:31.186421    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2989390-a19e-408b-ac9e-ec01df76bf12-istio-token\") pod \"e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-rxdxm\" (UID: \"c2989390-a19e-408b-ac9e-ec01df76bf12\") " pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-rxdxm"
Nov 02 12:44:31.196191 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:31.195956    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-rxdxm"
Nov 02 12:44:31.899445 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:44:31.899418    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2989390_a19e_408b_ac9e_ec01df76bf12.slice/crio-32b8b7203ed9d02aeec1b4d4edd3c9fe6d27547f909ff1e04b792df4cfc5d3d1 WatchSource:0}: Error finding container 32b8b7203ed9d02aeec1b4d4edd3c9fe6d27547f909ff1e04b792df4cfc5d3d1: Status 404 returned error can't find the container with id 32b8b7203ed9d02aeec1b4d4edd3c9fe6d27547f909ff1e04b792df4cfc5d3d1
Nov 02 12:44:31.902208 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:31.902168    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:44:31.902603 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:31.902250    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:44:31.902603 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:31.902296    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:44:31.903816 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:31.903795    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-rxdxm"]
Nov 02 12:44:32.291618 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:32.291577    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-rxdxm" event={"ID":"c2989390-a19e-408b-ac9e-ec01df76bf12","Type":"ContainerStarted","Data":"0eafe970aeb286d1e6ba5e03bba870e417ca71634a3d1606425969ba2fad7d3e"}
Nov 02 12:44:32.291815 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:32.291675    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-rxdxm" event={"ID":"c2989390-a19e-408b-ac9e-ec01df76bf12","Type":"ContainerStarted","Data":"32b8b7203ed9d02aeec1b4d4edd3c9fe6d27547f909ff1e04b792df4cfc5d3d1"}
Nov 02 12:44:32.341594 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:32.341536    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-rxdxm" podStartSLOduration=2.341523531 podStartE2EDuration="2.341523531s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:44:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:44:32.337764834 +0000 UTC m=+1068.593163975" watchObservedRunningTime="2026-11-02 12:44:32.341523531 +0000 UTC m=+1068.596922626"
Nov 02 12:44:33.205266 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:33.205231    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-rxdxm"
Nov 02 12:44:33.210539 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:33.210517    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-rxdxm"
Nov 02 12:44:33.299762 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:33.299730    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-rxdxm"
Nov 02 12:44:33.300680 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:33.300625    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-rxdxm"
Nov 02 12:44:35.123778 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:35.123744    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-97e08c-69f98f57c9-gb4vp" podUID="9ee37559-86ab-44e9-b50f-e360036fa07c" containerName="payload-processing" containerID="cri-o://fba2eeb8842e0f52d7f056f99cc761089617123806bc8aff1dad1306ee8d180b" gracePeriod=25
Nov 02 12:44:35.319835 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:35.319621    4559 generic.go:356] "Generic (PLEG): container finished" podID="9ee37559-86ab-44e9-b50f-e360036fa07c" containerID="fba2eeb8842e0f52d7f056f99cc761089617123806bc8aff1dad1306ee8d180b" exitCode=0
Nov 02 12:44:35.319994 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:35.319832    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-97e08c-69f98f57c9-gb4vp" event={"ID":"9ee37559-86ab-44e9-b50f-e360036fa07c","Type":"ContainerDied","Data":"fba2eeb8842e0f52d7f056f99cc761089617123806bc8aff1dad1306ee8d180b"}
Nov 02 12:44:35.404272 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:35.404249    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-97e08c-69f98f57c9-gb4vp"
Nov 02 12:44:35.538086 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:35.538053    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ee37559-86ab-44e9-b50f-e360036fa07c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ee37559-86ab-44e9-b50f-e360036fa07c-plugins-config-volume\") pod \"9ee37559-86ab-44e9-b50f-e360036fa07c\" (UID: \"9ee37559-86ab-44e9-b50f-e360036fa07c\") "
Nov 02 12:44:35.538336 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:35.538322    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ee37559-86ab-44e9-b50f-e360036fa07c-kube-api-access-7wzgk\" (UniqueName: \"kubernetes.io/projected/9ee37559-86ab-44e9-b50f-e360036fa07c-kube-api-access-7wzgk\") pod \"9ee37559-86ab-44e9-b50f-e360036fa07c\" (UID: \"9ee37559-86ab-44e9-b50f-e360036fa07c\") "
Nov 02 12:44:35.538554 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:35.538530    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ee37559-86ab-44e9-b50f-e360036fa07c-plugins-config-volume" pod "9ee37559-86ab-44e9-b50f-e360036fa07c" (UID: "9ee37559-86ab-44e9-b50f-e360036fa07c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:44:35.538823 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:35.538803    4559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ee37559-86ab-44e9-b50f-e360036fa07c-plugins-config-volume\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:44:35.540350 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:35.540310    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ee37559-86ab-44e9-b50f-e360036fa07c-kube-api-access-7wzgk" pod "9ee37559-86ab-44e9-b50f-e360036fa07c" (UID: "9ee37559-86ab-44e9-b50f-e360036fa07c"). InnerVolumeSpecName "kube-api-access-7wzgk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:44:35.640739 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:35.640663    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7wzgk\" (UniqueName: \"kubernetes.io/projected/9ee37559-86ab-44e9-b50f-e360036fa07c-kube-api-access-7wzgk\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:44:35.858971 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:35.858935    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-cd1386cb-556c7759d7-fwckp"]
Nov 02 12:44:35.859594 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:35.859580    4559 cpu_manager.go:414] "containerMap: removing container" podUID="9ee37559-86ab-44e9-b50f-e360036fa07c" containerName="payload-processing"
Nov 02 12:44:35.859654 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:35.859596    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ee37559-86ab-44e9-b50f-e360036fa07c" containerName="payload-processing"
Nov 02 12:44:35.859824 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:35.859811    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ee37559-86ab-44e9-b50f-e360036fa07c" containerName="payload-processing"
Nov 02 12:44:35.863717 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:35.863698    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-cd1386cb-556c7759d7-fwckp"
Nov 02 12:44:35.879352 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:35.879330    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-cd1386cb\""
Nov 02 12:44:35.879468 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:35.879456    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-cd1386cb-dockercfg-qkk95\""
Nov 02 12:44:35.883606 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:35.883551    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-cd1386cb-556c7759d7-fwckp"]
Nov 02 12:44:35.938990 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:35.938953    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-cd1386cb-7b76784c55-g6lkv"]
Nov 02 12:44:35.951721 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:35.949953    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-cd1386cb-7b76784c55-g6lkv"
Nov 02 12:44:36.018937 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:36.018879    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cswbm\" (UniqueName: \"kubernetes.io/projected/0830a319-a5b3-4f40-a0ba-6e218c86466b-kube-api-access-cswbm\") pod \"payload-pre-processing-e2e-mt-cd1386cb-556c7759d7-fwckp\" (UID: \"0830a319-a5b3-4f40-a0ba-6e218c86466b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-cd1386cb-556c7759d7-fwckp"
Nov 02 12:44:36.049992 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:36.049955    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lm5lm\" (UniqueName: \"kubernetes.io/projected/d1eaa8f7-ef7a-47b0-b7cc-6100c6e858e8-kube-api-access-lm5lm\") pod \"payload-processing-e2e-mt-cd1386cb-7b76784c55-g6lkv\" (UID: \"d1eaa8f7-ef7a-47b0-b7cc-6100c6e858e8\") " pod="openshift-ingress/payload-processing-e2e-mt-cd1386cb-7b76784c55-g6lkv"
Nov 02 12:44:36.050302 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:36.050264    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0830a319-a5b3-4f40-a0ba-6e218c86466b-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-cd1386cb-556c7759d7-fwckp\" (UID: \"0830a319-a5b3-4f40-a0ba-6e218c86466b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-cd1386cb-556c7759d7-fwckp"
Nov 02 12:44:36.050520 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:36.050504    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1eaa8f7-ef7a-47b0-b7cc-6100c6e858e8-plugins-config-volume\") pod \"payload-processing-e2e-mt-cd1386cb-7b76784c55-g6lkv\" (UID: \"d1eaa8f7-ef7a-47b0-b7cc-6100c6e858e8\") " pod="openshift-ingress/payload-processing-e2e-mt-cd1386cb-7b76784c55-g6lkv"
Nov 02 12:44:36.157666 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:36.157334    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lm5lm\" (UniqueName: \"kubernetes.io/projected/d1eaa8f7-ef7a-47b0-b7cc-6100c6e858e8-kube-api-access-lm5lm\") pod \"payload-processing-e2e-mt-cd1386cb-7b76784c55-g6lkv\" (UID: \"d1eaa8f7-ef7a-47b0-b7cc-6100c6e858e8\") " pod="openshift-ingress/payload-processing-e2e-mt-cd1386cb-7b76784c55-g6lkv"
Nov 02 12:44:36.157666 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:36.157448    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0830a319-a5b3-4f40-a0ba-6e218c86466b-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-cd1386cb-556c7759d7-fwckp\" (UID: \"0830a319-a5b3-4f40-a0ba-6e218c86466b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-cd1386cb-556c7759d7-fwckp"
Nov 02 12:44:36.157666 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:36.157519    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1eaa8f7-ef7a-47b0-b7cc-6100c6e858e8-plugins-config-volume\") pod \"payload-processing-e2e-mt-cd1386cb-7b76784c55-g6lkv\" (UID: \"d1eaa8f7-ef7a-47b0-b7cc-6100c6e858e8\") " pod="openshift-ingress/payload-processing-e2e-mt-cd1386cb-7b76784c55-g6lkv"
Nov 02 12:44:36.158269 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:36.157713    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cswbm\" (UniqueName: \"kubernetes.io/projected/0830a319-a5b3-4f40-a0ba-6e218c86466b-kube-api-access-cswbm\") pod \"payload-pre-processing-e2e-mt-cd1386cb-556c7759d7-fwckp\" (UID: \"0830a319-a5b3-4f40-a0ba-6e218c86466b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-cd1386cb-556c7759d7-fwckp"
Nov 02 12:44:36.171727 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:36.159226    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0830a319-a5b3-4f40-a0ba-6e218c86466b-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-cd1386cb-556c7759d7-fwckp\" (UID: \"0830a319-a5b3-4f40-a0ba-6e218c86466b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-cd1386cb-556c7759d7-fwckp"
Nov 02 12:44:36.253708 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:36.247106    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1eaa8f7-ef7a-47b0-b7cc-6100c6e858e8-plugins-config-volume\") pod \"payload-processing-e2e-mt-cd1386cb-7b76784c55-g6lkv\" (UID: \"d1eaa8f7-ef7a-47b0-b7cc-6100c6e858e8\") " pod="openshift-ingress/payload-processing-e2e-mt-cd1386cb-7b76784c55-g6lkv"
Nov 02 12:44:36.272093 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:36.265029    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lm5lm\" (UniqueName: \"kubernetes.io/projected/d1eaa8f7-ef7a-47b0-b7cc-6100c6e858e8-kube-api-access-lm5lm\") pod \"payload-processing-e2e-mt-cd1386cb-7b76784c55-g6lkv\" (UID: \"d1eaa8f7-ef7a-47b0-b7cc-6100c6e858e8\") " pod="openshift-ingress/payload-processing-e2e-mt-cd1386cb-7b76784c55-g6lkv"
Nov 02 12:44:36.297688 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:36.292759    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cswbm\" (UniqueName: \"kubernetes.io/projected/0830a319-a5b3-4f40-a0ba-6e218c86466b-kube-api-access-cswbm\") pod \"payload-pre-processing-e2e-mt-cd1386cb-556c7759d7-fwckp\" (UID: \"0830a319-a5b3-4f40-a0ba-6e218c86466b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-cd1386cb-556c7759d7-fwckp"
Nov 02 12:44:36.303588 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:36.300699    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-cd1386cb-7b76784c55-g6lkv"]
Nov 02 12:44:36.393682 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:36.390058    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-cd1386cb-7b76784c55-g6lkv"
Nov 02 12:44:36.408678 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:36.403067    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-97e08c-69f98f57c9-gb4vp" event={"ID":"9ee37559-86ab-44e9-b50f-e360036fa07c","Type":"ContainerDied","Data":"a047efaec410b90e764e2fae189da7be01ad3d59837179715b952930136b4d2a"}
Nov 02 12:44:36.408678 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:36.403148    4559 scope.go:122] "RemoveContainer" containerID="fba2eeb8842e0f52d7f056f99cc761089617123806bc8aff1dad1306ee8d180b"
Nov 02 12:44:36.408678 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:36.403380    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-97e08c-69f98f57c9-gb4vp"
Nov 02 12:44:36.542710 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:36.539440    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-cd1386cb-556c7759d7-fwckp"
Nov 02 12:44:36.700980 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:36.700941    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-97e08c-69f98f57c9-gb4vp"]
Nov 02 12:44:36.702298 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:36.702252    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-97e08c-69f98f57c9-gb4vp"]
Nov 02 12:44:37.193383 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:44:37.174759    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1eaa8f7_ef7a_47b0_b7cc_6100c6e858e8.slice/crio-f6fc474c12de05049adf8624fdaf82c2e700b80f02808800e6e38cf416fac93b WatchSource:0}: Error finding container f6fc474c12de05049adf8624fdaf82c2e700b80f02808800e6e38cf416fac93b: Status 404 returned error can't find the container with id f6fc474c12de05049adf8624fdaf82c2e700b80f02808800e6e38cf416fac93b
Nov 02 12:44:37.318193 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:37.303552    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-cd1386cb-7b76784c55-g6lkv"]
Nov 02 12:44:37.382069 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:37.374528    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-cd1386cb-556c7759d7-fwckp"]
Nov 02 12:44:37.420660 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:37.420112    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-cd1386cb-7b76784c55-g6lkv" event={"ID":"d1eaa8f7-ef7a-47b0-b7cc-6100c6e858e8","Type":"ContainerStarted","Data":"f6fc474c12de05049adf8624fdaf82c2e700b80f02808800e6e38cf416fac93b"}
Nov 02 12:44:37.423085 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:37.422563    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-cd1386cb-556c7759d7-fwckp" event={"ID":"0830a319-a5b3-4f40-a0ba-6e218c86466b","Type":"ContainerStarted","Data":"97e76d710393993783395dd2295bb8f1c9ddd04b4650241d8acbe24dc554ee66"}
Nov 02 12:44:38.113656 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:38.113530    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ee37559-86ab-44e9-b50f-e360036fa07c" path="/var/lib/kubelet/pods/9ee37559-86ab-44e9-b50f-e360036fa07c/volumes"
Nov 02 12:44:38.440965 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:38.439827    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-cd1386cb-7b76784c55-g6lkv" event={"ID":"d1eaa8f7-ef7a-47b0-b7cc-6100c6e858e8","Type":"ContainerStarted","Data":"a102c76363df6fd1e22f21f30d378635f9008b72393bcad37083a1fbadcc0c7f"}
Nov 02 12:44:38.440965 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:38.440157    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-cd1386cb-7b76784c55-g6lkv"
Nov 02 12:44:38.443208 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:38.443181    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-cd1386cb-556c7759d7-fwckp" event={"ID":"0830a319-a5b3-4f40-a0ba-6e218c86466b","Type":"ContainerStarted","Data":"da99a43b4b2f08c38e21730f80c7fa736947bcdb08560df1027af6754ceadc5e"}
Nov 02 12:44:38.443538 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:38.443514    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-cd1386cb-556c7759d7-fwckp"
Nov 02 12:44:38.538660 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:38.538165    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-cd1386cb-7b76784c55-g6lkv" podStartSLOduration=3.538143596 podStartE2EDuration="3.538143596s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:44:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:44:38.505626083 +0000 UTC m=+1074.761025179" watchObservedRunningTime="2026-11-02 12:44:38.538143596 +0000 UTC m=+1074.793542692"
Nov 02 12:44:38.601671 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:38.598795    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-cd1386cb-556c7759d7-fwckp" podStartSLOduration=3.598729166 podStartE2EDuration="3.598729166s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:44:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:44:38.597835874 +0000 UTC m=+1074.853234970" watchObservedRunningTime="2026-11-02 12:44:38.598729166 +0000 UTC m=+1074.854128275"
Nov 02 12:44:38.988518 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:38.988454    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-9f7f01-openshift-default-64d8f9568c-pw8tx"]
Nov 02 12:44:38.988981 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:38.988946    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-9f7f01-openshift-default-64d8f9568c-pw8tx" podUID="26b966b3-2043-474a-95aa-3037982c3fc4" containerName="istio-proxy" containerID="cri-o://2e2bbe96217d9f6965c75c614cb63495b7c0e945cf5dc32aca4c7557f46e8f5f" gracePeriod=30
Nov 02 12:44:42.396620 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:42.396585    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-cd1386cb-b4fb76986-9zgpp"]
Nov 02 12:44:42.401952 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:42.401936    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cd1386cb-b4fb76986-9zgpp"
Nov 02 12:44:42.421200 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:42.421175    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-cd1386cb-b4fb76986-9zgpp"]
Nov 02 12:44:42.429127 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:42.429101    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-cd1386cb-serving-cert\""
Nov 02 12:44:42.495434 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:42.495394    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bd63532-f2d2-4317-a46a-955025664a34-openshift-service-ca\") pod \"maas-api-e2e-mt-cd1386cb-b4fb76986-9zgpp\" (UID: \"6bd63532-f2d2-4317-a46a-955025664a34\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cd1386cb-b4fb76986-9zgpp"
Nov 02 12:44:42.495593 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:42.495545    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6bd63532-f2d2-4317-a46a-955025664a34-metrics-certs\") pod \"maas-api-e2e-mt-cd1386cb-b4fb76986-9zgpp\" (UID: \"6bd63532-f2d2-4317-a46a-955025664a34\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cd1386cb-b4fb76986-9zgpp"
Nov 02 12:44:42.495593 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:42.495573    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6bd63532-f2d2-4317-a46a-955025664a34-maas-api-tls\") pod \"maas-api-e2e-mt-cd1386cb-b4fb76986-9zgpp\" (UID: \"6bd63532-f2d2-4317-a46a-955025664a34\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cd1386cb-b4fb76986-9zgpp"
Nov 02 12:44:42.495716 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:42.495678    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdvjf\" (UniqueName: \"kubernetes.io/projected/6bd63532-f2d2-4317-a46a-955025664a34-kube-api-access-gdvjf\") pod \"maas-api-e2e-mt-cd1386cb-b4fb76986-9zgpp\" (UID: \"6bd63532-f2d2-4317-a46a-955025664a34\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cd1386cb-b4fb76986-9zgpp"
Nov 02 12:44:42.597195 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:42.597164    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdvjf\" (UniqueName: \"kubernetes.io/projected/6bd63532-f2d2-4317-a46a-955025664a34-kube-api-access-gdvjf\") pod \"maas-api-e2e-mt-cd1386cb-b4fb76986-9zgpp\" (UID: \"6bd63532-f2d2-4317-a46a-955025664a34\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cd1386cb-b4fb76986-9zgpp"
Nov 02 12:44:42.597362 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:42.597345    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bd63532-f2d2-4317-a46a-955025664a34-openshift-service-ca\") pod \"maas-api-e2e-mt-cd1386cb-b4fb76986-9zgpp\" (UID: \"6bd63532-f2d2-4317-a46a-955025664a34\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cd1386cb-b4fb76986-9zgpp"
Nov 02 12:44:42.597509 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:42.597492    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6bd63532-f2d2-4317-a46a-955025664a34-metrics-certs\") pod \"maas-api-e2e-mt-cd1386cb-b4fb76986-9zgpp\" (UID: \"6bd63532-f2d2-4317-a46a-955025664a34\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cd1386cb-b4fb76986-9zgpp"
Nov 02 12:44:42.597547 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:42.597523    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6bd63532-f2d2-4317-a46a-955025664a34-maas-api-tls\") pod \"maas-api-e2e-mt-cd1386cb-b4fb76986-9zgpp\" (UID: \"6bd63532-f2d2-4317-a46a-955025664a34\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cd1386cb-b4fb76986-9zgpp"
Nov 02 12:44:42.598041 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:42.598018    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bd63532-f2d2-4317-a46a-955025664a34-openshift-service-ca\") pod \"maas-api-e2e-mt-cd1386cb-b4fb76986-9zgpp\" (UID: \"6bd63532-f2d2-4317-a46a-955025664a34\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cd1386cb-b4fb76986-9zgpp"
Nov 02 12:44:42.599968 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:42.599936    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6bd63532-f2d2-4317-a46a-955025664a34-metrics-certs\") pod \"maas-api-e2e-mt-cd1386cb-b4fb76986-9zgpp\" (UID: \"6bd63532-f2d2-4317-a46a-955025664a34\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cd1386cb-b4fb76986-9zgpp"
Nov 02 12:44:42.600055 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:42.600005    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6bd63532-f2d2-4317-a46a-955025664a34-maas-api-tls\") pod \"maas-api-e2e-mt-cd1386cb-b4fb76986-9zgpp\" (UID: \"6bd63532-f2d2-4317-a46a-955025664a34\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cd1386cb-b4fb76986-9zgpp"
Nov 02 12:44:42.675689 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:42.674314    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdvjf\" (UniqueName: \"kubernetes.io/projected/6bd63532-f2d2-4317-a46a-955025664a34-kube-api-access-gdvjf\") pod \"maas-api-e2e-mt-cd1386cb-b4fb76986-9zgpp\" (UID: \"6bd63532-f2d2-4317-a46a-955025664a34\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cd1386cb-b4fb76986-9zgpp"
Nov 02 12:44:42.729929 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:42.715553    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cd1386cb-b4fb76986-9zgpp"
Nov 02 12:44:43.092312 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:44:43.092285    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bd63532_f2d2_4317_a46a_955025664a34.slice/crio-b80a54d1ede6d26125ed30d6a0fa1bb9851a8d003d385fa8c8d62c8ee096be73 WatchSource:0}: Error finding container b80a54d1ede6d26125ed30d6a0fa1bb9851a8d003d385fa8c8d62c8ee096be73: Status 404 returned error can't find the container with id b80a54d1ede6d26125ed30d6a0fa1bb9851a8d003d385fa8c8d62c8ee096be73
Nov 02 12:44:43.093857 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:43.093833    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-cd1386cb-b4fb76986-9zgpp"]
Nov 02 12:44:43.481839 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:43.481816    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cd1386cb-b4fb76986-9zgpp" event={"ID":"6bd63532-f2d2-4317-a46a-955025664a34","Type":"ContainerStarted","Data":"b80a54d1ede6d26125ed30d6a0fa1bb9851a8d003d385fa8c8d62c8ee096be73"}
Nov 02 12:44:44.267314 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:44.267287    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-9f7f01-openshift-default-64d8f9568c-pw8tx"
Nov 02 12:44:44.352096 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:44.351462    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26b966b3-2043-474a-95aa-3037982c3fc4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26b966b3-2043-474a-95aa-3037982c3fc4-istio-data\") pod \"26b966b3-2043-474a-95aa-3037982c3fc4\" (UID: \"26b966b3-2043-474a-95aa-3037982c3fc4\") "
Nov 02 12:44:44.352096 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:44.351529    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26b966b3-2043-474a-95aa-3037982c3fc4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26b966b3-2043-474a-95aa-3037982c3fc4-workload-certs\") pod \"26b966b3-2043-474a-95aa-3037982c3fc4\" (UID: \"26b966b3-2043-474a-95aa-3037982c3fc4\") "
Nov 02 12:44:44.352096 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:44.351586    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26b966b3-2043-474a-95aa-3037982c3fc4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26b966b3-2043-474a-95aa-3037982c3fc4-workload-socket\") pod \"26b966b3-2043-474a-95aa-3037982c3fc4\" (UID: \"26b966b3-2043-474a-95aa-3037982c3fc4\") "
Nov 02 12:44:44.352096 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:44.351687    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26b966b3-2043-474a-95aa-3037982c3fc4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26b966b3-2043-474a-95aa-3037982c3fc4-istio-envoy\") pod \"26b966b3-2043-474a-95aa-3037982c3fc4\" (UID: \"26b966b3-2043-474a-95aa-3037982c3fc4\") "
Nov 02 12:44:44.352096 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:44.351728    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26b966b3-2043-474a-95aa-3037982c3fc4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26b966b3-2043-474a-95aa-3037982c3fc4-credential-socket\") pod \"26b966b3-2043-474a-95aa-3037982c3fc4\" (UID: \"26b966b3-2043-474a-95aa-3037982c3fc4\") "
Nov 02 12:44:44.352096 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:44.351790    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26b966b3-2043-474a-95aa-3037982c3fc4-kube-api-access-6hmdj\" (UniqueName: \"kubernetes.io/projected/26b966b3-2043-474a-95aa-3037982c3fc4-kube-api-access-6hmdj\") pod \"26b966b3-2043-474a-95aa-3037982c3fc4\" (UID: \"26b966b3-2043-474a-95aa-3037982c3fc4\") "
Nov 02 12:44:44.352096 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:44.351905    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26b966b3-2043-474a-95aa-3037982c3fc4-workload-certs" pod "26b966b3-2043-474a-95aa-3037982c3fc4" (UID: "26b966b3-2043-474a-95aa-3037982c3fc4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:44:44.353266 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:44.353235    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26b966b3-2043-474a-95aa-3037982c3fc4-istio-data" pod "26b966b3-2043-474a-95aa-3037982c3fc4" (UID: "26b966b3-2043-474a-95aa-3037982c3fc4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:44:44.353424 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:44.351961    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26b966b3-2043-474a-95aa-3037982c3fc4-istio-token\" (UniqueName: \"kubernetes.io/projected/26b966b3-2043-474a-95aa-3037982c3fc4-istio-token\") pod \"26b966b3-2043-474a-95aa-3037982c3fc4\" (UID: \"26b966b3-2043-474a-95aa-3037982c3fc4\") "
Nov 02 12:44:44.353580 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:44.353551    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26b966b3-2043-474a-95aa-3037982c3fc4-credential-socket" pod "26b966b3-2043-474a-95aa-3037982c3fc4" (UID: "26b966b3-2043-474a-95aa-3037982c3fc4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:44:44.353720 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:44.353700    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/26b966b3-2043-474a-95aa-3037982c3fc4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26b966b3-2043-474a-95aa-3037982c3fc4-istio-podinfo\") pod \"26b966b3-2043-474a-95aa-3037982c3fc4\" (UID: \"26b966b3-2043-474a-95aa-3037982c3fc4\") "
Nov 02 12:44:44.353881 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:44.353859    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26b966b3-2043-474a-95aa-3037982c3fc4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26b966b3-2043-474a-95aa-3037982c3fc4-istiod-ca-cert\") pod \"26b966b3-2043-474a-95aa-3037982c3fc4\" (UID: \"26b966b3-2043-474a-95aa-3037982c3fc4\") "
Nov 02 12:44:44.354409 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:44.353634    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26b966b3-2043-474a-95aa-3037982c3fc4-workload-socket" pod "26b966b3-2043-474a-95aa-3037982c3fc4" (UID: "26b966b3-2043-474a-95aa-3037982c3fc4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:44:44.354498 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:44.354314    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26b966b3-2043-474a-95aa-3037982c3fc4-istiod-ca-cert" pod "26b966b3-2043-474a-95aa-3037982c3fc4" (UID: "26b966b3-2043-474a-95aa-3037982c3fc4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:44:44.355309 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:44.355272    4559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26b966b3-2043-474a-95aa-3037982c3fc4-istiod-ca-cert\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:44:44.355414 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:44.355313    4559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26b966b3-2043-474a-95aa-3037982c3fc4-istio-data\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:44:44.355414 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:44.355328    4559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26b966b3-2043-474a-95aa-3037982c3fc4-workload-certs\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:44:44.355414 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:44.355340    4559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26b966b3-2043-474a-95aa-3037982c3fc4-workload-socket\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:44:44.355414 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:44.355360    4559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26b966b3-2043-474a-95aa-3037982c3fc4-credential-socket\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:44:44.381711 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:44.381683    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26b966b3-2043-474a-95aa-3037982c3fc4-istio-envoy" pod "26b966b3-2043-474a-95aa-3037982c3fc4" (UID: "26b966b3-2043-474a-95aa-3037982c3fc4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:44:44.381921 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:44.381882    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26b966b3-2043-474a-95aa-3037982c3fc4-kube-api-access-6hmdj" pod "26b966b3-2043-474a-95aa-3037982c3fc4" (UID: "26b966b3-2043-474a-95aa-3037982c3fc4"). InnerVolumeSpecName "kube-api-access-6hmdj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:44:44.382128 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:44.382104    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26b966b3-2043-474a-95aa-3037982c3fc4-istio-token" pod "26b966b3-2043-474a-95aa-3037982c3fc4" (UID: "26b966b3-2043-474a-95aa-3037982c3fc4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:44:44.384573 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:44.384541    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/26b966b3-2043-474a-95aa-3037982c3fc4-istio-podinfo" pod "26b966b3-2043-474a-95aa-3037982c3fc4" (UID: "26b966b3-2043-474a-95aa-3037982c3fc4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 12:44:44.460426 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:44.460352    4559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26b966b3-2043-474a-95aa-3037982c3fc4-istio-token\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:44:44.460685 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:44.460664    4559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26b966b3-2043-474a-95aa-3037982c3fc4-istio-podinfo\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:44:44.460828 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:44.460810    4559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26b966b3-2043-474a-95aa-3037982c3fc4-istio-envoy\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:44:44.461145 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:44.460991    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6hmdj\" (UniqueName: \"kubernetes.io/projected/26b966b3-2043-474a-95aa-3037982c3fc4-kube-api-access-6hmdj\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:44:44.535999 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:44.534721    4559 generic.go:356] "Generic (PLEG): container finished" podID="26b966b3-2043-474a-95aa-3037982c3fc4" containerID="2e2bbe96217d9f6965c75c614cb63495b7c0e945cf5dc32aca4c7557f46e8f5f" exitCode=0
Nov 02 12:44:44.535999 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:44.535009    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-9f7f01-openshift-default-64d8f9568c-pw8tx" event={"ID":"26b966b3-2043-474a-95aa-3037982c3fc4","Type":"ContainerDied","Data":"2e2bbe96217d9f6965c75c614cb63495b7c0e945cf5dc32aca4c7557f46e8f5f"}
Nov 02 12:44:44.535999 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:44.535097    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-9f7f01-openshift-default-64d8f9568c-pw8tx" event={"ID":"26b966b3-2043-474a-95aa-3037982c3fc4","Type":"ContainerDied","Data":"2124fa563fc8e9d6e80ec77eacfae49a93490f54ad34b6945a077413d5447cc8"}
Nov 02 12:44:44.535999 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:44.535127    4559 scope.go:122] "RemoveContainer" containerID="2e2bbe96217d9f6965c75c614cb63495b7c0e945cf5dc32aca4c7557f46e8f5f"
Nov 02 12:44:44.535999 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:44.535289    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-9f7f01-openshift-default-64d8f9568c-pw8tx"
Nov 02 12:44:44.539387 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:44.539279    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cd1386cb-b4fb76986-9zgpp" event={"ID":"6bd63532-f2d2-4317-a46a-955025664a34","Type":"ContainerStarted","Data":"659029b930309e951d85b60a7d80edb93af960d34bcd4d644c1662e5949ca7d0"}
Nov 02 12:44:44.540629 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:44.540611    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cd1386cb-b4fb76986-9zgpp"
Nov 02 12:44:44.556460 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:44.556439    4559 scope.go:122] "RemoveContainer" containerID="67baa54012d28b2eded48236c9ab1b20b43f10ba74555771f760756c4dba1ec7"
Nov 02 12:44:44.565749 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:44.565729    4559 scope.go:122] "RemoveContainer" containerID="2e2bbe96217d9f6965c75c614cb63495b7c0e945cf5dc32aca4c7557f46e8f5f"
Nov 02 12:44:44.566031 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:44:44.566012    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e2bbe96217d9f6965c75c614cb63495b7c0e945cf5dc32aca4c7557f46e8f5f\": container with ID starting with 2e2bbe96217d9f6965c75c614cb63495b7c0e945cf5dc32aca4c7557f46e8f5f not found: ID does not exist" containerID="2e2bbe96217d9f6965c75c614cb63495b7c0e945cf5dc32aca4c7557f46e8f5f"
Nov 02 12:44:44.566091 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:44.566040    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e2bbe96217d9f6965c75c614cb63495b7c0e945cf5dc32aca4c7557f46e8f5f"} err="failed to get container status \"2e2bbe96217d9f6965c75c614cb63495b7c0e945cf5dc32aca4c7557f46e8f5f\": rpc error: code = NotFound desc = could not find container \"2e2bbe96217d9f6965c75c614cb63495b7c0e945cf5dc32aca4c7557f46e8f5f\": container with ID starting with 2e2bbe96217d9f6965c75c614cb63495b7c0e945cf5dc32aca4c7557f46e8f5f not found: ID does not exist"
Nov 02 12:44:44.566091 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:44.566057    4559 scope.go:122] "RemoveContainer" containerID="67baa54012d28b2eded48236c9ab1b20b43f10ba74555771f760756c4dba1ec7"
Nov 02 12:44:44.566292 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:44:44.566272    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67baa54012d28b2eded48236c9ab1b20b43f10ba74555771f760756c4dba1ec7\": container with ID starting with 67baa54012d28b2eded48236c9ab1b20b43f10ba74555771f760756c4dba1ec7 not found: ID does not exist" containerID="67baa54012d28b2eded48236c9ab1b20b43f10ba74555771f760756c4dba1ec7"
Nov 02 12:44:44.566342 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:44.566299    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67baa54012d28b2eded48236c9ab1b20b43f10ba74555771f760756c4dba1ec7"} err="failed to get container status \"67baa54012d28b2eded48236c9ab1b20b43f10ba74555771f760756c4dba1ec7\": rpc error: code = NotFound desc = could not find container \"67baa54012d28b2eded48236c9ab1b20b43f10ba74555771f760756c4dba1ec7\": container with ID starting with 67baa54012d28b2eded48236c9ab1b20b43f10ba74555771f760756c4dba1ec7 not found: ID does not exist"
Nov 02 12:44:44.599973 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:44.599923    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cd1386cb-b4fb76986-9zgpp" podStartSLOduration=2.258933284 podStartE2EDuration="2.599906297s" totalImagesPullingTime="340.973013ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:44:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:44:44.597331913 +0000 UTC m=+1080.852731006" watchObservedRunningTime="2026-11-02 12:44:44.599906297 +0000 UTC m=+1080.855305392"
Nov 02 12:44:44.639183 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:44.639160    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-9f7f01-openshift-default-64d8f9568c-pw8tx"]
Nov 02 12:44:44.644821 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:44.644797    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-9f7f01-openshift-default-64d8f9568c-pw8tx"]
Nov 02 12:44:45.086907 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:44:45.086376    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqcmj" containerName="extractor"
Nov 02 12:44:45.563234 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:45.563204    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-c79cde-789d76c79-cnjxb"]
Nov 02 12:44:45.601432 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:45.601397    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-c79cde-76d5b754d7-2vhqc"]
Nov 02 12:44:45.895047 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:45.894975    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26b966b3-2043-474a-95aa-3037982c3fc4" path="/var/lib/kubelet/pods/26b966b3-2043-474a-95aa-3037982c3fc4/volumes"
Nov 02 12:44:49.456080 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:49.456048    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-cd1386cb-556c7759d7-fwckp"
Nov 02 12:44:49.456597 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:49.456135    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-cd1386cb-7b76784c55-g6lkv"
Nov 02 12:44:50.563595 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:50.563556    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-c79cde-789d76c79-cnjxb" podUID="730c9cc3-ba43-4ed4-8769-13999c58cf13" containerName="payload-processing" containerID="cri-o://3f6b2633003d3ce771ffc57324358907b056216f6282579d876e71b663dedcda" gracePeriod=25
Nov 02 12:44:50.602456 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:50.602419    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c79cde-76d5b754d7-2vhqc" podUID="9077a191-58b9-4dbd-aa96-b87eaa5398bf" containerName="payload-pre-processing" containerID="cri-o://33300948ccd06393ac439495af4550576623e716490d238d50fc5df3dc78af42" gracePeriod=25
Nov 02 12:44:50.889852 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:50.889822    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-c79cde-789d76c79-cnjxb"
Nov 02 12:44:50.935878 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:50.935852    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c79cde-76d5b754d7-2vhqc"
Nov 02 12:44:50.955453 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:50.955397    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/730c9cc3-ba43-4ed4-8769-13999c58cf13-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/730c9cc3-ba43-4ed4-8769-13999c58cf13-plugins-config-volume\") pod \"730c9cc3-ba43-4ed4-8769-13999c58cf13\" (UID: \"730c9cc3-ba43-4ed4-8769-13999c58cf13\") "
Nov 02 12:44:50.956098 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:50.956050    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/730c9cc3-ba43-4ed4-8769-13999c58cf13-kube-api-access-l4zrc\" (UniqueName: \"kubernetes.io/projected/730c9cc3-ba43-4ed4-8769-13999c58cf13-kube-api-access-l4zrc\") pod \"730c9cc3-ba43-4ed4-8769-13999c58cf13\" (UID: \"730c9cc3-ba43-4ed4-8769-13999c58cf13\") "
Nov 02 12:44:50.956768 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:50.956740    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/730c9cc3-ba43-4ed4-8769-13999c58cf13-plugins-config-volume" pod "730c9cc3-ba43-4ed4-8769-13999c58cf13" (UID: "730c9cc3-ba43-4ed4-8769-13999c58cf13"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:44:50.957699 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:50.957679    4559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/730c9cc3-ba43-4ed4-8769-13999c58cf13-plugins-config-volume\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:44:50.959160 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:50.959133    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/730c9cc3-ba43-4ed4-8769-13999c58cf13-kube-api-access-l4zrc" pod "730c9cc3-ba43-4ed4-8769-13999c58cf13" (UID: "730c9cc3-ba43-4ed4-8769-13999c58cf13"). InnerVolumeSpecName "kube-api-access-l4zrc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:44:51.058443 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:51.058414    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9077a191-58b9-4dbd-aa96-b87eaa5398bf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9077a191-58b9-4dbd-aa96-b87eaa5398bf-plugins-config-volume\") pod \"9077a191-58b9-4dbd-aa96-b87eaa5398bf\" (UID: \"9077a191-58b9-4dbd-aa96-b87eaa5398bf\") "
Nov 02 12:44:51.058619 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:51.058560    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9077a191-58b9-4dbd-aa96-b87eaa5398bf-kube-api-access-bq2ff\" (UniqueName: \"kubernetes.io/projected/9077a191-58b9-4dbd-aa96-b87eaa5398bf-kube-api-access-bq2ff\") pod \"9077a191-58b9-4dbd-aa96-b87eaa5398bf\" (UID: \"9077a191-58b9-4dbd-aa96-b87eaa5398bf\") "
Nov 02 12:44:51.058855 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:51.058833    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9077a191-58b9-4dbd-aa96-b87eaa5398bf-plugins-config-volume" pod "9077a191-58b9-4dbd-aa96-b87eaa5398bf" (UID: "9077a191-58b9-4dbd-aa96-b87eaa5398bf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:44:51.059184 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:51.059167    4559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9077a191-58b9-4dbd-aa96-b87eaa5398bf-plugins-config-volume\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:44:51.059224 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:51.059187    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l4zrc\" (UniqueName: \"kubernetes.io/projected/730c9cc3-ba43-4ed4-8769-13999c58cf13-kube-api-access-l4zrc\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:44:51.060519 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:51.060498    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9077a191-58b9-4dbd-aa96-b87eaa5398bf-kube-api-access-bq2ff" pod "9077a191-58b9-4dbd-aa96-b87eaa5398bf" (UID: "9077a191-58b9-4dbd-aa96-b87eaa5398bf"). InnerVolumeSpecName "kube-api-access-bq2ff". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:44:51.160320 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:51.160245    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bq2ff\" (UniqueName: \"kubernetes.io/projected/9077a191-58b9-4dbd-aa96-b87eaa5398bf-kube-api-access-bq2ff\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:44:51.606694 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:51.606663    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cd1386cb-b4fb76986-9zgpp"
Nov 02 12:44:51.623664 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:51.622067    4559 generic.go:356] "Generic (PLEG): container finished" podID="9077a191-58b9-4dbd-aa96-b87eaa5398bf" containerID="33300948ccd06393ac439495af4550576623e716490d238d50fc5df3dc78af42" exitCode=0
Nov 02 12:44:51.623664 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:51.622163    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c79cde-76d5b754d7-2vhqc" event={"ID":"9077a191-58b9-4dbd-aa96-b87eaa5398bf","Type":"ContainerDied","Data":"33300948ccd06393ac439495af4550576623e716490d238d50fc5df3dc78af42"}
Nov 02 12:44:51.623664 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:51.622214    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c79cde-76d5b754d7-2vhqc" event={"ID":"9077a191-58b9-4dbd-aa96-b87eaa5398bf","Type":"ContainerDied","Data":"5f3207dd88374acce49f9aee0787d843ee147d6b10cf8bbe17792ca869e90864"}
Nov 02 12:44:51.623664 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:51.622244    4559 scope.go:122] "RemoveContainer" containerID="33300948ccd06393ac439495af4550576623e716490d238d50fc5df3dc78af42"
Nov 02 12:44:51.623664 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:51.622422    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c79cde-76d5b754d7-2vhqc"
Nov 02 12:44:51.632729 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:51.632683    4559 generic.go:356] "Generic (PLEG): container finished" podID="730c9cc3-ba43-4ed4-8769-13999c58cf13" containerID="3f6b2633003d3ce771ffc57324358907b056216f6282579d876e71b663dedcda" exitCode=0
Nov 02 12:44:51.636922 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:51.636891    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-c79cde-789d76c79-cnjxb" event={"ID":"730c9cc3-ba43-4ed4-8769-13999c58cf13","Type":"ContainerDied","Data":"3f6b2633003d3ce771ffc57324358907b056216f6282579d876e71b663dedcda"}
Nov 02 12:44:51.637186 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:51.637146    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-c79cde-789d76c79-cnjxb" event={"ID":"730c9cc3-ba43-4ed4-8769-13999c58cf13","Type":"ContainerDied","Data":"0ba944fc5562971da3ac0cf0ded0f334fa31d5b8511e1d6695b74446e8c6acfd"}
Nov 02 12:44:51.637432 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:51.637407    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-c79cde-789d76c79-cnjxb"
Nov 02 12:44:51.711676 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:51.710977    4559 scope.go:122] "RemoveContainer" containerID="33300948ccd06393ac439495af4550576623e716490d238d50fc5df3dc78af42"
Nov 02 12:44:51.721618 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:44:51.719973    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33300948ccd06393ac439495af4550576623e716490d238d50fc5df3dc78af42\": container with ID starting with 33300948ccd06393ac439495af4550576623e716490d238d50fc5df3dc78af42 not found: ID does not exist" containerID="33300948ccd06393ac439495af4550576623e716490d238d50fc5df3dc78af42"
Nov 02 12:44:51.721618 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:51.720017    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33300948ccd06393ac439495af4550576623e716490d238d50fc5df3dc78af42"} err="failed to get container status \"33300948ccd06393ac439495af4550576623e716490d238d50fc5df3dc78af42\": rpc error: code = NotFound desc = could not find container \"33300948ccd06393ac439495af4550576623e716490d238d50fc5df3dc78af42\": container with ID starting with 33300948ccd06393ac439495af4550576623e716490d238d50fc5df3dc78af42 not found: ID does not exist"
Nov 02 12:44:51.721618 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:51.720041    4559 scope.go:122] "RemoveContainer" containerID="3f6b2633003d3ce771ffc57324358907b056216f6282579d876e71b663dedcda"
Nov 02 12:44:51.732588 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:51.732560    4559 scope.go:122] "RemoveContainer" containerID="3f6b2633003d3ce771ffc57324358907b056216f6282579d876e71b663dedcda"
Nov 02 12:44:51.733274 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:44:51.733253    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f6b2633003d3ce771ffc57324358907b056216f6282579d876e71b663dedcda\": container with ID starting with 3f6b2633003d3ce771ffc57324358907b056216f6282579d876e71b663dedcda not found: ID does not exist" containerID="3f6b2633003d3ce771ffc57324358907b056216f6282579d876e71b663dedcda"
Nov 02 12:44:51.733372 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:51.733281    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f6b2633003d3ce771ffc57324358907b056216f6282579d876e71b663dedcda"} err="failed to get container status \"3f6b2633003d3ce771ffc57324358907b056216f6282579d876e71b663dedcda\": rpc error: code = NotFound desc = could not find container \"3f6b2633003d3ce771ffc57324358907b056216f6282579d876e71b663dedcda\": container with ID starting with 3f6b2633003d3ce771ffc57324358907b056216f6282579d876e71b663dedcda not found: ID does not exist"
Nov 02 12:44:51.799738 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:51.799697    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-c79cde-76d5b754d7-2vhqc"]
Nov 02 12:44:51.813197 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:51.813171    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-c79cde-76d5b754d7-2vhqc"]
Nov 02 12:44:51.858159 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:51.858102    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-c79cde-789d76c79-cnjxb"]
Nov 02 12:44:51.871194 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:51.871174    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-c79cde-789d76c79-cnjxb"]
Nov 02 12:44:51.894742 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:51.894719    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="730c9cc3-ba43-4ed4-8769-13999c58cf13" path="/var/lib/kubelet/pods/730c9cc3-ba43-4ed4-8769-13999c58cf13/volumes"
Nov 02 12:44:51.895127 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:44:51.895113    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9077a191-58b9-4dbd-aa96-b87eaa5398bf" path="/var/lib/kubelet/pods/9077a191-58b9-4dbd-aa96-b87eaa5398bf/volumes"
Nov 02 12:45:00.228935 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:00.228521    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f0cf7e-29893725-22q6j"]
Nov 02 12:45:00.231676 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:00.229561    4559 cpu_manager.go:414] "containerMap: removing container" podUID="26b966b3-2043-474a-95aa-3037982c3fc4" containerName="istio-proxy"
Nov 02 12:45:00.231676 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:00.229584    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26b966b3-2043-474a-95aa-3037982c3fc4" containerName="istio-proxy"
Nov 02 12:45:00.231676 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:00.229610    4559 cpu_manager.go:414] "containerMap: removing container" podUID="9077a191-58b9-4dbd-aa96-b87eaa5398bf" containerName="payload-pre-processing"
Nov 02 12:45:00.231676 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:00.229618    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9077a191-58b9-4dbd-aa96-b87eaa5398bf" containerName="payload-pre-processing"
Nov 02 12:45:00.231676 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:00.229634    4559 cpu_manager.go:414] "containerMap: removing container" podUID="26b966b3-2043-474a-95aa-3037982c3fc4" containerName="istio-proxy"
Nov 02 12:45:00.231676 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:00.229657    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26b966b3-2043-474a-95aa-3037982c3fc4" containerName="istio-proxy"
Nov 02 12:45:00.231676 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:00.229687    4559 cpu_manager.go:414] "containerMap: removing container" podUID="730c9cc3-ba43-4ed4-8769-13999c58cf13" containerName="payload-processing"
Nov 02 12:45:00.231676 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:00.229694    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="730c9cc3-ba43-4ed4-8769-13999c58cf13" containerName="payload-processing"
Nov 02 12:45:00.231676 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:00.229863    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="26b966b3-2043-474a-95aa-3037982c3fc4" containerName="istio-proxy"
Nov 02 12:45:00.231676 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:00.229934    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="9077a191-58b9-4dbd-aa96-b87eaa5398bf" containerName="payload-pre-processing"
Nov 02 12:45:00.231676 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:00.229947    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="26b966b3-2043-474a-95aa-3037982c3fc4" containerName="istio-proxy"
Nov 02 12:45:00.231676 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:00.229961    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="730c9cc3-ba43-4ed4-8769-13999c58cf13" containerName="payload-processing"
Nov 02 12:45:00.236090 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:00.236058    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f0cf7e-29893725-22q6j"
Nov 02 12:45:00.248925 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:00.248782    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f0cf7e-29893725-22q6j"]
Nov 02 12:45:00.302539 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:00.300586    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w9cwp\""
Nov 02 12:45:00.352978 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:00.352932    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/531f3c3c-dbae-4e74-a7ae-27a21e2a9228-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-f0cf7e-29893725-22q6j\" (UID: \"531f3c3c-dbae-4e74-a7ae-27a21e2a9228\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f0cf7e-29893725-22q6j"
Nov 02 12:45:00.353323 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:00.353084    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrt9g\" (UniqueName: \"kubernetes.io/projected/531f3c3c-dbae-4e74-a7ae-27a21e2a9228-kube-api-access-nrt9g\") pod \"maas-api-key-cleanup-e2e-ipp-a-f0cf7e-29893725-22q6j\" (UID: \"531f3c3c-dbae-4e74-a7ae-27a21e2a9228\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f0cf7e-29893725-22q6j"
Nov 02 12:45:00.454541 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:00.454505    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrt9g\" (UniqueName: \"kubernetes.io/projected/531f3c3c-dbae-4e74-a7ae-27a21e2a9228-kube-api-access-nrt9g\") pod \"maas-api-key-cleanup-e2e-ipp-a-f0cf7e-29893725-22q6j\" (UID: \"531f3c3c-dbae-4e74-a7ae-27a21e2a9228\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f0cf7e-29893725-22q6j"
Nov 02 12:45:00.454811 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:00.454792    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/531f3c3c-dbae-4e74-a7ae-27a21e2a9228-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-f0cf7e-29893725-22q6j\" (UID: \"531f3c3c-dbae-4e74-a7ae-27a21e2a9228\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f0cf7e-29893725-22q6j"
Nov 02 12:45:00.455388 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:00.455368    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/531f3c3c-dbae-4e74-a7ae-27a21e2a9228-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-f0cf7e-29893725-22q6j\" (UID: \"531f3c3c-dbae-4e74-a7ae-27a21e2a9228\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f0cf7e-29893725-22q6j"
Nov 02 12:45:00.479133 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:00.479079    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrt9g\" (UniqueName: \"kubernetes.io/projected/531f3c3c-dbae-4e74-a7ae-27a21e2a9228-kube-api-access-nrt9g\") pod \"maas-api-key-cleanup-e2e-ipp-a-f0cf7e-29893725-22q6j\" (UID: \"531f3c3c-dbae-4e74-a7ae-27a21e2a9228\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f0cf7e-29893725-22q6j"
Nov 02 12:45:00.587931 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:00.587902    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f0cf7e-29893725-22q6j"
Nov 02 12:45:00.925169 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:45:00.925138    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod531f3c3c_dbae_4e74_a7ae_27a21e2a9228.slice/crio-4c8674aa0cf987b9c5d0ab6a54901618bba8ed112f4cb6de2f26f682f49360a9 WatchSource:0}: Error finding container 4c8674aa0cf987b9c5d0ab6a54901618bba8ed112f4cb6de2f26f682f49360a9: Status 404 returned error can't find the container with id 4c8674aa0cf987b9c5d0ab6a54901618bba8ed112f4cb6de2f26f682f49360a9
Nov 02 12:45:00.928371 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:00.928341    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f0cf7e-29893725-22q6j"]
Nov 02 12:45:01.078245 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:01.078209    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-c79cde-openshift-default-54dbf475f7-g56xt"]
Nov 02 12:45:01.078506 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:01.078480    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-c79cde-openshift-default-54dbf475f7-g56xt" podUID="ea9a3b60-d2ac-4837-b979-2ea31f79cee0" containerName="istio-proxy" containerID="cri-o://a67e9c2c37d610837c24c172d688bf1a2a32f784482327da22bcbdef0c4a7f4c" gracePeriod=30
Nov 02 12:45:01.849815 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:01.849702    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"]
Nov 02 12:45:01.853362 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:01.853333    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f0cf7e-29893725-22q6j" event={"ID":"531f3c3c-dbae-4e74-a7ae-27a21e2a9228","Type":"ContainerStarted","Data":"4c8674aa0cf987b9c5d0ab6a54901618bba8ed112f4cb6de2f26f682f49360a9"}
Nov 02 12:45:05.334104 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:05.334052    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-6a3af1-84b6d48b4b-rb9gx"]
Nov 02 12:45:05.334878 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:05.334846    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6a3af1-84b6d48b4b-rb9gx" podUID="227d2e4d-cc59-4cc6-b705-f257e3470f16" containerName="maas-api" containerID="cri-o://7c9aa713c0f86c6ada47f985a5972f1d4ff210053e1e477aab191a167f2b02ae" gracePeriod=30
Nov 02 12:45:05.982680 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:05.980509    4559 generic.go:356] "Generic (PLEG): container finished" podID="227d2e4d-cc59-4cc6-b705-f257e3470f16" containerID="7c9aa713c0f86c6ada47f985a5972f1d4ff210053e1e477aab191a167f2b02ae" exitCode=0
Nov 02 12:45:06.493663 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:06.489402    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6a3af1-84b6d48b4b-rb9gx" event={"ID":"227d2e4d-cc59-4cc6-b705-f257e3470f16","Type":"ContainerDied","Data":"7c9aa713c0f86c6ada47f985a5972f1d4ff210053e1e477aab191a167f2b02ae"}
Nov 02 12:45:06.579660 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:06.577849    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6a3af1-84b6d48b4b-rb9gx"
Nov 02 12:45:06.714663 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:06.711696    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/227d2e4d-cc59-4cc6-b705-f257e3470f16-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/227d2e4d-cc59-4cc6-b705-f257e3470f16-maas-api-tls\") pod \"227d2e4d-cc59-4cc6-b705-f257e3470f16\" (UID: \"227d2e4d-cc59-4cc6-b705-f257e3470f16\") "
Nov 02 12:45:06.714663 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:06.711860    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/227d2e4d-cc59-4cc6-b705-f257e3470f16-metrics-certs\" (UniqueName: \"kubernetes.io/secret/227d2e4d-cc59-4cc6-b705-f257e3470f16-metrics-certs\") pod \"227d2e4d-cc59-4cc6-b705-f257e3470f16\" (UID: \"227d2e4d-cc59-4cc6-b705-f257e3470f16\") "
Nov 02 12:45:06.714663 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:06.711944    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/227d2e4d-cc59-4cc6-b705-f257e3470f16-kube-api-access-mm54s\" (UniqueName: \"kubernetes.io/projected/227d2e4d-cc59-4cc6-b705-f257e3470f16-kube-api-access-mm54s\") pod \"227d2e4d-cc59-4cc6-b705-f257e3470f16\" (UID: \"227d2e4d-cc59-4cc6-b705-f257e3470f16\") "
Nov 02 12:45:06.714663 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:06.712162    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/227d2e4d-cc59-4cc6-b705-f257e3470f16-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/227d2e4d-cc59-4cc6-b705-f257e3470f16-openshift-service-ca\") pod \"227d2e4d-cc59-4cc6-b705-f257e3470f16\" (UID: \"227d2e4d-cc59-4cc6-b705-f257e3470f16\") "
Nov 02 12:45:06.726880 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:06.726846    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/227d2e4d-cc59-4cc6-b705-f257e3470f16-openshift-service-ca" pod "227d2e4d-cc59-4cc6-b705-f257e3470f16" (UID: "227d2e4d-cc59-4cc6-b705-f257e3470f16"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:45:06.753086 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:06.738558    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/227d2e4d-cc59-4cc6-b705-f257e3470f16-maas-api-tls" pod "227d2e4d-cc59-4cc6-b705-f257e3470f16" (UID: "227d2e4d-cc59-4cc6-b705-f257e3470f16"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:45:06.778791 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:06.778723    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/227d2e4d-cc59-4cc6-b705-f257e3470f16-kube-api-access-mm54s" pod "227d2e4d-cc59-4cc6-b705-f257e3470f16" (UID: "227d2e4d-cc59-4cc6-b705-f257e3470f16"). InnerVolumeSpecName "kube-api-access-mm54s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:45:06.827382 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:06.826981    4559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/227d2e4d-cc59-4cc6-b705-f257e3470f16-openshift-service-ca\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:45:06.827382 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:06.827014    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/227d2e4d-cc59-4cc6-b705-f257e3470f16-maas-api-tls\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:45:06.827382 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:06.827032    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mm54s\" (UniqueName: \"kubernetes.io/projected/227d2e4d-cc59-4cc6-b705-f257e3470f16-kube-api-access-mm54s\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:45:06.829939 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:06.829904    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/227d2e4d-cc59-4cc6-b705-f257e3470f16-metrics-certs" pod "227d2e4d-cc59-4cc6-b705-f257e3470f16" (UID: "227d2e4d-cc59-4cc6-b705-f257e3470f16"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:45:06.860100 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:06.860064    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-c79cde-openshift-default-54dbf475f7-g56xt"
Nov 02 12:45:06.929130 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:06.929095    4559 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/227d2e4d-cc59-4cc6-b705-f257e3470f16-metrics-certs\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:45:07.025166 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:07.025065    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6a3af1-84b6d48b4b-rb9gx" event={"ID":"227d2e4d-cc59-4cc6-b705-f257e3470f16","Type":"ContainerDied","Data":"76c4e83e9ab2495e42895e086f0cf9454028d566ea0b3ff60f9e7153e1070afc"}
Nov 02 12:45:07.025166 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:07.025145    4559 scope.go:122] "RemoveContainer" containerID="7c9aa713c0f86c6ada47f985a5972f1d4ff210053e1e477aab191a167f2b02ae"
Nov 02 12:45:07.025411 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:07.025326    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6a3af1-84b6d48b4b-rb9gx"
Nov 02 12:45:07.030097 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:07.030067    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea9a3b60-d2ac-4837-b979-2ea31f79cee0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea9a3b60-d2ac-4837-b979-2ea31f79cee0-credential-socket\") pod \"ea9a3b60-d2ac-4837-b979-2ea31f79cee0\" (UID: \"ea9a3b60-d2ac-4837-b979-2ea31f79cee0\") "
Nov 02 12:45:07.030304 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:07.030225    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea9a3b60-d2ac-4837-b979-2ea31f79cee0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea9a3b60-d2ac-4837-b979-2ea31f79cee0-istio-data\") pod \"ea9a3b60-d2ac-4837-b979-2ea31f79cee0\" (UID: \"ea9a3b60-d2ac-4837-b979-2ea31f79cee0\") "
Nov 02 12:45:07.030304 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:07.030267    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea9a3b60-d2ac-4837-b979-2ea31f79cee0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea9a3b60-d2ac-4837-b979-2ea31f79cee0-istiod-ca-cert\") pod \"ea9a3b60-d2ac-4837-b979-2ea31f79cee0\" (UID: \"ea9a3b60-d2ac-4837-b979-2ea31f79cee0\") "
Nov 02 12:45:07.030568 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:07.030345    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea9a3b60-d2ac-4837-b979-2ea31f79cee0-istio-token\" (UniqueName: \"kubernetes.io/projected/ea9a3b60-d2ac-4837-b979-2ea31f79cee0-istio-token\") pod \"ea9a3b60-d2ac-4837-b979-2ea31f79cee0\" (UID: \"ea9a3b60-d2ac-4837-b979-2ea31f79cee0\") "
Nov 02 12:45:07.030568 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:07.030402    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea9a3b60-d2ac-4837-b979-2ea31f79cee0-kube-api-access-2s6lw\" (UniqueName: \"kubernetes.io/projected/ea9a3b60-d2ac-4837-b979-2ea31f79cee0-kube-api-access-2s6lw\") pod \"ea9a3b60-d2ac-4837-b979-2ea31f79cee0\" (UID: \"ea9a3b60-d2ac-4837-b979-2ea31f79cee0\") "
Nov 02 12:45:07.030568 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:07.030444    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea9a3b60-d2ac-4837-b979-2ea31f79cee0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea9a3b60-d2ac-4837-b979-2ea31f79cee0-istio-envoy\") pod \"ea9a3b60-d2ac-4837-b979-2ea31f79cee0\" (UID: \"ea9a3b60-d2ac-4837-b979-2ea31f79cee0\") "
Nov 02 12:45:07.030568 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:07.030533    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ea9a3b60-d2ac-4837-b979-2ea31f79cee0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea9a3b60-d2ac-4837-b979-2ea31f79cee0-istio-podinfo\") pod \"ea9a3b60-d2ac-4837-b979-2ea31f79cee0\" (UID: \"ea9a3b60-d2ac-4837-b979-2ea31f79cee0\") "
Nov 02 12:45:07.030568 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:07.030559    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea9a3b60-d2ac-4837-b979-2ea31f79cee0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea9a3b60-d2ac-4837-b979-2ea31f79cee0-workload-certs\") pod \"ea9a3b60-d2ac-4837-b979-2ea31f79cee0\" (UID: \"ea9a3b60-d2ac-4837-b979-2ea31f79cee0\") "
Nov 02 12:45:07.030878 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:07.030717    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea9a3b60-d2ac-4837-b979-2ea31f79cee0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea9a3b60-d2ac-4837-b979-2ea31f79cee0-workload-socket\") pod \"ea9a3b60-d2ac-4837-b979-2ea31f79cee0\" (UID: \"ea9a3b60-d2ac-4837-b979-2ea31f79cee0\") "
Nov 02 12:45:07.031567 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:07.031542    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea9a3b60-d2ac-4837-b979-2ea31f79cee0-workload-socket" pod "ea9a3b60-d2ac-4837-b979-2ea31f79cee0" (UID: "ea9a3b60-d2ac-4837-b979-2ea31f79cee0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:45:07.032112 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:07.032083    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea9a3b60-d2ac-4837-b979-2ea31f79cee0-istio-data" pod "ea9a3b60-d2ac-4837-b979-2ea31f79cee0" (UID: "ea9a3b60-d2ac-4837-b979-2ea31f79cee0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:45:07.032209 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:07.032177    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea9a3b60-d2ac-4837-b979-2ea31f79cee0-istiod-ca-cert" pod "ea9a3b60-d2ac-4837-b979-2ea31f79cee0" (UID: "ea9a3b60-d2ac-4837-b979-2ea31f79cee0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:45:07.032321 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:07.032298    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea9a3b60-d2ac-4837-b979-2ea31f79cee0-credential-socket" pod "ea9a3b60-d2ac-4837-b979-2ea31f79cee0" (UID: "ea9a3b60-d2ac-4837-b979-2ea31f79cee0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:45:07.032841 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:07.032541    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea9a3b60-d2ac-4837-b979-2ea31f79cee0-workload-certs" pod "ea9a3b60-d2ac-4837-b979-2ea31f79cee0" (UID: "ea9a3b60-d2ac-4837-b979-2ea31f79cee0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:45:07.036676 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:07.036149    4559 generic.go:356] "Generic (PLEG): container finished" podID="ea9a3b60-d2ac-4837-b979-2ea31f79cee0" containerID="a67e9c2c37d610837c24c172d688bf1a2a32f784482327da22bcbdef0c4a7f4c" exitCode=0
Nov 02 12:45:07.036676 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:07.036253    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c79cde-openshift-default-54dbf475f7-g56xt" event={"ID":"ea9a3b60-d2ac-4837-b979-2ea31f79cee0","Type":"ContainerDied","Data":"a67e9c2c37d610837c24c172d688bf1a2a32f784482327da22bcbdef0c4a7f4c"}
Nov 02 12:45:07.036676 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:07.036298    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c79cde-openshift-default-54dbf475f7-g56xt" event={"ID":"ea9a3b60-d2ac-4837-b979-2ea31f79cee0","Type":"ContainerDied","Data":"d2d6c052637aac0649d8d12cddbe1f17bb9ea297fdf961faa0fd9f10e582ebd4"}
Nov 02 12:45:07.036676 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:07.036375    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-c79cde-openshift-default-54dbf475f7-g56xt"
Nov 02 12:45:07.038766 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:07.038721    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea9a3b60-d2ac-4837-b979-2ea31f79cee0-istio-token" pod "ea9a3b60-d2ac-4837-b979-2ea31f79cee0" (UID: "ea9a3b60-d2ac-4837-b979-2ea31f79cee0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:45:07.046011 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:07.045993    4559 scope.go:122] "RemoveContainer" containerID="a67e9c2c37d610837c24c172d688bf1a2a32f784482327da22bcbdef0c4a7f4c"
Nov 02 12:45:07.047857 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:07.047831    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ea9a3b60-d2ac-4837-b979-2ea31f79cee0-istio-podinfo" pod "ea9a3b60-d2ac-4837-b979-2ea31f79cee0" (UID: "ea9a3b60-d2ac-4837-b979-2ea31f79cee0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 12:45:07.047943 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:07.047831    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea9a3b60-d2ac-4837-b979-2ea31f79cee0-istio-envoy" pod "ea9a3b60-d2ac-4837-b979-2ea31f79cee0" (UID: "ea9a3b60-d2ac-4837-b979-2ea31f79cee0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:45:07.047943 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:07.047871    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea9a3b60-d2ac-4837-b979-2ea31f79cee0-kube-api-access-2s6lw" pod "ea9a3b60-d2ac-4837-b979-2ea31f79cee0" (UID: "ea9a3b60-d2ac-4837-b979-2ea31f79cee0"). InnerVolumeSpecName "kube-api-access-2s6lw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:45:07.057506 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:07.057412    4559 scope.go:122] "RemoveContainer" containerID="6618ad71a8db78e22aa9f5dfcbf9c761ac8e0f287f7d1924fa12329ce2ffb07f"
Nov 02 12:45:07.069723 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:07.069699    4559 scope.go:122] "RemoveContainer" containerID="a67e9c2c37d610837c24c172d688bf1a2a32f784482327da22bcbdef0c4a7f4c"
Nov 02 12:45:07.070057 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:45:07.070035    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a67e9c2c37d610837c24c172d688bf1a2a32f784482327da22bcbdef0c4a7f4c\": container with ID starting with a67e9c2c37d610837c24c172d688bf1a2a32f784482327da22bcbdef0c4a7f4c not found: ID does not exist" containerID="a67e9c2c37d610837c24c172d688bf1a2a32f784482327da22bcbdef0c4a7f4c"
Nov 02 12:45:07.070157 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:07.070068    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a67e9c2c37d610837c24c172d688bf1a2a32f784482327da22bcbdef0c4a7f4c"} err="failed to get container status \"a67e9c2c37d610837c24c172d688bf1a2a32f784482327da22bcbdef0c4a7f4c\": rpc error: code = NotFound desc = could not find container \"a67e9c2c37d610837c24c172d688bf1a2a32f784482327da22bcbdef0c4a7f4c\": container with ID starting with a67e9c2c37d610837c24c172d688bf1a2a32f784482327da22bcbdef0c4a7f4c not found: ID does not exist"
Nov 02 12:45:07.070157 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:07.070085    4559 scope.go:122] "RemoveContainer" containerID="6618ad71a8db78e22aa9f5dfcbf9c761ac8e0f287f7d1924fa12329ce2ffb07f"
Nov 02 12:45:07.070407 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:45:07.070375    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6618ad71a8db78e22aa9f5dfcbf9c761ac8e0f287f7d1924fa12329ce2ffb07f\": container with ID starting with 6618ad71a8db78e22aa9f5dfcbf9c761ac8e0f287f7d1924fa12329ce2ffb07f not found: ID does not exist" containerID="6618ad71a8db78e22aa9f5dfcbf9c761ac8e0f287f7d1924fa12329ce2ffb07f"
Nov 02 12:45:07.070478 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:07.070419    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6618ad71a8db78e22aa9f5dfcbf9c761ac8e0f287f7d1924fa12329ce2ffb07f"} err="failed to get container status \"6618ad71a8db78e22aa9f5dfcbf9c761ac8e0f287f7d1924fa12329ce2ffb07f\": rpc error: code = NotFound desc = could not find container \"6618ad71a8db78e22aa9f5dfcbf9c761ac8e0f287f7d1924fa12329ce2ffb07f\": container with ID starting with 6618ad71a8db78e22aa9f5dfcbf9c761ac8e0f287f7d1924fa12329ce2ffb07f not found: ID does not exist"
Nov 02 12:45:07.121985 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:07.121740    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-6a3af1-84b6d48b4b-rb9gx"]
Nov 02 12:45:07.128042 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:07.128007    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-6a3af1-84b6d48b4b-rb9gx"]
Nov 02 12:45:07.132677 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:07.132651    4559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea9a3b60-d2ac-4837-b979-2ea31f79cee0-istio-token\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:45:07.132797 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:07.132682    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2s6lw\" (UniqueName: \"kubernetes.io/projected/ea9a3b60-d2ac-4837-b979-2ea31f79cee0-kube-api-access-2s6lw\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:45:07.132797 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:07.132700    4559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea9a3b60-d2ac-4837-b979-2ea31f79cee0-istio-envoy\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:45:07.132797 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:07.132716    4559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea9a3b60-d2ac-4837-b979-2ea31f79cee0-istio-podinfo\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:45:07.132797 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:07.132731    4559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea9a3b60-d2ac-4837-b979-2ea31f79cee0-workload-certs\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:45:07.132797 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:07.132746    4559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea9a3b60-d2ac-4837-b979-2ea31f79cee0-workload-socket\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:45:07.132797 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:07.132760    4559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea9a3b60-d2ac-4837-b979-2ea31f79cee0-credential-socket\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:45:07.132797 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:07.132774    4559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea9a3b60-d2ac-4837-b979-2ea31f79cee0-istio-data\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:45:07.132797 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:07.132786    4559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea9a3b60-d2ac-4837-b979-2ea31f79cee0-istiod-ca-cert\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:45:07.422982 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:07.422865    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-c79cde-openshift-default-54dbf475f7-g56xt"]
Nov 02 12:45:07.428703 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:07.428675    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-c79cde-openshift-default-54dbf475f7-g56xt"]
Nov 02 12:45:08.125714 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:08.122517    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="227d2e4d-cc59-4cc6-b705-f257e3470f16" path="/var/lib/kubelet/pods/227d2e4d-cc59-4cc6-b705-f257e3470f16/volumes"
Nov 02 12:45:08.125714 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:08.123291    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea9a3b60-d2ac-4837-b979-2ea31f79cee0" path="/var/lib/kubelet/pods/ea9a3b60-d2ac-4837-b979-2ea31f79cee0/volumes"
Nov 02 12:45:08.125714 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:08.123745    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7004-5ffcd5"]
Nov 02 12:45:08.125714 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:08.124583    4559 cpu_manager.go:414] "containerMap: removing container" podUID="ea9a3b60-d2ac-4837-b979-2ea31f79cee0" containerName="istio-proxy"
Nov 02 12:45:08.125714 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:08.124598    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea9a3b60-d2ac-4837-b979-2ea31f79cee0" containerName="istio-proxy"
Nov 02 12:45:08.125714 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:08.124631    4559 cpu_manager.go:414] "containerMap: removing container" podUID="227d2e4d-cc59-4cc6-b705-f257e3470f16" containerName="maas-api"
Nov 02 12:45:08.125714 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:08.125035    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="227d2e4d-cc59-4cc6-b705-f257e3470f16" containerName="maas-api"
Nov 02 12:45:08.125714 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:08.125208    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea9a3b60-d2ac-4837-b979-2ea31f79cee0" containerName="istio-proxy"
Nov 02 12:45:08.125714 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:08.125223    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="227d2e4d-cc59-4cc6-b705-f257e3470f16" containerName="maas-api"
Nov 02 12:45:08.125714 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:08.125233    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea9a3b60-d2ac-4837-b979-2ea31f79cee0" containerName="istio-proxy"
Nov 02 12:45:08.130704 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:08.130680    4559 generic.go:356] "Generic (PLEG): container finished" podID="0992908f-5ad3-4600-b8d4-f26d07958ee2" containerID="3c680aa9299d50dbc18dc2288a5ba61232752ea9c393f42d1f8ab47b681c7714" exitCode=0
Nov 02 12:45:08.139792 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:08.139768    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7004-5ffcd5"]
Nov 02 12:45:08.139931 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:08.139800    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-6a3af1-openshift-default-666bcf8bbc-2hjz9" event={"ID":"0992908f-5ad3-4600-b8d4-f26d07958ee2","Type":"ContainerDied","Data":"3c680aa9299d50dbc18dc2288a5ba61232752ea9c393f42d1f8ab47b681c7714"}
Nov 02 12:45:08.140204 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:08.140185    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7004-5ffcd5"
Nov 02 12:45:08.140455 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:08.140210    4559 scope.go:122] "RemoveContainer" containerID="3c680aa9299d50dbc18dc2288a5ba61232752ea9c393f42d1f8ab47b681c7714"
Nov 02 12:45:08.145309 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:08.145276    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:45:08.149983 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:08.149941    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:45:08.150448 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:08.150421    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:45:08.241669 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:08.233997    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-hfcns\""
Nov 02 12:45:08.347664 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:08.347531    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4h5h\" (UniqueName: \"kubernetes.io/projected/09198b81-42f1-4ae2-9e17-517a352ceaf6-kube-api-access-v4h5h\") pod \"test-curl-7004-5ffcd5\" (UID: \"09198b81-42f1-4ae2-9e17-517a352ceaf6\") " pod="opendatahub/test-curl-7004-5ffcd5"
Nov 02 12:45:08.449904 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:08.449870    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4h5h\" (UniqueName: \"kubernetes.io/projected/09198b81-42f1-4ae2-9e17-517a352ceaf6-kube-api-access-v4h5h\") pod \"test-curl-7004-5ffcd5\" (UID: \"09198b81-42f1-4ae2-9e17-517a352ceaf6\") " pod="opendatahub/test-curl-7004-5ffcd5"
Nov 02 12:45:08.492669 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:08.477840    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4h5h\" (UniqueName: \"kubernetes.io/projected/09198b81-42f1-4ae2-9e17-517a352ceaf6-kube-api-access-v4h5h\") pod \"test-curl-7004-5ffcd5\" (UID: \"09198b81-42f1-4ae2-9e17-517a352ceaf6\") " pod="opendatahub/test-curl-7004-5ffcd5"
Nov 02 12:45:08.674436 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:08.674340    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7004-5ffcd5"
Nov 02 12:45:09.111131 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:09.111096    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7004-5ffcd5"]
Nov 02 12:45:09.113130 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:45:09.113080    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod09198b81_42f1_4ae2_9e17_517a352ceaf6.slice/crio-34dd9bb25fa86d79b6c3beb54ecf147678fcc5f66022f716ea8534dfab53f7e4 WatchSource:0}: Error finding container 34dd9bb25fa86d79b6c3beb54ecf147678fcc5f66022f716ea8534dfab53f7e4: Status 404 returned error can't find the container with id 34dd9bb25fa86d79b6c3beb54ecf147678fcc5f66022f716ea8534dfab53f7e4
Nov 02 12:45:09.141502 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:09.141477    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f0cf7e-29893725-22q6j" event={"ID":"531f3c3c-dbae-4e74-a7ae-27a21e2a9228","Type":"ContainerStarted","Data":"53bbf3ac11e190b870b7fa8b70ce65534fb842c5842add50c94cfb023c37d101"}
Nov 02 12:45:09.143622 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:09.143599    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-6a3af1-openshift-default-666bcf8bbc-2hjz9" event={"ID":"0992908f-5ad3-4600-b8d4-f26d07958ee2","Type":"ContainerStarted","Data":"10113f71438d69865cfe93a7947741f9f87b9e9201ff8a4bf8dad0e71a872faf"}
Nov 02 12:45:09.145044 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:09.145013    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7004-5ffcd5" event={"ID":"09198b81-42f1-4ae2-9e17-517a352ceaf6","Type":"ContainerStarted","Data":"34dd9bb25fa86d79b6c3beb54ecf147678fcc5f66022f716ea8534dfab53f7e4"}
Nov 02 12:45:09.208285 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:09.208011    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6a3af1-84b6d48b4b-rb9gx" podUID="227d2e4d-cc59-4cc6-b705-f257e3470f16" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.96:8443/health\": context deadline exceeded"
Nov 02 12:45:09.214187 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:09.214116    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f0cf7e-29893725-22q6j" podStartSLOduration=1.470391976 podStartE2EDuration="9.214103216s" totalImagesPullingTime="7.74371124s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:45:09.207191808 +0000 UTC m=+1105.462590906" watchObservedRunningTime="2026-11-02 12:45:09.214103216 +0000 UTC m=+1105.469502311"
Nov 02 12:45:09.490792 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:09.490760    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-6a3af1-openshift-default-666bcf8bbc-2hjz9"
Nov 02 12:45:09.518734 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:09.518432    4559 patch_prober.go:29] interesting pod/e2e-api-b-6a3af1-openshift-default-666bcf8bbc-2hjz9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Nov 02 12:45:09.518734 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:09.518492    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-6a3af1-openshift-default-666bcf8bbc-2hjz9" podUID="0992908f-5ad3-4600-b8d4-f26d07958ee2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Nov 02 12:45:10.153802 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:10.153704    4559 generic.go:356] "Generic (PLEG): container finished" podID="531f3c3c-dbae-4e74-a7ae-27a21e2a9228" containerID="53bbf3ac11e190b870b7fa8b70ce65534fb842c5842add50c94cfb023c37d101" exitCode=60
Nov 02 12:45:10.154262 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:10.153805    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f0cf7e-29893725-22q6j" event={"ID":"531f3c3c-dbae-4e74-a7ae-27a21e2a9228","Type":"ContainerDied","Data":"53bbf3ac11e190b870b7fa8b70ce65534fb842c5842add50c94cfb023c37d101"}
Nov 02 12:45:10.154262 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:10.154104    4559 scope.go:122] "RemoveContainer" containerID="53bbf3ac11e190b870b7fa8b70ce65534fb842c5842add50c94cfb023c37d101"
Nov 02 12:45:10.156959 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:10.156933    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7004-5ffcd5" event={"ID":"09198b81-42f1-4ae2-9e17-517a352ceaf6","Type":"ContainerStarted","Data":"c592f5c0195536d8f8ac329d76f14238acae84cb9dc1eb0685ff44105c8faafd"}
Nov 02 12:45:10.219746 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:10.218853    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e6ceda-dbd545695-ggztc"]
Nov 02 12:45:10.220256 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:10.220199    4559 cpu_manager.go:414] "containerMap: removing container" podUID="ea9a3b60-d2ac-4837-b979-2ea31f79cee0" containerName="istio-proxy"
Nov 02 12:45:10.220256 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:10.220228    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea9a3b60-d2ac-4837-b979-2ea31f79cee0" containerName="istio-proxy"
Nov 02 12:45:10.225888 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:10.225863    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e6ceda-dbd545695-ggztc"
Nov 02 12:45:10.245655 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:10.242854    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-e6ceda-dockercfg-5666f\""
Nov 02 12:45:10.245655 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:10.243103    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-e6ceda\""
Nov 02 12:45:10.255915 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:10.253501    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-6a3af1-openshift-default-666bcf8bbc-2hjz9"
Nov 02 12:45:10.256168 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:10.256113    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e6ceda-dbd545695-ggztc"]
Nov 02 12:45:10.256508 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:10.256461    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7004-5ffcd5" podStartSLOduration=3.256429106 podStartE2EDuration="3.256429106s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:45:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:45:10.242887077 +0000 UTC m=+1106.498286174" watchObservedRunningTime="2026-11-02 12:45:10.256429106 +0000 UTC m=+1106.511828202"
Nov 02 12:45:10.310409 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:10.310036    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb4ea3c3-befb-4ee2-b46b-93ce5e4a7110-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-e6ceda-dbd545695-ggztc\" (UID: \"bb4ea3c3-befb-4ee2-b46b-93ce5e4a7110\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e6ceda-dbd545695-ggztc"
Nov 02 12:45:10.310409 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:10.310197    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69qcj\" (UniqueName: \"kubernetes.io/projected/bb4ea3c3-befb-4ee2-b46b-93ce5e4a7110-kube-api-access-69qcj\") pod \"payload-pre-processing-e2e-ipp-a-e6ceda-dbd545695-ggztc\" (UID: \"bb4ea3c3-befb-4ee2-b46b-93ce5e4a7110\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e6ceda-dbd545695-ggztc"
Nov 02 12:45:10.481664 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:10.443530    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb4ea3c3-befb-4ee2-b46b-93ce5e4a7110-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-e6ceda-dbd545695-ggztc\" (UID: \"bb4ea3c3-befb-4ee2-b46b-93ce5e4a7110\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e6ceda-dbd545695-ggztc"
Nov 02 12:45:10.481664 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:10.443653    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69qcj\" (UniqueName: \"kubernetes.io/projected/bb4ea3c3-befb-4ee2-b46b-93ce5e4a7110-kube-api-access-69qcj\") pod \"payload-pre-processing-e2e-ipp-a-e6ceda-dbd545695-ggztc\" (UID: \"bb4ea3c3-befb-4ee2-b46b-93ce5e4a7110\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e6ceda-dbd545695-ggztc"
Nov 02 12:45:10.661663 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:10.506022    4559 patch_prober.go:29] interesting pod/e2e-api-b-6a3af1-openshift-default-666bcf8bbc-2hjz9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Nov 02 12:45:10.661663 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:10.509497    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb4ea3c3-befb-4ee2-b46b-93ce5e4a7110-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-e6ceda-dbd545695-ggztc\" (UID: \"bb4ea3c3-befb-4ee2-b46b-93ce5e4a7110\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e6ceda-dbd545695-ggztc"
Nov 02 12:45:10.661663 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:10.549303    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e6ceda-779dbb5897-lqnhx"]
Nov 02 12:45:10.661663 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:10.565056    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-6a3af1-openshift-default-666bcf8bbc-2hjz9" podUID="0992908f-5ad3-4600-b8d4-f26d07958ee2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Nov 02 12:45:10.661663 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:10.591664    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-e6ceda-779dbb5897-lqnhx"
Nov 02 12:45:10.661663 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:10.597706    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e6ceda-779dbb5897-lqnhx"]
Nov 02 12:45:10.780492 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:10.780208    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69qcj\" (UniqueName: \"kubernetes.io/projected/bb4ea3c3-befb-4ee2-b46b-93ce5e4a7110-kube-api-access-69qcj\") pod \"payload-pre-processing-e2e-ipp-a-e6ceda-dbd545695-ggztc\" (UID: \"bb4ea3c3-befb-4ee2-b46b-93ce5e4a7110\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e6ceda-dbd545695-ggztc"
Nov 02 12:45:10.824712 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:10.806371    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkxh8\" (UniqueName: \"kubernetes.io/projected/ae88d18c-c12e-4c53-a0e0-b012a2d624ff-kube-api-access-wkxh8\") pod \"payload-processing-e2e-ipp-a-e6ceda-779dbb5897-lqnhx\" (UID: \"ae88d18c-c12e-4c53-a0e0-b012a2d624ff\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e6ceda-779dbb5897-lqnhx"
Nov 02 12:45:10.824712 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:10.823703    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae88d18c-c12e-4c53-a0e0-b012a2d624ff-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-e6ceda-779dbb5897-lqnhx\" (UID: \"ae88d18c-c12e-4c53-a0e0-b012a2d624ff\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e6ceda-779dbb5897-lqnhx"
Nov 02 12:45:10.854516 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:10.854488    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e6ceda-dbd545695-ggztc"
Nov 02 12:45:10.925332 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:10.925298    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkxh8\" (UniqueName: \"kubernetes.io/projected/ae88d18c-c12e-4c53-a0e0-b012a2d624ff-kube-api-access-wkxh8\") pod \"payload-processing-e2e-ipp-a-e6ceda-779dbb5897-lqnhx\" (UID: \"ae88d18c-c12e-4c53-a0e0-b012a2d624ff\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e6ceda-779dbb5897-lqnhx"
Nov 02 12:45:10.925502 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:10.925473    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae88d18c-c12e-4c53-a0e0-b012a2d624ff-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-e6ceda-779dbb5897-lqnhx\" (UID: \"ae88d18c-c12e-4c53-a0e0-b012a2d624ff\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e6ceda-779dbb5897-lqnhx"
Nov 02 12:45:10.928592 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:10.926384    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae88d18c-c12e-4c53-a0e0-b012a2d624ff-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-e6ceda-779dbb5897-lqnhx\" (UID: \"ae88d18c-c12e-4c53-a0e0-b012a2d624ff\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e6ceda-779dbb5897-lqnhx"
Nov 02 12:45:10.951272 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:10.951217    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkxh8\" (UniqueName: \"kubernetes.io/projected/ae88d18c-c12e-4c53-a0e0-b012a2d624ff-kube-api-access-wkxh8\") pod \"payload-processing-e2e-ipp-a-e6ceda-779dbb5897-lqnhx\" (UID: \"ae88d18c-c12e-4c53-a0e0-b012a2d624ff\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e6ceda-779dbb5897-lqnhx"
Nov 02 12:45:11.154715 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:11.150906    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-e6ceda-779dbb5897-lqnhx"
Nov 02 12:45:11.319700 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:11.312988    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e6ceda-dbd545695-ggztc"]
Nov 02 12:45:11.336746 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:11.336117    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f0cf7e-29893725-22q6j" event={"ID":"531f3c3c-dbae-4e74-a7ae-27a21e2a9228","Type":"ContainerStarted","Data":"ae90ba1d7e53ff1afdd43c4373dffd2d3df6d654f6eb9166985816f1d98e1ca5"}
Nov 02 12:45:11.518126 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:11.518075    4559 patch_prober.go:29] interesting pod/e2e-api-b-6a3af1-openshift-default-666bcf8bbc-2hjz9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Nov 02 12:45:11.518459 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:11.518136    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-6a3af1-openshift-default-666bcf8bbc-2hjz9" podUID="0992908f-5ad3-4600-b8d4-f26d07958ee2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Nov 02 12:45:11.709992 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:11.709957    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7004-5ffcd5"]
Nov 02 12:45:11.790668 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:11.778125    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7004-5ffcd5"]
Nov 02 12:45:12.661366 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:12.657034    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-61e341/distinct-w1-61e341-kserve-787cd566f4-cjj8s" podUID="6886ce28-5800-402d-b893-f92109e289a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:45:12.661366 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:12.657059    4559 patch_prober.go:29] interesting pod/e2e-api-b-6a3af1-openshift-default-666bcf8bbc-2hjz9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Nov 02 12:45:12.661366 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:12.657118    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-6a3af1-openshift-default-666bcf8bbc-2hjz9" podUID="0992908f-5ad3-4600-b8d4-f26d07958ee2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Nov 02 12:45:12.661366 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:45:12.658173    4559 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae88d18c_c12e_4c53_a0e0_b012a2d624ff.slice/crio-3565744111bfc9fbdba8b071a5af61b1f1f47fad3a43b21f6664fb7488e75c72\": RecentStats: unable to find data in memory cache]"
Nov 02 12:45:12.677608 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:12.677032    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e6ceda-779dbb5897-lqnhx"]
Nov 02 12:45:12.677608 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:12.677098    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-77cf70-openshift-default-d8f7564b9-rftzg"]
Nov 02 12:45:12.678069 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:12.678043    4559 cpu_manager.go:414] "containerMap: removing container" podUID="09198b81-42f1-4ae2-9e17-517a352ceaf6" containerName="test-curl-7004-5ffcd5"
Nov 02 12:45:12.678163 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:12.678071    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09198b81-42f1-4ae2-9e17-517a352ceaf6" containerName="test-curl-7004-5ffcd5"
Nov 02 12:45:12.678258 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:12.678245    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="09198b81-42f1-4ae2-9e17-517a352ceaf6" containerName="test-curl-7004-5ffcd5"
Nov 02 12:45:12.721514 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:12.721482    4559 generic.go:356] "Generic (PLEG): container finished" podID="531f3c3c-dbae-4e74-a7ae-27a21e2a9228" containerID="ae90ba1d7e53ff1afdd43c4373dffd2d3df6d654f6eb9166985816f1d98e1ca5" exitCode=60
Nov 02 12:45:12.722226 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:12.722210    4559 scope.go:122] "RemoveContainer" containerID="ae90ba1d7e53ff1afdd43c4373dffd2d3df6d654f6eb9166985816f1d98e1ca5"
Nov 02 12:45:12.722757 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:45:12.722733    4559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-f0cf7e-29893725-22q6j_odh-ai-gateway-infra(531f3c3c-dbae-4e74-a7ae-27a21e2a9228)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f0cf7e-29893725-22q6j" podUID="531f3c3c-dbae-4e74-a7ae-27a21e2a9228"
Nov 02 12:45:12.728221 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:12.727813    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f0cf7e-29893725-22q6j" event={"ID":"531f3c3c-dbae-4e74-a7ae-27a21e2a9228","Type":"ContainerDied","Data":"ae90ba1d7e53ff1afdd43c4373dffd2d3df6d654f6eb9166985816f1d98e1ca5"}
Nov 02 12:45:12.728221 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:12.727869    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-77cf70-openshift-default-d8f7564b9-rftzg"]
Nov 02 12:45:12.728221 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:12.727902    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e6ceda-dbd545695-ggztc" event={"ID":"bb4ea3c3-befb-4ee2-b46b-93ce5e4a7110","Type":"ContainerStarted","Data":"0bda04b55a799c6f508cb4c8b62e72c2097c6260e3740d8b32dc315e9d971161"}
Nov 02 12:45:12.728221 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:12.727926    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-77cf70-openshift-default-d8f7564b9-rftzg"
Nov 02 12:45:12.728221 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:12.727948    4559 scope.go:122] "RemoveContainer" containerID="53bbf3ac11e190b870b7fa8b70ce65534fb842c5842add50c94cfb023c37d101"
Nov 02 12:45:12.728221 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:12.727928    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e6ceda-dbd545695-ggztc" event={"ID":"bb4ea3c3-befb-4ee2-b46b-93ce5e4a7110","Type":"ContainerStarted","Data":"539ed763ae891654f18d31f04282b2ec2095145ae1d00c6d1b280486b4e6e110"}
Nov 02 12:45:12.728221 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:12.728069    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e6ceda-dbd545695-ggztc"
Nov 02 12:45:12.730217 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:12.730068    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7004-5ffcd5" podUID="09198b81-42f1-4ae2-9e17-517a352ceaf6" containerName="test-curl-7004-5ffcd5" containerID="cri-o://c592f5c0195536d8f8ac329d76f14238acae84cb9dc1eb0685ff44105c8faafd" gracePeriod=2
Nov 02 12:45:12.730217 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:12.730174    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-e6ceda-779dbb5897-lqnhx" event={"ID":"ae88d18c-c12e-4c53-a0e0-b012a2d624ff","Type":"ContainerStarted","Data":"3565744111bfc9fbdba8b071a5af61b1f1f47fad3a43b21f6664fb7488e75c72"}
Nov 02 12:45:12.753274 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:12.753246    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-77cf70-openshift-default-dockercfg-m88pp\""
Nov 02 12:45:12.754370 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:12.754329    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60be7b9e-07a0-4cc6-99d1-1278b6bad791-workload-certs\") pod \"e2e-ipp-b-77cf70-openshift-default-d8f7564b9-rftzg\" (UID: \"60be7b9e-07a0-4cc6-99d1-1278b6bad791\") " pod="openshift-ingress/e2e-ipp-b-77cf70-openshift-default-d8f7564b9-rftzg"
Nov 02 12:45:12.754454 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:12.754381    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60be7b9e-07a0-4cc6-99d1-1278b6bad791-workload-socket\") pod \"e2e-ipp-b-77cf70-openshift-default-d8f7564b9-rftzg\" (UID: \"60be7b9e-07a0-4cc6-99d1-1278b6bad791\") " pod="openshift-ingress/e2e-ipp-b-77cf70-openshift-default-d8f7564b9-rftzg"
Nov 02 12:45:12.754454 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:12.754427    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60be7b9e-07a0-4cc6-99d1-1278b6bad791-istio-data\") pod \"e2e-ipp-b-77cf70-openshift-default-d8f7564b9-rftzg\" (UID: \"60be7b9e-07a0-4cc6-99d1-1278b6bad791\") " pod="openshift-ingress/e2e-ipp-b-77cf70-openshift-default-d8f7564b9-rftzg"
Nov 02 12:45:12.754569 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:12.754492    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60be7b9e-07a0-4cc6-99d1-1278b6bad791-istio-podinfo\") pod \"e2e-ipp-b-77cf70-openshift-default-d8f7564b9-rftzg\" (UID: \"60be7b9e-07a0-4cc6-99d1-1278b6bad791\") " pod="openshift-ingress/e2e-ipp-b-77cf70-openshift-default-d8f7564b9-rftzg"
Nov 02 12:45:12.754569 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:12.754524    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60be7b9e-07a0-4cc6-99d1-1278b6bad791-istiod-ca-cert\") pod \"e2e-ipp-b-77cf70-openshift-default-d8f7564b9-rftzg\" (UID: \"60be7b9e-07a0-4cc6-99d1-1278b6bad791\") " pod="openshift-ingress/e2e-ipp-b-77cf70-openshift-default-d8f7564b9-rftzg"
Nov 02 12:45:12.754569 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:12.754563    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60be7b9e-07a0-4cc6-99d1-1278b6bad791-credential-socket\") pod \"e2e-ipp-b-77cf70-openshift-default-d8f7564b9-rftzg\" (UID: \"60be7b9e-07a0-4cc6-99d1-1278b6bad791\") " pod="openshift-ingress/e2e-ipp-b-77cf70-openshift-default-d8f7564b9-rftzg"
Nov 02 12:45:12.754745 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:12.754607    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60be7b9e-07a0-4cc6-99d1-1278b6bad791-istio-envoy\") pod \"e2e-ipp-b-77cf70-openshift-default-d8f7564b9-rftzg\" (UID: \"60be7b9e-07a0-4cc6-99d1-1278b6bad791\") " pod="openshift-ingress/e2e-ipp-b-77cf70-openshift-default-d8f7564b9-rftzg"
Nov 02 12:45:12.755116 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:12.755091    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60be7b9e-07a0-4cc6-99d1-1278b6bad791-istio-token\") pod \"e2e-ipp-b-77cf70-openshift-default-d8f7564b9-rftzg\" (UID: \"60be7b9e-07a0-4cc6-99d1-1278b6bad791\") " pod="openshift-ingress/e2e-ipp-b-77cf70-openshift-default-d8f7564b9-rftzg"
Nov 02 12:45:12.755229 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:12.755177    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfgdl\" (UniqueName: \"kubernetes.io/projected/60be7b9e-07a0-4cc6-99d1-1278b6bad791-kube-api-access-xfgdl\") pod \"e2e-ipp-b-77cf70-openshift-default-d8f7564b9-rftzg\" (UID: \"60be7b9e-07a0-4cc6-99d1-1278b6bad791\") " pod="openshift-ingress/e2e-ipp-b-77cf70-openshift-default-d8f7564b9-rftzg"
Nov 02 12:45:12.860850 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:12.858697    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60be7b9e-07a0-4cc6-99d1-1278b6bad791-istio-podinfo\") pod \"e2e-ipp-b-77cf70-openshift-default-d8f7564b9-rftzg\" (UID: \"60be7b9e-07a0-4cc6-99d1-1278b6bad791\") " pod="openshift-ingress/e2e-ipp-b-77cf70-openshift-default-d8f7564b9-rftzg"
Nov 02 12:45:12.860850 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:12.858757    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60be7b9e-07a0-4cc6-99d1-1278b6bad791-istiod-ca-cert\") pod \"e2e-ipp-b-77cf70-openshift-default-d8f7564b9-rftzg\" (UID: \"60be7b9e-07a0-4cc6-99d1-1278b6bad791\") " pod="openshift-ingress/e2e-ipp-b-77cf70-openshift-default-d8f7564b9-rftzg"
Nov 02 12:45:12.860850 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:12.858848    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60be7b9e-07a0-4cc6-99d1-1278b6bad791-credential-socket\") pod \"e2e-ipp-b-77cf70-openshift-default-d8f7564b9-rftzg\" (UID: \"60be7b9e-07a0-4cc6-99d1-1278b6bad791\") " pod="openshift-ingress/e2e-ipp-b-77cf70-openshift-default-d8f7564b9-rftzg"
Nov 02 12:45:12.860850 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:12.858913    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60be7b9e-07a0-4cc6-99d1-1278b6bad791-istio-envoy\") pod \"e2e-ipp-b-77cf70-openshift-default-d8f7564b9-rftzg\" (UID: \"60be7b9e-07a0-4cc6-99d1-1278b6bad791\") " pod="openshift-ingress/e2e-ipp-b-77cf70-openshift-default-d8f7564b9-rftzg"
Nov 02 12:45:12.860850 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:12.859005    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60be7b9e-07a0-4cc6-99d1-1278b6bad791-istio-token\") pod \"e2e-ipp-b-77cf70-openshift-default-d8f7564b9-rftzg\" (UID: \"60be7b9e-07a0-4cc6-99d1-1278b6bad791\") " pod="openshift-ingress/e2e-ipp-b-77cf70-openshift-default-d8f7564b9-rftzg"
Nov 02 12:45:12.860850 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:12.859050    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfgdl\" (UniqueName: \"kubernetes.io/projected/60be7b9e-07a0-4cc6-99d1-1278b6bad791-kube-api-access-xfgdl\") pod \"e2e-ipp-b-77cf70-openshift-default-d8f7564b9-rftzg\" (UID: \"60be7b9e-07a0-4cc6-99d1-1278b6bad791\") " pod="openshift-ingress/e2e-ipp-b-77cf70-openshift-default-d8f7564b9-rftzg"
Nov 02 12:45:12.860850 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:12.859229    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60be7b9e-07a0-4cc6-99d1-1278b6bad791-workload-certs\") pod \"e2e-ipp-b-77cf70-openshift-default-d8f7564b9-rftzg\" (UID: \"60be7b9e-07a0-4cc6-99d1-1278b6bad791\") " pod="openshift-ingress/e2e-ipp-b-77cf70-openshift-default-d8f7564b9-rftzg"
Nov 02 12:45:12.860850 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:12.859268    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60be7b9e-07a0-4cc6-99d1-1278b6bad791-workload-socket\") pod \"e2e-ipp-b-77cf70-openshift-default-d8f7564b9-rftzg\" (UID: \"60be7b9e-07a0-4cc6-99d1-1278b6bad791\") " pod="openshift-ingress/e2e-ipp-b-77cf70-openshift-default-d8f7564b9-rftzg"
Nov 02 12:45:12.860850 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:12.859321    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60be7b9e-07a0-4cc6-99d1-1278b6bad791-istio-data\") pod \"e2e-ipp-b-77cf70-openshift-default-d8f7564b9-rftzg\" (UID: \"60be7b9e-07a0-4cc6-99d1-1278b6bad791\") " pod="openshift-ingress/e2e-ipp-b-77cf70-openshift-default-d8f7564b9-rftzg"
Nov 02 12:45:12.860850 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:12.859880    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60be7b9e-07a0-4cc6-99d1-1278b6bad791-istio-data\") pod \"e2e-ipp-b-77cf70-openshift-default-d8f7564b9-rftzg\" (UID: \"60be7b9e-07a0-4cc6-99d1-1278b6bad791\") " pod="openshift-ingress/e2e-ipp-b-77cf70-openshift-default-d8f7564b9-rftzg"
Nov 02 12:45:12.872967 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:12.872176    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60be7b9e-07a0-4cc6-99d1-1278b6bad791-istiod-ca-cert\") pod \"e2e-ipp-b-77cf70-openshift-default-d8f7564b9-rftzg\" (UID: \"60be7b9e-07a0-4cc6-99d1-1278b6bad791\") " pod="openshift-ingress/e2e-ipp-b-77cf70-openshift-default-d8f7564b9-rftzg"
Nov 02 12:45:12.872967 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:12.872454    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60be7b9e-07a0-4cc6-99d1-1278b6bad791-credential-socket\") pod \"e2e-ipp-b-77cf70-openshift-default-d8f7564b9-rftzg\" (UID: \"60be7b9e-07a0-4cc6-99d1-1278b6bad791\") " pod="openshift-ingress/e2e-ipp-b-77cf70-openshift-default-d8f7564b9-rftzg"
Nov 02 12:45:12.873211 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:12.873138    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e6ceda-dbd545695-ggztc" podStartSLOduration=2.873121929 podStartE2EDuration="2.873121929s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:45:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:45:12.815177622 +0000 UTC m=+1109.070576719" watchObservedRunningTime="2026-11-02 12:45:12.873121929 +0000 UTC m=+1109.128521027"
Nov 02 12:45:12.874332 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:12.873748    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60be7b9e-07a0-4cc6-99d1-1278b6bad791-workload-certs\") pod \"e2e-ipp-b-77cf70-openshift-default-d8f7564b9-rftzg\" (UID: \"60be7b9e-07a0-4cc6-99d1-1278b6bad791\") " pod="openshift-ingress/e2e-ipp-b-77cf70-openshift-default-d8f7564b9-rftzg"
Nov 02 12:45:12.874332 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:12.874267    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60be7b9e-07a0-4cc6-99d1-1278b6bad791-workload-socket\") pod \"e2e-ipp-b-77cf70-openshift-default-d8f7564b9-rftzg\" (UID: \"60be7b9e-07a0-4cc6-99d1-1278b6bad791\") " pod="openshift-ingress/e2e-ipp-b-77cf70-openshift-default-d8f7564b9-rftzg"
Nov 02 12:45:12.941974 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:12.941906    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60be7b9e-07a0-4cc6-99d1-1278b6bad791-istio-podinfo\") pod \"e2e-ipp-b-77cf70-openshift-default-d8f7564b9-rftzg\" (UID: \"60be7b9e-07a0-4cc6-99d1-1278b6bad791\") " pod="openshift-ingress/e2e-ipp-b-77cf70-openshift-default-d8f7564b9-rftzg"
Nov 02 12:45:12.945667 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:12.944729    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfgdl\" (UniqueName: \"kubernetes.io/projected/60be7b9e-07a0-4cc6-99d1-1278b6bad791-kube-api-access-xfgdl\") pod \"e2e-ipp-b-77cf70-openshift-default-d8f7564b9-rftzg\" (UID: \"60be7b9e-07a0-4cc6-99d1-1278b6bad791\") " pod="openshift-ingress/e2e-ipp-b-77cf70-openshift-default-d8f7564b9-rftzg"
Nov 02 12:45:12.946897 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:12.946760    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60be7b9e-07a0-4cc6-99d1-1278b6bad791-istio-envoy\") pod \"e2e-ipp-b-77cf70-openshift-default-d8f7564b9-rftzg\" (UID: \"60be7b9e-07a0-4cc6-99d1-1278b6bad791\") " pod="openshift-ingress/e2e-ipp-b-77cf70-openshift-default-d8f7564b9-rftzg"
Nov 02 12:45:12.949136 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:12.949111    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60be7b9e-07a0-4cc6-99d1-1278b6bad791-istio-token\") pod \"e2e-ipp-b-77cf70-openshift-default-d8f7564b9-rftzg\" (UID: \"60be7b9e-07a0-4cc6-99d1-1278b6bad791\") " pod="openshift-ingress/e2e-ipp-b-77cf70-openshift-default-d8f7564b9-rftzg"
Nov 02 12:45:13.073889 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:13.073121    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-77cf70-openshift-default-d8f7564b9-rftzg"
Nov 02 12:45:13.534159 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:13.529066    4559 patch_prober.go:29] interesting pod/e2e-api-b-6a3af1-openshift-default-666bcf8bbc-2hjz9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Nov 02 12:45:13.534159 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:13.529123    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-6a3af1-openshift-default-666bcf8bbc-2hjz9" podUID="0992908f-5ad3-4600-b8d4-f26d07958ee2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Nov 02 12:45:13.548498 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:13.548368    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-77cf70-openshift-default-d8f7564b9-rftzg"]
Nov 02 12:45:13.558614 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:45:13.558572    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60be7b9e_07a0_4cc6_99d1_1278b6bad791.slice/crio-0503d6d411b77548038f4b74adec0711b53127292d7180cdcef710a29606b729 WatchSource:0}: Error finding container 0503d6d411b77548038f4b74adec0711b53127292d7180cdcef710a29606b729: Status 404 returned error can't find the container with id 0503d6d411b77548038f4b74adec0711b53127292d7180cdcef710a29606b729
Nov 02 12:45:13.566116 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:13.565610    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:45:13.566116 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:13.565715    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:45:13.566116 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:13.565747    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:45:13.771708 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:13.771677    4559 scope.go:122] "RemoveContainer" containerID="ae90ba1d7e53ff1afdd43c4373dffd2d3df6d654f6eb9166985816f1d98e1ca5"
Nov 02 12:45:13.773849 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:45:13.773813    4559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-f0cf7e-29893725-22q6j_odh-ai-gateway-infra(531f3c3c-dbae-4e74-a7ae-27a21e2a9228)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f0cf7e-29893725-22q6j" podUID="531f3c3c-dbae-4e74-a7ae-27a21e2a9228"
Nov 02 12:45:13.779312 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:13.777103    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-77cf70-openshift-default-d8f7564b9-rftzg" event={"ID":"60be7b9e-07a0-4cc6-99d1-1278b6bad791","Type":"ContainerStarted","Data":"fd000b435ffe806cb1b71a659d652a87fc1284ebe313cb5576983e842e971154"}
Nov 02 12:45:13.779572 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:13.779551    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-77cf70-openshift-default-d8f7564b9-rftzg" event={"ID":"60be7b9e-07a0-4cc6-99d1-1278b6bad791","Type":"ContainerStarted","Data":"0503d6d411b77548038f4b74adec0711b53127292d7180cdcef710a29606b729"}
Nov 02 12:45:13.808704 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:13.805429    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-e6ceda-779dbb5897-lqnhx" event={"ID":"ae88d18c-c12e-4c53-a0e0-b012a2d624ff","Type":"ContainerStarted","Data":"32d9f8b61a8d60432fa5d9681571b3f89751ba2e1b4c7a1fb08c6dff6e612330"}
Nov 02 12:45:13.821760 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:13.821734    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-e6ceda-779dbb5897-lqnhx"
Nov 02 12:45:13.910712 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:13.910634    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-77cf70-openshift-default-d8f7564b9-rftzg" podStartSLOduration=2.91061286 podStartE2EDuration="2.91061286s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:45:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:45:13.877885888 +0000 UTC m=+1110.133284987" watchObservedRunningTime="2026-11-02 12:45:13.91061286 +0000 UTC m=+1110.166011957"
Nov 02 12:45:13.963321 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:13.963253    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-e6ceda-779dbb5897-lqnhx" podStartSLOduration=3.963226743 podStartE2EDuration="3.963226743s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:45:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:45:13.946046806 +0000 UTC m=+1110.201445902" watchObservedRunningTime="2026-11-02 12:45:13.963226743 +0000 UTC m=+1110.218625840"
Nov 02 12:45:14.094221 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:14.092295    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-77cf70-openshift-default-d8f7564b9-rftzg"
Nov 02 12:45:14.094221 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:14.092444    4559 patch_prober.go:29] interesting pod/e2e-ipp-b-77cf70-openshift-default-d8f7564b9-rftzg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body=
Nov 02 12:45:14.094221 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:14.092491    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-77cf70-openshift-default-d8f7564b9-rftzg" podUID="60be7b9e-07a0-4cc6-99d1-1278b6bad791" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused"
Nov 02 12:45:14.516664 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:14.508844    4559 patch_prober.go:29] interesting pod/e2e-api-b-6a3af1-openshift-default-666bcf8bbc-2hjz9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Nov 02 12:45:14.516664 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:14.508910    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-6a3af1-openshift-default-666bcf8bbc-2hjz9" podUID="0992908f-5ad3-4600-b8d4-f26d07958ee2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Nov 02 12:45:14.830752 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:14.830676    4559 generic.go:356] "Generic (PLEG): container finished" podID="09198b81-42f1-4ae2-9e17-517a352ceaf6" containerID="c592f5c0195536d8f8ac329d76f14238acae84cb9dc1eb0685ff44105c8faafd" exitCode=137
Nov 02 12:45:15.044990 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:15.044756    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7004-5ffcd5"
Nov 02 12:45:15.118390 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:15.117995    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-77cf70-openshift-default-d8f7564b9-rftzg"
Nov 02 12:45:15.149820 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:15.148225    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09198b81-42f1-4ae2-9e17-517a352ceaf6-kube-api-access-v4h5h\" (UniqueName: \"kubernetes.io/projected/09198b81-42f1-4ae2-9e17-517a352ceaf6-kube-api-access-v4h5h\") pod \"09198b81-42f1-4ae2-9e17-517a352ceaf6\" (UID: \"09198b81-42f1-4ae2-9e17-517a352ceaf6\") "
Nov 02 12:45:15.189300 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:15.189237    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09198b81-42f1-4ae2-9e17-517a352ceaf6-kube-api-access-v4h5h" pod "09198b81-42f1-4ae2-9e17-517a352ceaf6" (UID: "09198b81-42f1-4ae2-9e17-517a352ceaf6"). InnerVolumeSpecName "kube-api-access-v4h5h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:45:15.234864 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:45:15.225162    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqcmj" containerName="extractor"
Nov 02 12:45:15.251170 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:15.251136    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v4h5h\" (UniqueName: \"kubernetes.io/projected/09198b81-42f1-4ae2-9e17-517a352ceaf6-kube-api-access-v4h5h\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:45:15.489944 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:15.489909    4559 patch_prober.go:29] interesting pod/e2e-api-b-6a3af1-openshift-default-666bcf8bbc-2hjz9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Nov 02 12:45:15.490100 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:15.489998    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-6a3af1-openshift-default-666bcf8bbc-2hjz9" podUID="0992908f-5ad3-4600-b8d4-f26d07958ee2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Nov 02 12:45:15.843601 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:15.843519    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7004-5ffcd5"
Nov 02 12:45:15.843601 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:15.843582    4559 scope.go:122] "RemoveContainer" containerID="c592f5c0195536d8f8ac329d76f14238acae84cb9dc1eb0685ff44105c8faafd"
Nov 02 12:45:15.844266 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:15.844233    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-77cf70-openshift-default-d8f7564b9-rftzg"
Nov 02 12:45:15.851122 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:15.851098    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-77cf70-openshift-default-d8f7564b9-rftzg"
Nov 02 12:45:15.967671 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:15.967062    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09198b81-42f1-4ae2-9e17-517a352ceaf6" path="/var/lib/kubelet/pods/09198b81-42f1-4ae2-9e17-517a352ceaf6/volumes"
Nov 02 12:45:15.971669 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:45:15.968954    4559 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7004-5ffcd5\" is forbidden: User \"system:node:ip-10-0-128-132.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-132.ec2.internal' and this object" podUID="09198b81-42f1-4ae2-9e17-517a352ceaf6" pod="opendatahub/test-curl-7004-5ffcd5"
Nov 02 12:45:16.088618 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:16.088586    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-nsg75"]
Nov 02 12:45:16.107763 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:16.107688    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-nsg75"
Nov 02 12:45:16.146306 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:16.146268    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-nsg75"]
Nov 02 12:45:16.176437 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:16.176377    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f0dd625-e644-4b6a-b697-c7b66b68e6ca-istio-podinfo\") pod \"e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-nsg75\" (UID: \"7f0dd625-e644-4b6a-b697-c7b66b68e6ca\") " pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-nsg75"
Nov 02 12:45:16.176660 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:16.176457    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f0dd625-e644-4b6a-b697-c7b66b68e6ca-istio-token\") pod \"e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-nsg75\" (UID: \"7f0dd625-e644-4b6a-b697-c7b66b68e6ca\") " pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-nsg75"
Nov 02 12:45:16.176660 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:16.176491    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f0dd625-e644-4b6a-b697-c7b66b68e6ca-istiod-ca-cert\") pod \"e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-nsg75\" (UID: \"7f0dd625-e644-4b6a-b697-c7b66b68e6ca\") " pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-nsg75"
Nov 02 12:45:16.176660 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:16.176560    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f0dd625-e644-4b6a-b697-c7b66b68e6ca-istio-envoy\") pod \"e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-nsg75\" (UID: \"7f0dd625-e644-4b6a-b697-c7b66b68e6ca\") " pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-nsg75"
Nov 02 12:45:16.176876 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:16.176683    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f0dd625-e644-4b6a-b697-c7b66b68e6ca-workload-socket\") pod \"e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-nsg75\" (UID: \"7f0dd625-e644-4b6a-b697-c7b66b68e6ca\") " pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-nsg75"
Nov 02 12:45:16.176876 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:16.176752    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f0dd625-e644-4b6a-b697-c7b66b68e6ca-credential-socket\") pod \"e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-nsg75\" (UID: \"7f0dd625-e644-4b6a-b697-c7b66b68e6ca\") " pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-nsg75"
Nov 02 12:45:16.176982 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:16.176890    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f0dd625-e644-4b6a-b697-c7b66b68e6ca-istio-data\") pod \"e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-nsg75\" (UID: \"7f0dd625-e644-4b6a-b697-c7b66b68e6ca\") " pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-nsg75"
Nov 02 12:45:16.176982 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:16.176926    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f0dd625-e644-4b6a-b697-c7b66b68e6ca-workload-certs\") pod \"e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-nsg75\" (UID: \"7f0dd625-e644-4b6a-b697-c7b66b68e6ca\") " pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-nsg75"
Nov 02 12:45:16.176982 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:16.176966    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4sxgt\" (UniqueName: \"kubernetes.io/projected/7f0dd625-e644-4b6a-b697-c7b66b68e6ca-kube-api-access-4sxgt\") pod \"e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-nsg75\" (UID: \"7f0dd625-e644-4b6a-b697-c7b66b68e6ca\") " pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-nsg75"
Nov 02 12:45:16.278626 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:16.278584    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f0dd625-e644-4b6a-b697-c7b66b68e6ca-workload-socket\") pod \"e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-nsg75\" (UID: \"7f0dd625-e644-4b6a-b697-c7b66b68e6ca\") " pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-nsg75"
Nov 02 12:45:16.278860 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:16.278674    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f0dd625-e644-4b6a-b697-c7b66b68e6ca-credential-socket\") pod \"e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-nsg75\" (UID: \"7f0dd625-e644-4b6a-b697-c7b66b68e6ca\") " pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-nsg75"
Nov 02 12:45:16.278860 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:16.278787    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f0dd625-e644-4b6a-b697-c7b66b68e6ca-istio-data\") pod \"e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-nsg75\" (UID: \"7f0dd625-e644-4b6a-b697-c7b66b68e6ca\") " pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-nsg75"
Nov 02 12:45:16.278860 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:16.278821    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f0dd625-e644-4b6a-b697-c7b66b68e6ca-workload-certs\") pod \"e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-nsg75\" (UID: \"7f0dd625-e644-4b6a-b697-c7b66b68e6ca\") " pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-nsg75"
Nov 02 12:45:16.278860 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:16.278849    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4sxgt\" (UniqueName: \"kubernetes.io/projected/7f0dd625-e644-4b6a-b697-c7b66b68e6ca-kube-api-access-4sxgt\") pod \"e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-nsg75\" (UID: \"7f0dd625-e644-4b6a-b697-c7b66b68e6ca\") " pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-nsg75"
Nov 02 12:45:16.279130 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:16.279081    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f0dd625-e644-4b6a-b697-c7b66b68e6ca-credential-socket\") pod \"e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-nsg75\" (UID: \"7f0dd625-e644-4b6a-b697-c7b66b68e6ca\") " pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-nsg75"
Nov 02 12:45:16.279130 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:16.279108    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f0dd625-e644-4b6a-b697-c7b66b68e6ca-istio-podinfo\") pod \"e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-nsg75\" (UID: \"7f0dd625-e644-4b6a-b697-c7b66b68e6ca\") " pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-nsg75"
Nov 02 12:45:16.279253 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:16.279150    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f0dd625-e644-4b6a-b697-c7b66b68e6ca-istio-token\") pod \"e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-nsg75\" (UID: \"7f0dd625-e644-4b6a-b697-c7b66b68e6ca\") " pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-nsg75"
Nov 02 12:45:16.279253 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:16.279180    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f0dd625-e644-4b6a-b697-c7b66b68e6ca-istiod-ca-cert\") pod \"e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-nsg75\" (UID: \"7f0dd625-e644-4b6a-b697-c7b66b68e6ca\") " pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-nsg75"
Nov 02 12:45:16.279253 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:16.279223    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f0dd625-e644-4b6a-b697-c7b66b68e6ca-istio-envoy\") pod \"e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-nsg75\" (UID: \"7f0dd625-e644-4b6a-b697-c7b66b68e6ca\") " pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-nsg75"
Nov 02 12:45:16.279485 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:16.279330    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f0dd625-e644-4b6a-b697-c7b66b68e6ca-workload-certs\") pod \"e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-nsg75\" (UID: \"7f0dd625-e644-4b6a-b697-c7b66b68e6ca\") " pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-nsg75"
Nov 02 12:45:16.279608 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:16.279587    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f0dd625-e644-4b6a-b697-c7b66b68e6ca-istio-data\") pod \"e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-nsg75\" (UID: \"7f0dd625-e644-4b6a-b697-c7b66b68e6ca\") " pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-nsg75"
Nov 02 12:45:16.281173 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:16.281144    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f0dd625-e644-4b6a-b697-c7b66b68e6ca-istiod-ca-cert\") pod \"e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-nsg75\" (UID: \"7f0dd625-e644-4b6a-b697-c7b66b68e6ca\") " pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-nsg75"
Nov 02 12:45:16.281477 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:16.281452    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f0dd625-e644-4b6a-b697-c7b66b68e6ca-workload-socket\") pod \"e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-nsg75\" (UID: \"7f0dd625-e644-4b6a-b697-c7b66b68e6ca\") " pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-nsg75"
Nov 02 12:45:16.282184 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:16.282160    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f0dd625-e644-4b6a-b697-c7b66b68e6ca-istio-envoy\") pod \"e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-nsg75\" (UID: \"7f0dd625-e644-4b6a-b697-c7b66b68e6ca\") " pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-nsg75"
Nov 02 12:45:16.287279 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:16.287254    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f0dd625-e644-4b6a-b697-c7b66b68e6ca-istio-podinfo\") pod \"e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-nsg75\" (UID: \"7f0dd625-e644-4b6a-b697-c7b66b68e6ca\") " pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-nsg75"
Nov 02 12:45:16.308444 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:16.308416    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4sxgt\" (UniqueName: \"kubernetes.io/projected/7f0dd625-e644-4b6a-b697-c7b66b68e6ca-kube-api-access-4sxgt\") pod \"e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-nsg75\" (UID: \"7f0dd625-e644-4b6a-b697-c7b66b68e6ca\") " pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-nsg75"
Nov 02 12:45:16.309751 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:16.309726    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f0dd625-e644-4b6a-b697-c7b66b68e6ca-istio-token\") pod \"e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-nsg75\" (UID: \"7f0dd625-e644-4b6a-b697-c7b66b68e6ca\") " pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-nsg75"
Nov 02 12:45:16.451903 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:16.451870    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-nsg75"
Nov 02 12:45:16.495499 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:16.495476    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-6a3af1-openshift-default-666bcf8bbc-2hjz9"
Nov 02 12:45:16.496653 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:16.496614    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-6a3af1-openshift-default-666bcf8bbc-2hjz9"
Nov 02 12:45:16.799390 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:45:16.799363    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f0dd625_e644_4b6a_b697_c7b66b68e6ca.slice/crio-cca1564adb48724f6609b65d3d771b566f800c54113c74e5a44539b4df73258a WatchSource:0}: Error finding container cca1564adb48724f6609b65d3d771b566f800c54113c74e5a44539b4df73258a: Status 404 returned error can't find the container with id cca1564adb48724f6609b65d3d771b566f800c54113c74e5a44539b4df73258a
Nov 02 12:45:16.801649 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:16.801615    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-nsg75"]
Nov 02 12:45:16.802515 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:16.802484    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:45:16.802578 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:16.802560    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:45:16.802616 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:16.802595    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:45:16.867404 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:16.867369    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-nsg75" event={"ID":"7f0dd625-e644-4b6a-b697-c7b66b68e6ca","Type":"ContainerStarted","Data":"cca1564adb48724f6609b65d3d771b566f800c54113c74e5a44539b4df73258a"}
Nov 02 12:45:17.175185 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:17.163389    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-cd1386cb/e2e-lifecycle-model-cd1386cb-kserve-7d9d698845-2579r"]
Nov 02 12:45:17.877110 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:17.877026    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-nsg75" event={"ID":"7f0dd625-e644-4b6a-b697-c7b66b68e6ca","Type":"ContainerStarted","Data":"54d254365c0ad15d60a825a2d5ee152c233fce1ea4d2e496e4b4bcc256a9dc2e"}
Nov 02 12:45:17.879001 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:17.878975    4559 generic.go:356] "Generic (PLEG): container finished" podID="c2989390-a19e-408b-ac9e-ec01df76bf12" containerID="0eafe970aeb286d1e6ba5e03bba870e417ca71634a3d1606425969ba2fad7d3e" exitCode=0
Nov 02 12:45:17.879133 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:17.879050    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-rxdxm" event={"ID":"c2989390-a19e-408b-ac9e-ec01df76bf12","Type":"ContainerDied","Data":"0eafe970aeb286d1e6ba5e03bba870e417ca71634a3d1606425969ba2fad7d3e"}
Nov 02 12:45:17.879592 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:17.879575    4559 scope.go:122] "RemoveContainer" containerID="0eafe970aeb286d1e6ba5e03bba870e417ca71634a3d1606425969ba2fad7d3e"
Nov 02 12:45:17.880982 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:17.880952    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:45:17.881087 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:17.881033    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:45:17.881087 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:17.881081    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:45:17.937157 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:17.937111    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-nsg75" podStartSLOduration=1.937097053 podStartE2EDuration="1.937097053s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:45:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:45:17.933978974 +0000 UTC m=+1114.189378069" watchObservedRunningTime="2026-11-02 12:45:17.937097053 +0000 UTC m=+1114.192496148"
Nov 02 12:45:18.313918 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:18.313762    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-rxdxm"
Nov 02 12:45:18.452328 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:18.452301    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-nsg75"
Nov 02 12:45:18.453891 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:18.453864    4559 patch_prober.go:29] interesting pod/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-nsg75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body=
Nov 02 12:45:18.453891 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:18.453950    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-nsg75" podUID="7f0dd625-e644-4b6a-b697-c7b66b68e6ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused"
Nov 02 12:45:18.763667 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:18.763578    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-6a3af1-openshift-default-666bcf8bbc-2hjz9"]
Nov 02 12:45:18.788669 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:18.763854    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-6a3af1-openshift-default-666bcf8bbc-2hjz9" podUID="0992908f-5ad3-4600-b8d4-f26d07958ee2" containerName="istio-proxy" containerID="cri-o://10113f71438d69865cfe93a7947741f9f87b9e9201ff8a4bf8dad0e71a872faf" gracePeriod=30
Nov 02 12:45:19.362718 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:19.279953    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-rxdxm" event={"ID":"c2989390-a19e-408b-ac9e-ec01df76bf12","Type":"ContainerStarted","Data":"3e3df72cf5e7359ef77f2f5048e63529635037abf749c2a0de01c73b4c1c00c8"}
Nov 02 12:45:19.482080 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:19.482045    4559 patch_prober.go:29] interesting pod/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-nsg75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body=
Nov 02 12:45:19.482368 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:19.482337    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-nsg75" podUID="7f0dd625-e644-4b6a-b697-c7b66b68e6ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused"
Nov 02 12:45:19.716620 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:19.716579    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-61e341/distinct-2-w1-61e341-kserve-954cf956f-n7jmg" podUID="c5e1a5f8-97b4-469d-be9a-eae7ca995283" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded"
Nov 02 12:45:20.215013 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:20.214978    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-rxdxm"
Nov 02 12:45:20.215369 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:20.215341    4559 patch_prober.go:29] interesting pod/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-rxdxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Nov 02 12:45:20.215545 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:20.215518    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-rxdxm" podUID="c2989390-a19e-408b-ac9e-ec01df76bf12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Nov 02 12:45:20.413268 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:20.413229    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-59060a-openshift-default-8666cdf765-hfz77"]
Nov 02 12:45:20.496309 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:20.496219    4559 patch_prober.go:29] interesting pod/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-nsg75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body=
Nov 02 12:45:20.496565 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:20.496534    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-nsg75" podUID="7f0dd625-e644-4b6a-b697-c7b66b68e6ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused"
Nov 02 12:45:20.878716 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:20.874336    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-59060a-openshift-default-8666cdf765-hfz77"]
Nov 02 12:45:20.878716 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:20.874483    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-59060a-openshift-default-8666cdf765-hfz77"
Nov 02 12:45:20.952362 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:20.948842    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-59060a-openshift-default-dockercfg-pvnxb\""
Nov 02 12:45:21.057727 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:21.027193    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ed1a149-3a01-4af1-9b01-e059ff121bed-workload-socket\") pod \"e2e-ipp-a-59060a-openshift-default-8666cdf765-hfz77\" (UID: \"9ed1a149-3a01-4af1-9b01-e059ff121bed\") " pod="openshift-ingress/e2e-ipp-a-59060a-openshift-default-8666cdf765-hfz77"
Nov 02 12:45:21.058423 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:21.058390    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ed1a149-3a01-4af1-9b01-e059ff121bed-istio-token\") pod \"e2e-ipp-a-59060a-openshift-default-8666cdf765-hfz77\" (UID: \"9ed1a149-3a01-4af1-9b01-e059ff121bed\") " pod="openshift-ingress/e2e-ipp-a-59060a-openshift-default-8666cdf765-hfz77"
Nov 02 12:45:21.061052 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:21.058682    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ed1a149-3a01-4af1-9b01-e059ff121bed-istiod-ca-cert\") pod \"e2e-ipp-a-59060a-openshift-default-8666cdf765-hfz77\" (UID: \"9ed1a149-3a01-4af1-9b01-e059ff121bed\") " pod="openshift-ingress/e2e-ipp-a-59060a-openshift-default-8666cdf765-hfz77"
Nov 02 12:45:21.061243 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:21.061223    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ed1a149-3a01-4af1-9b01-e059ff121bed-istio-envoy\") pod \"e2e-ipp-a-59060a-openshift-default-8666cdf765-hfz77\" (UID: \"9ed1a149-3a01-4af1-9b01-e059ff121bed\") " pod="openshift-ingress/e2e-ipp-a-59060a-openshift-default-8666cdf765-hfz77"
Nov 02 12:45:21.062874 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:21.062847    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ed1a149-3a01-4af1-9b01-e059ff121bed-credential-socket\") pod \"e2e-ipp-a-59060a-openshift-default-8666cdf765-hfz77\" (UID: \"9ed1a149-3a01-4af1-9b01-e059ff121bed\") " pod="openshift-ingress/e2e-ipp-a-59060a-openshift-default-8666cdf765-hfz77"
Nov 02 12:45:21.063250 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:21.063228    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ed1a149-3a01-4af1-9b01-e059ff121bed-workload-certs\") pod \"e2e-ipp-a-59060a-openshift-default-8666cdf765-hfz77\" (UID: \"9ed1a149-3a01-4af1-9b01-e059ff121bed\") " pod="openshift-ingress/e2e-ipp-a-59060a-openshift-default-8666cdf765-hfz77"
Nov 02 12:45:21.063443 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:21.063426    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ed1a149-3a01-4af1-9b01-e059ff121bed-istio-podinfo\") pod \"e2e-ipp-a-59060a-openshift-default-8666cdf765-hfz77\" (UID: \"9ed1a149-3a01-4af1-9b01-e059ff121bed\") " pod="openshift-ingress/e2e-ipp-a-59060a-openshift-default-8666cdf765-hfz77"
Nov 02 12:45:21.063664 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:21.063630    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tr8md\" (UniqueName: \"kubernetes.io/projected/9ed1a149-3a01-4af1-9b01-e059ff121bed-kube-api-access-tr8md\") pod \"e2e-ipp-a-59060a-openshift-default-8666cdf765-hfz77\" (UID: \"9ed1a149-3a01-4af1-9b01-e059ff121bed\") " pod="openshift-ingress/e2e-ipp-a-59060a-openshift-default-8666cdf765-hfz77"
Nov 02 12:45:21.063969 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:21.063952    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ed1a149-3a01-4af1-9b01-e059ff121bed-istio-data\") pod \"e2e-ipp-a-59060a-openshift-default-8666cdf765-hfz77\" (UID: \"9ed1a149-3a01-4af1-9b01-e059ff121bed\") " pod="openshift-ingress/e2e-ipp-a-59060a-openshift-default-8666cdf765-hfz77"
Nov 02 12:45:21.165842 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:21.165741    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ed1a149-3a01-4af1-9b01-e059ff121bed-istio-data\") pod \"e2e-ipp-a-59060a-openshift-default-8666cdf765-hfz77\" (UID: \"9ed1a149-3a01-4af1-9b01-e059ff121bed\") " pod="openshift-ingress/e2e-ipp-a-59060a-openshift-default-8666cdf765-hfz77"
Nov 02 12:45:21.165991 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:21.165872    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ed1a149-3a01-4af1-9b01-e059ff121bed-workload-socket\") pod \"e2e-ipp-a-59060a-openshift-default-8666cdf765-hfz77\" (UID: \"9ed1a149-3a01-4af1-9b01-e059ff121bed\") " pod="openshift-ingress/e2e-ipp-a-59060a-openshift-default-8666cdf765-hfz77"
Nov 02 12:45:21.165991 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:21.165909    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ed1a149-3a01-4af1-9b01-e059ff121bed-istio-token\") pod \"e2e-ipp-a-59060a-openshift-default-8666cdf765-hfz77\" (UID: \"9ed1a149-3a01-4af1-9b01-e059ff121bed\") " pod="openshift-ingress/e2e-ipp-a-59060a-openshift-default-8666cdf765-hfz77"
Nov 02 12:45:21.165991 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:21.165956    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ed1a149-3a01-4af1-9b01-e059ff121bed-istiod-ca-cert\") pod \"e2e-ipp-a-59060a-openshift-default-8666cdf765-hfz77\" (UID: \"9ed1a149-3a01-4af1-9b01-e059ff121bed\") " pod="openshift-ingress/e2e-ipp-a-59060a-openshift-default-8666cdf765-hfz77"
Nov 02 12:45:21.165991 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:21.165982    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ed1a149-3a01-4af1-9b01-e059ff121bed-istio-envoy\") pod \"e2e-ipp-a-59060a-openshift-default-8666cdf765-hfz77\" (UID: \"9ed1a149-3a01-4af1-9b01-e059ff121bed\") " pod="openshift-ingress/e2e-ipp-a-59060a-openshift-default-8666cdf765-hfz77"
Nov 02 12:45:21.166132 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:21.166017    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ed1a149-3a01-4af1-9b01-e059ff121bed-credential-socket\") pod \"e2e-ipp-a-59060a-openshift-default-8666cdf765-hfz77\" (UID: \"9ed1a149-3a01-4af1-9b01-e059ff121bed\") " pod="openshift-ingress/e2e-ipp-a-59060a-openshift-default-8666cdf765-hfz77"
Nov 02 12:45:21.166233 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:21.166191    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ed1a149-3a01-4af1-9b01-e059ff121bed-workload-certs\") pod \"e2e-ipp-a-59060a-openshift-default-8666cdf765-hfz77\" (UID: \"9ed1a149-3a01-4af1-9b01-e059ff121bed\") " pod="openshift-ingress/e2e-ipp-a-59060a-openshift-default-8666cdf765-hfz77"
Nov 02 12:45:21.166309 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:21.166232    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ed1a149-3a01-4af1-9b01-e059ff121bed-istio-podinfo\") pod \"e2e-ipp-a-59060a-openshift-default-8666cdf765-hfz77\" (UID: \"9ed1a149-3a01-4af1-9b01-e059ff121bed\") " pod="openshift-ingress/e2e-ipp-a-59060a-openshift-default-8666cdf765-hfz77"
Nov 02 12:45:21.166309 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:21.166282    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tr8md\" (UniqueName: \"kubernetes.io/projected/9ed1a149-3a01-4af1-9b01-e059ff121bed-kube-api-access-tr8md\") pod \"e2e-ipp-a-59060a-openshift-default-8666cdf765-hfz77\" (UID: \"9ed1a149-3a01-4af1-9b01-e059ff121bed\") " pod="openshift-ingress/e2e-ipp-a-59060a-openshift-default-8666cdf765-hfz77"
Nov 02 12:45:21.167168 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:21.167028    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ed1a149-3a01-4af1-9b01-e059ff121bed-istio-data\") pod \"e2e-ipp-a-59060a-openshift-default-8666cdf765-hfz77\" (UID: \"9ed1a149-3a01-4af1-9b01-e059ff121bed\") " pod="openshift-ingress/e2e-ipp-a-59060a-openshift-default-8666cdf765-hfz77"
Nov 02 12:45:21.167331 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:21.167310    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ed1a149-3a01-4af1-9b01-e059ff121bed-credential-socket\") pod \"e2e-ipp-a-59060a-openshift-default-8666cdf765-hfz77\" (UID: \"9ed1a149-3a01-4af1-9b01-e059ff121bed\") " pod="openshift-ingress/e2e-ipp-a-59060a-openshift-default-8666cdf765-hfz77"
Nov 02 12:45:21.167406 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:21.167375    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ed1a149-3a01-4af1-9b01-e059ff121bed-workload-certs\") pod \"e2e-ipp-a-59060a-openshift-default-8666cdf765-hfz77\" (UID: \"9ed1a149-3a01-4af1-9b01-e059ff121bed\") " pod="openshift-ingress/e2e-ipp-a-59060a-openshift-default-8666cdf765-hfz77"
Nov 02 12:45:21.167859 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:21.167841    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ed1a149-3a01-4af1-9b01-e059ff121bed-workload-socket\") pod \"e2e-ipp-a-59060a-openshift-default-8666cdf765-hfz77\" (UID: \"9ed1a149-3a01-4af1-9b01-e059ff121bed\") " pod="openshift-ingress/e2e-ipp-a-59060a-openshift-default-8666cdf765-hfz77"
Nov 02 12:45:21.168776 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:21.168754    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ed1a149-3a01-4af1-9b01-e059ff121bed-istiod-ca-cert\") pod \"e2e-ipp-a-59060a-openshift-default-8666cdf765-hfz77\" (UID: \"9ed1a149-3a01-4af1-9b01-e059ff121bed\") " pod="openshift-ingress/e2e-ipp-a-59060a-openshift-default-8666cdf765-hfz77"
Nov 02 12:45:21.170608 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:21.170585    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ed1a149-3a01-4af1-9b01-e059ff121bed-istio-podinfo\") pod \"e2e-ipp-a-59060a-openshift-default-8666cdf765-hfz77\" (UID: \"9ed1a149-3a01-4af1-9b01-e059ff121bed\") " pod="openshift-ingress/e2e-ipp-a-59060a-openshift-default-8666cdf765-hfz77"
Nov 02 12:45:21.171074 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:21.171052    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ed1a149-3a01-4af1-9b01-e059ff121bed-istio-envoy\") pod \"e2e-ipp-a-59060a-openshift-default-8666cdf765-hfz77\" (UID: \"9ed1a149-3a01-4af1-9b01-e059ff121bed\") " pod="openshift-ingress/e2e-ipp-a-59060a-openshift-default-8666cdf765-hfz77"
Nov 02 12:45:21.235661 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:21.231934    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tr8md\" (UniqueName: \"kubernetes.io/projected/9ed1a149-3a01-4af1-9b01-e059ff121bed-kube-api-access-tr8md\") pod \"e2e-ipp-a-59060a-openshift-default-8666cdf765-hfz77\" (UID: \"9ed1a149-3a01-4af1-9b01-e059ff121bed\") " pod="openshift-ingress/e2e-ipp-a-59060a-openshift-default-8666cdf765-hfz77"
Nov 02 12:45:21.235661 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:21.234108    4559 patch_prober.go:29] interesting pod/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-rxdxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Nov 02 12:45:21.235661 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:21.234157    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-rxdxm" podUID="c2989390-a19e-408b-ac9e-ec01df76bf12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Nov 02 12:45:21.256697 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:21.253266    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ed1a149-3a01-4af1-9b01-e059ff121bed-istio-token\") pod \"e2e-ipp-a-59060a-openshift-default-8666cdf765-hfz77\" (UID: \"9ed1a149-3a01-4af1-9b01-e059ff121bed\") " pod="openshift-ingress/e2e-ipp-a-59060a-openshift-default-8666cdf765-hfz77"
Nov 02 12:45:21.288038 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:21.287784    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-59060a-openshift-default-8666cdf765-hfz77"
Nov 02 12:45:21.468853 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:21.468815    4559 patch_prober.go:29] interesting pod/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-nsg75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body=
Nov 02 12:45:21.474764 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:21.474720    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-nsg75" podUID="7f0dd625-e644-4b6a-b697-c7b66b68e6ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused"
Nov 02 12:45:21.940873 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:45:21.940831    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ed1a149_3a01_4af1_9b01_e059ff121bed.slice/crio-2616f33b24e8ea4555d9d4444d083d2f499bb0221abb9070afbb38284e9fd340 WatchSource:0}: Error finding container 2616f33b24e8ea4555d9d4444d083d2f499bb0221abb9070afbb38284e9fd340: Status 404 returned error can't find the container with id 2616f33b24e8ea4555d9d4444d083d2f499bb0221abb9070afbb38284e9fd340
Nov 02 12:45:21.944549 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:21.944513    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:45:21.944677 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:21.944581    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:45:21.944677 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:21.944610    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:45:21.946609 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:21.946589    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-59060a-openshift-default-8666cdf765-hfz77"]
Nov 02 12:45:22.222622 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:22.216293    4559 patch_prober.go:29] interesting pod/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-rxdxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Nov 02 12:45:22.222622 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:22.216354    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-rxdxm" podUID="c2989390-a19e-408b-ac9e-ec01df76bf12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Nov 02 12:45:22.388106 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:22.388028    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-59060a-openshift-default-8666cdf765-hfz77" event={"ID":"9ed1a149-3a01-4af1-9b01-e059ff121bed","Type":"ContainerStarted","Data":"2616f33b24e8ea4555d9d4444d083d2f499bb0221abb9070afbb38284e9fd340"}
Nov 02 12:45:22.455660 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:22.455058    4559 patch_prober.go:29] interesting pod/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-nsg75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body=
Nov 02 12:45:22.455660 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:22.455123    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-nsg75" podUID="7f0dd625-e644-4b6a-b697-c7b66b68e6ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused"
Nov 02 12:45:23.212363 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:23.212327    4559 patch_prober.go:29] interesting pod/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-rxdxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Nov 02 12:45:23.212988 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:23.212945    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-rxdxm" podUID="c2989390-a19e-408b-ac9e-ec01df76bf12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Nov 02 12:45:23.437753 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:23.430801    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-59060a-openshift-default-8666cdf765-hfz77" event={"ID":"9ed1a149-3a01-4af1-9b01-e059ff121bed","Type":"ContainerStarted","Data":"53df5ae504ad7e4d8df77fb7760dd73b569c12cb28a56f5a0cb1bb2f5402e97e"}
Nov 02 12:45:23.452903 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:23.452874    4559 patch_prober.go:29] interesting pod/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-nsg75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body=
Nov 02 12:45:23.453041 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:23.452934    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-nsg75" podUID="7f0dd625-e644-4b6a-b697-c7b66b68e6ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused"
Nov 02 12:45:23.537499 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:23.537375    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-59060a-openshift-default-8666cdf765-hfz77" podStartSLOduration=3.53735533 podStartE2EDuration="3.53735533s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:45:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:45:23.501281396 +0000 UTC m=+1119.756680494" watchObservedRunningTime="2026-11-02 12:45:23.53735533 +0000 UTC m=+1119.792754416"
Nov 02 12:45:24.231735 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:24.229298    4559 patch_prober.go:29] interesting pod/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-rxdxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Nov 02 12:45:24.231735 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:24.229362    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-rxdxm" podUID="c2989390-a19e-408b-ac9e-ec01df76bf12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Nov 02 12:45:24.275879 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:24.275848    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-6a3af1-openshift-default-666bcf8bbc-2hjz9"
Nov 02 12:45:24.316662 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:24.315907    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-59060a-openshift-default-8666cdf765-hfz77"
Nov 02 12:45:24.332734 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:24.329976    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-59060a-openshift-default-8666cdf765-hfz77"
Nov 02 12:45:24.344003 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:24.343967    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0992908f-5ad3-4600-b8d4-f26d07958ee2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0992908f-5ad3-4600-b8d4-f26d07958ee2-workload-socket\") pod \"0992908f-5ad3-4600-b8d4-f26d07958ee2\" (UID: \"0992908f-5ad3-4600-b8d4-f26d07958ee2\") "
Nov 02 12:45:24.344213 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:24.344036    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0992908f-5ad3-4600-b8d4-f26d07958ee2-kube-api-access-jjdcj\" (UniqueName: \"kubernetes.io/projected/0992908f-5ad3-4600-b8d4-f26d07958ee2-kube-api-access-jjdcj\") pod \"0992908f-5ad3-4600-b8d4-f26d07958ee2\" (UID: \"0992908f-5ad3-4600-b8d4-f26d07958ee2\") "
Nov 02 12:45:24.344213 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:24.344093    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0992908f-5ad3-4600-b8d4-f26d07958ee2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0992908f-5ad3-4600-b8d4-f26d07958ee2-istio-data\") pod \"0992908f-5ad3-4600-b8d4-f26d07958ee2\" (UID: \"0992908f-5ad3-4600-b8d4-f26d07958ee2\") "
Nov 02 12:45:24.344213 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:24.344149    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0992908f-5ad3-4600-b8d4-f26d07958ee2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0992908f-5ad3-4600-b8d4-f26d07958ee2-credential-socket\") pod \"0992908f-5ad3-4600-b8d4-f26d07958ee2\" (UID: \"0992908f-5ad3-4600-b8d4-f26d07958ee2\") "
Nov 02 12:45:24.344377 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:24.344337    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0992908f-5ad3-4600-b8d4-f26d07958ee2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0992908f-5ad3-4600-b8d4-f26d07958ee2-istio-podinfo\") pod \"0992908f-5ad3-4600-b8d4-f26d07958ee2\" (UID: \"0992908f-5ad3-4600-b8d4-f26d07958ee2\") "
Nov 02 12:45:24.344432 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:24.344404    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0992908f-5ad3-4600-b8d4-f26d07958ee2-istio-token\" (UniqueName: \"kubernetes.io/projected/0992908f-5ad3-4600-b8d4-f26d07958ee2-istio-token\") pod \"0992908f-5ad3-4600-b8d4-f26d07958ee2\" (UID: \"0992908f-5ad3-4600-b8d4-f26d07958ee2\") "
Nov 02 12:45:24.344588 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:24.344474    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0992908f-5ad3-4600-b8d4-f26d07958ee2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0992908f-5ad3-4600-b8d4-f26d07958ee2-istiod-ca-cert\") pod \"0992908f-5ad3-4600-b8d4-f26d07958ee2\" (UID: \"0992908f-5ad3-4600-b8d4-f26d07958ee2\") "
Nov 02 12:45:24.344810 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:24.344782    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0992908f-5ad3-4600-b8d4-f26d07958ee2-workload-socket" pod "0992908f-5ad3-4600-b8d4-f26d07958ee2" (UID: "0992908f-5ad3-4600-b8d4-f26d07958ee2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:45:24.346416 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:24.346386    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0992908f-5ad3-4600-b8d4-f26d07958ee2-istiod-ca-cert" pod "0992908f-5ad3-4600-b8d4-f26d07958ee2" (UID: "0992908f-5ad3-4600-b8d4-f26d07958ee2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:45:24.346517 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:24.346483    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0992908f-5ad3-4600-b8d4-f26d07958ee2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0992908f-5ad3-4600-b8d4-f26d07958ee2-workload-certs\") pod \"0992908f-5ad3-4600-b8d4-f26d07958ee2\" (UID: \"0992908f-5ad3-4600-b8d4-f26d07958ee2\") "
Nov 02 12:45:24.346578 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:24.346550    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0992908f-5ad3-4600-b8d4-f26d07958ee2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0992908f-5ad3-4600-b8d4-f26d07958ee2-istio-envoy\") pod \"0992908f-5ad3-4600-b8d4-f26d07958ee2\" (UID: \"0992908f-5ad3-4600-b8d4-f26d07958ee2\") "
Nov 02 12:45:24.347448 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:24.347260    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0992908f-5ad3-4600-b8d4-f26d07958ee2-istio-data" pod "0992908f-5ad3-4600-b8d4-f26d07958ee2" (UID: "0992908f-5ad3-4600-b8d4-f26d07958ee2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:45:24.347448 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:24.347438    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0992908f-5ad3-4600-b8d4-f26d07958ee2-istio-podinfo" pod "0992908f-5ad3-4600-b8d4-f26d07958ee2" (UID: "0992908f-5ad3-4600-b8d4-f26d07958ee2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 12:45:24.347700 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:24.347628    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0992908f-5ad3-4600-b8d4-f26d07958ee2-workload-certs" pod "0992908f-5ad3-4600-b8d4-f26d07958ee2" (UID: "0992908f-5ad3-4600-b8d4-f26d07958ee2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:45:24.348111 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:24.348078    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0992908f-5ad3-4600-b8d4-f26d07958ee2-istio-token" pod "0992908f-5ad3-4600-b8d4-f26d07958ee2" (UID: "0992908f-5ad3-4600-b8d4-f26d07958ee2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:45:24.350065 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:24.350021    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0992908f-5ad3-4600-b8d4-f26d07958ee2-istio-envoy" pod "0992908f-5ad3-4600-b8d4-f26d07958ee2" (UID: "0992908f-5ad3-4600-b8d4-f26d07958ee2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:45:24.350529 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:24.350419    4559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0992908f-5ad3-4600-b8d4-f26d07958ee2-istio-podinfo\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:45:24.350529 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:24.350444    4559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0992908f-5ad3-4600-b8d4-f26d07958ee2-istiod-ca-cert\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:45:24.350529 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:24.350460    4559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0992908f-5ad3-4600-b8d4-f26d07958ee2-workload-certs\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:45:24.350529 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:24.350472    4559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0992908f-5ad3-4600-b8d4-f26d07958ee2-istio-envoy\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:45:24.350529 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:24.350482    4559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0992908f-5ad3-4600-b8d4-f26d07958ee2-workload-socket\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:45:24.350529 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:24.350494    4559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0992908f-5ad3-4600-b8d4-f26d07958ee2-istio-data\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:45:24.350529 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:24.350501    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0992908f-5ad3-4600-b8d4-f26d07958ee2-kube-api-access-jjdcj" pod "0992908f-5ad3-4600-b8d4-f26d07958ee2" (UID: "0992908f-5ad3-4600-b8d4-f26d07958ee2"). InnerVolumeSpecName "kube-api-access-jjdcj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:45:24.352242 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:24.352221    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0992908f-5ad3-4600-b8d4-f26d07958ee2-credential-socket" pod "0992908f-5ad3-4600-b8d4-f26d07958ee2" (UID: "0992908f-5ad3-4600-b8d4-f26d07958ee2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:45:24.465116 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:24.460433    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jjdcj\" (UniqueName: \"kubernetes.io/projected/0992908f-5ad3-4600-b8d4-f26d07958ee2-kube-api-access-jjdcj\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:45:24.465116 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:24.460473    4559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0992908f-5ad3-4600-b8d4-f26d07958ee2-credential-socket\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:45:24.465116 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:24.460488    4559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0992908f-5ad3-4600-b8d4-f26d07958ee2-istio-token\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:45:24.490744 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:24.485134    4559 patch_prober.go:29] interesting pod/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-nsg75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body=
Nov 02 12:45:24.490744 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:24.485192    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-nsg75" podUID="7f0dd625-e644-4b6a-b697-c7b66b68e6ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused"
Nov 02 12:45:24.758325 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:24.758244    4559 generic.go:356] "Generic (PLEG): container finished" podID="0992908f-5ad3-4600-b8d4-f26d07958ee2" containerID="10113f71438d69865cfe93a7947741f9f87b9e9201ff8a4bf8dad0e71a872faf" exitCode=0
Nov 02 12:45:24.759911 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:24.759870    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-6a3af1-openshift-default-666bcf8bbc-2hjz9"
Nov 02 12:45:24.824539 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:24.824460    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-6a3af1-openshift-default-666bcf8bbc-2hjz9" event={"ID":"0992908f-5ad3-4600-b8d4-f26d07958ee2","Type":"ContainerDied","Data":"10113f71438d69865cfe93a7947741f9f87b9e9201ff8a4bf8dad0e71a872faf"}
Nov 02 12:45:24.824539 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:24.824537    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-6a3af1-openshift-default-666bcf8bbc-2hjz9" event={"ID":"0992908f-5ad3-4600-b8d4-f26d07958ee2","Type":"ContainerDied","Data":"7bf0822da13088100666569a3d151001bc54e5fb2866e48502e17c8510bac656"}
Nov 02 12:45:24.835161 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:24.824566    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-59060a-openshift-default-8666cdf765-hfz77"
Nov 02 12:45:24.835161 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:24.824596    4559 scope.go:122] "RemoveContainer" containerID="10113f71438d69865cfe93a7947741f9f87b9e9201ff8a4bf8dad0e71a872faf"
Nov 02 12:45:24.835161 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:24.827749    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-59060a-openshift-default-8666cdf765-hfz77"
Nov 02 12:45:24.845781 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:24.845753    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-e6ceda-779dbb5897-lqnhx"
Nov 02 12:45:24.846561 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:24.846537    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e6ceda-dbd545695-ggztc"
Nov 02 12:45:24.847385 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:24.847362    4559 scope.go:122] "RemoveContainer" containerID="3c680aa9299d50dbc18dc2288a5ba61232752ea9c393f42d1f8ab47b681c7714"
Nov 02 12:45:24.870104 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:24.870085    4559 scope.go:122] "RemoveContainer" containerID="10113f71438d69865cfe93a7947741f9f87b9e9201ff8a4bf8dad0e71a872faf"
Nov 02 12:45:24.870515 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:45:24.870466    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10113f71438d69865cfe93a7947741f9f87b9e9201ff8a4bf8dad0e71a872faf\": container with ID starting with 10113f71438d69865cfe93a7947741f9f87b9e9201ff8a4bf8dad0e71a872faf not found: ID does not exist" containerID="10113f71438d69865cfe93a7947741f9f87b9e9201ff8a4bf8dad0e71a872faf"
Nov 02 12:45:24.870515 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:24.870507    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10113f71438d69865cfe93a7947741f9f87b9e9201ff8a4bf8dad0e71a872faf"} err="failed to get container status \"10113f71438d69865cfe93a7947741f9f87b9e9201ff8a4bf8dad0e71a872faf\": rpc error: code = NotFound desc = could not find container \"10113f71438d69865cfe93a7947741f9f87b9e9201ff8a4bf8dad0e71a872faf\": container with ID starting with 10113f71438d69865cfe93a7947741f9f87b9e9201ff8a4bf8dad0e71a872faf not found: ID does not exist"
Nov 02 12:45:24.870515 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:24.870532    4559 scope.go:122] "RemoveContainer" containerID="3c680aa9299d50dbc18dc2288a5ba61232752ea9c393f42d1f8ab47b681c7714"
Nov 02 12:45:24.873725 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:45:24.871625    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c680aa9299d50dbc18dc2288a5ba61232752ea9c393f42d1f8ab47b681c7714\": container with ID starting with 3c680aa9299d50dbc18dc2288a5ba61232752ea9c393f42d1f8ab47b681c7714 not found: ID does not exist" containerID="3c680aa9299d50dbc18dc2288a5ba61232752ea9c393f42d1f8ab47b681c7714"
Nov 02 12:45:24.873821 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:24.873736    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c680aa9299d50dbc18dc2288a5ba61232752ea9c393f42d1f8ab47b681c7714"} err="failed to get container status \"3c680aa9299d50dbc18dc2288a5ba61232752ea9c393f42d1f8ab47b681c7714\": rpc error: code = NotFound desc = could not find container \"3c680aa9299d50dbc18dc2288a5ba61232752ea9c393f42d1f8ab47b681c7714\": container with ID starting with 3c680aa9299d50dbc18dc2288a5ba61232752ea9c393f42d1f8ab47b681c7714 not found: ID does not exist"
Nov 02 12:45:25.050726 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:25.050500    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-6a3af1-openshift-default-666bcf8bbc-2hjz9"]
Nov 02 12:45:25.057571 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:25.057457    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-6a3af1-openshift-default-666bcf8bbc-2hjz9"]
Nov 02 12:45:25.213661 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:25.206632    4559 patch_prober.go:29] interesting pod/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-rxdxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Nov 02 12:45:25.213661 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:25.210795    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-rxdxm" podUID="c2989390-a19e-408b-ac9e-ec01df76bf12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Nov 02 12:45:25.478843 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:25.478812    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-nsg75"
Nov 02 12:45:25.480563 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:25.480419    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-nsg75"
Nov 02 12:45:25.488929 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:25.488873    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-nsg75"
Nov 02 12:45:25.890491 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:25.890406    4559 scope.go:122] "RemoveContainer" containerID="ae90ba1d7e53ff1afdd43c4373dffd2d3df6d654f6eb9166985816f1d98e1ca5"
Nov 02 12:45:25.901361 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:25.901323    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0992908f-5ad3-4600-b8d4-f26d07958ee2" path="/var/lib/kubelet/pods/0992908f-5ad3-4600-b8d4-f26d07958ee2/volumes"
Nov 02 12:45:26.318361 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:26.318326    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-rxdxm"
Nov 02 12:45:26.318819 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:26.318789    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-rxdxm"
Nov 02 12:45:26.319786 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:26.319766    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-rxdxm"
Nov 02 12:45:26.775184 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:26.775154    4559 generic.go:356] "Generic (PLEG): container finished" podID="531f3c3c-dbae-4e74-a7ae-27a21e2a9228" containerID="f916558239dba7625fe81cf59592e2159d9bb3baa222885f3bc8a12bf84229b5" exitCode=60
Nov 02 12:45:26.775600 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:26.775226    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f0cf7e-29893725-22q6j" event={"ID":"531f3c3c-dbae-4e74-a7ae-27a21e2a9228","Type":"ContainerDied","Data":"f916558239dba7625fe81cf59592e2159d9bb3baa222885f3bc8a12bf84229b5"}
Nov 02 12:45:26.775600 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:26.775282    4559 scope.go:122] "RemoveContainer" containerID="ae90ba1d7e53ff1afdd43c4373dffd2d3df6d654f6eb9166985816f1d98e1ca5"
Nov 02 12:45:26.775751 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:26.775691    4559 scope.go:122] "RemoveContainer" containerID="f916558239dba7625fe81cf59592e2159d9bb3baa222885f3bc8a12bf84229b5"
Nov 02 12:45:26.776112 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:45:26.776085    4559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-f0cf7e-29893725-22q6j_odh-ai-gateway-infra(531f3c3c-dbae-4e74-a7ae-27a21e2a9228)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f0cf7e-29893725-22q6j" podUID="531f3c3c-dbae-4e74-a7ae-27a21e2a9228"
Nov 02 12:45:27.828031 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:27.828001    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f0cf7e-29893725-22q6j"]
Nov 02 12:45:27.932898 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:27.932860    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-59060a-75f98c4c8c-kln2x"]
Nov 02 12:45:27.933988 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:27.933967    4559 cpu_manager.go:414] "containerMap: removing container" podUID="0992908f-5ad3-4600-b8d4-f26d07958ee2" containerName="istio-proxy"
Nov 02 12:45:27.934062 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:27.933991    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0992908f-5ad3-4600-b8d4-f26d07958ee2" containerName="istio-proxy"
Nov 02 12:45:27.934062 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:27.934058    4559 cpu_manager.go:414] "containerMap: removing container" podUID="0992908f-5ad3-4600-b8d4-f26d07958ee2" containerName="istio-proxy"
Nov 02 12:45:27.934130 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:27.934068    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0992908f-5ad3-4600-b8d4-f26d07958ee2" containerName="istio-proxy"
Nov 02 12:45:27.934239 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:27.934223    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="0992908f-5ad3-4600-b8d4-f26d07958ee2" containerName="istio-proxy"
Nov 02 12:45:27.940184 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:27.940159    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-59060a-75f98c4c8c-kln2x"
Nov 02 12:45:27.963691 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:27.955906    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-59060a\""
Nov 02 12:45:27.963691 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:27.955971    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-59060a-dockercfg-shnbr\""
Nov 02 12:45:27.963691 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:27.959264    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-59060a-75f98c4c8c-kln2x"]
Nov 02 12:45:27.981019 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:27.980995    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f0cf7e-29893725-22q6j"
Nov 02 12:45:28.139715 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:28.013655    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b90ad6d8-0ab8-4e18-9c5e-e74569e0d0c5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-59060a-75f98c4c8c-kln2x\" (UID: \"b90ad6d8-0ab8-4e18-9c5e-e74569e0d0c5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-59060a-75f98c4c8c-kln2x"
Nov 02 12:45:28.139715 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:28.013970    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtw6c\" (UniqueName: \"kubernetes.io/projected/b90ad6d8-0ab8-4e18-9c5e-e74569e0d0c5-kube-api-access-vtw6c\") pod \"payload-pre-processing-e2e-ipp-a-59060a-75f98c4c8c-kln2x\" (UID: \"b90ad6d8-0ab8-4e18-9c5e-e74569e0d0c5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-59060a-75f98c4c8c-kln2x"
Nov 02 12:45:28.180538 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:28.142179    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b90ad6d8-0ab8-4e18-9c5e-e74569e0d0c5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-59060a-75f98c4c8c-kln2x\" (UID: \"b90ad6d8-0ab8-4e18-9c5e-e74569e0d0c5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-59060a-75f98c4c8c-kln2x"
Nov 02 12:45:28.180538 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:28.142489    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtw6c\" (UniqueName: \"kubernetes.io/projected/b90ad6d8-0ab8-4e18-9c5e-e74569e0d0c5-kube-api-access-vtw6c\") pod \"payload-pre-processing-e2e-ipp-a-59060a-75f98c4c8c-kln2x\" (UID: \"b90ad6d8-0ab8-4e18-9c5e-e74569e0d0c5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-59060a-75f98c4c8c-kln2x"
Nov 02 12:45:28.228661 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:28.225948    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-59060a-dc47f5fff-g5cgn"]
Nov 02 12:45:28.270704 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:28.247007    4559 cpu_manager.go:414] "containerMap: removing container" podUID="531f3c3c-dbae-4e74-a7ae-27a21e2a9228" containerName="cleanup"
Nov 02 12:45:28.270704 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:28.247037    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="531f3c3c-dbae-4e74-a7ae-27a21e2a9228" containerName="cleanup"
Nov 02 12:45:28.270704 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:28.247093    4559 cpu_manager.go:414] "containerMap: removing container" podUID="531f3c3c-dbae-4e74-a7ae-27a21e2a9228" containerName="cleanup"
Nov 02 12:45:28.270704 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:28.247101    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="531f3c3c-dbae-4e74-a7ae-27a21e2a9228" containerName="cleanup"
Nov 02 12:45:28.270704 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:28.247482    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="531f3c3c-dbae-4e74-a7ae-27a21e2a9228" containerName="cleanup"
Nov 02 12:45:28.270704 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:28.247495    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="0992908f-5ad3-4600-b8d4-f26d07958ee2" containerName="istio-proxy"
Nov 02 12:45:28.270704 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:28.247519    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="531f3c3c-dbae-4e74-a7ae-27a21e2a9228" containerName="cleanup"
Nov 02 12:45:28.270704 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:28.247538    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="531f3c3c-dbae-4e74-a7ae-27a21e2a9228" containerName="cleanup"
Nov 02 12:45:28.270704 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:28.250416    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b90ad6d8-0ab8-4e18-9c5e-e74569e0d0c5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-59060a-75f98c4c8c-kln2x\" (UID: \"b90ad6d8-0ab8-4e18-9c5e-e74569e0d0c5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-59060a-75f98c4c8c-kln2x"
Nov 02 12:45:28.270704 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:28.252457    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/531f3c3c-dbae-4e74-a7ae-27a21e2a9228-kube-api-access-nrt9g\" (UniqueName: \"kubernetes.io/projected/531f3c3c-dbae-4e74-a7ae-27a21e2a9228-kube-api-access-nrt9g\") pod \"531f3c3c-dbae-4e74-a7ae-27a21e2a9228\" (UID: \"531f3c3c-dbae-4e74-a7ae-27a21e2a9228\") "
Nov 02 12:45:28.330155 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:28.324420    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-59060a-dc47f5fff-g5cgn"
Nov 02 12:45:28.371735 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:28.366468    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/531f3c3c-dbae-4e74-a7ae-27a21e2a9228-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/531f3c3c-dbae-4e74-a7ae-27a21e2a9228-maas-api-service-ca\") pod \"531f3c3c-dbae-4e74-a7ae-27a21e2a9228\" (UID: \"531f3c3c-dbae-4e74-a7ae-27a21e2a9228\") "
Nov 02 12:45:28.371735 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:28.368696    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/531f3c3c-dbae-4e74-a7ae-27a21e2a9228-maas-api-service-ca" pod "531f3c3c-dbae-4e74-a7ae-27a21e2a9228" (UID: "531f3c3c-dbae-4e74-a7ae-27a21e2a9228"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:45:28.397782 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:28.392785    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtw6c\" (UniqueName: \"kubernetes.io/projected/b90ad6d8-0ab8-4e18-9c5e-e74569e0d0c5-kube-api-access-vtw6c\") pod \"payload-pre-processing-e2e-ipp-a-59060a-75f98c4c8c-kln2x\" (UID: \"b90ad6d8-0ab8-4e18-9c5e-e74569e0d0c5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-59060a-75f98c4c8c-kln2x"
Nov 02 12:45:28.397782 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:28.392872    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-59060a-dc47f5fff-g5cgn"]
Nov 02 12:45:28.426042 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:28.425982    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/531f3c3c-dbae-4e74-a7ae-27a21e2a9228-kube-api-access-nrt9g" pod "531f3c3c-dbae-4e74-a7ae-27a21e2a9228" (UID: "531f3c3c-dbae-4e74-a7ae-27a21e2a9228"). InnerVolumeSpecName "kube-api-access-nrt9g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:45:28.469178 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:28.469145    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/531f3c3c-dbae-4e74-a7ae-27a21e2a9228-maas-api-service-ca\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:45:28.469178 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:28.469174    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nrt9g\" (UniqueName: \"kubernetes.io/projected/531f3c3c-dbae-4e74-a7ae-27a21e2a9228-kube-api-access-nrt9g\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:45:28.562395 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:28.562358    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-59060a-75f98c4c8c-kln2x"
Nov 02 12:45:28.570693 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:28.570658    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f28d03d2-8b87-43a0-b75a-ba0aa7ba664d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-59060a-dc47f5fff-g5cgn\" (UID: \"f28d03d2-8b87-43a0-b75a-ba0aa7ba664d\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-59060a-dc47f5fff-g5cgn"
Nov 02 12:45:28.570819 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:28.570795    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9dn6\" (UniqueName: \"kubernetes.io/projected/f28d03d2-8b87-43a0-b75a-ba0aa7ba664d-kube-api-access-k9dn6\") pod \"payload-processing-e2e-ipp-a-59060a-dc47f5fff-g5cgn\" (UID: \"f28d03d2-8b87-43a0-b75a-ba0aa7ba664d\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-59060a-dc47f5fff-g5cgn"
Nov 02 12:45:28.672294 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:28.672264    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f28d03d2-8b87-43a0-b75a-ba0aa7ba664d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-59060a-dc47f5fff-g5cgn\" (UID: \"f28d03d2-8b87-43a0-b75a-ba0aa7ba664d\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-59060a-dc47f5fff-g5cgn"
Nov 02 12:45:28.672483 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:28.672329    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9dn6\" (UniqueName: \"kubernetes.io/projected/f28d03d2-8b87-43a0-b75a-ba0aa7ba664d-kube-api-access-k9dn6\") pod \"payload-processing-e2e-ipp-a-59060a-dc47f5fff-g5cgn\" (UID: \"f28d03d2-8b87-43a0-b75a-ba0aa7ba664d\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-59060a-dc47f5fff-g5cgn"
Nov 02 12:45:28.673045 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:28.673019    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f28d03d2-8b87-43a0-b75a-ba0aa7ba664d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-59060a-dc47f5fff-g5cgn\" (UID: \"f28d03d2-8b87-43a0-b75a-ba0aa7ba664d\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-59060a-dc47f5fff-g5cgn"
Nov 02 12:45:28.696825 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:28.696788    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9dn6\" (UniqueName: \"kubernetes.io/projected/f28d03d2-8b87-43a0-b75a-ba0aa7ba664d-kube-api-access-k9dn6\") pod \"payload-processing-e2e-ipp-a-59060a-dc47f5fff-g5cgn\" (UID: \"f28d03d2-8b87-43a0-b75a-ba0aa7ba664d\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-59060a-dc47f5fff-g5cgn"
Nov 02 12:45:28.704727 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:28.704694    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-59060a-dc47f5fff-g5cgn"
Nov 02 12:45:28.803228 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:28.803195    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f0cf7e-29893725-22q6j"
Nov 02 12:45:28.803466 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:28.803200    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f0cf7e-29893725-22q6j" event={"ID":"531f3c3c-dbae-4e74-a7ae-27a21e2a9228","Type":"ContainerDied","Data":"4c8674aa0cf987b9c5d0ab6a54901618bba8ed112f4cb6de2f26f682f49360a9"}
Nov 02 12:45:28.803466 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:28.803335    4559 scope.go:122] "RemoveContainer" containerID="f916558239dba7625fe81cf59592e2159d9bb3baa222885f3bc8a12bf84229b5"
Nov 02 12:45:28.859387 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:28.859336    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f0cf7e-29893725-22q6j"]
Nov 02 12:45:28.863020 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:28.862998    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f0cf7e-29893725-22q6j"]
Nov 02 12:45:28.871926 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:45:28.871900    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb90ad6d8_0ab8_4e18_9c5e_e74569e0d0c5.slice/crio-d4b2d57ad7b89c4abd1c76a5b33f41c5ad61930e92884882db4f3e1508cac834 WatchSource:0}: Error finding container d4b2d57ad7b89c4abd1c76a5b33f41c5ad61930e92884882db4f3e1508cac834: Status 404 returned error can't find the container with id d4b2d57ad7b89c4abd1c76a5b33f41c5ad61930e92884882db4f3e1508cac834
Nov 02 12:45:28.872223 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:28.872203    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-59060a-75f98c4c8c-kln2x"]
Nov 02 12:45:29.011125 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:45:29.011093    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf28d03d2_8b87_43a0_b75a_ba0aa7ba664d.slice/crio-e5ecfd6e4580b73158fb759846dae05373c906b705783a2b1c54eda39d962386 WatchSource:0}: Error finding container e5ecfd6e4580b73158fb759846dae05373c906b705783a2b1c54eda39d962386: Status 404 returned error can't find the container with id e5ecfd6e4580b73158fb759846dae05373c906b705783a2b1c54eda39d962386
Nov 02 12:45:29.013505 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:29.013481    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-59060a-dc47f5fff-g5cgn"]
Nov 02 12:45:29.813234 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:29.813200    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-59060a-dc47f5fff-g5cgn" event={"ID":"f28d03d2-8b87-43a0-b75a-ba0aa7ba664d","Type":"ContainerStarted","Data":"b08273f54cc27f897557b302fc8b2727eae1152f87aa4197889f3f6c38e9153d"}
Nov 02 12:45:29.813468 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:29.813249    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-59060a-dc47f5fff-g5cgn" event={"ID":"f28d03d2-8b87-43a0-b75a-ba0aa7ba664d","Type":"ContainerStarted","Data":"e5ecfd6e4580b73158fb759846dae05373c906b705783a2b1c54eda39d962386"}
Nov 02 12:45:29.813468 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:29.813399    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-59060a-dc47f5fff-g5cgn"
Nov 02 12:45:29.814991 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:29.814968    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-59060a-75f98c4c8c-kln2x" event={"ID":"b90ad6d8-0ab8-4e18-9c5e-e74569e0d0c5","Type":"ContainerStarted","Data":"acd50389f58746f8bd2cdacb86a3a6a89c198968c0226ed91eae61e78b8c0449"}
Nov 02 12:45:29.815084 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:29.815010    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-59060a-75f98c4c8c-kln2x"
Nov 02 12:45:29.815084 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:29.815026    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-59060a-75f98c4c8c-kln2x" event={"ID":"b90ad6d8-0ab8-4e18-9c5e-e74569e0d0c5","Type":"ContainerStarted","Data":"d4b2d57ad7b89c4abd1c76a5b33f41c5ad61930e92884882db4f3e1508cac834"}
Nov 02 12:45:29.859352 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:29.859301    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-59060a-dc47f5fff-g5cgn" podStartSLOduration=2.85928788 podStartE2EDuration="2.85928788s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:45:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:45:29.854157773 +0000 UTC m=+1126.109556868" watchObservedRunningTime="2026-11-02 12:45:29.85928788 +0000 UTC m=+1126.114686975"
Nov 02 12:45:29.894560 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:29.894530    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="531f3c3c-dbae-4e74-a7ae-27a21e2a9228" path="/var/lib/kubelet/pods/531f3c3c-dbae-4e74-a7ae-27a21e2a9228/volumes"
Nov 02 12:45:29.902548 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:29.902500    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-59060a-75f98c4c8c-kln2x" podStartSLOduration=2.902486948 podStartE2EDuration="2.902486948s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:45:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:45:29.899470995 +0000 UTC m=+1126.154870090" watchObservedRunningTime="2026-11-02 12:45:29.902486948 +0000 UTC m=+1126.157886093"
Nov 02 12:45:32.195688 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:32.195634    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-mt-cd1386cb/e2e-lifecycle-model-cd1386cb-kserve-7d9d698845-2579r" podUID="572f58c8-cca9-48c0-99d3-0ebde6b82476" containerName="main" containerID="cri-o://177a2ce7afdb94b482f18b9e727d3c7b26727efc418ffd2328a392e6148b6a68" gracePeriod=45
Nov 02 12:45:32.766438 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:32.761172    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-cd1386cb/e2e-lifecycle-model-cd1386cb-kserve-7d9d698845-2579r"
Nov 02 12:45:32.841487 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:32.841450    4559 generic.go:356] "Generic (PLEG): container finished" podID="572f58c8-cca9-48c0-99d3-0ebde6b82476" containerID="177a2ce7afdb94b482f18b9e727d3c7b26727efc418ffd2328a392e6148b6a68" exitCode=0
Nov 02 12:45:32.841677 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:32.841497    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-cd1386cb/e2e-lifecycle-model-cd1386cb-kserve-7d9d698845-2579r"
Nov 02 12:45:32.841728 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:32.841699    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-cd1386cb/e2e-lifecycle-model-cd1386cb-kserve-7d9d698845-2579r" event={"ID":"572f58c8-cca9-48c0-99d3-0ebde6b82476","Type":"ContainerDied","Data":"177a2ce7afdb94b482f18b9e727d3c7b26727efc418ffd2328a392e6148b6a68"}
Nov 02 12:45:32.841765 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:32.841736    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-cd1386cb/e2e-lifecycle-model-cd1386cb-kserve-7d9d698845-2579r" event={"ID":"572f58c8-cca9-48c0-99d3-0ebde6b82476","Type":"ContainerDied","Data":"8b744c566dfe165060511d1e4a7599c67c46bbe3ca8eed11c4ef995752448061"}
Nov 02 12:45:32.841765 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:32.841757    4559 scope.go:122] "RemoveContainer" containerID="177a2ce7afdb94b482f18b9e727d3c7b26727efc418ffd2328a392e6148b6a68"
Nov 02 12:45:32.856451 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:32.856427    4559 scope.go:122] "RemoveContainer" containerID="177a2ce7afdb94b482f18b9e727d3c7b26727efc418ffd2328a392e6148b6a68"
Nov 02 12:45:32.856866 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:45:32.856834    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"177a2ce7afdb94b482f18b9e727d3c7b26727efc418ffd2328a392e6148b6a68\": container with ID starting with 177a2ce7afdb94b482f18b9e727d3c7b26727efc418ffd2328a392e6148b6a68 not found: ID does not exist" containerID="177a2ce7afdb94b482f18b9e727d3c7b26727efc418ffd2328a392e6148b6a68"
Nov 02 12:45:32.856986 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:32.856878    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"177a2ce7afdb94b482f18b9e727d3c7b26727efc418ffd2328a392e6148b6a68"} err="failed to get container status \"177a2ce7afdb94b482f18b9e727d3c7b26727efc418ffd2328a392e6148b6a68\": rpc error: code = NotFound desc = could not find container \"177a2ce7afdb94b482f18b9e727d3c7b26727efc418ffd2328a392e6148b6a68\": container with ID starting with 177a2ce7afdb94b482f18b9e727d3c7b26727efc418ffd2328a392e6148b6a68 not found: ID does not exist"
Nov 02 12:45:32.879606 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:32.879583    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/572f58c8-cca9-48c0-99d3-0ebde6b82476-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/572f58c8-cca9-48c0-99d3-0ebde6b82476-model-cache\") pod \"572f58c8-cca9-48c0-99d3-0ebde6b82476\" (UID: \"572f58c8-cca9-48c0-99d3-0ebde6b82476\") "
Nov 02 12:45:32.879749 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:32.879663    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/572f58c8-cca9-48c0-99d3-0ebde6b82476-home\" (UniqueName: \"kubernetes.io/empty-dir/572f58c8-cca9-48c0-99d3-0ebde6b82476-home\") pod \"572f58c8-cca9-48c0-99d3-0ebde6b82476\" (UID: \"572f58c8-cca9-48c0-99d3-0ebde6b82476\") "
Nov 02 12:45:32.879749 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:32.879689    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/572f58c8-cca9-48c0-99d3-0ebde6b82476-kube-api-access-lt5z5\" (UniqueName: \"kubernetes.io/projected/572f58c8-cca9-48c0-99d3-0ebde6b82476-kube-api-access-lt5z5\") pod \"572f58c8-cca9-48c0-99d3-0ebde6b82476\" (UID: \"572f58c8-cca9-48c0-99d3-0ebde6b82476\") "
Nov 02 12:45:32.879868 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:32.879818    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/572f58c8-cca9-48c0-99d3-0ebde6b82476-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/572f58c8-cca9-48c0-99d3-0ebde6b82476-tmp-dir\") pod \"572f58c8-cca9-48c0-99d3-0ebde6b82476\" (UID: \"572f58c8-cca9-48c0-99d3-0ebde6b82476\") "
Nov 02 12:45:32.879868 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:32.879856    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/572f58c8-cca9-48c0-99d3-0ebde6b82476-tls-certs\" (UniqueName: \"kubernetes.io/secret/572f58c8-cca9-48c0-99d3-0ebde6b82476-tls-certs\") pod \"572f58c8-cca9-48c0-99d3-0ebde6b82476\" (UID: \"572f58c8-cca9-48c0-99d3-0ebde6b82476\") "
Nov 02 12:45:32.879971 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:32.879887    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/572f58c8-cca9-48c0-99d3-0ebde6b82476-model-cache" pod "572f58c8-cca9-48c0-99d3-0ebde6b82476" (UID: "572f58c8-cca9-48c0-99d3-0ebde6b82476"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:45:32.879971 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:32.879963    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/572f58c8-cca9-48c0-99d3-0ebde6b82476-dshm\" (UniqueName: \"kubernetes.io/empty-dir/572f58c8-cca9-48c0-99d3-0ebde6b82476-dshm\") pod \"572f58c8-cca9-48c0-99d3-0ebde6b82476\" (UID: \"572f58c8-cca9-48c0-99d3-0ebde6b82476\") "
Nov 02 12:45:32.880091 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:32.879971    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/572f58c8-cca9-48c0-99d3-0ebde6b82476-home" pod "572f58c8-cca9-48c0-99d3-0ebde6b82476" (UID: "572f58c8-cca9-48c0-99d3-0ebde6b82476"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:45:32.880145 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:32.880109    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/572f58c8-cca9-48c0-99d3-0ebde6b82476-tmp-dir" pod "572f58c8-cca9-48c0-99d3-0ebde6b82476" (UID: "572f58c8-cca9-48c0-99d3-0ebde6b82476"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:45:32.880715 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:32.880688    4559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/572f58c8-cca9-48c0-99d3-0ebde6b82476-home\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:45:32.880715 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:32.880717    4559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/572f58c8-cca9-48c0-99d3-0ebde6b82476-tmp-dir\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:45:32.880901 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:32.880733    4559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/572f58c8-cca9-48c0-99d3-0ebde6b82476-model-cache\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:45:32.882286 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:32.882254    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/572f58c8-cca9-48c0-99d3-0ebde6b82476-tls-certs" pod "572f58c8-cca9-48c0-99d3-0ebde6b82476" (UID: "572f58c8-cca9-48c0-99d3-0ebde6b82476"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:45:32.882501 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:32.882318    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/572f58c8-cca9-48c0-99d3-0ebde6b82476-dshm" pod "572f58c8-cca9-48c0-99d3-0ebde6b82476" (UID: "572f58c8-cca9-48c0-99d3-0ebde6b82476"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:45:32.882501 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:32.882377    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/572f58c8-cca9-48c0-99d3-0ebde6b82476-kube-api-access-lt5z5" pod "572f58c8-cca9-48c0-99d3-0ebde6b82476" (UID: "572f58c8-cca9-48c0-99d3-0ebde6b82476"). InnerVolumeSpecName "kube-api-access-lt5z5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:45:32.982665 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:32.982598    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lt5z5\" (UniqueName: \"kubernetes.io/projected/572f58c8-cca9-48c0-99d3-0ebde6b82476-kube-api-access-lt5z5\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:45:32.982845 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:32.982661    4559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/572f58c8-cca9-48c0-99d3-0ebde6b82476-tls-certs\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:45:32.982845 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:32.982693    4559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/572f58c8-cca9-48c0-99d3-0ebde6b82476-dshm\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:45:33.190095 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:33.190066    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-cd1386cb/e2e-lifecycle-model-cd1386cb-kserve-7d9d698845-2579r"]
Nov 02 12:45:33.202485 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:33.202456    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-cd1386cb/e2e-lifecycle-model-cd1386cb-kserve-7d9d698845-2579r"]
Nov 02 12:45:33.620318 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:33.618703    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-77cf70-99dbd47d7-4rbjg"]
Nov 02 12:45:33.620318 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:33.619828    4559 cpu_manager.go:414] "containerMap: removing container" podUID="531f3c3c-dbae-4e74-a7ae-27a21e2a9228" containerName="cleanup"
Nov 02 12:45:33.620318 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:33.619850    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="531f3c3c-dbae-4e74-a7ae-27a21e2a9228" containerName="cleanup"
Nov 02 12:45:33.620318 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:33.619905    4559 cpu_manager.go:414] "containerMap: removing container" podUID="572f58c8-cca9-48c0-99d3-0ebde6b82476" containerName="main"
Nov 02 12:45:33.620318 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:33.619913    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="572f58c8-cca9-48c0-99d3-0ebde6b82476" containerName="main"
Nov 02 12:45:33.620318 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:33.620043    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="572f58c8-cca9-48c0-99d3-0ebde6b82476" containerName="main"
Nov 02 12:45:33.640685 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:33.629447    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-77cf70-99dbd47d7-4rbjg"
Nov 02 12:45:33.659115 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:33.658811    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-77cf70-serving-cert\""
Nov 02 12:45:33.696446 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:33.696407    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2v4ll\" (UniqueName: \"kubernetes.io/projected/630a0e10-db9e-42e4-80a9-2268163c591a-kube-api-access-2v4ll\") pod \"maas-api-e2e-ipp-b-77cf70-99dbd47d7-4rbjg\" (UID: \"630a0e10-db9e-42e4-80a9-2268163c591a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-77cf70-99dbd47d7-4rbjg"
Nov 02 12:45:33.696619 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:33.696596    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/630a0e10-db9e-42e4-80a9-2268163c591a-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-77cf70-99dbd47d7-4rbjg\" (UID: \"630a0e10-db9e-42e4-80a9-2268163c591a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-77cf70-99dbd47d7-4rbjg"
Nov 02 12:45:33.696711 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:33.696676    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/630a0e10-db9e-42e4-80a9-2268163c591a-maas-api-tls\") pod \"maas-api-e2e-ipp-b-77cf70-99dbd47d7-4rbjg\" (UID: \"630a0e10-db9e-42e4-80a9-2268163c591a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-77cf70-99dbd47d7-4rbjg"
Nov 02 12:45:33.696776 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:33.696762    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/630a0e10-db9e-42e4-80a9-2268163c591a-metrics-certs\") pod \"maas-api-e2e-ipp-b-77cf70-99dbd47d7-4rbjg\" (UID: \"630a0e10-db9e-42e4-80a9-2268163c591a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-77cf70-99dbd47d7-4rbjg"
Nov 02 12:45:33.811686 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:33.805770    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2v4ll\" (UniqueName: \"kubernetes.io/projected/630a0e10-db9e-42e4-80a9-2268163c591a-kube-api-access-2v4ll\") pod \"maas-api-e2e-ipp-b-77cf70-99dbd47d7-4rbjg\" (UID: \"630a0e10-db9e-42e4-80a9-2268163c591a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-77cf70-99dbd47d7-4rbjg"
Nov 02 12:45:33.811686 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:33.805942    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/630a0e10-db9e-42e4-80a9-2268163c591a-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-77cf70-99dbd47d7-4rbjg\" (UID: \"630a0e10-db9e-42e4-80a9-2268163c591a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-77cf70-99dbd47d7-4rbjg"
Nov 02 12:45:33.811686 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:33.806006    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/630a0e10-db9e-42e4-80a9-2268163c591a-maas-api-tls\") pod \"maas-api-e2e-ipp-b-77cf70-99dbd47d7-4rbjg\" (UID: \"630a0e10-db9e-42e4-80a9-2268163c591a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-77cf70-99dbd47d7-4rbjg"
Nov 02 12:45:33.811686 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:33.806089    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/630a0e10-db9e-42e4-80a9-2268163c591a-metrics-certs\") pod \"maas-api-e2e-ipp-b-77cf70-99dbd47d7-4rbjg\" (UID: \"630a0e10-db9e-42e4-80a9-2268163c591a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-77cf70-99dbd47d7-4rbjg"
Nov 02 12:45:33.811686 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:33.810195    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/630a0e10-db9e-42e4-80a9-2268163c591a-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-77cf70-99dbd47d7-4rbjg\" (UID: \"630a0e10-db9e-42e4-80a9-2268163c591a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-77cf70-99dbd47d7-4rbjg"
Nov 02 12:45:33.811686 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:45:33.810592    4559 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-77cf70-serving-cert: secret "maas-api-e2e-ipp-b-77cf70-serving-cert" not found
Nov 02 12:45:33.811686 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:45:33.810676    4559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/630a0e10-db9e-42e4-80a9-2268163c591a-maas-api-tls podName:630a0e10-db9e-42e4-80a9-2268163c591a nodeName:}" failed. No retries permitted until 2026-11-02 12:45:34.310655868 +0000 UTC m=+1130.566054955 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/630a0e10-db9e-42e4-80a9-2268163c591a-maas-api-tls") pod "maas-api-e2e-ipp-b-77cf70-99dbd47d7-4rbjg" (UID: "630a0e10-db9e-42e4-80a9-2268163c591a") : secret "maas-api-e2e-ipp-b-77cf70-serving-cert" not found
Nov 02 12:45:33.861670 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:33.853442    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-77cf70-99dbd47d7-4rbjg"]
Nov 02 12:45:33.940980 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:33.940946    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="572f58c8-cca9-48c0-99d3-0ebde6b82476" path="/var/lib/kubelet/pods/572f58c8-cca9-48c0-99d3-0ebde6b82476/volumes"
Nov 02 12:45:33.943731 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:33.943673    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/630a0e10-db9e-42e4-80a9-2268163c591a-metrics-certs\") pod \"maas-api-e2e-ipp-b-77cf70-99dbd47d7-4rbjg\" (UID: \"630a0e10-db9e-42e4-80a9-2268163c591a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-77cf70-99dbd47d7-4rbjg"
Nov 02 12:45:33.946869 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:33.946845    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2v4ll\" (UniqueName: \"kubernetes.io/projected/630a0e10-db9e-42e4-80a9-2268163c591a-kube-api-access-2v4ll\") pod \"maas-api-e2e-ipp-b-77cf70-99dbd47d7-4rbjg\" (UID: \"630a0e10-db9e-42e4-80a9-2268163c591a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-77cf70-99dbd47d7-4rbjg"
Nov 02 12:45:34.313555 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:34.313475    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/630a0e10-db9e-42e4-80a9-2268163c591a-maas-api-tls\") pod \"maas-api-e2e-ipp-b-77cf70-99dbd47d7-4rbjg\" (UID: \"630a0e10-db9e-42e4-80a9-2268163c591a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-77cf70-99dbd47d7-4rbjg"
Nov 02 12:45:34.316073 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:34.316041    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/630a0e10-db9e-42e4-80a9-2268163c591a-maas-api-tls\") pod \"maas-api-e2e-ipp-b-77cf70-99dbd47d7-4rbjg\" (UID: \"630a0e10-db9e-42e4-80a9-2268163c591a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-77cf70-99dbd47d7-4rbjg"
Nov 02 12:45:34.506935 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:34.506908    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-77cf70-99dbd47d7-4rbjg"
Nov 02 12:45:34.878488 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:34.878396    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-77cf70-99dbd47d7-4rbjg"]
Nov 02 12:45:34.927295 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:34.927261    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-77cf70-99dbd47d7-4rbjg" event={"ID":"630a0e10-db9e-42e4-80a9-2268163c591a","Type":"ContainerStarted","Data":"0b8a4bdda3cd79f3ce2120ab753b4cd8895e44543a60811116c4259a1674eb66"}
Nov 02 12:45:35.935498 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:35.935463    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-77cf70-99dbd47d7-4rbjg" event={"ID":"630a0e10-db9e-42e4-80a9-2268163c591a","Type":"ContainerStarted","Data":"ae229b6d4b81f3b0dc2c8a60559d000be699e7bed4e6aa1fd9f81132da2e5b88"}
Nov 02 12:45:35.935884 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:35.935521    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-77cf70-99dbd47d7-4rbjg"
Nov 02 12:45:35.982958 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:35.982904    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-77cf70-99dbd47d7-4rbjg" podStartSLOduration=2.7147822169999998 podStartE2EDuration="2.982891687s" totalImagesPullingTime="268.10947ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:45:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:45:35.979388474 +0000 UTC m=+1132.234787569" watchObservedRunningTime="2026-11-02 12:45:35.982891687 +0000 UTC m=+1132.238290806"
Nov 02 12:45:40.827437 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:40.827405    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-59060a-75f98c4c8c-kln2x"
Nov 02 12:45:40.827930 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:40.827703    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-59060a-dc47f5fff-g5cgn"
Nov 02 12:45:41.947913 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:41.947880    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-77cf70-99dbd47d7-4rbjg"
Nov 02 12:45:42.176723 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:42.176689    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e6ceda-779dbb5897-lqnhx"]
Nov 02 12:45:42.211331 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:42.211244    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e6ceda-dbd545695-ggztc"]
Nov 02 12:45:42.502076 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:42.501095    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-59060a-cfb6d4595-zt7rh"]
Nov 02 12:45:42.507669 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:42.506910    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-59060a-cfb6d4595-zt7rh"
Nov 02 12:45:42.524664 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:42.523947    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-59060a-serving-cert\""
Nov 02 12:45:42.556676 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:42.552557    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-59060a-cfb6d4595-zt7rh"]
Nov 02 12:45:42.707723 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:42.707633    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9wkt\" (UniqueName: \"kubernetes.io/projected/2265e1d5-bae9-495b-b51d-b3e638c128ea-kube-api-access-t9wkt\") pod \"maas-api-e2e-ipp-a-59060a-cfb6d4595-zt7rh\" (UID: \"2265e1d5-bae9-495b-b51d-b3e638c128ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-59060a-cfb6d4595-zt7rh"
Nov 02 12:45:42.707923 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:42.707798    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2265e1d5-bae9-495b-b51d-b3e638c128ea-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-59060a-cfb6d4595-zt7rh\" (UID: \"2265e1d5-bae9-495b-b51d-b3e638c128ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-59060a-cfb6d4595-zt7rh"
Nov 02 12:45:42.708049 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:42.708026    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2265e1d5-bae9-495b-b51d-b3e638c128ea-metrics-certs\") pod \"maas-api-e2e-ipp-a-59060a-cfb6d4595-zt7rh\" (UID: \"2265e1d5-bae9-495b-b51d-b3e638c128ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-59060a-cfb6d4595-zt7rh"
Nov 02 12:45:42.708109 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:42.708095    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2265e1d5-bae9-495b-b51d-b3e638c128ea-maas-api-tls\") pod \"maas-api-e2e-ipp-a-59060a-cfb6d4595-zt7rh\" (UID: \"2265e1d5-bae9-495b-b51d-b3e638c128ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-59060a-cfb6d4595-zt7rh"
Nov 02 12:45:42.810159 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:42.810077    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9wkt\" (UniqueName: \"kubernetes.io/projected/2265e1d5-bae9-495b-b51d-b3e638c128ea-kube-api-access-t9wkt\") pod \"maas-api-e2e-ipp-a-59060a-cfb6d4595-zt7rh\" (UID: \"2265e1d5-bae9-495b-b51d-b3e638c128ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-59060a-cfb6d4595-zt7rh"
Nov 02 12:45:42.810159 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:42.810142    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2265e1d5-bae9-495b-b51d-b3e638c128ea-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-59060a-cfb6d4595-zt7rh\" (UID: \"2265e1d5-bae9-495b-b51d-b3e638c128ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-59060a-cfb6d4595-zt7rh"
Nov 02 12:45:42.810377 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:42.810317    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2265e1d5-bae9-495b-b51d-b3e638c128ea-metrics-certs\") pod \"maas-api-e2e-ipp-a-59060a-cfb6d4595-zt7rh\" (UID: \"2265e1d5-bae9-495b-b51d-b3e638c128ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-59060a-cfb6d4595-zt7rh"
Nov 02 12:45:42.810436 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:42.810383    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2265e1d5-bae9-495b-b51d-b3e638c128ea-maas-api-tls\") pod \"maas-api-e2e-ipp-a-59060a-cfb6d4595-zt7rh\" (UID: \"2265e1d5-bae9-495b-b51d-b3e638c128ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-59060a-cfb6d4595-zt7rh"
Nov 02 12:45:42.810869 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:42.810839    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2265e1d5-bae9-495b-b51d-b3e638c128ea-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-59060a-cfb6d4595-zt7rh\" (UID: \"2265e1d5-bae9-495b-b51d-b3e638c128ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-59060a-cfb6d4595-zt7rh"
Nov 02 12:45:42.812771 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:42.812741    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2265e1d5-bae9-495b-b51d-b3e638c128ea-maas-api-tls\") pod \"maas-api-e2e-ipp-a-59060a-cfb6d4595-zt7rh\" (UID: \"2265e1d5-bae9-495b-b51d-b3e638c128ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-59060a-cfb6d4595-zt7rh"
Nov 02 12:45:42.812877 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:42.812787    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2265e1d5-bae9-495b-b51d-b3e638c128ea-metrics-certs\") pod \"maas-api-e2e-ipp-a-59060a-cfb6d4595-zt7rh\" (UID: \"2265e1d5-bae9-495b-b51d-b3e638c128ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-59060a-cfb6d4595-zt7rh"
Nov 02 12:45:42.833671 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:42.833629    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9wkt\" (UniqueName: \"kubernetes.io/projected/2265e1d5-bae9-495b-b51d-b3e638c128ea-kube-api-access-t9wkt\") pod \"maas-api-e2e-ipp-a-59060a-cfb6d4595-zt7rh\" (UID: \"2265e1d5-bae9-495b-b51d-b3e638c128ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-59060a-cfb6d4595-zt7rh"
Nov 02 12:45:42.902901 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:42.902872    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-59060a-cfb6d4595-zt7rh"
Nov 02 12:45:43.661979 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:43.661771    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-59060a-cfb6d4595-zt7rh"]
Nov 02 12:45:44.148765 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:44.148725    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-59060a-cfb6d4595-zt7rh" event={"ID":"2265e1d5-bae9-495b-b51d-b3e638c128ea","Type":"ContainerStarted","Data":"5249048539e9edfb82c9a8ba02fbe9f17fcb0a1ab3ec77bd34fef810bb779e5e"}
Nov 02 12:45:44.148952 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:44.148778    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-59060a-cfb6d4595-zt7rh" event={"ID":"2265e1d5-bae9-495b-b51d-b3e638c128ea","Type":"ContainerStarted","Data":"517988dc30077f2e8c3da04ce6b11c47e5a554f60a6dfe125c52a5b14fbe2028"}
Nov 02 12:45:44.148952 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:44.148842    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-59060a-cfb6d4595-zt7rh"
Nov 02 12:45:45.086786 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:45:45.086740    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqcmj" containerName="extractor"
Nov 02 12:45:47.177470 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:47.177410    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-e6ceda-779dbb5897-lqnhx" podUID="ae88d18c-c12e-4c53-a0e0-b012a2d624ff" containerName="payload-processing" containerID="cri-o://32d9f8b61a8d60432fa5d9681571b3f89751ba2e1b4c7a1fb08c6dff6e612330" gracePeriod=25
Nov 02 12:45:47.212270 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:47.212230    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e6ceda-dbd545695-ggztc" podUID="bb4ea3c3-befb-4ee2-b46b-93ce5e4a7110" containerName="payload-pre-processing" containerID="cri-o://0bda04b55a799c6f508cb4c8b62e72c2097c6260e3740d8b32dc315e9d971161" gracePeriod=25
Nov 02 12:45:47.592865 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:47.592835    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-e6ceda-779dbb5897-lqnhx"
Nov 02 12:45:47.639491 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:47.639439    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-59060a-cfb6d4595-zt7rh" podStartSLOduration=5.183054048 podStartE2EDuration="5.639422974s" totalImagesPullingTime="456.368926ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:45:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:45:44.20796978 +0000 UTC m=+1140.463368879" watchObservedRunningTime="2026-11-02 12:45:47.639422974 +0000 UTC m=+1143.894822068"
Nov 02 12:45:47.649007 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:47.648982    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e6ceda-dbd545695-ggztc"
Nov 02 12:45:47.731016 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:47.730985    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae88d18c-c12e-4c53-a0e0-b012a2d624ff-kube-api-access-wkxh8\" (UniqueName: \"kubernetes.io/projected/ae88d18c-c12e-4c53-a0e0-b012a2d624ff-kube-api-access-wkxh8\") pod \"ae88d18c-c12e-4c53-a0e0-b012a2d624ff\" (UID: \"ae88d18c-c12e-4c53-a0e0-b012a2d624ff\") "
Nov 02 12:45:47.731213 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:47.731028    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae88d18c-c12e-4c53-a0e0-b012a2d624ff-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae88d18c-c12e-4c53-a0e0-b012a2d624ff-plugins-config-volume\") pod \"ae88d18c-c12e-4c53-a0e0-b012a2d624ff\" (UID: \"ae88d18c-c12e-4c53-a0e0-b012a2d624ff\") "
Nov 02 12:45:47.731532 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:47.731501    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae88d18c-c12e-4c53-a0e0-b012a2d624ff-plugins-config-volume" pod "ae88d18c-c12e-4c53-a0e0-b012a2d624ff" (UID: "ae88d18c-c12e-4c53-a0e0-b012a2d624ff"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:45:47.731760 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:47.731739    4559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae88d18c-c12e-4c53-a0e0-b012a2d624ff-plugins-config-volume\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:45:47.733182 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:47.733153    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae88d18c-c12e-4c53-a0e0-b012a2d624ff-kube-api-access-wkxh8" pod "ae88d18c-c12e-4c53-a0e0-b012a2d624ff" (UID: "ae88d18c-c12e-4c53-a0e0-b012a2d624ff"). InnerVolumeSpecName "kube-api-access-wkxh8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:45:47.833172 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:47.833097    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb4ea3c3-befb-4ee2-b46b-93ce5e4a7110-kube-api-access-69qcj\" (UniqueName: \"kubernetes.io/projected/bb4ea3c3-befb-4ee2-b46b-93ce5e4a7110-kube-api-access-69qcj\") pod \"bb4ea3c3-befb-4ee2-b46b-93ce5e4a7110\" (UID: \"bb4ea3c3-befb-4ee2-b46b-93ce5e4a7110\") "
Nov 02 12:45:47.833331 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:47.833207    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb4ea3c3-befb-4ee2-b46b-93ce5e4a7110-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb4ea3c3-befb-4ee2-b46b-93ce5e4a7110-plugins-config-volume\") pod \"bb4ea3c3-befb-4ee2-b46b-93ce5e4a7110\" (UID: \"bb4ea3c3-befb-4ee2-b46b-93ce5e4a7110\") "
Nov 02 12:45:47.833628 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:47.833605    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb4ea3c3-befb-4ee2-b46b-93ce5e4a7110-plugins-config-volume" pod "bb4ea3c3-befb-4ee2-b46b-93ce5e4a7110" (UID: "bb4ea3c3-befb-4ee2-b46b-93ce5e4a7110"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:45:47.833918 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:47.833902    4559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb4ea3c3-befb-4ee2-b46b-93ce5e4a7110-plugins-config-volume\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:45:47.833959 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:47.833922    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wkxh8\" (UniqueName: \"kubernetes.io/projected/ae88d18c-c12e-4c53-a0e0-b012a2d624ff-kube-api-access-wkxh8\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:45:47.835315 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:47.835293    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb4ea3c3-befb-4ee2-b46b-93ce5e4a7110-kube-api-access-69qcj" pod "bb4ea3c3-befb-4ee2-b46b-93ce5e4a7110" (UID: "bb4ea3c3-befb-4ee2-b46b-93ce5e4a7110"). InnerVolumeSpecName "kube-api-access-69qcj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:45:47.935441 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:47.935403    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-69qcj\" (UniqueName: \"kubernetes.io/projected/bb4ea3c3-befb-4ee2-b46b-93ce5e4a7110-kube-api-access-69qcj\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:45:48.177798 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:48.177766    4559 generic.go:356] "Generic (PLEG): container finished" podID="bb4ea3c3-befb-4ee2-b46b-93ce5e4a7110" containerID="0bda04b55a799c6f508cb4c8b62e72c2097c6260e3740d8b32dc315e9d971161" exitCode=0
Nov 02 12:45:48.178311 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:48.177817    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e6ceda-dbd545695-ggztc"
Nov 02 12:45:48.178311 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:48.177857    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e6ceda-dbd545695-ggztc" event={"ID":"bb4ea3c3-befb-4ee2-b46b-93ce5e4a7110","Type":"ContainerDied","Data":"0bda04b55a799c6f508cb4c8b62e72c2097c6260e3740d8b32dc315e9d971161"}
Nov 02 12:45:48.178311 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:48.177914    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e6ceda-dbd545695-ggztc" event={"ID":"bb4ea3c3-befb-4ee2-b46b-93ce5e4a7110","Type":"ContainerDied","Data":"539ed763ae891654f18d31f04282b2ec2095145ae1d00c6d1b280486b4e6e110"}
Nov 02 12:45:48.178311 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:48.177941    4559 scope.go:122] "RemoveContainer" containerID="0bda04b55a799c6f508cb4c8b62e72c2097c6260e3740d8b32dc315e9d971161"
Nov 02 12:45:48.179360 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:48.179341    4559 generic.go:356] "Generic (PLEG): container finished" podID="ae88d18c-c12e-4c53-a0e0-b012a2d624ff" containerID="32d9f8b61a8d60432fa5d9681571b3f89751ba2e1b4c7a1fb08c6dff6e612330" exitCode=0
Nov 02 12:45:48.179466 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:48.179367    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-e6ceda-779dbb5897-lqnhx" event={"ID":"ae88d18c-c12e-4c53-a0e0-b012a2d624ff","Type":"ContainerDied","Data":"32d9f8b61a8d60432fa5d9681571b3f89751ba2e1b4c7a1fb08c6dff6e612330"}
Nov 02 12:45:48.179466 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:48.179396    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-e6ceda-779dbb5897-lqnhx"
Nov 02 12:45:48.179466 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:48.179416    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-e6ceda-779dbb5897-lqnhx" event={"ID":"ae88d18c-c12e-4c53-a0e0-b012a2d624ff","Type":"ContainerDied","Data":"3565744111bfc9fbdba8b071a5af61b1f1f47fad3a43b21f6664fb7488e75c72"}
Nov 02 12:45:48.188822 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:48.188796    4559 scope.go:122] "RemoveContainer" containerID="0bda04b55a799c6f508cb4c8b62e72c2097c6260e3740d8b32dc315e9d971161"
Nov 02 12:45:48.189111 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:45:48.189089    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0bda04b55a799c6f508cb4c8b62e72c2097c6260e3740d8b32dc315e9d971161\": container with ID starting with 0bda04b55a799c6f508cb4c8b62e72c2097c6260e3740d8b32dc315e9d971161 not found: ID does not exist" containerID="0bda04b55a799c6f508cb4c8b62e72c2097c6260e3740d8b32dc315e9d971161"
Nov 02 12:45:48.189212 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:48.189122    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0bda04b55a799c6f508cb4c8b62e72c2097c6260e3740d8b32dc315e9d971161"} err="failed to get container status \"0bda04b55a799c6f508cb4c8b62e72c2097c6260e3740d8b32dc315e9d971161\": rpc error: code = NotFound desc = could not find container \"0bda04b55a799c6f508cb4c8b62e72c2097c6260e3740d8b32dc315e9d971161\": container with ID starting with 0bda04b55a799c6f508cb4c8b62e72c2097c6260e3740d8b32dc315e9d971161 not found: ID does not exist"
Nov 02 12:45:48.189212 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:48.189146    4559 scope.go:122] "RemoveContainer" containerID="32d9f8b61a8d60432fa5d9681571b3f89751ba2e1b4c7a1fb08c6dff6e612330"
Nov 02 12:45:48.206911 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:48.206894    4559 scope.go:122] "RemoveContainer" containerID="32d9f8b61a8d60432fa5d9681571b3f89751ba2e1b4c7a1fb08c6dff6e612330"
Nov 02 12:45:48.207236 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:45:48.207215    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32d9f8b61a8d60432fa5d9681571b3f89751ba2e1b4c7a1fb08c6dff6e612330\": container with ID starting with 32d9f8b61a8d60432fa5d9681571b3f89751ba2e1b4c7a1fb08c6dff6e612330 not found: ID does not exist" containerID="32d9f8b61a8d60432fa5d9681571b3f89751ba2e1b4c7a1fb08c6dff6e612330"
Nov 02 12:45:48.207312 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:48.207247    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32d9f8b61a8d60432fa5d9681571b3f89751ba2e1b4c7a1fb08c6dff6e612330"} err="failed to get container status \"32d9f8b61a8d60432fa5d9681571b3f89751ba2e1b4c7a1fb08c6dff6e612330\": rpc error: code = NotFound desc = could not find container \"32d9f8b61a8d60432fa5d9681571b3f89751ba2e1b4c7a1fb08c6dff6e612330\": container with ID starting with 32d9f8b61a8d60432fa5d9681571b3f89751ba2e1b4c7a1fb08c6dff6e612330 not found: ID does not exist"
Nov 02 12:45:48.233117 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:48.233093    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e6ceda-dbd545695-ggztc"]
Nov 02 12:45:48.247373 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:48.247344    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e6ceda-dbd545695-ggztc"]
Nov 02 12:45:48.275837 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:48.275809    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e6ceda-779dbb5897-lqnhx"]
Nov 02 12:45:48.279885 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:48.279863    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e6ceda-779dbb5897-lqnhx"]
Nov 02 12:45:49.894983 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:49.894947    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae88d18c-c12e-4c53-a0e0-b012a2d624ff" path="/var/lib/kubelet/pods/ae88d18c-c12e-4c53-a0e0-b012a2d624ff/volumes"
Nov 02 12:45:49.895610 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:49.895585    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb4ea3c3-befb-4ee2-b46b-93ce5e4a7110" path="/var/lib/kubelet/pods/bb4ea3c3-befb-4ee2-b46b-93ce5e4a7110/volumes"
Nov 02 12:45:50.163161 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:50.163088    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-59060a-cfb6d4595-zt7rh"
Nov 02 12:45:54.523826 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:54.523791    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-cd1386cb-7b76784c55-g6lkv"]
Nov 02 12:45:54.559287 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:54.559255    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-cd1386cb-556c7759d7-fwckp"]
Nov 02 12:45:56.280398 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:56.279252    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"]
Nov 02 12:45:59.531665 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:59.531201    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-cd1386cb-7b76784c55-g6lkv" podUID="d1eaa8f7-ef7a-47b0-b7cc-6100c6e858e8" containerName="payload-processing" containerID="cri-o://a102c76363df6fd1e22f21f30d378635f9008b72393bcad37083a1fbadcc0c7f" gracePeriod=25
Nov 02 12:45:59.560253 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:45:59.560220    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-cd1386cb-556c7759d7-fwckp" podUID="0830a319-a5b3-4f40-a0ba-6e218c86466b" containerName="payload-pre-processing" containerID="cri-o://da99a43b4b2f08c38e21730f80c7fa736947bcdb08560df1027af6754ceadc5e" gracePeriod=25
Nov 02 12:46:00.260367 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:00.260336    4559 generic.go:356] "Generic (PLEG): container finished" podID="d1eaa8f7-ef7a-47b0-b7cc-6100c6e858e8" containerID="a102c76363df6fd1e22f21f30d378635f9008b72393bcad37083a1fbadcc0c7f" exitCode=0
Nov 02 12:46:00.260476 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:00.260425    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-cd1386cb-7b76784c55-g6lkv" event={"ID":"d1eaa8f7-ef7a-47b0-b7cc-6100c6e858e8","Type":"ContainerDied","Data":"a102c76363df6fd1e22f21f30d378635f9008b72393bcad37083a1fbadcc0c7f"}
Nov 02 12:46:00.266584 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:00.266551    4559 generic.go:356] "Generic (PLEG): container finished" podID="0830a319-a5b3-4f40-a0ba-6e218c86466b" containerID="da99a43b4b2f08c38e21730f80c7fa736947bcdb08560df1027af6754ceadc5e" exitCode=0
Nov 02 12:46:00.266733 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:00.266693    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-cd1386cb-556c7759d7-fwckp" event={"ID":"0830a319-a5b3-4f40-a0ba-6e218c86466b","Type":"ContainerDied","Data":"da99a43b4b2f08c38e21730f80c7fa736947bcdb08560df1027af6754ceadc5e"}
Nov 02 12:46:00.405895 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:00.405857    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-cd1386cb-7b76784c55-g6lkv"
Nov 02 12:46:00.461948 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:00.461923    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1eaa8f7-ef7a-47b0-b7cc-6100c6e858e8-kube-api-access-lm5lm\" (UniqueName: \"kubernetes.io/projected/d1eaa8f7-ef7a-47b0-b7cc-6100c6e858e8-kube-api-access-lm5lm\") pod \"d1eaa8f7-ef7a-47b0-b7cc-6100c6e858e8\" (UID: \"d1eaa8f7-ef7a-47b0-b7cc-6100c6e858e8\") "
Nov 02 12:46:00.462129 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:00.462094    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1eaa8f7-ef7a-47b0-b7cc-6100c6e858e8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1eaa8f7-ef7a-47b0-b7cc-6100c6e858e8-plugins-config-volume\") pod \"d1eaa8f7-ef7a-47b0-b7cc-6100c6e858e8\" (UID: \"d1eaa8f7-ef7a-47b0-b7cc-6100c6e858e8\") "
Nov 02 12:46:00.462460 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:00.462433    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1eaa8f7-ef7a-47b0-b7cc-6100c6e858e8-plugins-config-volume" pod "d1eaa8f7-ef7a-47b0-b7cc-6100c6e858e8" (UID: "d1eaa8f7-ef7a-47b0-b7cc-6100c6e858e8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:46:00.462719 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:00.462698    4559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1eaa8f7-ef7a-47b0-b7cc-6100c6e858e8-plugins-config-volume\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:46:00.462851 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:00.462837    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-cd1386cb-556c7759d7-fwckp"
Nov 02 12:46:00.463917 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:00.463894    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1eaa8f7-ef7a-47b0-b7cc-6100c6e858e8-kube-api-access-lm5lm" pod "d1eaa8f7-ef7a-47b0-b7cc-6100c6e858e8" (UID: "d1eaa8f7-ef7a-47b0-b7cc-6100c6e858e8"). InnerVolumeSpecName "kube-api-access-lm5lm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:46:00.563502 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:00.563473    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0830a319-a5b3-4f40-a0ba-6e218c86466b-kube-api-access-cswbm\" (UniqueName: \"kubernetes.io/projected/0830a319-a5b3-4f40-a0ba-6e218c86466b-kube-api-access-cswbm\") pod \"0830a319-a5b3-4f40-a0ba-6e218c86466b\" (UID: \"0830a319-a5b3-4f40-a0ba-6e218c86466b\") "
Nov 02 12:46:00.563877 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:00.563597    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0830a319-a5b3-4f40-a0ba-6e218c86466b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0830a319-a5b3-4f40-a0ba-6e218c86466b-plugins-config-volume\") pod \"0830a319-a5b3-4f40-a0ba-6e218c86466b\" (UID: \"0830a319-a5b3-4f40-a0ba-6e218c86466b\") "
Nov 02 12:46:00.563983 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:00.563959    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0830a319-a5b3-4f40-a0ba-6e218c86466b-plugins-config-volume" pod "0830a319-a5b3-4f40-a0ba-6e218c86466b" (UID: "0830a319-a5b3-4f40-a0ba-6e218c86466b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:46:00.564075 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:00.564060    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lm5lm\" (UniqueName: \"kubernetes.io/projected/d1eaa8f7-ef7a-47b0-b7cc-6100c6e858e8-kube-api-access-lm5lm\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:46:00.564108 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:00.564079    4559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0830a319-a5b3-4f40-a0ba-6e218c86466b-plugins-config-volume\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:46:00.566195 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:00.566177    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0830a319-a5b3-4f40-a0ba-6e218c86466b-kube-api-access-cswbm" pod "0830a319-a5b3-4f40-a0ba-6e218c86466b" (UID: "0830a319-a5b3-4f40-a0ba-6e218c86466b"). InnerVolumeSpecName "kube-api-access-cswbm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:46:00.665745 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:00.665707    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cswbm\" (UniqueName: \"kubernetes.io/projected/0830a319-a5b3-4f40-a0ba-6e218c86466b-kube-api-access-cswbm\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:46:01.275418 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:01.275387    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-cd1386cb-7b76784c55-g6lkv"
Nov 02 12:46:01.275602 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:01.275390    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-cd1386cb-7b76784c55-g6lkv" event={"ID":"d1eaa8f7-ef7a-47b0-b7cc-6100c6e858e8","Type":"ContainerDied","Data":"f6fc474c12de05049adf8624fdaf82c2e700b80f02808800e6e38cf416fac93b"}
Nov 02 12:46:01.275602 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:01.275546    4559 scope.go:122] "RemoveContainer" containerID="a102c76363df6fd1e22f21f30d378635f9008b72393bcad37083a1fbadcc0c7f"
Nov 02 12:46:01.277000 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:01.276945    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-cd1386cb-556c7759d7-fwckp"
Nov 02 12:46:01.277000 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:01.276968    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-cd1386cb-556c7759d7-fwckp" event={"ID":"0830a319-a5b3-4f40-a0ba-6e218c86466b","Type":"ContainerDied","Data":"97e76d710393993783395dd2295bb8f1c9ddd04b4650241d8acbe24dc554ee66"}
Nov 02 12:46:01.341216 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:01.341157    4559 scope.go:122] "RemoveContainer" containerID="da99a43b4b2f08c38e21730f80c7fa736947bcdb08560df1027af6754ceadc5e"
Nov 02 12:46:01.397488 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:01.384192    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-cd1386cb-7b76784c55-g6lkv"]
Nov 02 12:46:01.411931 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:01.411896    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-cd1386cb-7b76784c55-g6lkv"]
Nov 02 12:46:01.447049 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:01.446847    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-cd1386cb-556c7759d7-fwckp"]
Nov 02 12:46:01.454224 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:01.454188    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-cd1386cb-556c7759d7-fwckp"]
Nov 02 12:46:02.003660 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:01.997025    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0830a319-a5b3-4f40-a0ba-6e218c86466b" path="/var/lib/kubelet/pods/0830a319-a5b3-4f40-a0ba-6e218c86466b/volumes"
Nov 02 12:46:02.003660 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:01.998019    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1eaa8f7-ef7a-47b0-b7cc-6100c6e858e8" path="/var/lib/kubelet/pods/d1eaa8f7-ef7a-47b0-b7cc-6100c6e858e8/volumes"
Nov 02 12:46:06.421112 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:06.421078    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-59060a-dc47f5fff-g5cgn"]
Nov 02 12:46:06.455170 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:06.455140    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-59060a-75f98c4c8c-kln2x"]
Nov 02 12:46:08.513780 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:08.513744    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-cd1386cb-b4fb76986-9zgpp"]
Nov 02 12:46:08.514284 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:08.513962    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cd1386cb-b4fb76986-9zgpp" podUID="6bd63532-f2d2-4317-a46a-955025664a34" containerName="maas-api" containerID="cri-o://659029b930309e951d85b60a7d80edb93af960d34bcd4d644c1662e5949ca7d0" gracePeriod=30
Nov 02 12:46:08.899240 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:08.880722    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cd1386cb-b4fb76986-9zgpp"
Nov 02 12:46:08.926997 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:08.919334    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6bd63532-f2d2-4317-a46a-955025664a34-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6bd63532-f2d2-4317-a46a-955025664a34-metrics-certs\") pod \"6bd63532-f2d2-4317-a46a-955025664a34\" (UID: \"6bd63532-f2d2-4317-a46a-955025664a34\") "
Nov 02 12:46:08.926997 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:08.919440    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6bd63532-f2d2-4317-a46a-955025664a34-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bd63532-f2d2-4317-a46a-955025664a34-openshift-service-ca\") pod \"6bd63532-f2d2-4317-a46a-955025664a34\" (UID: \"6bd63532-f2d2-4317-a46a-955025664a34\") "
Nov 02 12:46:08.926997 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:08.919629    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6bd63532-f2d2-4317-a46a-955025664a34-kube-api-access-gdvjf\" (UniqueName: \"kubernetes.io/projected/6bd63532-f2d2-4317-a46a-955025664a34-kube-api-access-gdvjf\") pod \"6bd63532-f2d2-4317-a46a-955025664a34\" (UID: \"6bd63532-f2d2-4317-a46a-955025664a34\") "
Nov 02 12:46:08.926997 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:08.919816    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6bd63532-f2d2-4317-a46a-955025664a34-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6bd63532-f2d2-4317-a46a-955025664a34-maas-api-tls\") pod \"6bd63532-f2d2-4317-a46a-955025664a34\" (UID: \"6bd63532-f2d2-4317-a46a-955025664a34\") "
Nov 02 12:46:08.926997 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:08.919906    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6bd63532-f2d2-4317-a46a-955025664a34-openshift-service-ca" pod "6bd63532-f2d2-4317-a46a-955025664a34" (UID: "6bd63532-f2d2-4317-a46a-955025664a34"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:46:08.938996 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:08.937781    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6bd63532-f2d2-4317-a46a-955025664a34-metrics-certs" pod "6bd63532-f2d2-4317-a46a-955025664a34" (UID: "6bd63532-f2d2-4317-a46a-955025664a34"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:46:08.938996 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:08.938420    4559 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6bd63532-f2d2-4317-a46a-955025664a34-metrics-certs\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:46:08.938996 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:08.938449    4559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bd63532-f2d2-4317-a46a-955025664a34-openshift-service-ca\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:46:09.041038 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:09.041004    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bd63532-f2d2-4317-a46a-955025664a34-kube-api-access-gdvjf" pod "6bd63532-f2d2-4317-a46a-955025664a34" (UID: "6bd63532-f2d2-4317-a46a-955025664a34"). InnerVolumeSpecName "kube-api-access-gdvjf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:46:09.077456 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:09.042189    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gdvjf\" (UniqueName: \"kubernetes.io/projected/6bd63532-f2d2-4317-a46a-955025664a34-kube-api-access-gdvjf\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:46:09.077456 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:09.042475    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6bd63532-f2d2-4317-a46a-955025664a34-maas-api-tls" pod "6bd63532-f2d2-4317-a46a-955025664a34" (UID: "6bd63532-f2d2-4317-a46a-955025664a34"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:46:09.157658 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:09.149091    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6bd63532-f2d2-4317-a46a-955025664a34-maas-api-tls\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:46:09.473438 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:09.473406    4559 generic.go:356] "Generic (PLEG): container finished" podID="6bd63532-f2d2-4317-a46a-955025664a34" containerID="659029b930309e951d85b60a7d80edb93af960d34bcd4d644c1662e5949ca7d0" exitCode=0
Nov 02 12:46:09.473627 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:09.473478    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cd1386cb-b4fb76986-9zgpp"
Nov 02 12:46:09.473627 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:09.473497    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cd1386cb-b4fb76986-9zgpp" event={"ID":"6bd63532-f2d2-4317-a46a-955025664a34","Type":"ContainerDied","Data":"659029b930309e951d85b60a7d80edb93af960d34bcd4d644c1662e5949ca7d0"}
Nov 02 12:46:09.473627 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:09.473557    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cd1386cb-b4fb76986-9zgpp" event={"ID":"6bd63532-f2d2-4317-a46a-955025664a34","Type":"ContainerDied","Data":"b80a54d1ede6d26125ed30d6a0fa1bb9851a8d003d385fa8c8d62c8ee096be73"}
Nov 02 12:46:09.473627 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:09.473585    4559 scope.go:122] "RemoveContainer" containerID="659029b930309e951d85b60a7d80edb93af960d34bcd4d644c1662e5949ca7d0"
Nov 02 12:46:09.484549 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:09.484524    4559 scope.go:122] "RemoveContainer" containerID="659029b930309e951d85b60a7d80edb93af960d34bcd4d644c1662e5949ca7d0"
Nov 02 12:46:09.486242 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:46:09.486181    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"659029b930309e951d85b60a7d80edb93af960d34bcd4d644c1662e5949ca7d0\": container with ID starting with 659029b930309e951d85b60a7d80edb93af960d34bcd4d644c1662e5949ca7d0 not found: ID does not exist" containerID="659029b930309e951d85b60a7d80edb93af960d34bcd4d644c1662e5949ca7d0"
Nov 02 12:46:09.486242 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:09.486219    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"659029b930309e951d85b60a7d80edb93af960d34bcd4d644c1662e5949ca7d0"} err="failed to get container status \"659029b930309e951d85b60a7d80edb93af960d34bcd4d644c1662e5949ca7d0\": rpc error: code = NotFound desc = could not find container \"659029b930309e951d85b60a7d80edb93af960d34bcd4d644c1662e5949ca7d0\": container with ID starting with 659029b930309e951d85b60a7d80edb93af960d34bcd4d644c1662e5949ca7d0 not found: ID does not exist"
Nov 02 12:46:09.530775 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:09.530740    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-cd1386cb-b4fb76986-9zgpp"]
Nov 02 12:46:09.543286 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:09.543258    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-cd1386cb-b4fb76986-9zgpp"]
Nov 02 12:46:09.895016 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:09.894927    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6bd63532-f2d2-4317-a46a-955025664a34" path="/var/lib/kubelet/pods/6bd63532-f2d2-4317-a46a-955025664a34/volumes"
Nov 02 12:46:10.801777 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:10.801743    4559 patch_prober.go:29] interesting pod/e2e-ipp-a-59060a-openshift-default-8666cdf765-hfz77 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 12:46:10.820849 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:10.801805    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-59060a-openshift-default-8666cdf765-hfz77" podUID="9ed1a149-3a01-4af1-9b01-e059ff121bed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:46:11.422048 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:11.421989    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-59060a-dc47f5fff-g5cgn" podUID="f28d03d2-8b87-43a0-b75a-ba0aa7ba664d" containerName="payload-processing" containerID="cri-o://b08273f54cc27f897557b302fc8b2727eae1152f87aa4197889f3f6c38e9153d" gracePeriod=25
Nov 02 12:46:11.456092 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:11.456057    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-59060a-75f98c4c8c-kln2x" podUID="b90ad6d8-0ab8-4e18-9c5e-e74569e0d0c5" containerName="payload-pre-processing" containerID="cri-o://acd50389f58746f8bd2cdacb86a3a6a89c198968c0226ed91eae61e78b8c0449" gracePeriod=25
Nov 02 12:46:11.487555 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:11.487527    4559 generic.go:356] "Generic (PLEG): container finished" podID="9ed1a149-3a01-4af1-9b01-e059ff121bed" containerID="53df5ae504ad7e4d8df77fb7760dd73b569c12cb28a56f5a0cb1bb2f5402e97e" exitCode=0
Nov 02 12:46:11.487751 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:11.487603    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-59060a-openshift-default-8666cdf765-hfz77" event={"ID":"9ed1a149-3a01-4af1-9b01-e059ff121bed","Type":"ContainerDied","Data":"53df5ae504ad7e4d8df77fb7760dd73b569c12cb28a56f5a0cb1bb2f5402e97e"}
Nov 02 12:46:11.488201 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:11.488169    4559 scope.go:122] "RemoveContainer" containerID="53df5ae504ad7e4d8df77fb7760dd73b569c12cb28a56f5a0cb1bb2f5402e97e"
Nov 02 12:46:11.489501 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:11.489475    4559 generic.go:356] "Generic (PLEG): container finished" podID="60be7b9e-07a0-4cc6-99d1-1278b6bad791" containerID="fd000b435ffe806cb1b71a659d652a87fc1284ebe313cb5576983e842e971154" exitCode=0
Nov 02 12:46:11.489622 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:11.489510    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-77cf70-openshift-default-d8f7564b9-rftzg" event={"ID":"60be7b9e-07a0-4cc6-99d1-1278b6bad791","Type":"ContainerDied","Data":"fd000b435ffe806cb1b71a659d652a87fc1284ebe313cb5576983e842e971154"}
Nov 02 12:46:11.489896 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:11.489865    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:46:11.489977 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:11.489905    4559 scope.go:122] "RemoveContainer" containerID="fd000b435ffe806cb1b71a659d652a87fc1284ebe313cb5576983e842e971154"
Nov 02 12:46:11.489977 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:11.489954    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:46:11.490086 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:11.490007    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:46:11.491251 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:11.491219    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:46:11.491343 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:11.491296    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:46:11.491403 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:11.491344    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:46:11.821837 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:11.820985    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-59060a-dc47f5fff-g5cgn"
Nov 02 12:46:11.909317 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:11.906850    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f28d03d2-8b87-43a0-b75a-ba0aa7ba664d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f28d03d2-8b87-43a0-b75a-ba0aa7ba664d-plugins-config-volume\") pod \"f28d03d2-8b87-43a0-b75a-ba0aa7ba664d\" (UID: \"f28d03d2-8b87-43a0-b75a-ba0aa7ba664d\") "
Nov 02 12:46:11.909317 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:11.906907    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f28d03d2-8b87-43a0-b75a-ba0aa7ba664d-kube-api-access-k9dn6\" (UniqueName: \"kubernetes.io/projected/f28d03d2-8b87-43a0-b75a-ba0aa7ba664d-kube-api-access-k9dn6\") pod \"f28d03d2-8b87-43a0-b75a-ba0aa7ba664d\" (UID: \"f28d03d2-8b87-43a0-b75a-ba0aa7ba664d\") "
Nov 02 12:46:11.909317 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:11.908275    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f28d03d2-8b87-43a0-b75a-ba0aa7ba664d-plugins-config-volume" pod "f28d03d2-8b87-43a0-b75a-ba0aa7ba664d" (UID: "f28d03d2-8b87-43a0-b75a-ba0aa7ba664d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:46:11.914687 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:11.914626    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f28d03d2-8b87-43a0-b75a-ba0aa7ba664d-kube-api-access-k9dn6" pod "f28d03d2-8b87-43a0-b75a-ba0aa7ba664d" (UID: "f28d03d2-8b87-43a0-b75a-ba0aa7ba664d"). InnerVolumeSpecName "kube-api-access-k9dn6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:46:11.931876 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:11.931802    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-59060a-75f98c4c8c-kln2x"
Nov 02 12:46:12.008586 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:12.008556    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b90ad6d8-0ab8-4e18-9c5e-e74569e0d0c5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b90ad6d8-0ab8-4e18-9c5e-e74569e0d0c5-plugins-config-volume\") pod \"b90ad6d8-0ab8-4e18-9c5e-e74569e0d0c5\" (UID: \"b90ad6d8-0ab8-4e18-9c5e-e74569e0d0c5\") "
Nov 02 12:46:12.008779 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:12.008618    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b90ad6d8-0ab8-4e18-9c5e-e74569e0d0c5-kube-api-access-vtw6c\" (UniqueName: \"kubernetes.io/projected/b90ad6d8-0ab8-4e18-9c5e-e74569e0d0c5-kube-api-access-vtw6c\") pod \"b90ad6d8-0ab8-4e18-9c5e-e74569e0d0c5\" (UID: \"b90ad6d8-0ab8-4e18-9c5e-e74569e0d0c5\") "
Nov 02 12:46:12.009463 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:12.009184    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b90ad6d8-0ab8-4e18-9c5e-e74569e0d0c5-plugins-config-volume" pod "b90ad6d8-0ab8-4e18-9c5e-e74569e0d0c5" (UID: "b90ad6d8-0ab8-4e18-9c5e-e74569e0d0c5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:46:12.009619 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:12.009585    4559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b90ad6d8-0ab8-4e18-9c5e-e74569e0d0c5-plugins-config-volume\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:46:12.009706 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:12.009628    4559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f28d03d2-8b87-43a0-b75a-ba0aa7ba664d-plugins-config-volume\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:46:12.009706 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:12.009686    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9dn6\" (UniqueName: \"kubernetes.io/projected/f28d03d2-8b87-43a0-b75a-ba0aa7ba664d-kube-api-access-k9dn6\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:46:12.011596 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:12.011571    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b90ad6d8-0ab8-4e18-9c5e-e74569e0d0c5-kube-api-access-vtw6c" pod "b90ad6d8-0ab8-4e18-9c5e-e74569e0d0c5" (UID: "b90ad6d8-0ab8-4e18-9c5e-e74569e0d0c5"). InnerVolumeSpecName "kube-api-access-vtw6c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:46:12.111493 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:12.111459    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vtw6c\" (UniqueName: \"kubernetes.io/projected/b90ad6d8-0ab8-4e18-9c5e-e74569e0d0c5-kube-api-access-vtw6c\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:46:12.523039 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:12.523002    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-77cf70-openshift-default-d8f7564b9-rftzg" event={"ID":"60be7b9e-07a0-4cc6-99d1-1278b6bad791","Type":"ContainerStarted","Data":"2c0e95211f34c0ba2ab371f9f6e7b430bf725216b1e73f988eb5a2ae1327fed1"}
Nov 02 12:46:12.538704 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:12.527750    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-59060a-openshift-default-8666cdf765-hfz77" event={"ID":"9ed1a149-3a01-4af1-9b01-e059ff121bed","Type":"ContainerStarted","Data":"73d0524d7341a61dfeeda451f1a2a72043fc9e3d3f045d0d0fe49a0b6eae043d"}
Nov 02 12:46:12.561104 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:12.561069    4559 generic.go:356] "Generic (PLEG): container finished" podID="f28d03d2-8b87-43a0-b75a-ba0aa7ba664d" containerID="b08273f54cc27f897557b302fc8b2727eae1152f87aa4197889f3f6c38e9153d" exitCode=0
Nov 02 12:46:12.561477 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:12.561454    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-59060a-dc47f5fff-g5cgn" event={"ID":"f28d03d2-8b87-43a0-b75a-ba0aa7ba664d","Type":"ContainerDied","Data":"b08273f54cc27f897557b302fc8b2727eae1152f87aa4197889f3f6c38e9153d"}
Nov 02 12:46:12.561705 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:12.561686    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-59060a-dc47f5fff-g5cgn" event={"ID":"f28d03d2-8b87-43a0-b75a-ba0aa7ba664d","Type":"ContainerDied","Data":"e5ecfd6e4580b73158fb759846dae05373c906b705783a2b1c54eda39d962386"}
Nov 02 12:46:12.561860 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:12.561845    4559 scope.go:122] "RemoveContainer" containerID="b08273f54cc27f897557b302fc8b2727eae1152f87aa4197889f3f6c38e9153d"
Nov 02 12:46:12.562151 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:12.562135    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-59060a-dc47f5fff-g5cgn"
Nov 02 12:46:13.043760 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:13.042987    4559 generic.go:356] "Generic (PLEG): container finished" podID="b90ad6d8-0ab8-4e18-9c5e-e74569e0d0c5" containerID="acd50389f58746f8bd2cdacb86a3a6a89c198968c0226ed91eae61e78b8c0449" exitCode=0
Nov 02 12:46:13.043760 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:13.043572    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-59060a-75f98c4c8c-kln2x"
Nov 02 12:46:13.065682 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:13.065009    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-59060a-75f98c4c8c-kln2x" event={"ID":"b90ad6d8-0ab8-4e18-9c5e-e74569e0d0c5","Type":"ContainerDied","Data":"acd50389f58746f8bd2cdacb86a3a6a89c198968c0226ed91eae61e78b8c0449"}
Nov 02 12:46:13.065682 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:13.065070    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-59060a-75f98c4c8c-kln2x" event={"ID":"b90ad6d8-0ab8-4e18-9c5e-e74569e0d0c5","Type":"ContainerDied","Data":"d4b2d57ad7b89c4abd1c76a5b33f41c5ad61930e92884882db4f3e1508cac834"}
Nov 02 12:46:13.065682 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:13.065354    4559 scope.go:122] "RemoveContainer" containerID="b08273f54cc27f897557b302fc8b2727eae1152f87aa4197889f3f6c38e9153d"
Nov 02 12:46:13.072951 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:46:13.072816    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b08273f54cc27f897557b302fc8b2727eae1152f87aa4197889f3f6c38e9153d\": container with ID starting with b08273f54cc27f897557b302fc8b2727eae1152f87aa4197889f3f6c38e9153d not found: ID does not exist" containerID="b08273f54cc27f897557b302fc8b2727eae1152f87aa4197889f3f6c38e9153d"
Nov 02 12:46:13.073261 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:13.073222    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b08273f54cc27f897557b302fc8b2727eae1152f87aa4197889f3f6c38e9153d"} err="failed to get container status \"b08273f54cc27f897557b302fc8b2727eae1152f87aa4197889f3f6c38e9153d\": rpc error: code = NotFound desc = could not find container \"b08273f54cc27f897557b302fc8b2727eae1152f87aa4197889f3f6c38e9153d\": container with ID starting with b08273f54cc27f897557b302fc8b2727eae1152f87aa4197889f3f6c38e9153d not found: ID does not exist"
Nov 02 12:46:13.073390 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:13.073376    4559 scope.go:122] "RemoveContainer" containerID="acd50389f58746f8bd2cdacb86a3a6a89c198968c0226ed91eae61e78b8c0449"
Nov 02 12:46:13.080039 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:13.080007    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-77cf70-86cf7b694c-45888"]
Nov 02 12:46:13.115492 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:13.099771    4559 patch_prober.go:29] interesting pod/e2e-ipp-b-77cf70-openshift-default-d8f7564b9-rftzg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body=
Nov 02 12:46:13.115492 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:13.099846    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-77cf70-openshift-default-d8f7564b9-rftzg" podUID="60be7b9e-07a0-4cc6-99d1-1278b6bad791" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused"
Nov 02 12:46:13.147994 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:13.146966    4559 cpu_manager.go:414] "containerMap: removing container" podUID="d1eaa8f7-ef7a-47b0-b7cc-6100c6e858e8" containerName="payload-processing"
Nov 02 12:46:13.147994 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:13.146997    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1eaa8f7-ef7a-47b0-b7cc-6100c6e858e8" containerName="payload-processing"
Nov 02 12:46:13.147994 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:13.147031    4559 cpu_manager.go:414] "containerMap: removing container" podUID="0830a319-a5b3-4f40-a0ba-6e218c86466b" containerName="payload-pre-processing"
Nov 02 12:46:13.147994 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:13.147040    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0830a319-a5b3-4f40-a0ba-6e218c86466b" containerName="payload-pre-processing"
Nov 02 12:46:13.147994 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:13.147074    4559 cpu_manager.go:414] "containerMap: removing container" podUID="bb4ea3c3-befb-4ee2-b46b-93ce5e4a7110" containerName="payload-pre-processing"
Nov 02 12:46:13.147994 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:13.147084    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb4ea3c3-befb-4ee2-b46b-93ce5e4a7110" containerName="payload-pre-processing"
Nov 02 12:46:13.147994 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:13.147108    4559 cpu_manager.go:414] "containerMap: removing container" podUID="f28d03d2-8b87-43a0-b75a-ba0aa7ba664d" containerName="payload-processing"
Nov 02 12:46:13.147994 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:13.147116    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f28d03d2-8b87-43a0-b75a-ba0aa7ba664d" containerName="payload-processing"
Nov 02 12:46:13.147994 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:13.147171    4559 cpu_manager.go:414] "containerMap: removing container" podUID="6bd63532-f2d2-4317-a46a-955025664a34" containerName="maas-api"
Nov 02 12:46:13.147994 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:13.147179    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6bd63532-f2d2-4317-a46a-955025664a34" containerName="maas-api"
Nov 02 12:46:13.147994 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:13.147233    4559 cpu_manager.go:414] "containerMap: removing container" podUID="b90ad6d8-0ab8-4e18-9c5e-e74569e0d0c5" containerName="payload-pre-processing"
Nov 02 12:46:13.147994 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:13.147242    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b90ad6d8-0ab8-4e18-9c5e-e74569e0d0c5" containerName="payload-pre-processing"
Nov 02 12:46:13.147994 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:13.147272    4559 cpu_manager.go:414] "containerMap: removing container" podUID="ae88d18c-c12e-4c53-a0e0-b012a2d624ff" containerName="payload-processing"
Nov 02 12:46:13.147994 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:13.147288    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae88d18c-c12e-4c53-a0e0-b012a2d624ff" containerName="payload-processing"
Nov 02 12:46:13.149291 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:13.147693    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="0830a319-a5b3-4f40-a0ba-6e218c86466b" containerName="payload-pre-processing"
Nov 02 12:46:13.149291 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:13.149113    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb4ea3c3-befb-4ee2-b46b-93ce5e4a7110" containerName="payload-pre-processing"
Nov 02 12:46:13.149291 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:13.149141    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="f28d03d2-8b87-43a0-b75a-ba0aa7ba664d" containerName="payload-processing"
Nov 02 12:46:13.149291 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:13.149166    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="6bd63532-f2d2-4317-a46a-955025664a34" containerName="maas-api"
Nov 02 12:46:13.149291 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:13.149185    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="b90ad6d8-0ab8-4e18-9c5e-e74569e0d0c5" containerName="payload-pre-processing"
Nov 02 12:46:13.150420 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:13.149628    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae88d18c-c12e-4c53-a0e0-b012a2d624ff" containerName="payload-processing"
Nov 02 12:46:13.150420 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:13.149679    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1eaa8f7-ef7a-47b0-b7cc-6100c6e858e8" containerName="payload-processing"
Nov 02 12:46:13.161791 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:13.161769    4559 scope.go:122] "RemoveContainer" containerID="acd50389f58746f8bd2cdacb86a3a6a89c198968c0226ed91eae61e78b8c0449"
Nov 02 12:46:13.162105 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:46:13.162078    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"acd50389f58746f8bd2cdacb86a3a6a89c198968c0226ed91eae61e78b8c0449\": container with ID starting with acd50389f58746f8bd2cdacb86a3a6a89c198968c0226ed91eae61e78b8c0449 not found: ID does not exist" containerID="acd50389f58746f8bd2cdacb86a3a6a89c198968c0226ed91eae61e78b8c0449"
Nov 02 12:46:13.162185 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:13.162113    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"acd50389f58746f8bd2cdacb86a3a6a89c198968c0226ed91eae61e78b8c0449"} err="failed to get container status \"acd50389f58746f8bd2cdacb86a3a6a89c198968c0226ed91eae61e78b8c0449\": rpc error: code = NotFound desc = could not find container \"acd50389f58746f8bd2cdacb86a3a6a89c198968c0226ed91eae61e78b8c0449\": container with ID starting with acd50389f58746f8bd2cdacb86a3a6a89c198968c0226ed91eae61e78b8c0449 not found: ID does not exist"
Nov 02 12:46:13.165141 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:13.165112    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-77cf70-86cf7b694c-45888"]
Nov 02 12:46:13.165237 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:13.165156    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-77cf70-openshift-default-d8f7564b9-rftzg"
Nov 02 12:46:13.165237 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:13.165189    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-77cf70-8686479b7b-mcd6f"]
Nov 02 12:46:13.165348 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:13.165268    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-77cf70-86cf7b694c-45888"
Nov 02 12:46:13.170966 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:13.170937    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-77cf70-8686479b7b-mcd6f"]
Nov 02 12:46:13.171386 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:13.171321    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-77cf70-8686479b7b-mcd6f"
Nov 02 12:46:13.180907 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:13.180819    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-77cf70\""
Nov 02 12:46:13.181156 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:13.181136    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-77cf70-dockercfg-b8czh\""
Nov 02 12:46:13.245415 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:13.245376    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-59060a-dc47f5fff-g5cgn"]
Nov 02 12:46:13.251237 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:13.251211    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-59060a-dc47f5fff-g5cgn"]
Nov 02 12:46:13.303734 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:13.296673    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-59060a-openshift-default-8666cdf765-hfz77"
Nov 02 12:46:13.335669 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:13.320805    4559 patch_prober.go:29] interesting pod/e2e-ipp-a-59060a-openshift-default-8666cdf765-hfz77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body=
Nov 02 12:46:13.335669 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:13.320908    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-59060a-openshift-default-8666cdf765-hfz77" podUID="9ed1a149-3a01-4af1-9b01-e059ff121bed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused"
Nov 02 12:46:13.356666 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:13.353075    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ee04a57-e5e9-48fa-a5a8-c89bcf917616-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-77cf70-8686479b7b-mcd6f\" (UID: \"6ee04a57-e5e9-48fa-a5a8-c89bcf917616\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-77cf70-8686479b7b-mcd6f"
Nov 02 12:46:13.372562 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:13.372522    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gchq\" (UniqueName: \"kubernetes.io/projected/6ee04a57-e5e9-48fa-a5a8-c89bcf917616-kube-api-access-8gchq\") pod \"payload-processing-e2e-ipp-b-77cf70-8686479b7b-mcd6f\" (UID: \"6ee04a57-e5e9-48fa-a5a8-c89bcf917616\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-77cf70-8686479b7b-mcd6f"
Nov 02 12:46:13.373033 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:13.373003    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc18257f-4549-4cf1-9fe5-227496db7c80-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-77cf70-86cf7b694c-45888\" (UID: \"bc18257f-4549-4cf1-9fe5-227496db7c80\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-77cf70-86cf7b694c-45888"
Nov 02 12:46:13.374795 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:13.374762    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dj6qf\" (UniqueName: \"kubernetes.io/projected/bc18257f-4549-4cf1-9fe5-227496db7c80-kube-api-access-dj6qf\") pod \"payload-pre-processing-e2e-ipp-b-77cf70-86cf7b694c-45888\" (UID: \"bc18257f-4549-4cf1-9fe5-227496db7c80\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-77cf70-86cf7b694c-45888"
Nov 02 12:46:13.473723 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:13.468708    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-59060a-75f98c4c8c-kln2x"]
Nov 02 12:46:13.499109 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:13.479538    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-59060a-75f98c4c8c-kln2x"]
Nov 02 12:46:13.500523 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:13.500489    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dj6qf\" (UniqueName: \"kubernetes.io/projected/bc18257f-4549-4cf1-9fe5-227496db7c80-kube-api-access-dj6qf\") pod \"payload-pre-processing-e2e-ipp-b-77cf70-86cf7b694c-45888\" (UID: \"bc18257f-4549-4cf1-9fe5-227496db7c80\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-77cf70-86cf7b694c-45888"
Nov 02 12:46:13.507663 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:13.501062    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ee04a57-e5e9-48fa-a5a8-c89bcf917616-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-77cf70-8686479b7b-mcd6f\" (UID: \"6ee04a57-e5e9-48fa-a5a8-c89bcf917616\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-77cf70-8686479b7b-mcd6f"
Nov 02 12:46:13.507993 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:13.507966    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gchq\" (UniqueName: \"kubernetes.io/projected/6ee04a57-e5e9-48fa-a5a8-c89bcf917616-kube-api-access-8gchq\") pod \"payload-processing-e2e-ipp-b-77cf70-8686479b7b-mcd6f\" (UID: \"6ee04a57-e5e9-48fa-a5a8-c89bcf917616\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-77cf70-8686479b7b-mcd6f"
Nov 02 12:46:13.508269 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:13.508221    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc18257f-4549-4cf1-9fe5-227496db7c80-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-77cf70-86cf7b694c-45888\" (UID: \"bc18257f-4549-4cf1-9fe5-227496db7c80\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-77cf70-86cf7b694c-45888"
Nov 02 12:46:13.529556 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:13.529522    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ee04a57-e5e9-48fa-a5a8-c89bcf917616-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-77cf70-8686479b7b-mcd6f\" (UID: \"6ee04a57-e5e9-48fa-a5a8-c89bcf917616\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-77cf70-8686479b7b-mcd6f"
Nov 02 12:46:13.533723 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:13.530437    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc18257f-4549-4cf1-9fe5-227496db7c80-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-77cf70-86cf7b694c-45888\" (UID: \"bc18257f-4549-4cf1-9fe5-227496db7c80\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-77cf70-86cf7b694c-45888"
Nov 02 12:46:13.546768 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:13.543586    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dj6qf\" (UniqueName: \"kubernetes.io/projected/bc18257f-4549-4cf1-9fe5-227496db7c80-kube-api-access-dj6qf\") pod \"payload-pre-processing-e2e-ipp-b-77cf70-86cf7b694c-45888\" (UID: \"bc18257f-4549-4cf1-9fe5-227496db7c80\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-77cf70-86cf7b694c-45888"
Nov 02 12:46:13.546980 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:13.545705    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gchq\" (UniqueName: \"kubernetes.io/projected/6ee04a57-e5e9-48fa-a5a8-c89bcf917616-kube-api-access-8gchq\") pod \"payload-processing-e2e-ipp-b-77cf70-8686479b7b-mcd6f\" (UID: \"6ee04a57-e5e9-48fa-a5a8-c89bcf917616\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-77cf70-8686479b7b-mcd6f"
Nov 02 12:46:13.550857 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:13.550835    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-77cf70-8686479b7b-mcd6f"
Nov 02 12:46:13.807028 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:13.806932    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-77cf70-86cf7b694c-45888"
Nov 02 12:46:13.951697 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:13.951664    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b90ad6d8-0ab8-4e18-9c5e-e74569e0d0c5" path="/var/lib/kubelet/pods/b90ad6d8-0ab8-4e18-9c5e-e74569e0d0c5/volumes"
Nov 02 12:46:13.952992 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:13.952448    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f28d03d2-8b87-43a0-b75a-ba0aa7ba664d" path="/var/lib/kubelet/pods/f28d03d2-8b87-43a0-b75a-ba0aa7ba664d/volumes"
Nov 02 12:46:13.952992 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:13.952893    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-77cf70-8686479b7b-mcd6f"]
Nov 02 12:46:14.055507 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:14.055384    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-77cf70-8686479b7b-mcd6f" event={"ID":"6ee04a57-e5e9-48fa-a5a8-c89bcf917616","Type":"ContainerStarted","Data":"7bca123cb8bca1e9f4a93957626fe9d8256fd1cca29846f150ef7a1cfc974722"}
Nov 02 12:46:14.075711 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:14.074135    4559 patch_prober.go:29] interesting pod/e2e-ipp-b-77cf70-openshift-default-d8f7564b9-rftzg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body=
Nov 02 12:46:14.075711 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:14.074183    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-77cf70-openshift-default-d8f7564b9-rftzg" podUID="60be7b9e-07a0-4cc6-99d1-1278b6bad791" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused"
Nov 02 12:46:14.238222 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:14.238122    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-77cf70-86cf7b694c-45888"]
Nov 02 12:46:14.240345 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:46:14.240316    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc18257f_4549_4cf1_9fe5_227496db7c80.slice/crio-103a1295884f274165fd4fe01692e934a624a19dfa3a9d5a3bcb4ea36116a3ea WatchSource:0}: Error finding container 103a1295884f274165fd4fe01692e934a624a19dfa3a9d5a3bcb4ea36116a3ea: Status 404 returned error can't find the container with id 103a1295884f274165fd4fe01692e934a624a19dfa3a9d5a3bcb4ea36116a3ea
Nov 02 12:46:14.312419 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:14.312378    4559 patch_prober.go:29] interesting pod/e2e-ipp-a-59060a-openshift-default-8666cdf765-hfz77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body=
Nov 02 12:46:14.312935 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:14.312901    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-59060a-openshift-default-8666cdf765-hfz77" podUID="9ed1a149-3a01-4af1-9b01-e059ff121bed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused"
Nov 02 12:46:14.664434 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:14.664400    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-97de36-openshift-default-77899d747c-cq4bc"]
Nov 02 12:46:14.673058 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:14.673019    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-97de36-openshift-default-77899d747c-cq4bc"
Nov 02 12:46:14.733657 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:14.714879    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-97de36-openshift-default-dockercfg-bxm9b\""
Nov 02 12:46:14.747160 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:14.747123    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-97de36-openshift-default-77899d747c-cq4bc"]
Nov 02 12:46:14.882678 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:14.870901    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b55da6b-89cb-46d6-a7a0-3a09e7d7d17e-istio-token\") pod \"e2e-ipp-b-97de36-openshift-default-77899d747c-cq4bc\" (UID: \"5b55da6b-89cb-46d6-a7a0-3a09e7d7d17e\") " pod="openshift-ingress/e2e-ipp-b-97de36-openshift-default-77899d747c-cq4bc"
Nov 02 12:46:14.882678 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:14.871106    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b55da6b-89cb-46d6-a7a0-3a09e7d7d17e-istio-data\") pod \"e2e-ipp-b-97de36-openshift-default-77899d747c-cq4bc\" (UID: \"5b55da6b-89cb-46d6-a7a0-3a09e7d7d17e\") " pod="openshift-ingress/e2e-ipp-b-97de36-openshift-default-77899d747c-cq4bc"
Nov 02 12:46:14.882678 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:14.871164    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b55da6b-89cb-46d6-a7a0-3a09e7d7d17e-istio-envoy\") pod \"e2e-ipp-b-97de36-openshift-default-77899d747c-cq4bc\" (UID: \"5b55da6b-89cb-46d6-a7a0-3a09e7d7d17e\") " pod="openshift-ingress/e2e-ipp-b-97de36-openshift-default-77899d747c-cq4bc"
Nov 02 12:46:14.882678 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:14.871308    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b55da6b-89cb-46d6-a7a0-3a09e7d7d17e-credential-socket\") pod \"e2e-ipp-b-97de36-openshift-default-77899d747c-cq4bc\" (UID: \"5b55da6b-89cb-46d6-a7a0-3a09e7d7d17e\") " pod="openshift-ingress/e2e-ipp-b-97de36-openshift-default-77899d747c-cq4bc"
Nov 02 12:46:14.882678 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:14.871531    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b55da6b-89cb-46d6-a7a0-3a09e7d7d17e-workload-socket\") pod \"e2e-ipp-b-97de36-openshift-default-77899d747c-cq4bc\" (UID: \"5b55da6b-89cb-46d6-a7a0-3a09e7d7d17e\") " pod="openshift-ingress/e2e-ipp-b-97de36-openshift-default-77899d747c-cq4bc"
Nov 02 12:46:14.882678 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:14.871606    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b55da6b-89cb-46d6-a7a0-3a09e7d7d17e-istiod-ca-cert\") pod \"e2e-ipp-b-97de36-openshift-default-77899d747c-cq4bc\" (UID: \"5b55da6b-89cb-46d6-a7a0-3a09e7d7d17e\") " pod="openshift-ingress/e2e-ipp-b-97de36-openshift-default-77899d747c-cq4bc"
Nov 02 12:46:15.093824 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:15.048075    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8wrg\" (UniqueName: \"kubernetes.io/projected/5b55da6b-89cb-46d6-a7a0-3a09e7d7d17e-kube-api-access-h8wrg\") pod \"e2e-ipp-b-97de36-openshift-default-77899d747c-cq4bc\" (UID: \"5b55da6b-89cb-46d6-a7a0-3a09e7d7d17e\") " pod="openshift-ingress/e2e-ipp-b-97de36-openshift-default-77899d747c-cq4bc"
Nov 02 12:46:15.093824 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:15.048162    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b55da6b-89cb-46d6-a7a0-3a09e7d7d17e-istio-podinfo\") pod \"e2e-ipp-b-97de36-openshift-default-77899d747c-cq4bc\" (UID: \"5b55da6b-89cb-46d6-a7a0-3a09e7d7d17e\") " pod="openshift-ingress/e2e-ipp-b-97de36-openshift-default-77899d747c-cq4bc"
Nov 02 12:46:15.093824 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:15.048248    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b55da6b-89cb-46d6-a7a0-3a09e7d7d17e-workload-certs\") pod \"e2e-ipp-b-97de36-openshift-default-77899d747c-cq4bc\" (UID: \"5b55da6b-89cb-46d6-a7a0-3a09e7d7d17e\") " pod="openshift-ingress/e2e-ipp-b-97de36-openshift-default-77899d747c-cq4bc"
Nov 02 12:46:15.093824 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:15.077172    4559 patch_prober.go:29] interesting pod/e2e-ipp-b-77cf70-openshift-default-d8f7564b9-rftzg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body=
Nov 02 12:46:15.093824 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:15.077263    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-77cf70-openshift-default-d8f7564b9-rftzg" podUID="60be7b9e-07a0-4cc6-99d1-1278b6bad791" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused"
Nov 02 12:46:15.152700 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:15.152658    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b55da6b-89cb-46d6-a7a0-3a09e7d7d17e-workload-socket\") pod \"e2e-ipp-b-97de36-openshift-default-77899d747c-cq4bc\" (UID: \"5b55da6b-89cb-46d6-a7a0-3a09e7d7d17e\") " pod="openshift-ingress/e2e-ipp-b-97de36-openshift-default-77899d747c-cq4bc"
Nov 02 12:46:15.152889 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:15.152736    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b55da6b-89cb-46d6-a7a0-3a09e7d7d17e-istiod-ca-cert\") pod \"e2e-ipp-b-97de36-openshift-default-77899d747c-cq4bc\" (UID: \"5b55da6b-89cb-46d6-a7a0-3a09e7d7d17e\") " pod="openshift-ingress/e2e-ipp-b-97de36-openshift-default-77899d747c-cq4bc"
Nov 02 12:46:15.152972 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:15.152943    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8wrg\" (UniqueName: \"kubernetes.io/projected/5b55da6b-89cb-46d6-a7a0-3a09e7d7d17e-kube-api-access-h8wrg\") pod \"e2e-ipp-b-97de36-openshift-default-77899d747c-cq4bc\" (UID: \"5b55da6b-89cb-46d6-a7a0-3a09e7d7d17e\") " pod="openshift-ingress/e2e-ipp-b-97de36-openshift-default-77899d747c-cq4bc"
Nov 02 12:46:15.153032 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:15.152990    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b55da6b-89cb-46d6-a7a0-3a09e7d7d17e-istio-podinfo\") pod \"e2e-ipp-b-97de36-openshift-default-77899d747c-cq4bc\" (UID: \"5b55da6b-89cb-46d6-a7a0-3a09e7d7d17e\") " pod="openshift-ingress/e2e-ipp-b-97de36-openshift-default-77899d747c-cq4bc"
Nov 02 12:46:15.153099 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:15.153065    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b55da6b-89cb-46d6-a7a0-3a09e7d7d17e-workload-certs\") pod \"e2e-ipp-b-97de36-openshift-default-77899d747c-cq4bc\" (UID: \"5b55da6b-89cb-46d6-a7a0-3a09e7d7d17e\") " pod="openshift-ingress/e2e-ipp-b-97de36-openshift-default-77899d747c-cq4bc"
Nov 02 12:46:15.153172 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:15.153157    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b55da6b-89cb-46d6-a7a0-3a09e7d7d17e-istio-token\") pod \"e2e-ipp-b-97de36-openshift-default-77899d747c-cq4bc\" (UID: \"5b55da6b-89cb-46d6-a7a0-3a09e7d7d17e\") " pod="openshift-ingress/e2e-ipp-b-97de36-openshift-default-77899d747c-cq4bc"
Nov 02 12:46:15.153226 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:15.153209    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b55da6b-89cb-46d6-a7a0-3a09e7d7d17e-istio-data\") pod \"e2e-ipp-b-97de36-openshift-default-77899d747c-cq4bc\" (UID: \"5b55da6b-89cb-46d6-a7a0-3a09e7d7d17e\") " pod="openshift-ingress/e2e-ipp-b-97de36-openshift-default-77899d747c-cq4bc"
Nov 02 12:46:15.153276 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:15.153256    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b55da6b-89cb-46d6-a7a0-3a09e7d7d17e-istio-envoy\") pod \"e2e-ipp-b-97de36-openshift-default-77899d747c-cq4bc\" (UID: \"5b55da6b-89cb-46d6-a7a0-3a09e7d7d17e\") " pod="openshift-ingress/e2e-ipp-b-97de36-openshift-default-77899d747c-cq4bc"
Nov 02 12:46:15.153353 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:15.153338    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b55da6b-89cb-46d6-a7a0-3a09e7d7d17e-credential-socket\") pod \"e2e-ipp-b-97de36-openshift-default-77899d747c-cq4bc\" (UID: \"5b55da6b-89cb-46d6-a7a0-3a09e7d7d17e\") " pod="openshift-ingress/e2e-ipp-b-97de36-openshift-default-77899d747c-cq4bc"
Nov 02 12:46:15.153960 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:15.153931    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b55da6b-89cb-46d6-a7a0-3a09e7d7d17e-credential-socket\") pod \"e2e-ipp-b-97de36-openshift-default-77899d747c-cq4bc\" (UID: \"5b55da6b-89cb-46d6-a7a0-3a09e7d7d17e\") " pod="openshift-ingress/e2e-ipp-b-97de36-openshift-default-77899d747c-cq4bc"
Nov 02 12:46:15.154146 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:15.154124    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b55da6b-89cb-46d6-a7a0-3a09e7d7d17e-workload-certs\") pod \"e2e-ipp-b-97de36-openshift-default-77899d747c-cq4bc\" (UID: \"5b55da6b-89cb-46d6-a7a0-3a09e7d7d17e\") " pod="openshift-ingress/e2e-ipp-b-97de36-openshift-default-77899d747c-cq4bc"
Nov 02 12:46:15.154476 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:15.154447    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b55da6b-89cb-46d6-a7a0-3a09e7d7d17e-workload-socket\") pod \"e2e-ipp-b-97de36-openshift-default-77899d747c-cq4bc\" (UID: \"5b55da6b-89cb-46d6-a7a0-3a09e7d7d17e\") " pod="openshift-ingress/e2e-ipp-b-97de36-openshift-default-77899d747c-cq4bc"
Nov 02 12:46:15.173183 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:15.173153    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b55da6b-89cb-46d6-a7a0-3a09e7d7d17e-istio-data\") pod \"e2e-ipp-b-97de36-openshift-default-77899d747c-cq4bc\" (UID: \"5b55da6b-89cb-46d6-a7a0-3a09e7d7d17e\") " pod="openshift-ingress/e2e-ipp-b-97de36-openshift-default-77899d747c-cq4bc"
Nov 02 12:46:15.174199 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:15.174136    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b55da6b-89cb-46d6-a7a0-3a09e7d7d17e-istiod-ca-cert\") pod \"e2e-ipp-b-97de36-openshift-default-77899d747c-cq4bc\" (UID: \"5b55da6b-89cb-46d6-a7a0-3a09e7d7d17e\") " pod="openshift-ingress/e2e-ipp-b-97de36-openshift-default-77899d747c-cq4bc"
Nov 02 12:46:15.235658 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:15.227279    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b55da6b-89cb-46d6-a7a0-3a09e7d7d17e-istio-token\") pod \"e2e-ipp-b-97de36-openshift-default-77899d747c-cq4bc\" (UID: \"5b55da6b-89cb-46d6-a7a0-3a09e7d7d17e\") " pod="openshift-ingress/e2e-ipp-b-97de36-openshift-default-77899d747c-cq4bc"
Nov 02 12:46:15.235658 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:15.227941    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8wrg\" (UniqueName: \"kubernetes.io/projected/5b55da6b-89cb-46d6-a7a0-3a09e7d7d17e-kube-api-access-h8wrg\") pod \"e2e-ipp-b-97de36-openshift-default-77899d747c-cq4bc\" (UID: \"5b55da6b-89cb-46d6-a7a0-3a09e7d7d17e\") " pod="openshift-ingress/e2e-ipp-b-97de36-openshift-default-77899d747c-cq4bc"
Nov 02 12:46:15.317806 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:15.316781    4559 patch_prober.go:29] interesting pod/e2e-ipp-a-59060a-openshift-default-8666cdf765-hfz77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body=
Nov 02 12:46:15.317806 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:15.316846    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-59060a-openshift-default-8666cdf765-hfz77" podUID="9ed1a149-3a01-4af1-9b01-e059ff121bed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused"
Nov 02 12:46:15.435179 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:15.422434    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b55da6b-89cb-46d6-a7a0-3a09e7d7d17e-istio-podinfo\") pod \"e2e-ipp-b-97de36-openshift-default-77899d747c-cq4bc\" (UID: \"5b55da6b-89cb-46d6-a7a0-3a09e7d7d17e\") " pod="openshift-ingress/e2e-ipp-b-97de36-openshift-default-77899d747c-cq4bc"
Nov 02 12:46:15.450668 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:15.438930    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b55da6b-89cb-46d6-a7a0-3a09e7d7d17e-istio-envoy\") pod \"e2e-ipp-b-97de36-openshift-default-77899d747c-cq4bc\" (UID: \"5b55da6b-89cb-46d6-a7a0-3a09e7d7d17e\") " pod="openshift-ingress/e2e-ipp-b-97de36-openshift-default-77899d747c-cq4bc"
Nov 02 12:46:15.465744 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:15.464606    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-77cf70-86cf7b694c-45888" event={"ID":"bc18257f-4549-4cf1-9fe5-227496db7c80","Type":"ContainerStarted","Data":"103a1295884f274165fd4fe01692e934a624a19dfa3a9d5a3bcb4ea36116a3ea"}
Nov 02 12:46:15.574022 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:46:15.573973    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqcmj" containerName="extractor"
Nov 02 12:46:15.674542 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:15.674511    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-97de36-openshift-default-77899d747c-cq4bc"
Nov 02 12:46:15.844380 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:15.844338    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-77cf70-openshift-default-d8f7564b9-rftzg"
Nov 02 12:46:16.074888 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:16.074804    4559 patch_prober.go:29] interesting pod/e2e-ipp-b-77cf70-openshift-default-d8f7564b9-rftzg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body=
Nov 02 12:46:16.075058 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:16.074877    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-77cf70-openshift-default-d8f7564b9-rftzg" podUID="60be7b9e-07a0-4cc6-99d1-1278b6bad791" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused"
Nov 02 12:46:16.142407 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:46:16.141037    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b55da6b_89cb_46d6_a7a0_3a09e7d7d17e.slice/crio-c7f8060de2a53e3b8849723a538a4b650d1214f1c2774e5ab7fdf317842f501e WatchSource:0}: Error finding container c7f8060de2a53e3b8849723a538a4b650d1214f1c2774e5ab7fdf317842f501e: Status 404 returned error can't find the container with id c7f8060de2a53e3b8849723a538a4b650d1214f1c2774e5ab7fdf317842f501e
Nov 02 12:46:16.144013 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:16.143986    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-97de36-openshift-default-77899d747c-cq4bc"]
Nov 02 12:46:16.145244 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:16.145203    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:46:16.145327 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:16.145289    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:46:16.145377 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:16.145330    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:46:16.315903 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:16.310033    4559 patch_prober.go:29] interesting pod/e2e-ipp-a-59060a-openshift-default-8666cdf765-hfz77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body=
Nov 02 12:46:16.315903 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:16.310092    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-59060a-openshift-default-8666cdf765-hfz77" podUID="9ed1a149-3a01-4af1-9b01-e059ff121bed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused"
Nov 02 12:46:16.663627 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:16.663531    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-97de36-openshift-default-77899d747c-cq4bc" event={"ID":"5b55da6b-89cb-46d6-a7a0-3a09e7d7d17e","Type":"ContainerStarted","Data":"ef26164c25b358f05ae65310109400f62dc5283758f5989525cebc2709f8e391"}
Nov 02 12:46:16.663900 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:16.663874    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-97de36-openshift-default-77899d747c-cq4bc" event={"ID":"5b55da6b-89cb-46d6-a7a0-3a09e7d7d17e","Type":"ContainerStarted","Data":"c7f8060de2a53e3b8849723a538a4b650d1214f1c2774e5ab7fdf317842f501e"}
Nov 02 12:46:16.688836 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:16.688694    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-97de36-openshift-default-77899d747c-cq4bc"
Nov 02 12:46:17.110694 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:17.104792    4559 patch_prober.go:29] interesting pod/e2e-ipp-b-77cf70-openshift-default-d8f7564b9-rftzg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body=
Nov 02 12:46:17.110694 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:17.104857    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-77cf70-openshift-default-d8f7564b9-rftzg" podUID="60be7b9e-07a0-4cc6-99d1-1278b6bad791" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused"
Nov 02 12:46:17.304138 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:17.293133    4559 patch_prober.go:29] interesting pod/e2e-ipp-a-59060a-openshift-default-8666cdf765-hfz77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body=
Nov 02 12:46:17.304138 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:17.293199    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-59060a-openshift-default-8666cdf765-hfz77" podUID="9ed1a149-3a01-4af1-9b01-e059ff121bed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused"
Nov 02 12:46:17.682615 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:17.682585    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-97de36-openshift-default-77899d747c-cq4bc"
Nov 02 12:46:17.745107 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:17.745044    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-97de36-openshift-default-77899d747c-cq4bc" podStartSLOduration=3.745022764 podStartE2EDuration="3.745022764s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:46:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:46:16.740902996 +0000 UTC m=+1172.996302091" watchObservedRunningTime="2026-11-02 12:46:17.745022764 +0000 UTC m=+1174.000421860"
Nov 02 12:46:18.074605 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:18.074512    4559 patch_prober.go:29] interesting pod/e2e-ipp-b-77cf70-openshift-default-d8f7564b9-rftzg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body=
Nov 02 12:46:18.074605 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:18.074573    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-77cf70-openshift-default-d8f7564b9-rftzg" podUID="60be7b9e-07a0-4cc6-99d1-1278b6bad791" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused"
Nov 02 12:46:18.307665 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:18.305125    4559 patch_prober.go:29] interesting pod/e2e-ipp-a-59060a-openshift-default-8666cdf765-hfz77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body=
Nov 02 12:46:18.307665 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:18.305190    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-59060a-openshift-default-8666cdf765-hfz77" podUID="9ed1a149-3a01-4af1-9b01-e059ff121bed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused"
Nov 02 12:46:18.676663 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:18.669699    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-cd1386cb-openshift-default-5979d457c7-2msnf"]
Nov 02 12:46:18.676663 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:18.669980    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-cd1386cb-openshift-default-5979d457c7-2msnf" podUID="154e3aa3-c520-484e-be0a-74ea246b763b" containerName="istio-proxy" containerID="cri-o://4c3090e0a62f5efd133b12cd690636736ae43a8dc6e3a0ffd00a5b92159629b9" gracePeriod=30
Nov 02 12:46:18.811444 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:18.780143    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"]
Nov 02 12:46:19.025989 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:19.012348    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-97de36-openshift-default-77899d747c-cq4bc"
Nov 02 12:46:19.053531 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:19.053495    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-97de36-openshift-default-77899d747c-cq4bc"
Nov 02 12:46:19.097755 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:19.079380    4559 patch_prober.go:29] interesting pod/e2e-ipp-b-77cf70-openshift-default-d8f7564b9-rftzg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body=
Nov 02 12:46:19.108431 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:19.108344    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-77cf70-openshift-default-d8f7564b9-rftzg" podUID="60be7b9e-07a0-4cc6-99d1-1278b6bad791" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused"
Nov 02 12:46:19.291606 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:19.291520    4559 patch_prober.go:29] interesting pod/e2e-ipp-a-59060a-openshift-default-8666cdf765-hfz77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body=
Nov 02 12:46:19.291606 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:19.291583    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-59060a-openshift-default-8666cdf765-hfz77" podUID="9ed1a149-3a01-4af1-9b01-e059ff121bed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused"
Nov 02 12:46:19.292274 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:19.292192    4559 patch_prober.go:29] interesting pod/e2e-mt-cd1386cb-openshift-default-5979d457c7-2msnf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 12:46:19.292274 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:19.292237    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-cd1386cb-openshift-default-5979d457c7-2msnf" podUID="154e3aa3-c520-484e-be0a-74ea246b763b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 12:46:19.614253 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:19.613339    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-b2ec05-openshift-default-5f4c648f59-bldmv"]
Nov 02 12:46:19.624901 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:19.624876    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-b2ec05-openshift-default-5f4c648f59-bldmv"
Nov 02 12:46:19.644727 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:19.644629    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f74d2ef-c796-4c91-b763-7593c16cfe71-istiod-ca-cert\") pod \"e2e-api-a-b2ec05-openshift-default-5f4c648f59-bldmv\" (UID: \"0f74d2ef-c796-4c91-b763-7593c16cfe71\") " pod="openshift-ingress/e2e-api-a-b2ec05-openshift-default-5f4c648f59-bldmv"
Nov 02 12:46:19.644727 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:19.644713    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f74d2ef-c796-4c91-b763-7593c16cfe71-workload-socket\") pod \"e2e-api-a-b2ec05-openshift-default-5f4c648f59-bldmv\" (UID: \"0f74d2ef-c796-4c91-b763-7593c16cfe71\") " pod="openshift-ingress/e2e-api-a-b2ec05-openshift-default-5f4c648f59-bldmv"
Nov 02 12:46:19.644943 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:19.644829    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwv9f\" (UniqueName: \"kubernetes.io/projected/0f74d2ef-c796-4c91-b763-7593c16cfe71-kube-api-access-wwv9f\") pod \"e2e-api-a-b2ec05-openshift-default-5f4c648f59-bldmv\" (UID: \"0f74d2ef-c796-4c91-b763-7593c16cfe71\") " pod="openshift-ingress/e2e-api-a-b2ec05-openshift-default-5f4c648f59-bldmv"
Nov 02 12:46:19.644943 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:19.644888    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f74d2ef-c796-4c91-b763-7593c16cfe71-credential-socket\") pod \"e2e-api-a-b2ec05-openshift-default-5f4c648f59-bldmv\" (UID: \"0f74d2ef-c796-4c91-b763-7593c16cfe71\") " pod="openshift-ingress/e2e-api-a-b2ec05-openshift-default-5f4c648f59-bldmv"
Nov 02 12:46:19.645053 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:19.644961    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f74d2ef-c796-4c91-b763-7593c16cfe71-workload-certs\") pod \"e2e-api-a-b2ec05-openshift-default-5f4c648f59-bldmv\" (UID: \"0f74d2ef-c796-4c91-b763-7593c16cfe71\") " pod="openshift-ingress/e2e-api-a-b2ec05-openshift-default-5f4c648f59-bldmv"
Nov 02 12:46:19.645053 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:19.645003    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f74d2ef-c796-4c91-b763-7593c16cfe71-istio-token\") pod \"e2e-api-a-b2ec05-openshift-default-5f4c648f59-bldmv\" (UID: \"0f74d2ef-c796-4c91-b763-7593c16cfe71\") " pod="openshift-ingress/e2e-api-a-b2ec05-openshift-default-5f4c648f59-bldmv"
Nov 02 12:46:19.645149 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:19.645055    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f74d2ef-c796-4c91-b763-7593c16cfe71-istio-data\") pod \"e2e-api-a-b2ec05-openshift-default-5f4c648f59-bldmv\" (UID: \"0f74d2ef-c796-4c91-b763-7593c16cfe71\") " pod="openshift-ingress/e2e-api-a-b2ec05-openshift-default-5f4c648f59-bldmv"
Nov 02 12:46:19.645149 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:19.645111    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f74d2ef-c796-4c91-b763-7593c16cfe71-istio-podinfo\") pod \"e2e-api-a-b2ec05-openshift-default-5f4c648f59-bldmv\" (UID: \"0f74d2ef-c796-4c91-b763-7593c16cfe71\") " pod="openshift-ingress/e2e-api-a-b2ec05-openshift-default-5f4c648f59-bldmv"
Nov 02 12:46:19.645318 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:19.645203    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f74d2ef-c796-4c91-b763-7593c16cfe71-istio-envoy\") pod \"e2e-api-a-b2ec05-openshift-default-5f4c648f59-bldmv\" (UID: \"0f74d2ef-c796-4c91-b763-7593c16cfe71\") " pod="openshift-ingress/e2e-api-a-b2ec05-openshift-default-5f4c648f59-bldmv"
Nov 02 12:46:19.655004 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:19.654845    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-b2ec05-openshift-default-dockercfg-58smv\""
Nov 02 12:46:19.677196 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:19.677166    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-b2ec05-openshift-default-5f4c648f59-bldmv"]
Nov 02 12:46:19.813664 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:19.799831    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f74d2ef-c796-4c91-b763-7593c16cfe71-istiod-ca-cert\") pod \"e2e-api-a-b2ec05-openshift-default-5f4c648f59-bldmv\" (UID: \"0f74d2ef-c796-4c91-b763-7593c16cfe71\") " pod="openshift-ingress/e2e-api-a-b2ec05-openshift-default-5f4c648f59-bldmv"
Nov 02 12:46:19.813664 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:19.799910    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f74d2ef-c796-4c91-b763-7593c16cfe71-workload-socket\") pod \"e2e-api-a-b2ec05-openshift-default-5f4c648f59-bldmv\" (UID: \"0f74d2ef-c796-4c91-b763-7593c16cfe71\") " pod="openshift-ingress/e2e-api-a-b2ec05-openshift-default-5f4c648f59-bldmv"
Nov 02 12:46:19.813664 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:19.800035    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwv9f\" (UniqueName: \"kubernetes.io/projected/0f74d2ef-c796-4c91-b763-7593c16cfe71-kube-api-access-wwv9f\") pod \"e2e-api-a-b2ec05-openshift-default-5f4c648f59-bldmv\" (UID: \"0f74d2ef-c796-4c91-b763-7593c16cfe71\") " pod="openshift-ingress/e2e-api-a-b2ec05-openshift-default-5f4c648f59-bldmv"
Nov 02 12:46:19.813664 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:19.800092    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f74d2ef-c796-4c91-b763-7593c16cfe71-credential-socket\") pod \"e2e-api-a-b2ec05-openshift-default-5f4c648f59-bldmv\" (UID: \"0f74d2ef-c796-4c91-b763-7593c16cfe71\") " pod="openshift-ingress/e2e-api-a-b2ec05-openshift-default-5f4c648f59-bldmv"
Nov 02 12:46:19.813664 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:19.800162    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f74d2ef-c796-4c91-b763-7593c16cfe71-workload-certs\") pod \"e2e-api-a-b2ec05-openshift-default-5f4c648f59-bldmv\" (UID: \"0f74d2ef-c796-4c91-b763-7593c16cfe71\") " pod="openshift-ingress/e2e-api-a-b2ec05-openshift-default-5f4c648f59-bldmv"
Nov 02 12:46:19.813664 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:19.800202    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f74d2ef-c796-4c91-b763-7593c16cfe71-istio-token\") pod \"e2e-api-a-b2ec05-openshift-default-5f4c648f59-bldmv\" (UID: \"0f74d2ef-c796-4c91-b763-7593c16cfe71\") " pod="openshift-ingress/e2e-api-a-b2ec05-openshift-default-5f4c648f59-bldmv"
Nov 02 12:46:19.813664 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:19.800258    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f74d2ef-c796-4c91-b763-7593c16cfe71-istio-data\") pod \"e2e-api-a-b2ec05-openshift-default-5f4c648f59-bldmv\" (UID: \"0f74d2ef-c796-4c91-b763-7593c16cfe71\") " pod="openshift-ingress/e2e-api-a-b2ec05-openshift-default-5f4c648f59-bldmv"
Nov 02 12:46:19.813664 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:19.800317    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f74d2ef-c796-4c91-b763-7593c16cfe71-istio-podinfo\") pod \"e2e-api-a-b2ec05-openshift-default-5f4c648f59-bldmv\" (UID: \"0f74d2ef-c796-4c91-b763-7593c16cfe71\") " pod="openshift-ingress/e2e-api-a-b2ec05-openshift-default-5f4c648f59-bldmv"
Nov 02 12:46:19.813664 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:19.800410    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f74d2ef-c796-4c91-b763-7593c16cfe71-istio-envoy\") pod \"e2e-api-a-b2ec05-openshift-default-5f4c648f59-bldmv\" (UID: \"0f74d2ef-c796-4c91-b763-7593c16cfe71\") " pod="openshift-ingress/e2e-api-a-b2ec05-openshift-default-5f4c648f59-bldmv"
Nov 02 12:46:19.869751 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:19.855907    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f74d2ef-c796-4c91-b763-7593c16cfe71-istiod-ca-cert\") pod \"e2e-api-a-b2ec05-openshift-default-5f4c648f59-bldmv\" (UID: \"0f74d2ef-c796-4c91-b763-7593c16cfe71\") " pod="openshift-ingress/e2e-api-a-b2ec05-openshift-default-5f4c648f59-bldmv"
Nov 02 12:46:19.869751 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:19.856191    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f74d2ef-c796-4c91-b763-7593c16cfe71-istio-envoy\") pod \"e2e-api-a-b2ec05-openshift-default-5f4c648f59-bldmv\" (UID: \"0f74d2ef-c796-4c91-b763-7593c16cfe71\") " pod="openshift-ingress/e2e-api-a-b2ec05-openshift-default-5f4c648f59-bldmv"
Nov 02 12:46:19.895662 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:19.872172    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f74d2ef-c796-4c91-b763-7593c16cfe71-credential-socket\") pod \"e2e-api-a-b2ec05-openshift-default-5f4c648f59-bldmv\" (UID: \"0f74d2ef-c796-4c91-b763-7593c16cfe71\") " pod="openshift-ingress/e2e-api-a-b2ec05-openshift-default-5f4c648f59-bldmv"
Nov 02 12:46:19.895662 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:19.874004    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f74d2ef-c796-4c91-b763-7593c16cfe71-workload-socket\") pod \"e2e-api-a-b2ec05-openshift-default-5f4c648f59-bldmv\" (UID: \"0f74d2ef-c796-4c91-b763-7593c16cfe71\") " pod="openshift-ingress/e2e-api-a-b2ec05-openshift-default-5f4c648f59-bldmv"
Nov 02 12:46:19.895662 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:19.884974    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f74d2ef-c796-4c91-b763-7593c16cfe71-istio-data\") pod \"e2e-api-a-b2ec05-openshift-default-5f4c648f59-bldmv\" (UID: \"0f74d2ef-c796-4c91-b763-7593c16cfe71\") " pod="openshift-ingress/e2e-api-a-b2ec05-openshift-default-5f4c648f59-bldmv"
Nov 02 12:46:19.895662 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:19.895066    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f74d2ef-c796-4c91-b763-7593c16cfe71-workload-certs\") pod \"e2e-api-a-b2ec05-openshift-default-5f4c648f59-bldmv\" (UID: \"0f74d2ef-c796-4c91-b763-7593c16cfe71\") " pod="openshift-ingress/e2e-api-a-b2ec05-openshift-default-5f4c648f59-bldmv"
Nov 02 12:46:19.980025 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:19.979986    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwv9f\" (UniqueName: \"kubernetes.io/projected/0f74d2ef-c796-4c91-b763-7593c16cfe71-kube-api-access-wwv9f\") pod \"e2e-api-a-b2ec05-openshift-default-5f4c648f59-bldmv\" (UID: \"0f74d2ef-c796-4c91-b763-7593c16cfe71\") " pod="openshift-ingress/e2e-api-a-b2ec05-openshift-default-5f4c648f59-bldmv"
Nov 02 12:46:20.012255 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:19.993333    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f74d2ef-c796-4c91-b763-7593c16cfe71-istio-token\") pod \"e2e-api-a-b2ec05-openshift-default-5f4c648f59-bldmv\" (UID: \"0f74d2ef-c796-4c91-b763-7593c16cfe71\") " pod="openshift-ingress/e2e-api-a-b2ec05-openshift-default-5f4c648f59-bldmv"
Nov 02 12:46:20.012255 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:20.005239    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f74d2ef-c796-4c91-b763-7593c16cfe71-istio-podinfo\") pod \"e2e-api-a-b2ec05-openshift-default-5f4c648f59-bldmv\" (UID: \"0f74d2ef-c796-4c91-b763-7593c16cfe71\") " pod="openshift-ingress/e2e-api-a-b2ec05-openshift-default-5f4c648f59-bldmv"
Nov 02 12:46:20.146726 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:20.146609    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-77cf70-openshift-default-d8f7564b9-rftzg"
Nov 02 12:46:20.154632 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:20.154571    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-77cf70-openshift-default-d8f7564b9-rftzg"
Nov 02 12:46:20.267532 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:20.267489    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-b2ec05-openshift-default-5f4c648f59-bldmv"
Nov 02 12:46:20.297271 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:20.297242    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-59060a-openshift-default-8666cdf765-hfz77"
Nov 02 12:46:20.298940 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:20.298795    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-59060a-openshift-default-8666cdf765-hfz77"
Nov 02 12:46:20.309340 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:20.309267    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-59060a-openshift-default-8666cdf765-hfz77"
Nov 02 12:46:20.664199 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:20.661218    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:46:20.664199 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:20.661414    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:46:20.664199 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:20.661464    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-b2ec05-openshift-default-5f4c648f59-bldmv"]
Nov 02 12:46:20.664199 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:20.661506    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:46:21.115423 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:21.115382    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-77cf70-8686479b7b-mcd6f" event={"ID":"6ee04a57-e5e9-48fa-a5a8-c89bcf917616","Type":"ContainerStarted","Data":"dd9e4200d5332ef941b46088cb0070d1c6d40ee4933dd60bedd4e88737c2e1a8"}
Nov 02 12:46:21.116789 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:21.116762    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-77cf70-8686479b7b-mcd6f"
Nov 02 12:46:21.118926 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:21.118900    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-77cf70-86cf7b694c-45888" event={"ID":"bc18257f-4549-4cf1-9fe5-227496db7c80","Type":"ContainerStarted","Data":"a1d8ea82c378d3fbb5c3c1ca0a57a8200213b14ba9a1d88cf8b784efe0cd4252"}
Nov 02 12:46:21.119698 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:21.119677    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-77cf70-86cf7b694c-45888"
Nov 02 12:46:21.123419 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:21.123396    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b2ec05-openshift-default-5f4c648f59-bldmv" event={"ID":"0f74d2ef-c796-4c91-b763-7593c16cfe71","Type":"ContainerStarted","Data":"c0efdec28f98e63e1d763aae8b0b3dfd8090e3c17e7641565e99d42b96efcd88"}
Nov 02 12:46:21.123596 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:21.123577    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b2ec05-openshift-default-5f4c648f59-bldmv" event={"ID":"0f74d2ef-c796-4c91-b763-7593c16cfe71","Type":"ContainerStarted","Data":"bca71ad2adc636aeec68fd95e82bb351112d33b4381596fe9e51c64698c99e98"}
Nov 02 12:46:21.163136 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:21.163039    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-77cf70-8686479b7b-mcd6f" podStartSLOduration=2.287896237 podStartE2EDuration="9.163021215s" totalImagesPullingTime="6.875124978s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:46:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:46:21.162379779 +0000 UTC m=+1177.417778879" watchObservedRunningTime="2026-11-02 12:46:21.163021215 +0000 UTC m=+1177.418420312"
Nov 02 12:46:21.207248 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:21.207192    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-77cf70-86cf7b694c-45888" podStartSLOduration=2.618230031 podStartE2EDuration="9.20717811s" totalImagesPullingTime="6.588948079s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:46:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:46:21.204141322 +0000 UTC m=+1177.459540416" watchObservedRunningTime="2026-11-02 12:46:21.20717811 +0000 UTC m=+1177.462577204"
Nov 02 12:46:21.252095 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:21.252032    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-b2ec05-openshift-default-5f4c648f59-bldmv" podStartSLOduration=2.252017236 podStartE2EDuration="2.252017236s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:46:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:46:21.250040702 +0000 UTC m=+1177.505439808" watchObservedRunningTime="2026-11-02 12:46:21.252017236 +0000 UTC m=+1177.507416405"
Nov 02 12:46:21.268690 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:21.268620    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-b2ec05-openshift-default-5f4c648f59-bldmv"
Nov 02 12:46:21.274133 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:21.274108    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-b2ec05-openshift-default-5f4c648f59-bldmv"
Nov 02 12:46:21.336078 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:21.336043    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-97de36-76c57bb9ff-9dv2z"]
Nov 02 12:46:21.341944 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:21.341917    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-97de36-76c57bb9ff-9dv2z"
Nov 02 12:46:21.361035 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:21.361010    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-97de36-dockercfg-2ddzm\""
Nov 02 12:46:21.363719 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:21.363692    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-97de36-76c57bb9ff-9dv2z"]
Nov 02 12:46:21.395171 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:21.395137    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-97de36\""
Nov 02 12:46:21.522669 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:21.499961    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ll7nf\" (UniqueName: \"kubernetes.io/projected/7affa7ad-9c8c-4896-8ad8-2030fdc323fd-kube-api-access-ll7nf\") pod \"payload-pre-processing-e2e-ipp-b-97de36-76c57bb9ff-9dv2z\" (UID: \"7affa7ad-9c8c-4896-8ad8-2030fdc323fd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-97de36-76c57bb9ff-9dv2z"
Nov 02 12:46:21.589661 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:21.582116    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7affa7ad-9c8c-4896-8ad8-2030fdc323fd-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-97de36-76c57bb9ff-9dv2z\" (UID: \"7affa7ad-9c8c-4896-8ad8-2030fdc323fd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-97de36-76c57bb9ff-9dv2z"
Nov 02 12:46:21.705661 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:21.697576    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ll7nf\" (UniqueName: \"kubernetes.io/projected/7affa7ad-9c8c-4896-8ad8-2030fdc323fd-kube-api-access-ll7nf\") pod \"payload-pre-processing-e2e-ipp-b-97de36-76c57bb9ff-9dv2z\" (UID: \"7affa7ad-9c8c-4896-8ad8-2030fdc323fd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-97de36-76c57bb9ff-9dv2z"
Nov 02 12:46:21.706859 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:21.706822    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7affa7ad-9c8c-4896-8ad8-2030fdc323fd-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-97de36-76c57bb9ff-9dv2z\" (UID: \"7affa7ad-9c8c-4896-8ad8-2030fdc323fd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-97de36-76c57bb9ff-9dv2z"
Nov 02 12:46:21.722600 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:21.714730    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7affa7ad-9c8c-4896-8ad8-2030fdc323fd-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-97de36-76c57bb9ff-9dv2z\" (UID: \"7affa7ad-9c8c-4896-8ad8-2030fdc323fd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-97de36-76c57bb9ff-9dv2z"
Nov 02 12:46:21.826707 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:21.813587    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ll7nf\" (UniqueName: \"kubernetes.io/projected/7affa7ad-9c8c-4896-8ad8-2030fdc323fd-kube-api-access-ll7nf\") pod \"payload-pre-processing-e2e-ipp-b-97de36-76c57bb9ff-9dv2z\" (UID: \"7affa7ad-9c8c-4896-8ad8-2030fdc323fd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-97de36-76c57bb9ff-9dv2z"
Nov 02 12:46:21.962845 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:21.962809    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-97de36-76c57bb9ff-9dv2z"
Nov 02 12:46:22.132121 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:22.132032    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-b2ec05-openshift-default-5f4c648f59-bldmv"
Nov 02 12:46:22.132923 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:22.132901    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-b2ec05-openshift-default-5f4c648f59-bldmv"
Nov 02 12:46:22.280223 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:46:22.280196    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7affa7ad_9c8c_4896_8ad8_2030fdc323fd.slice/crio-e0758f725a5d37a91c2b5d148b78d06aecb7c933994ce134788db7b7baeedf7e WatchSource:0}: Error finding container e0758f725a5d37a91c2b5d148b78d06aecb7c933994ce134788db7b7baeedf7e: Status 404 returned error can't find the container with id e0758f725a5d37a91c2b5d148b78d06aecb7c933994ce134788db7b7baeedf7e
Nov 02 12:46:22.282471 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:22.282447    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-97de36-76c57bb9ff-9dv2z"]
Nov 02 12:46:23.150082 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:23.150048    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-97de36-76c57bb9ff-9dv2z" event={"ID":"7affa7ad-9c8c-4896-8ad8-2030fdc323fd","Type":"ContainerStarted","Data":"8ceae3a58addfbc12560a31a4a65029cf1e30cc051f3a1c187aec1947d38b3bb"}
Nov 02 12:46:23.150805 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:23.150766    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-97de36-76c57bb9ff-9dv2z" event={"ID":"7affa7ad-9c8c-4896-8ad8-2030fdc323fd","Type":"ContainerStarted","Data":"e0758f725a5d37a91c2b5d148b78d06aecb7c933994ce134788db7b7baeedf7e"}
Nov 02 12:46:23.152359 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:23.152336    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-97de36-76c57bb9ff-9dv2z"
Nov 02 12:46:23.227922 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:23.227859    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-97de36-76c57bb9ff-9dv2z" podStartSLOduration=2.2278383489999998 podStartE2EDuration="2.227838349s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:46:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:46:23.206934214 +0000 UTC m=+1179.462333322" watchObservedRunningTime="2026-11-02 12:46:23.227838349 +0000 UTC m=+1179.483237445"
Nov 02 12:46:24.199343 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:46:24.199315    4559 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod154e3aa3_c520_484e_be0a_74ea246b763b.slice/crio-conmon-4c3090e0a62f5efd133b12cd690636736ae43a8dc6e3a0ffd00a5b92159629b9.scope\": RecentStats: unable to find data in memory cache]"
Nov 02 12:46:24.438497 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:24.438437    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-cd1386cb-openshift-default-5979d457c7-2msnf"
Nov 02 12:46:24.510406 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:24.510376    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/154e3aa3-c520-484e-be0a-74ea246b763b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/154e3aa3-c520-484e-be0a-74ea246b763b-istio-data\") pod \"154e3aa3-c520-484e-be0a-74ea246b763b\" (UID: \"154e3aa3-c520-484e-be0a-74ea246b763b\") "
Nov 02 12:46:24.510698 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:24.510527    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/154e3aa3-c520-484e-be0a-74ea246b763b-istio-token\" (UniqueName: \"kubernetes.io/projected/154e3aa3-c520-484e-be0a-74ea246b763b-istio-token\") pod \"154e3aa3-c520-484e-be0a-74ea246b763b\" (UID: \"154e3aa3-c520-484e-be0a-74ea246b763b\") "
Nov 02 12:46:24.510698 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:24.510678    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/154e3aa3-c520-484e-be0a-74ea246b763b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/154e3aa3-c520-484e-be0a-74ea246b763b-istiod-ca-cert\") pod \"154e3aa3-c520-484e-be0a-74ea246b763b\" (UID: \"154e3aa3-c520-484e-be0a-74ea246b763b\") "
Nov 02 12:46:24.511300 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:24.510723    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/154e3aa3-c520-484e-be0a-74ea246b763b-kube-api-access-4fjgx\" (UniqueName: \"kubernetes.io/projected/154e3aa3-c520-484e-be0a-74ea246b763b-kube-api-access-4fjgx\") pod \"154e3aa3-c520-484e-be0a-74ea246b763b\" (UID: \"154e3aa3-c520-484e-be0a-74ea246b763b\") "
Nov 02 12:46:24.511300 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:24.510851    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/154e3aa3-c520-484e-be0a-74ea246b763b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/154e3aa3-c520-484e-be0a-74ea246b763b-credential-socket\") pod \"154e3aa3-c520-484e-be0a-74ea246b763b\" (UID: \"154e3aa3-c520-484e-be0a-74ea246b763b\") "
Nov 02 12:46:24.511300 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:24.510887    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/154e3aa3-c520-484e-be0a-74ea246b763b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/154e3aa3-c520-484e-be0a-74ea246b763b-istio-envoy\") pod \"154e3aa3-c520-484e-be0a-74ea246b763b\" (UID: \"154e3aa3-c520-484e-be0a-74ea246b763b\") "
Nov 02 12:46:24.511300 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:24.510922    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/154e3aa3-c520-484e-be0a-74ea246b763b-istio-data" pod "154e3aa3-c520-484e-be0a-74ea246b763b" (UID: "154e3aa3-c520-484e-be0a-74ea246b763b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:46:24.511300 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:24.510936    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/154e3aa3-c520-484e-be0a-74ea246b763b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/154e3aa3-c520-484e-be0a-74ea246b763b-workload-certs\") pod \"154e3aa3-c520-484e-be0a-74ea246b763b\" (UID: \"154e3aa3-c520-484e-be0a-74ea246b763b\") "
Nov 02 12:46:24.511300 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:24.511008    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/154e3aa3-c520-484e-be0a-74ea246b763b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/154e3aa3-c520-484e-be0a-74ea246b763b-istio-podinfo\") pod \"154e3aa3-c520-484e-be0a-74ea246b763b\" (UID: \"154e3aa3-c520-484e-be0a-74ea246b763b\") "
Nov 02 12:46:24.511300 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:24.511040    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/154e3aa3-c520-484e-be0a-74ea246b763b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/154e3aa3-c520-484e-be0a-74ea246b763b-workload-socket\") pod \"154e3aa3-c520-484e-be0a-74ea246b763b\" (UID: \"154e3aa3-c520-484e-be0a-74ea246b763b\") "
Nov 02 12:46:24.511802 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:24.511696    4559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/154e3aa3-c520-484e-be0a-74ea246b763b-istio-data\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:46:24.512225 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:24.512203    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/154e3aa3-c520-484e-be0a-74ea246b763b-istiod-ca-cert" pod "154e3aa3-c520-484e-be0a-74ea246b763b" (UID: "154e3aa3-c520-484e-be0a-74ea246b763b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:46:24.512575 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:24.512553    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/154e3aa3-c520-484e-be0a-74ea246b763b-workload-certs" pod "154e3aa3-c520-484e-be0a-74ea246b763b" (UID: "154e3aa3-c520-484e-be0a-74ea246b763b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:46:24.512990 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:24.512962    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/154e3aa3-c520-484e-be0a-74ea246b763b-credential-socket" pod "154e3aa3-c520-484e-be0a-74ea246b763b" (UID: "154e3aa3-c520-484e-be0a-74ea246b763b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:46:24.513829 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:24.513799    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/154e3aa3-c520-484e-be0a-74ea246b763b-workload-socket" pod "154e3aa3-c520-484e-be0a-74ea246b763b" (UID: "154e3aa3-c520-484e-be0a-74ea246b763b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:46:24.514423 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:24.514398    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/154e3aa3-c520-484e-be0a-74ea246b763b-istio-envoy" pod "154e3aa3-c520-484e-be0a-74ea246b763b" (UID: "154e3aa3-c520-484e-be0a-74ea246b763b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:46:24.514507 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:24.514436    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/154e3aa3-c520-484e-be0a-74ea246b763b-istio-token" pod "154e3aa3-c520-484e-be0a-74ea246b763b" (UID: "154e3aa3-c520-484e-be0a-74ea246b763b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:46:24.514943 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:24.514918    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/154e3aa3-c520-484e-be0a-74ea246b763b-istio-podinfo" pod "154e3aa3-c520-484e-be0a-74ea246b763b" (UID: "154e3aa3-c520-484e-be0a-74ea246b763b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 12:46:24.516290 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:24.516262    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/154e3aa3-c520-484e-be0a-74ea246b763b-kube-api-access-4fjgx" pod "154e3aa3-c520-484e-be0a-74ea246b763b" (UID: "154e3aa3-c520-484e-be0a-74ea246b763b"). InnerVolumeSpecName "kube-api-access-4fjgx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:46:24.612787 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:24.612756    4559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/154e3aa3-c520-484e-be0a-74ea246b763b-istio-token\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:46:24.612787 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:24.612781    4559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/154e3aa3-c520-484e-be0a-74ea246b763b-istiod-ca-cert\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:46:24.612787 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:24.612791    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4fjgx\" (UniqueName: \"kubernetes.io/projected/154e3aa3-c520-484e-be0a-74ea246b763b-kube-api-access-4fjgx\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:46:24.613014 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:24.612801    4559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/154e3aa3-c520-484e-be0a-74ea246b763b-credential-socket\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:46:24.613014 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:24.612809    4559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/154e3aa3-c520-484e-be0a-74ea246b763b-istio-envoy\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:46:24.613014 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:24.612818    4559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/154e3aa3-c520-484e-be0a-74ea246b763b-workload-certs\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:46:24.613014 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:24.612826    4559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/154e3aa3-c520-484e-be0a-74ea246b763b-istio-podinfo\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:46:24.613014 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:24.612833    4559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/154e3aa3-c520-484e-be0a-74ea246b763b-workload-socket\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:46:25.211665 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:25.210350    4559 generic.go:356] "Generic (PLEG): container finished" podID="154e3aa3-c520-484e-be0a-74ea246b763b" containerID="4c3090e0a62f5efd133b12cd690636736ae43a8dc6e3a0ffd00a5b92159629b9" exitCode=0
Nov 02 12:46:25.211665 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:25.211604    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-cd1386cb-openshift-default-5979d457c7-2msnf"
Nov 02 12:46:25.218666 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:25.215962    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-cd1386cb-openshift-default-5979d457c7-2msnf" event={"ID":"154e3aa3-c520-484e-be0a-74ea246b763b","Type":"ContainerDied","Data":"4c3090e0a62f5efd133b12cd690636736ae43a8dc6e3a0ffd00a5b92159629b9"}
Nov 02 12:46:25.218666 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:25.216036    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-cd1386cb-openshift-default-5979d457c7-2msnf" event={"ID":"154e3aa3-c520-484e-be0a-74ea246b763b","Type":"ContainerDied","Data":"ea7df6d548f1da643bce894a5d550aa62179997b66352f99e7408c643c700c42"}
Nov 02 12:46:25.218666 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:25.216071    4559 scope.go:122] "RemoveContainer" containerID="4c3090e0a62f5efd133b12cd690636736ae43a8dc6e3a0ffd00a5b92159629b9"
Nov 02 12:46:25.257251 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:25.257232    4559 scope.go:122] "RemoveContainer" containerID="4c3090e0a62f5efd133b12cd690636736ae43a8dc6e3a0ffd00a5b92159629b9"
Nov 02 12:46:25.257471 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:46:25.257448    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c3090e0a62f5efd133b12cd690636736ae43a8dc6e3a0ffd00a5b92159629b9\": container with ID starting with 4c3090e0a62f5efd133b12cd690636736ae43a8dc6e3a0ffd00a5b92159629b9 not found: ID does not exist" containerID="4c3090e0a62f5efd133b12cd690636736ae43a8dc6e3a0ffd00a5b92159629b9"
Nov 02 12:46:25.257546 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:25.257478    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c3090e0a62f5efd133b12cd690636736ae43a8dc6e3a0ffd00a5b92159629b9"} err="failed to get container status \"4c3090e0a62f5efd133b12cd690636736ae43a8dc6e3a0ffd00a5b92159629b9\": rpc error: code = NotFound desc = could not find container \"4c3090e0a62f5efd133b12cd690636736ae43a8dc6e3a0ffd00a5b92159629b9\": container with ID starting with 4c3090e0a62f5efd133b12cd690636736ae43a8dc6e3a0ffd00a5b92159629b9 not found: ID does not exist"
Nov 02 12:46:25.295566 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:25.295536    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-cd1386cb-openshift-default-5979d457c7-2msnf"]
Nov 02 12:46:25.308767 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:25.308710    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-cd1386cb-openshift-default-5979d457c7-2msnf"]
Nov 02 12:46:25.491092 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:25.491017    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-b2ec05-57ff8d5965-9k4gl"]
Nov 02 12:46:25.491718 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:25.491680    4559 cpu_manager.go:414] "containerMap: removing container" podUID="154e3aa3-c520-484e-be0a-74ea246b763b" containerName="istio-proxy"
Nov 02 12:46:25.491718 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:25.491696    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="154e3aa3-c520-484e-be0a-74ea246b763b" containerName="istio-proxy"
Nov 02 12:46:25.491821 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:25.491803    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="154e3aa3-c520-484e-be0a-74ea246b763b" containerName="istio-proxy"
Nov 02 12:46:25.496799 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:25.496777    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-b2ec05-57ff8d5965-9k4gl"
Nov 02 12:46:25.514970 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:25.514940    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-b2ec05\""
Nov 02 12:46:25.515245 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:25.515215    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-b2ec05-57ff8d5965-9k4gl"]
Nov 02 12:46:25.652662 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:25.651667    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tsmg\" (UniqueName: \"kubernetes.io/projected/d3f2ed27-59d4-4eba-933a-bea46a18d385-kube-api-access-8tsmg\") pod \"payload-processing-e2e-api-a-b2ec05-57ff8d5965-9k4gl\" (UID: \"d3f2ed27-59d4-4eba-933a-bea46a18d385\") " pod="openshift-ingress/payload-processing-e2e-api-a-b2ec05-57ff8d5965-9k4gl"
Nov 02 12:46:25.652662 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:25.651929    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3f2ed27-59d4-4eba-933a-bea46a18d385-plugins-config-volume\") pod \"payload-processing-e2e-api-a-b2ec05-57ff8d5965-9k4gl\" (UID: \"d3f2ed27-59d4-4eba-933a-bea46a18d385\") " pod="openshift-ingress/payload-processing-e2e-api-a-b2ec05-57ff8d5965-9k4gl"
Nov 02 12:46:25.773707 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:25.770126    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tsmg\" (UniqueName: \"kubernetes.io/projected/d3f2ed27-59d4-4eba-933a-bea46a18d385-kube-api-access-8tsmg\") pod \"payload-processing-e2e-api-a-b2ec05-57ff8d5965-9k4gl\" (UID: \"d3f2ed27-59d4-4eba-933a-bea46a18d385\") " pod="openshift-ingress/payload-processing-e2e-api-a-b2ec05-57ff8d5965-9k4gl"
Nov 02 12:46:25.773707 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:25.770363    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3f2ed27-59d4-4eba-933a-bea46a18d385-plugins-config-volume\") pod \"payload-processing-e2e-api-a-b2ec05-57ff8d5965-9k4gl\" (UID: \"d3f2ed27-59d4-4eba-933a-bea46a18d385\") " pod="openshift-ingress/payload-processing-e2e-api-a-b2ec05-57ff8d5965-9k4gl"
Nov 02 12:46:25.773707 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:25.771443    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3f2ed27-59d4-4eba-933a-bea46a18d385-plugins-config-volume\") pod \"payload-processing-e2e-api-a-b2ec05-57ff8d5965-9k4gl\" (UID: \"d3f2ed27-59d4-4eba-933a-bea46a18d385\") " pod="openshift-ingress/payload-processing-e2e-api-a-b2ec05-57ff8d5965-9k4gl"
Nov 02 12:46:25.842663 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:25.842404    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tsmg\" (UniqueName: \"kubernetes.io/projected/d3f2ed27-59d4-4eba-933a-bea46a18d385-kube-api-access-8tsmg\") pod \"payload-processing-e2e-api-a-b2ec05-57ff8d5965-9k4gl\" (UID: \"d3f2ed27-59d4-4eba-933a-bea46a18d385\") " pod="openshift-ingress/payload-processing-e2e-api-a-b2ec05-57ff8d5965-9k4gl"
Nov 02 12:46:25.911398 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:25.911349    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="154e3aa3-c520-484e-be0a-74ea246b763b" path="/var/lib/kubelet/pods/154e3aa3-c520-484e-be0a-74ea246b763b/volumes"
Nov 02 12:46:26.111727 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:26.111635    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-b2ec05-57ff8d5965-9k4gl"
Nov 02 12:46:26.420407 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:46:26.420379    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3f2ed27_59d4_4eba_933a_bea46a18d385.slice/crio-acacf2ee9417de7eb9635d20d10f94a6d7eec49b9bae14202415cdc221781a60 WatchSource:0}: Error finding container acacf2ee9417de7eb9635d20d10f94a6d7eec49b9bae14202415cdc221781a60: Status 404 returned error can't find the container with id acacf2ee9417de7eb9635d20d10f94a6d7eec49b9bae14202415cdc221781a60
Nov 02 12:46:26.423262 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:26.423206    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-b2ec05-57ff8d5965-9k4gl"]
Nov 02 12:46:27.228575 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:27.228540    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-b2ec05-57ff8d5965-9k4gl" event={"ID":"d3f2ed27-59d4-4eba-933a-bea46a18d385","Type":"ContainerStarted","Data":"e16362647b3e3af3aa3f13e13a6de2dd8b285ccb9ee155661df99d2a919d1d3a"}
Nov 02 12:46:27.228788 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:27.228598    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-b2ec05-57ff8d5965-9k4gl" event={"ID":"d3f2ed27-59d4-4eba-933a-bea46a18d385","Type":"ContainerStarted","Data":"acacf2ee9417de7eb9635d20d10f94a6d7eec49b9bae14202415cdc221781a60"}
Nov 02 12:46:27.228788 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:27.228622    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-b2ec05-57ff8d5965-9k4gl"
Nov 02 12:46:27.283722 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:27.283653    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-b2ec05-57ff8d5965-9k4gl" podStartSLOduration=2.283622216 podStartE2EDuration="2.283622216s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:46:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:46:27.271259536 +0000 UTC m=+1183.526658623" watchObservedRunningTime="2026-11-02 12:46:27.283622216 +0000 UTC m=+1183.539021321"
Nov 02 12:46:33.154103 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:33.154070    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-77cf70-86cf7b694c-45888"
Nov 02 12:46:33.154982 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:33.154961    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-77cf70-8686479b7b-mcd6f"
Nov 02 12:46:35.108523 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:35.108492    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-729108-5c8649cbff-9xhdl"]
Nov 02 12:46:35.114136 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:35.114107    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-729108-5c8649cbff-9xhdl"
Nov 02 12:46:35.133182 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:35.133153    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-729108-5c8649cbff-9xhdl"]
Nov 02 12:46:35.140223 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:35.140165    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-729108\""
Nov 02 12:46:35.156490 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:35.156462    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-729108-6df46f4ff9-qw5p5"]
Nov 02 12:46:35.160969 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:35.160951    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-729108-6df46f4ff9-qw5p5"
Nov 02 12:46:35.178951 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:35.178927    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-729108-6df46f4ff9-qw5p5"]
Nov 02 12:46:35.179179 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:35.179159    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-729108-dockercfg-8hbq4\""
Nov 02 12:46:35.205249 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:35.205218    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ed81343-a77a-4401-bc65-21c804cfbead-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-729108-5c8649cbff-9xhdl\" (UID: \"0ed81343-a77a-4401-bc65-21c804cfbead\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-729108-5c8649cbff-9xhdl"
Nov 02 12:46:35.205381 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:35.205328    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jf2c\" (UniqueName: \"kubernetes.io/projected/b98cdf4a-b4e0-46f3-b791-5cc63707a366-kube-api-access-8jf2c\") pod \"payload-pre-processing-e2e-ipp-a-729108-6df46f4ff9-qw5p5\" (UID: \"b98cdf4a-b4e0-46f3-b791-5cc63707a366\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-729108-6df46f4ff9-qw5p5"
Nov 02 12:46:35.205428 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:35.205398    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j85d2\" (UniqueName: \"kubernetes.io/projected/0ed81343-a77a-4401-bc65-21c804cfbead-kube-api-access-j85d2\") pod \"payload-processing-e2e-ipp-a-729108-5c8649cbff-9xhdl\" (UID: \"0ed81343-a77a-4401-bc65-21c804cfbead\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-729108-5c8649cbff-9xhdl"
Nov 02 12:46:35.205569 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:35.205548    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b98cdf4a-b4e0-46f3-b791-5cc63707a366-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-729108-6df46f4ff9-qw5p5\" (UID: \"b98cdf4a-b4e0-46f3-b791-5cc63707a366\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-729108-6df46f4ff9-qw5p5"
Nov 02 12:46:35.214394 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:35.214372    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-97de36-76c57bb9ff-9dv2z"
Nov 02 12:46:35.317663 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:35.312223    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ed81343-a77a-4401-bc65-21c804cfbead-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-729108-5c8649cbff-9xhdl\" (UID: \"0ed81343-a77a-4401-bc65-21c804cfbead\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-729108-5c8649cbff-9xhdl"
Nov 02 12:46:35.317663 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:35.312308    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jf2c\" (UniqueName: \"kubernetes.io/projected/b98cdf4a-b4e0-46f3-b791-5cc63707a366-kube-api-access-8jf2c\") pod \"payload-pre-processing-e2e-ipp-a-729108-6df46f4ff9-qw5p5\" (UID: \"b98cdf4a-b4e0-46f3-b791-5cc63707a366\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-729108-6df46f4ff9-qw5p5"
Nov 02 12:46:35.317663 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:35.312392    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j85d2\" (UniqueName: \"kubernetes.io/projected/0ed81343-a77a-4401-bc65-21c804cfbead-kube-api-access-j85d2\") pod \"payload-processing-e2e-ipp-a-729108-5c8649cbff-9xhdl\" (UID: \"0ed81343-a77a-4401-bc65-21c804cfbead\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-729108-5c8649cbff-9xhdl"
Nov 02 12:46:35.317663 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:35.312502    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b98cdf4a-b4e0-46f3-b791-5cc63707a366-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-729108-6df46f4ff9-qw5p5\" (UID: \"b98cdf4a-b4e0-46f3-b791-5cc63707a366\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-729108-6df46f4ff9-qw5p5"
Nov 02 12:46:35.317663 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:35.313619    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b98cdf4a-b4e0-46f3-b791-5cc63707a366-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-729108-6df46f4ff9-qw5p5\" (UID: \"b98cdf4a-b4e0-46f3-b791-5cc63707a366\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-729108-6df46f4ff9-qw5p5"
Nov 02 12:46:35.330369 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:35.323764    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ed81343-a77a-4401-bc65-21c804cfbead-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-729108-5c8649cbff-9xhdl\" (UID: \"0ed81343-a77a-4401-bc65-21c804cfbead\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-729108-5c8649cbff-9xhdl"
Nov 02 12:46:35.411721 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:35.403165    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j85d2\" (UniqueName: \"kubernetes.io/projected/0ed81343-a77a-4401-bc65-21c804cfbead-kube-api-access-j85d2\") pod \"payload-processing-e2e-ipp-a-729108-5c8649cbff-9xhdl\" (UID: \"0ed81343-a77a-4401-bc65-21c804cfbead\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-729108-5c8649cbff-9xhdl"
Nov 02 12:46:35.492838 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:35.454481    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-729108-5c8649cbff-9xhdl"
Nov 02 12:46:35.673676 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:35.672612    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jf2c\" (UniqueName: \"kubernetes.io/projected/b98cdf4a-b4e0-46f3-b791-5cc63707a366-kube-api-access-8jf2c\") pod \"payload-pre-processing-e2e-ipp-a-729108-6df46f4ff9-qw5p5\" (UID: \"b98cdf4a-b4e0-46f3-b791-5cc63707a366\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-729108-6df46f4ff9-qw5p5"
Nov 02 12:46:35.778282 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:35.778077    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-729108-6df46f4ff9-qw5p5"
Nov 02 12:46:35.979309 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:46:35.979281    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ed81343_a77a_4401_bc65_21c804cfbead.slice/crio-c8fb1940d6c7564239c13cce1b67cf59eebf409f954f005b3a503e2164476aaa WatchSource:0}: Error finding container c8fb1940d6c7564239c13cce1b67cf59eebf409f954f005b3a503e2164476aaa: Status 404 returned error can't find the container with id c8fb1940d6c7564239c13cce1b67cf59eebf409f954f005b3a503e2164476aaa
Nov 02 12:46:35.980393 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:35.980367    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-729108-5c8649cbff-9xhdl"]
Nov 02 12:46:36.091886 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:46:36.091854    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb98cdf4a_b4e0_46f3_b791_5cc63707a366.slice/crio-b21e0524a87c05807f664334e92d80960f1221fa5cae609edd0c637cccd11322 WatchSource:0}: Error finding container b21e0524a87c05807f664334e92d80960f1221fa5cae609edd0c637cccd11322: Status 404 returned error can't find the container with id b21e0524a87c05807f664334e92d80960f1221fa5cae609edd0c637cccd11322
Nov 02 12:46:36.094427 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:36.094406    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-729108-6df46f4ff9-qw5p5"]
Nov 02 12:46:36.681556 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:36.681520    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-729108-6df46f4ff9-qw5p5" event={"ID":"b98cdf4a-b4e0-46f3-b791-5cc63707a366","Type":"ContainerStarted","Data":"7bdffcb5acd75f5193bc9ab46b5343c6b9855224d00bd74d5ee35881af05ae7f"}
Nov 02 12:46:36.682040 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:36.681579    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-729108-6df46f4ff9-qw5p5"
Nov 02 12:46:36.682040 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:36.681598    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-729108-6df46f4ff9-qw5p5" event={"ID":"b98cdf4a-b4e0-46f3-b791-5cc63707a366","Type":"ContainerStarted","Data":"b21e0524a87c05807f664334e92d80960f1221fa5cae609edd0c637cccd11322"}
Nov 02 12:46:36.683270 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:36.683239    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-729108-5c8649cbff-9xhdl" event={"ID":"0ed81343-a77a-4401-bc65-21c804cfbead","Type":"ContainerStarted","Data":"09d862a8ca9b796daf03cf836eedc91dba22752c8e9be1e80d18c3c05d05eecb"}
Nov 02 12:46:36.683389 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:36.683292    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-729108-5c8649cbff-9xhdl" event={"ID":"0ed81343-a77a-4401-bc65-21c804cfbead","Type":"ContainerStarted","Data":"c8fb1940d6c7564239c13cce1b67cf59eebf409f954f005b3a503e2164476aaa"}
Nov 02 12:46:36.683389 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:36.683322    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-729108-5c8649cbff-9xhdl"
Nov 02 12:46:36.726075 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:36.726025    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-729108-6df46f4ff9-qw5p5" podStartSLOduration=1.7260110819999999 podStartE2EDuration="1.726011082s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:46:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:46:36.72362789 +0000 UTC m=+1192.979026997" watchObservedRunningTime="2026-11-02 12:46:36.726011082 +0000 UTC m=+1192.981410179"
Nov 02 12:46:36.766313 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:36.766258    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-729108-5c8649cbff-9xhdl" podStartSLOduration=1.766241053 podStartE2EDuration="1.766241053s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:46:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:46:36.76413302 +0000 UTC m=+1193.019532115" watchObservedRunningTime="2026-11-02 12:46:36.766241053 +0000 UTC m=+1193.021640149"
Nov 02 12:46:38.199954 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:38.199919    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-97de36-58bfcc8d45-xpkdt"]
Nov 02 12:46:38.204388 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:38.204371    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-97de36-58bfcc8d45-xpkdt"
Nov 02 12:46:38.231809 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:38.231783    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-97de36-serving-cert\""
Nov 02 12:46:38.247835 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:38.247806    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-97de36-58bfcc8d45-xpkdt"]
Nov 02 12:46:38.261670 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:38.261616    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-b2ec05-57ff8d5965-9k4gl"
Nov 02 12:46:38.295588 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:38.295558    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b68f0680-158b-4a2e-970c-44e2d0604262-metrics-certs\") pod \"maas-api-e2e-ipp-b-97de36-58bfcc8d45-xpkdt\" (UID: \"b68f0680-158b-4a2e-970c-44e2d0604262\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-97de36-58bfcc8d45-xpkdt"
Nov 02 12:46:38.295925 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:38.295871    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmtnb\" (UniqueName: \"kubernetes.io/projected/b68f0680-158b-4a2e-970c-44e2d0604262-kube-api-access-cmtnb\") pod \"maas-api-e2e-ipp-b-97de36-58bfcc8d45-xpkdt\" (UID: \"b68f0680-158b-4a2e-970c-44e2d0604262\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-97de36-58bfcc8d45-xpkdt"
Nov 02 12:46:38.296014 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:38.295989    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b68f0680-158b-4a2e-970c-44e2d0604262-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-97de36-58bfcc8d45-xpkdt\" (UID: \"b68f0680-158b-4a2e-970c-44e2d0604262\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-97de36-58bfcc8d45-xpkdt"
Nov 02 12:46:38.296074 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:38.296049    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b68f0680-158b-4a2e-970c-44e2d0604262-maas-api-tls\") pod \"maas-api-e2e-ipp-b-97de36-58bfcc8d45-xpkdt\" (UID: \"b68f0680-158b-4a2e-970c-44e2d0604262\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-97de36-58bfcc8d45-xpkdt"
Nov 02 12:46:38.396909 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:38.396873    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b68f0680-158b-4a2e-970c-44e2d0604262-maas-api-tls\") pod \"maas-api-e2e-ipp-b-97de36-58bfcc8d45-xpkdt\" (UID: \"b68f0680-158b-4a2e-970c-44e2d0604262\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-97de36-58bfcc8d45-xpkdt"
Nov 02 12:46:38.397084 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:38.396926    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b68f0680-158b-4a2e-970c-44e2d0604262-metrics-certs\") pod \"maas-api-e2e-ipp-b-97de36-58bfcc8d45-xpkdt\" (UID: \"b68f0680-158b-4a2e-970c-44e2d0604262\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-97de36-58bfcc8d45-xpkdt"
Nov 02 12:46:38.397221 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:38.397205    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmtnb\" (UniqueName: \"kubernetes.io/projected/b68f0680-158b-4a2e-970c-44e2d0604262-kube-api-access-cmtnb\") pod \"maas-api-e2e-ipp-b-97de36-58bfcc8d45-xpkdt\" (UID: \"b68f0680-158b-4a2e-970c-44e2d0604262\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-97de36-58bfcc8d45-xpkdt"
Nov 02 12:46:38.397283 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:38.397269    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b68f0680-158b-4a2e-970c-44e2d0604262-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-97de36-58bfcc8d45-xpkdt\" (UID: \"b68f0680-158b-4a2e-970c-44e2d0604262\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-97de36-58bfcc8d45-xpkdt"
Nov 02 12:46:38.397967 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:38.397941    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b68f0680-158b-4a2e-970c-44e2d0604262-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-97de36-58bfcc8d45-xpkdt\" (UID: \"b68f0680-158b-4a2e-970c-44e2d0604262\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-97de36-58bfcc8d45-xpkdt"
Nov 02 12:46:38.399237 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:38.399216    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b68f0680-158b-4a2e-970c-44e2d0604262-maas-api-tls\") pod \"maas-api-e2e-ipp-b-97de36-58bfcc8d45-xpkdt\" (UID: \"b68f0680-158b-4a2e-970c-44e2d0604262\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-97de36-58bfcc8d45-xpkdt"
Nov 02 12:46:38.399345 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:38.399269    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b68f0680-158b-4a2e-970c-44e2d0604262-metrics-certs\") pod \"maas-api-e2e-ipp-b-97de36-58bfcc8d45-xpkdt\" (UID: \"b68f0680-158b-4a2e-970c-44e2d0604262\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-97de36-58bfcc8d45-xpkdt"
Nov 02 12:46:38.424680 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:38.424654    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmtnb\" (UniqueName: \"kubernetes.io/projected/b68f0680-158b-4a2e-970c-44e2d0604262-kube-api-access-cmtnb\") pod \"maas-api-e2e-ipp-b-97de36-58bfcc8d45-xpkdt\" (UID: \"b68f0680-158b-4a2e-970c-44e2d0604262\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-97de36-58bfcc8d45-xpkdt"
Nov 02 12:46:38.521065 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:38.520973    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-97de36-58bfcc8d45-xpkdt"
Nov 02 12:46:38.909505 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:46:38.909478    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb68f0680_158b_4a2e_970c_44e2d0604262.slice/crio-7c1bb3badaf94c64a30f2274a5cab68ea5d7796d4fceb28f0d6c6d41fbe2fd7f WatchSource:0}: Error finding container 7c1bb3badaf94c64a30f2274a5cab68ea5d7796d4fceb28f0d6c6d41fbe2fd7f: Status 404 returned error can't find the container with id 7c1bb3badaf94c64a30f2274a5cab68ea5d7796d4fceb28f0d6c6d41fbe2fd7f
Nov 02 12:46:38.910942 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:38.910918    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-97de36-58bfcc8d45-xpkdt"]
Nov 02 12:46:39.722666 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:39.715950    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-97de36-58bfcc8d45-xpkdt" event={"ID":"b68f0680-158b-4a2e-970c-44e2d0604262","Type":"ContainerStarted","Data":"7c1bb3badaf94c64a30f2274a5cab68ea5d7796d4fceb28f0d6c6d41fbe2fd7f"}
Nov 02 12:46:40.724044 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:40.724012    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-97de36-58bfcc8d45-xpkdt" event={"ID":"b68f0680-158b-4a2e-970c-44e2d0604262","Type":"ContainerStarted","Data":"f7b6d4d0cda80a464f7a2135db1a04491b0639220c5a3484526a72cb8d6da7b5"}
Nov 02 12:46:40.724470 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:40.724099    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-97de36-58bfcc8d45-xpkdt"
Nov 02 12:46:40.768486 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:40.768436    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-97de36-58bfcc8d45-xpkdt" podStartSLOduration=2.358796174 podStartE2EDuration="2.768422784s" totalImagesPullingTime="409.62661ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:46:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:46:40.766839273 +0000 UTC m=+1197.022238368" watchObservedRunningTime="2026-11-02 12:46:40.768422784 +0000 UTC m=+1197.023821878"
Nov 02 12:46:44.020107 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:44.020075    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-dvzvq"]
Nov 02 12:46:44.020705 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:44.020296    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-dvzvq" podUID="79ae08e3-f7c3-4e28-aabb-246f42f45b2e" containerName="istio-proxy" containerID="cri-o://ef92179081982d7a3d24dff3c68e9558db8a59fdf9424e883f26899570e86ba6" gracePeriod=30
Nov 02 12:46:44.035561 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:44.035531    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-nrntd"]
Nov 02 12:46:44.035819 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:44.035767    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-nrntd" podUID="35f5e006-f020-4166-ab03-aedc1fbf2ce4" containerName="istio-proxy" containerID="cri-o://3c630d665d2dbfd303d7ab36682f3d1f8a42ce8c41ac079f7978833d3799d53b" gracePeriod=30
Nov 02 12:46:44.406513 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:44.406428    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-b2ec05-5c56f74f78-jnq6q"]
Nov 02 12:46:44.412553 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:44.412535    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b2ec05-5c56f74f78-jnq6q"
Nov 02 12:46:44.428809 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:44.428780    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-b2ec05-serving-cert\""
Nov 02 12:46:44.430864 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:44.430840    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-b2ec05-5c56f74f78-jnq6q"]
Nov 02 12:46:44.467231 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:44.467194    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2kzgk\" (UniqueName: \"kubernetes.io/projected/00c5fac7-35bf-426e-b9f2-d1900ac09c83-kube-api-access-2kzgk\") pod \"maas-api-e2e-api-a-b2ec05-5c56f74f78-jnq6q\" (UID: \"00c5fac7-35bf-426e-b9f2-d1900ac09c83\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b2ec05-5c56f74f78-jnq6q"
Nov 02 12:46:44.467393 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:44.467276    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/00c5fac7-35bf-426e-b9f2-d1900ac09c83-metrics-certs\") pod \"maas-api-e2e-api-a-b2ec05-5c56f74f78-jnq6q\" (UID: \"00c5fac7-35bf-426e-b9f2-d1900ac09c83\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b2ec05-5c56f74f78-jnq6q"
Nov 02 12:46:44.467568 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:44.467545    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00c5fac7-35bf-426e-b9f2-d1900ac09c83-maas-api-tls\") pod \"maas-api-e2e-api-a-b2ec05-5c56f74f78-jnq6q\" (UID: \"00c5fac7-35bf-426e-b9f2-d1900ac09c83\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b2ec05-5c56f74f78-jnq6q"
Nov 02 12:46:44.467629 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:44.467615    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/00c5fac7-35bf-426e-b9f2-d1900ac09c83-openshift-service-ca\") pod \"maas-api-e2e-api-a-b2ec05-5c56f74f78-jnq6q\" (UID: \"00c5fac7-35bf-426e-b9f2-d1900ac09c83\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b2ec05-5c56f74f78-jnq6q"
Nov 02 12:46:44.569649 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:44.569616    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00c5fac7-35bf-426e-b9f2-d1900ac09c83-maas-api-tls\") pod \"maas-api-e2e-api-a-b2ec05-5c56f74f78-jnq6q\" (UID: \"00c5fac7-35bf-426e-b9f2-d1900ac09c83\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b2ec05-5c56f74f78-jnq6q"
Nov 02 12:46:44.569836 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:44.569694    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/00c5fac7-35bf-426e-b9f2-d1900ac09c83-openshift-service-ca\") pod \"maas-api-e2e-api-a-b2ec05-5c56f74f78-jnq6q\" (UID: \"00c5fac7-35bf-426e-b9f2-d1900ac09c83\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b2ec05-5c56f74f78-jnq6q"
Nov 02 12:46:44.569836 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:46:44.569727    4559 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-b2ec05-serving-cert: secret "maas-api-e2e-api-a-b2ec05-serving-cert" not found
Nov 02 12:46:44.569836 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:46:44.569781    4559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/00c5fac7-35bf-426e-b9f2-d1900ac09c83-maas-api-tls podName:00c5fac7-35bf-426e-b9f2-d1900ac09c83 nodeName:}" failed. No retries permitted until 2026-11-02 12:46:45.069766163 +0000 UTC m=+1201.325165240 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/00c5fac7-35bf-426e-b9f2-d1900ac09c83-maas-api-tls") pod "maas-api-e2e-api-a-b2ec05-5c56f74f78-jnq6q" (UID: "00c5fac7-35bf-426e-b9f2-d1900ac09c83") : secret "maas-api-e2e-api-a-b2ec05-serving-cert" not found
Nov 02 12:46:44.569836 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:44.569824    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kzgk\" (UniqueName: \"kubernetes.io/projected/00c5fac7-35bf-426e-b9f2-d1900ac09c83-kube-api-access-2kzgk\") pod \"maas-api-e2e-api-a-b2ec05-5c56f74f78-jnq6q\" (UID: \"00c5fac7-35bf-426e-b9f2-d1900ac09c83\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b2ec05-5c56f74f78-jnq6q"
Nov 02 12:46:44.570073 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:44.569873    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/00c5fac7-35bf-426e-b9f2-d1900ac09c83-metrics-certs\") pod \"maas-api-e2e-api-a-b2ec05-5c56f74f78-jnq6q\" (UID: \"00c5fac7-35bf-426e-b9f2-d1900ac09c83\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b2ec05-5c56f74f78-jnq6q"
Nov 02 12:46:44.570346 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:44.570326    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/00c5fac7-35bf-426e-b9f2-d1900ac09c83-openshift-service-ca\") pod \"maas-api-e2e-api-a-b2ec05-5c56f74f78-jnq6q\" (UID: \"00c5fac7-35bf-426e-b9f2-d1900ac09c83\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b2ec05-5c56f74f78-jnq6q"
Nov 02 12:46:44.572334 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:44.572315    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/00c5fac7-35bf-426e-b9f2-d1900ac09c83-metrics-certs\") pod \"maas-api-e2e-api-a-b2ec05-5c56f74f78-jnq6q\" (UID: \"00c5fac7-35bf-426e-b9f2-d1900ac09c83\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b2ec05-5c56f74f78-jnq6q"
Nov 02 12:46:44.593851 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:44.593823    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kzgk\" (UniqueName: \"kubernetes.io/projected/00c5fac7-35bf-426e-b9f2-d1900ac09c83-kube-api-access-2kzgk\") pod \"maas-api-e2e-api-a-b2ec05-5c56f74f78-jnq6q\" (UID: \"00c5fac7-35bf-426e-b9f2-d1900ac09c83\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b2ec05-5c56f74f78-jnq6q"
Nov 02 12:46:45.078872 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:45.078824    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00c5fac7-35bf-426e-b9f2-d1900ac09c83-maas-api-tls\") pod \"maas-api-e2e-api-a-b2ec05-5c56f74f78-jnq6q\" (UID: \"00c5fac7-35bf-426e-b9f2-d1900ac09c83\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b2ec05-5c56f74f78-jnq6q"
Nov 02 12:46:45.081359 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:45.081325    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00c5fac7-35bf-426e-b9f2-d1900ac09c83-maas-api-tls\") pod \"maas-api-e2e-api-a-b2ec05-5c56f74f78-jnq6q\" (UID: \"00c5fac7-35bf-426e-b9f2-d1900ac09c83\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b2ec05-5c56f74f78-jnq6q"
Nov 02 12:46:45.088022 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:46:45.087990    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqcmj" containerName="extractor"
Nov 02 12:46:45.327030 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:45.326949    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b2ec05-5c56f74f78-jnq6q"
Nov 02 12:46:45.710534 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:45.710257    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-b2ec05-5c56f74f78-jnq6q"]
Nov 02 12:46:45.781125 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:45.781088    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b2ec05-5c56f74f78-jnq6q" event={"ID":"00c5fac7-35bf-426e-b9f2-d1900ac09c83","Type":"ContainerStarted","Data":"2b0ca30455ee6582b6533ae8d0ff3b8ab101e32937e6b1a1dc8789f8099ecb36"}
Nov 02 12:46:46.736595 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:46.736565    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-97de36-58bfcc8d45-xpkdt"
Nov 02 12:46:46.791725 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:46.791684    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b2ec05-5c56f74f78-jnq6q" event={"ID":"00c5fac7-35bf-426e-b9f2-d1900ac09c83","Type":"ContainerStarted","Data":"d7454aba9ed7cd2c3350eed1b4ef1b967228ed7daa8540a741e2202013290ca9"}
Nov 02 12:46:46.791905 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:46.791753    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b2ec05-5c56f74f78-jnq6q"
Nov 02 12:46:46.855115 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:46.855062    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b2ec05-5c56f74f78-jnq6q" podStartSLOduration=2.255974417 podStartE2EDuration="2.85504775s" totalImagesPullingTime="599.073333ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:46:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:46:46.850972715 +0000 UTC m=+1203.106371810" watchObservedRunningTime="2026-11-02 12:46:46.85504775 +0000 UTC m=+1203.110446844"
Nov 02 12:46:47.692894 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:47.692869    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-729108-5c8649cbff-9xhdl"
Nov 02 12:46:47.693077 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:47.692934    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-729108-6df46f4ff9-qw5p5"
Nov 02 12:46:49.333870 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:49.333841    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-dvzvq"
Nov 02 12:46:49.367393 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:49.367367    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79ae08e3-f7c3-4e28-aabb-246f42f45b2e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79ae08e3-f7c3-4e28-aabb-246f42f45b2e-workload-socket\") pod \"79ae08e3-f7c3-4e28-aabb-246f42f45b2e\" (UID: \"79ae08e3-f7c3-4e28-aabb-246f42f45b2e\") "
Nov 02 12:46:49.367548 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:49.367436    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79ae08e3-f7c3-4e28-aabb-246f42f45b2e-istio-token\" (UniqueName: \"kubernetes.io/projected/79ae08e3-f7c3-4e28-aabb-246f42f45b2e-istio-token\") pod \"79ae08e3-f7c3-4e28-aabb-246f42f45b2e\" (UID: \"79ae08e3-f7c3-4e28-aabb-246f42f45b2e\") "
Nov 02 12:46:49.367548 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:49.367507    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79ae08e3-f7c3-4e28-aabb-246f42f45b2e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79ae08e3-f7c3-4e28-aabb-246f42f45b2e-credential-socket\") pod \"79ae08e3-f7c3-4e28-aabb-246f42f45b2e\" (UID: \"79ae08e3-f7c3-4e28-aabb-246f42f45b2e\") "
Nov 02 12:46:49.367701 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:49.367619    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79ae08e3-f7c3-4e28-aabb-246f42f45b2e-workload-socket" pod "79ae08e3-f7c3-4e28-aabb-246f42f45b2e" (UID: "79ae08e3-f7c3-4e28-aabb-246f42f45b2e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:46:49.367701 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:49.367659    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79ae08e3-f7c3-4e28-aabb-246f42f45b2e-kube-api-access-l6qw8\" (UniqueName: \"kubernetes.io/projected/79ae08e3-f7c3-4e28-aabb-246f42f45b2e-kube-api-access-l6qw8\") pod \"79ae08e3-f7c3-4e28-aabb-246f42f45b2e\" (UID: \"79ae08e3-f7c3-4e28-aabb-246f42f45b2e\") "
Nov 02 12:46:49.367822 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:49.367774    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79ae08e3-f7c3-4e28-aabb-246f42f45b2e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79ae08e3-f7c3-4e28-aabb-246f42f45b2e-workload-certs\") pod \"79ae08e3-f7c3-4e28-aabb-246f42f45b2e\" (UID: \"79ae08e3-f7c3-4e28-aabb-246f42f45b2e\") "
Nov 02 12:46:49.367822 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:49.367798    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79ae08e3-f7c3-4e28-aabb-246f42f45b2e-credential-socket" pod "79ae08e3-f7c3-4e28-aabb-246f42f45b2e" (UID: "79ae08e3-f7c3-4e28-aabb-246f42f45b2e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:46:49.367926 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:49.367866    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79ae08e3-f7c3-4e28-aabb-246f42f45b2e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79ae08e3-f7c3-4e28-aabb-246f42f45b2e-istio-data\") pod \"79ae08e3-f7c3-4e28-aabb-246f42f45b2e\" (UID: \"79ae08e3-f7c3-4e28-aabb-246f42f45b2e\") "
Nov 02 12:46:49.368003 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:49.367986    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/79ae08e3-f7c3-4e28-aabb-246f42f45b2e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79ae08e3-f7c3-4e28-aabb-246f42f45b2e-istio-podinfo\") pod \"79ae08e3-f7c3-4e28-aabb-246f42f45b2e\" (UID: \"79ae08e3-f7c3-4e28-aabb-246f42f45b2e\") "
Nov 02 12:46:49.368063 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:49.368030    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79ae08e3-f7c3-4e28-aabb-246f42f45b2e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79ae08e3-f7c3-4e28-aabb-246f42f45b2e-istio-envoy\") pod \"79ae08e3-f7c3-4e28-aabb-246f42f45b2e\" (UID: \"79ae08e3-f7c3-4e28-aabb-246f42f45b2e\") "
Nov 02 12:46:49.368118 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:49.367983    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79ae08e3-f7c3-4e28-aabb-246f42f45b2e-workload-certs" pod "79ae08e3-f7c3-4e28-aabb-246f42f45b2e" (UID: "79ae08e3-f7c3-4e28-aabb-246f42f45b2e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:46:49.368286 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:49.368225    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79ae08e3-f7c3-4e28-aabb-246f42f45b2e-istio-data" pod "79ae08e3-f7c3-4e28-aabb-246f42f45b2e" (UID: "79ae08e3-f7c3-4e28-aabb-246f42f45b2e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:46:49.368419 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:49.368323    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79ae08e3-f7c3-4e28-aabb-246f42f45b2e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79ae08e3-f7c3-4e28-aabb-246f42f45b2e-istiod-ca-cert\") pod \"79ae08e3-f7c3-4e28-aabb-246f42f45b2e\" (UID: \"79ae08e3-f7c3-4e28-aabb-246f42f45b2e\") "
Nov 02 12:46:49.369347 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:49.368760    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79ae08e3-f7c3-4e28-aabb-246f42f45b2e-istiod-ca-cert" pod "79ae08e3-f7c3-4e28-aabb-246f42f45b2e" (UID: "79ae08e3-f7c3-4e28-aabb-246f42f45b2e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:46:49.369678 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:49.369524    4559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79ae08e3-f7c3-4e28-aabb-246f42f45b2e-istiod-ca-cert\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:46:49.369678 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:49.369554    4559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79ae08e3-f7c3-4e28-aabb-246f42f45b2e-workload-socket\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:46:49.369678 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:49.369570    4559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79ae08e3-f7c3-4e28-aabb-246f42f45b2e-credential-socket\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:46:49.369678 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:49.369585    4559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79ae08e3-f7c3-4e28-aabb-246f42f45b2e-workload-certs\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:46:49.369678 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:49.369608    4559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79ae08e3-f7c3-4e28-aabb-246f42f45b2e-istio-data\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:46:49.370196 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:49.370177    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79ae08e3-f7c3-4e28-aabb-246f42f45b2e-istio-token" pod "79ae08e3-f7c3-4e28-aabb-246f42f45b2e" (UID: "79ae08e3-f7c3-4e28-aabb-246f42f45b2e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:46:49.370262 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:49.370195    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/79ae08e3-f7c3-4e28-aabb-246f42f45b2e-istio-podinfo" pod "79ae08e3-f7c3-4e28-aabb-246f42f45b2e" (UID: "79ae08e3-f7c3-4e28-aabb-246f42f45b2e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 12:46:49.370262 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:49.370204    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79ae08e3-f7c3-4e28-aabb-246f42f45b2e-kube-api-access-l6qw8" pod "79ae08e3-f7c3-4e28-aabb-246f42f45b2e" (UID: "79ae08e3-f7c3-4e28-aabb-246f42f45b2e"). InnerVolumeSpecName "kube-api-access-l6qw8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:46:49.370814 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:49.370796    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79ae08e3-f7c3-4e28-aabb-246f42f45b2e-istio-envoy" pod "79ae08e3-f7c3-4e28-aabb-246f42f45b2e" (UID: "79ae08e3-f7c3-4e28-aabb-246f42f45b2e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:46:49.374325 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:49.374307    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-nrntd"
Nov 02 12:46:49.470543 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:49.470509    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35f5e006-f020-4166-ab03-aedc1fbf2ce4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35f5e006-f020-4166-ab03-aedc1fbf2ce4-istio-data\") pod \"35f5e006-f020-4166-ab03-aedc1fbf2ce4\" (UID: \"35f5e006-f020-4166-ab03-aedc1fbf2ce4\") "
Nov 02 12:46:49.470731 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:49.470552    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35f5e006-f020-4166-ab03-aedc1fbf2ce4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35f5e006-f020-4166-ab03-aedc1fbf2ce4-istio-envoy\") pod \"35f5e006-f020-4166-ab03-aedc1fbf2ce4\" (UID: \"35f5e006-f020-4166-ab03-aedc1fbf2ce4\") "
Nov 02 12:46:49.470731 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:49.470576    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35f5e006-f020-4166-ab03-aedc1fbf2ce4-istio-token\" (UniqueName: \"kubernetes.io/projected/35f5e006-f020-4166-ab03-aedc1fbf2ce4-istio-token\") pod \"35f5e006-f020-4166-ab03-aedc1fbf2ce4\" (UID: \"35f5e006-f020-4166-ab03-aedc1fbf2ce4\") "
Nov 02 12:46:49.470731 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:49.470608    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35f5e006-f020-4166-ab03-aedc1fbf2ce4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35f5e006-f020-4166-ab03-aedc1fbf2ce4-workload-socket\") pod \"35f5e006-f020-4166-ab03-aedc1fbf2ce4\" (UID: \"35f5e006-f020-4166-ab03-aedc1fbf2ce4\") "
Nov 02 12:46:49.470731 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:49.470670    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35f5e006-f020-4166-ab03-aedc1fbf2ce4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35f5e006-f020-4166-ab03-aedc1fbf2ce4-istiod-ca-cert\") pod \"35f5e006-f020-4166-ab03-aedc1fbf2ce4\" (UID: \"35f5e006-f020-4166-ab03-aedc1fbf2ce4\") "
Nov 02 12:46:49.470938 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:49.470751    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35f5e006-f020-4166-ab03-aedc1fbf2ce4-kube-api-access-9tk6p\" (UniqueName: \"kubernetes.io/projected/35f5e006-f020-4166-ab03-aedc1fbf2ce4-kube-api-access-9tk6p\") pod \"35f5e006-f020-4166-ab03-aedc1fbf2ce4\" (UID: \"35f5e006-f020-4166-ab03-aedc1fbf2ce4\") "
Nov 02 12:46:49.470938 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:49.470789    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35f5e006-f020-4166-ab03-aedc1fbf2ce4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35f5e006-f020-4166-ab03-aedc1fbf2ce4-credential-socket\") pod \"35f5e006-f020-4166-ab03-aedc1fbf2ce4\" (UID: \"35f5e006-f020-4166-ab03-aedc1fbf2ce4\") "
Nov 02 12:46:49.470938 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:49.470910    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/35f5e006-f020-4166-ab03-aedc1fbf2ce4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35f5e006-f020-4166-ab03-aedc1fbf2ce4-istio-podinfo\") pod \"35f5e006-f020-4166-ab03-aedc1fbf2ce4\" (UID: \"35f5e006-f020-4166-ab03-aedc1fbf2ce4\") "
Nov 02 12:46:49.470938 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:49.470889    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35f5e006-f020-4166-ab03-aedc1fbf2ce4-istio-data" pod "35f5e006-f020-4166-ab03-aedc1fbf2ce4" (UID: "35f5e006-f020-4166-ab03-aedc1fbf2ce4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:46:49.471157 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:49.471092    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35f5e006-f020-4166-ab03-aedc1fbf2ce4-credential-socket" pod "35f5e006-f020-4166-ab03-aedc1fbf2ce4" (UID: "35f5e006-f020-4166-ab03-aedc1fbf2ce4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:46:49.471157 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:49.471111    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35f5e006-f020-4166-ab03-aedc1fbf2ce4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35f5e006-f020-4166-ab03-aedc1fbf2ce4-workload-certs\") pod \"35f5e006-f020-4166-ab03-aedc1fbf2ce4\" (UID: \"35f5e006-f020-4166-ab03-aedc1fbf2ce4\") "
Nov 02 12:46:49.471335 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:49.471311    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35f5e006-f020-4166-ab03-aedc1fbf2ce4-workload-certs" pod "35f5e006-f020-4166-ab03-aedc1fbf2ce4" (UID: "35f5e006-f020-4166-ab03-aedc1fbf2ce4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:46:49.471335 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:49.471321    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35f5e006-f020-4166-ab03-aedc1fbf2ce4-istiod-ca-cert" pod "35f5e006-f020-4166-ab03-aedc1fbf2ce4" (UID: "35f5e006-f020-4166-ab03-aedc1fbf2ce4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:46:49.471501 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:49.471386    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35f5e006-f020-4166-ab03-aedc1fbf2ce4-workload-socket" pod "35f5e006-f020-4166-ab03-aedc1fbf2ce4" (UID: "35f5e006-f020-4166-ab03-aedc1fbf2ce4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:46:49.471974 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:49.471951    4559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79ae08e3-f7c3-4e28-aabb-246f42f45b2e-istio-podinfo\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:46:49.472061 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:49.471982    4559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79ae08e3-f7c3-4e28-aabb-246f42f45b2e-istio-envoy\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:46:49.472061 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:49.471996    4559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35f5e006-f020-4166-ab03-aedc1fbf2ce4-workload-certs\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:46:49.472061 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:49.472008    4559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35f5e006-f020-4166-ab03-aedc1fbf2ce4-istio-data\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:46:49.472061 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:49.472020    4559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79ae08e3-f7c3-4e28-aabb-246f42f45b2e-istio-token\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:46:49.472061 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:49.472032    4559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35f5e006-f020-4166-ab03-aedc1fbf2ce4-workload-socket\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:46:49.472061 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:49.472045    4559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35f5e006-f020-4166-ab03-aedc1fbf2ce4-istiod-ca-cert\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:46:49.472061 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:49.472059    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l6qw8\" (UniqueName: \"kubernetes.io/projected/79ae08e3-f7c3-4e28-aabb-246f42f45b2e-kube-api-access-l6qw8\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:46:49.472301 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:49.472073    4559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35f5e006-f020-4166-ab03-aedc1fbf2ce4-credential-socket\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:46:49.473404 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:49.473376    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35f5e006-f020-4166-ab03-aedc1fbf2ce4-istio-token" pod "35f5e006-f020-4166-ab03-aedc1fbf2ce4" (UID: "35f5e006-f020-4166-ab03-aedc1fbf2ce4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:46:49.473537 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:49.473516    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/35f5e006-f020-4166-ab03-aedc1fbf2ce4-istio-podinfo" pod "35f5e006-f020-4166-ab03-aedc1fbf2ce4" (UID: "35f5e006-f020-4166-ab03-aedc1fbf2ce4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 12:46:49.473631 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:49.473606    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35f5e006-f020-4166-ab03-aedc1fbf2ce4-kube-api-access-9tk6p" pod "35f5e006-f020-4166-ab03-aedc1fbf2ce4" (UID: "35f5e006-f020-4166-ab03-aedc1fbf2ce4"). InnerVolumeSpecName "kube-api-access-9tk6p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:46:49.474548 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:49.474522    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35f5e006-f020-4166-ab03-aedc1fbf2ce4-istio-envoy" pod "35f5e006-f020-4166-ab03-aedc1fbf2ce4" (UID: "35f5e006-f020-4166-ab03-aedc1fbf2ce4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:46:49.573621 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:49.573534    4559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35f5e006-f020-4166-ab03-aedc1fbf2ce4-istio-envoy\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:46:49.573621 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:49.573565    4559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35f5e006-f020-4166-ab03-aedc1fbf2ce4-istio-token\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:46:49.573621 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:49.573575    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9tk6p\" (UniqueName: \"kubernetes.io/projected/35f5e006-f020-4166-ab03-aedc1fbf2ce4-kube-api-access-9tk6p\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:46:49.573621 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:49.573584    4559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35f5e006-f020-4166-ab03-aedc1fbf2ce4-istio-podinfo\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:46:49.815597 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:49.815564    4559 generic.go:356] "Generic (PLEG): container finished" podID="35f5e006-f020-4166-ab03-aedc1fbf2ce4" containerID="3c630d665d2dbfd303d7ab36682f3d1f8a42ce8c41ac079f7978833d3799d53b" exitCode=0
Nov 02 12:46:49.815812 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:49.815614    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-nrntd"
Nov 02 12:46:49.815812 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:49.815664    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-nrntd" event={"ID":"35f5e006-f020-4166-ab03-aedc1fbf2ce4","Type":"ContainerDied","Data":"3c630d665d2dbfd303d7ab36682f3d1f8a42ce8c41ac079f7978833d3799d53b"}
Nov 02 12:46:49.815812 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:49.815738    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-nrntd" event={"ID":"35f5e006-f020-4166-ab03-aedc1fbf2ce4","Type":"ContainerDied","Data":"f7a3cf0cc5613dce317c0eb15e6862604e6c3e23563d98b1f184063e6b80888c"}
Nov 02 12:46:49.815812 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:49.815761    4559 scope.go:122] "RemoveContainer" containerID="3c630d665d2dbfd303d7ab36682f3d1f8a42ce8c41ac079f7978833d3799d53b"
Nov 02 12:46:49.817226 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:49.817205    4559 generic.go:356] "Generic (PLEG): container finished" podID="79ae08e3-f7c3-4e28-aabb-246f42f45b2e" containerID="ef92179081982d7a3d24dff3c68e9558db8a59fdf9424e883f26899570e86ba6" exitCode=0
Nov 02 12:46:49.817342 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:49.817260    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-dvzvq"
Nov 02 12:46:49.817342 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:49.817282    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-dvzvq" event={"ID":"79ae08e3-f7c3-4e28-aabb-246f42f45b2e","Type":"ContainerDied","Data":"ef92179081982d7a3d24dff3c68e9558db8a59fdf9424e883f26899570e86ba6"}
Nov 02 12:46:49.817342 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:49.817326    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-dvzvq" event={"ID":"79ae08e3-f7c3-4e28-aabb-246f42f45b2e","Type":"ContainerDied","Data":"fac466686e80c7bbe068cd9d438fd552fea7a035932cbdcaf24e8a7be8bcef0b"}
Nov 02 12:46:49.826091 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:49.826077    4559 scope.go:122] "RemoveContainer" containerID="3c630d665d2dbfd303d7ab36682f3d1f8a42ce8c41ac079f7978833d3799d53b"
Nov 02 12:46:49.826394 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:46:49.826365    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c630d665d2dbfd303d7ab36682f3d1f8a42ce8c41ac079f7978833d3799d53b\": container with ID starting with 3c630d665d2dbfd303d7ab36682f3d1f8a42ce8c41ac079f7978833d3799d53b not found: ID does not exist" containerID="3c630d665d2dbfd303d7ab36682f3d1f8a42ce8c41ac079f7978833d3799d53b"
Nov 02 12:46:49.826511 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:49.826400    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c630d665d2dbfd303d7ab36682f3d1f8a42ce8c41ac079f7978833d3799d53b"} err="failed to get container status \"3c630d665d2dbfd303d7ab36682f3d1f8a42ce8c41ac079f7978833d3799d53b\": rpc error: code = NotFound desc = could not find container \"3c630d665d2dbfd303d7ab36682f3d1f8a42ce8c41ac079f7978833d3799d53b\": container with ID starting with 3c630d665d2dbfd303d7ab36682f3d1f8a42ce8c41ac079f7978833d3799d53b not found: ID does not exist"
Nov 02 12:46:49.826511 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:49.826424    4559 scope.go:122] "RemoveContainer" containerID="ef92179081982d7a3d24dff3c68e9558db8a59fdf9424e883f26899570e86ba6"
Nov 02 12:46:49.836340 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:49.836322    4559 scope.go:122] "RemoveContainer" containerID="ef92179081982d7a3d24dff3c68e9558db8a59fdf9424e883f26899570e86ba6"
Nov 02 12:46:49.836604 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:46:49.836585    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef92179081982d7a3d24dff3c68e9558db8a59fdf9424e883f26899570e86ba6\": container with ID starting with ef92179081982d7a3d24dff3c68e9558db8a59fdf9424e883f26899570e86ba6 not found: ID does not exist" containerID="ef92179081982d7a3d24dff3c68e9558db8a59fdf9424e883f26899570e86ba6"
Nov 02 12:46:49.836689 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:49.836613    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef92179081982d7a3d24dff3c68e9558db8a59fdf9424e883f26899570e86ba6"} err="failed to get container status \"ef92179081982d7a3d24dff3c68e9558db8a59fdf9424e883f26899570e86ba6\": rpc error: code = NotFound desc = could not find container \"ef92179081982d7a3d24dff3c68e9558db8a59fdf9424e883f26899570e86ba6\": container with ID starting with ef92179081982d7a3d24dff3c68e9558db8a59fdf9424e883f26899570e86ba6 not found: ID does not exist"
Nov 02 12:46:49.885412 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:49.885375    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-nrntd"]
Nov 02 12:46:49.895254 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:49.895224    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-nrntd"]
Nov 02 12:46:50.012765 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:50.008500    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-dvzvq"]
Nov 02 12:46:50.044677 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:50.042672    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-dvzvq"]
Nov 02 12:46:50.853223 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:50.853191    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-2hpjf"]
Nov 02 12:46:50.853774 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:50.853450    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-2hpjf" podUID="e590f1fe-f50e-4ca1-a7cf-ac07ae240dba" containerName="istio-proxy" containerID="cri-o://49f1142f2c555ccb65a3759cf33dc628efcec7ec6c26d37107392a0e764c636d" gracePeriod=30
Nov 02 12:46:50.868223 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:50.868194    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-tlgjz"]
Nov 02 12:46:50.868419 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:50.868397    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-tlgjz" podUID="0b5ce2e7-bb00-436e-bee3-2d87665168c3" containerName="istio-proxy" containerID="cri-o://75dc888d7d589386f2c45149c17b2d94d74071e7184e670f201ecd78ab04e560" gracePeriod=30
Nov 02 12:46:50.873338 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:50.873316    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-gr5tx"]
Nov 02 12:46:50.873517 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:50.873498    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-gr5tx" podUID="0d507435-7115-4950-93c5-da6c04d73874" containerName="istio-proxy" containerID="cri-o://cfe4f960ff871c43450fc355da9c07406069824e196b0dd85d5b0af9a7300eb4" gracePeriod=30
Nov 02 12:46:51.894494 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:51.894463    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35f5e006-f020-4166-ab03-aedc1fbf2ce4" path="/var/lib/kubelet/pods/35f5e006-f020-4166-ab03-aedc1fbf2ce4/volumes"
Nov 02 12:46:51.895052 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:51.895037    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79ae08e3-f7c3-4e28-aabb-246f42f45b2e" path="/var/lib/kubelet/pods/79ae08e3-f7c3-4e28-aabb-246f42f45b2e/volumes"
Nov 02 12:46:52.806296 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:52.806262    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b2ec05-5c56f74f78-jnq6q"
Nov 02 12:46:56.198364 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:56.198339    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-2hpjf"
Nov 02 12:46:56.219022 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:56.218997    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-tlgjz"
Nov 02 12:46:56.234131 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:56.234107    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-gr5tx"
Nov 02 12:46:56.291884 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:56.291847    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e590f1fe-f50e-4ca1-a7cf-ac07ae240dba-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e590f1fe-f50e-4ca1-a7cf-ac07ae240dba-istiod-ca-cert\") pod \"e590f1fe-f50e-4ca1-a7cf-ac07ae240dba\" (UID: \"e590f1fe-f50e-4ca1-a7cf-ac07ae240dba\") "
Nov 02 12:46:56.292054 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:56.291903    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e590f1fe-f50e-4ca1-a7cf-ac07ae240dba-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e590f1fe-f50e-4ca1-a7cf-ac07ae240dba-credential-socket\") pod \"e590f1fe-f50e-4ca1-a7cf-ac07ae240dba\" (UID: \"e590f1fe-f50e-4ca1-a7cf-ac07ae240dba\") "
Nov 02 12:46:56.292054 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:56.291987    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e590f1fe-f50e-4ca1-a7cf-ac07ae240dba-istio-token\" (UniqueName: \"kubernetes.io/projected/e590f1fe-f50e-4ca1-a7cf-ac07ae240dba-istio-token\") pod \"e590f1fe-f50e-4ca1-a7cf-ac07ae240dba\" (UID: \"e590f1fe-f50e-4ca1-a7cf-ac07ae240dba\") "
Nov 02 12:46:56.292054 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:56.292040    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d507435-7115-4950-93c5-da6c04d73874-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d507435-7115-4950-93c5-da6c04d73874-istio-data\") pod \"0d507435-7115-4950-93c5-da6c04d73874\" (UID: \"0d507435-7115-4950-93c5-da6c04d73874\") "
Nov 02 12:46:56.292246 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:56.292074    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e590f1fe-f50e-4ca1-a7cf-ac07ae240dba-kube-api-access-rvbh4\" (UniqueName: \"kubernetes.io/projected/e590f1fe-f50e-4ca1-a7cf-ac07ae240dba-kube-api-access-rvbh4\") pod \"e590f1fe-f50e-4ca1-a7cf-ac07ae240dba\" (UID: \"e590f1fe-f50e-4ca1-a7cf-ac07ae240dba\") "
Nov 02 12:46:56.292246 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:56.292103    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d507435-7115-4950-93c5-da6c04d73874-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d507435-7115-4950-93c5-da6c04d73874-workload-socket\") pod \"0d507435-7115-4950-93c5-da6c04d73874\" (UID: \"0d507435-7115-4950-93c5-da6c04d73874\") "
Nov 02 12:46:56.292246 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:56.292142    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d507435-7115-4950-93c5-da6c04d73874-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d507435-7115-4950-93c5-da6c04d73874-istiod-ca-cert\") pod \"0d507435-7115-4950-93c5-da6c04d73874\" (UID: \"0d507435-7115-4950-93c5-da6c04d73874\") "
Nov 02 12:46:56.292246 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:56.292145    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e590f1fe-f50e-4ca1-a7cf-ac07ae240dba-credential-socket" pod "e590f1fe-f50e-4ca1-a7cf-ac07ae240dba" (UID: "e590f1fe-f50e-4ca1-a7cf-ac07ae240dba"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:46:56.292246 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:56.292178    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d507435-7115-4950-93c5-da6c04d73874-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d507435-7115-4950-93c5-da6c04d73874-credential-socket\") pod \"0d507435-7115-4950-93c5-da6c04d73874\" (UID: \"0d507435-7115-4950-93c5-da6c04d73874\") "
Nov 02 12:46:56.292246 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:56.292219    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e590f1fe-f50e-4ca1-a7cf-ac07ae240dba-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e590f1fe-f50e-4ca1-a7cf-ac07ae240dba-istio-envoy\") pod \"e590f1fe-f50e-4ca1-a7cf-ac07ae240dba\" (UID: \"e590f1fe-f50e-4ca1-a7cf-ac07ae240dba\") "
Nov 02 12:46:56.292714 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:56.292271    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d507435-7115-4950-93c5-da6c04d73874-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d507435-7115-4950-93c5-da6c04d73874-workload-certs\") pod \"0d507435-7115-4950-93c5-da6c04d73874\" (UID: \"0d507435-7115-4950-93c5-da6c04d73874\") "
Nov 02 12:46:56.292714 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:56.292285    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e590f1fe-f50e-4ca1-a7cf-ac07ae240dba-istiod-ca-cert" pod "e590f1fe-f50e-4ca1-a7cf-ac07ae240dba" (UID: "e590f1fe-f50e-4ca1-a7cf-ac07ae240dba"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:46:56.292714 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:56.292323    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d507435-7115-4950-93c5-da6c04d73874-istio-token\" (UniqueName: \"kubernetes.io/projected/0d507435-7115-4950-93c5-da6c04d73874-istio-token\") pod \"0d507435-7115-4950-93c5-da6c04d73874\" (UID: \"0d507435-7115-4950-93c5-da6c04d73874\") "
Nov 02 12:46:56.292714 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:56.292373    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d507435-7115-4950-93c5-da6c04d73874-workload-socket" pod "0d507435-7115-4950-93c5-da6c04d73874" (UID: "0d507435-7115-4950-93c5-da6c04d73874"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:46:56.292714 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:56.292395    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e590f1fe-f50e-4ca1-a7cf-ac07ae240dba-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e590f1fe-f50e-4ca1-a7cf-ac07ae240dba-workload-certs\") pod \"e590f1fe-f50e-4ca1-a7cf-ac07ae240dba\" (UID: \"e590f1fe-f50e-4ca1-a7cf-ac07ae240dba\") "
Nov 02 12:46:56.292714 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:56.292431    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b5ce2e7-bb00-436e-bee3-2d87665168c3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b5ce2e7-bb00-436e-bee3-2d87665168c3-istiod-ca-cert\") pod \"0b5ce2e7-bb00-436e-bee3-2d87665168c3\" (UID: \"0b5ce2e7-bb00-436e-bee3-2d87665168c3\") "
Nov 02 12:46:56.292714 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:56.292490    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b5ce2e7-bb00-436e-bee3-2d87665168c3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b5ce2e7-bb00-436e-bee3-2d87665168c3-istio-data\") pod \"0b5ce2e7-bb00-436e-bee3-2d87665168c3\" (UID: \"0b5ce2e7-bb00-436e-bee3-2d87665168c3\") "
Nov 02 12:46:56.292714 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:56.292527    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b5ce2e7-bb00-436e-bee3-2d87665168c3-istio-token\" (UniqueName: \"kubernetes.io/projected/0b5ce2e7-bb00-436e-bee3-2d87665168c3-istio-token\") pod \"0b5ce2e7-bb00-436e-bee3-2d87665168c3\" (UID: \"0b5ce2e7-bb00-436e-bee3-2d87665168c3\") "
Nov 02 12:46:56.292714 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:56.292562    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b5ce2e7-bb00-436e-bee3-2d87665168c3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b5ce2e7-bb00-436e-bee3-2d87665168c3-istio-envoy\") pod \"0b5ce2e7-bb00-436e-bee3-2d87665168c3\" (UID: \"0b5ce2e7-bb00-436e-bee3-2d87665168c3\") "
Nov 02 12:46:56.292714 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:56.292571    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d507435-7115-4950-93c5-da6c04d73874-istio-data" pod "0d507435-7115-4950-93c5-da6c04d73874" (UID: "0d507435-7115-4950-93c5-da6c04d73874"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:46:56.292714 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:56.292593    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e590f1fe-f50e-4ca1-a7cf-ac07ae240dba-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e590f1fe-f50e-4ca1-a7cf-ac07ae240dba-workload-socket\") pod \"e590f1fe-f50e-4ca1-a7cf-ac07ae240dba\" (UID: \"e590f1fe-f50e-4ca1-a7cf-ac07ae240dba\") "
Nov 02 12:46:56.292714 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:56.292606    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d507435-7115-4950-93c5-da6c04d73874-workload-certs" pod "0d507435-7115-4950-93c5-da6c04d73874" (UID: "0d507435-7115-4950-93c5-da6c04d73874"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:46:56.292714 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:56.292623    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d507435-7115-4950-93c5-da6c04d73874-credential-socket" pod "0d507435-7115-4950-93c5-da6c04d73874" (UID: "0d507435-7115-4950-93c5-da6c04d73874"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:46:56.292714 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:56.292651    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e590f1fe-f50e-4ca1-a7cf-ac07ae240dba-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e590f1fe-f50e-4ca1-a7cf-ac07ae240dba-istio-podinfo\") pod \"e590f1fe-f50e-4ca1-a7cf-ac07ae240dba\" (UID: \"e590f1fe-f50e-4ca1-a7cf-ac07ae240dba\") "
Nov 02 12:46:56.292714 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:56.292713    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0b5ce2e7-bb00-436e-bee3-2d87665168c3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b5ce2e7-bb00-436e-bee3-2d87665168c3-istio-podinfo\") pod \"0b5ce2e7-bb00-436e-bee3-2d87665168c3\" (UID: \"0b5ce2e7-bb00-436e-bee3-2d87665168c3\") "
Nov 02 12:46:56.294545 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:56.292743    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b5ce2e7-bb00-436e-bee3-2d87665168c3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b5ce2e7-bb00-436e-bee3-2d87665168c3-credential-socket\") pod \"0b5ce2e7-bb00-436e-bee3-2d87665168c3\" (UID: \"0b5ce2e7-bb00-436e-bee3-2d87665168c3\") "
Nov 02 12:46:56.294545 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:56.292797    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b5ce2e7-bb00-436e-bee3-2d87665168c3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b5ce2e7-bb00-436e-bee3-2d87665168c3-workload-certs\") pod \"0b5ce2e7-bb00-436e-bee3-2d87665168c3\" (UID: \"0b5ce2e7-bb00-436e-bee3-2d87665168c3\") "
Nov 02 12:46:56.294545 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:56.292829    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b5ce2e7-bb00-436e-bee3-2d87665168c3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b5ce2e7-bb00-436e-bee3-2d87665168c3-workload-socket\") pod \"0b5ce2e7-bb00-436e-bee3-2d87665168c3\" (UID: \"0b5ce2e7-bb00-436e-bee3-2d87665168c3\") "
Nov 02 12:46:56.294545 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:56.292898    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d507435-7115-4950-93c5-da6c04d73874-kube-api-access-7zxfw\" (UniqueName: \"kubernetes.io/projected/0d507435-7115-4950-93c5-da6c04d73874-kube-api-access-7zxfw\") pod \"0d507435-7115-4950-93c5-da6c04d73874\" (UID: \"0d507435-7115-4950-93c5-da6c04d73874\") "
Nov 02 12:46:56.294545 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:56.292946    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d507435-7115-4950-93c5-da6c04d73874-istiod-ca-cert" pod "0d507435-7115-4950-93c5-da6c04d73874" (UID: "0d507435-7115-4950-93c5-da6c04d73874"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:46:56.294545 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:56.293094    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b5ce2e7-bb00-436e-bee3-2d87665168c3-istio-data" pod "0b5ce2e7-bb00-436e-bee3-2d87665168c3" (UID: "0b5ce2e7-bb00-436e-bee3-2d87665168c3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:46:56.294545 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:56.293121    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d507435-7115-4950-93c5-da6c04d73874-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d507435-7115-4950-93c5-da6c04d73874-istio-envoy\") pod \"0d507435-7115-4950-93c5-da6c04d73874\" (UID: \"0d507435-7115-4950-93c5-da6c04d73874\") "
Nov 02 12:46:56.294545 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:56.293200    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e590f1fe-f50e-4ca1-a7cf-ac07ae240dba-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e590f1fe-f50e-4ca1-a7cf-ac07ae240dba-istio-data\") pod \"e590f1fe-f50e-4ca1-a7cf-ac07ae240dba\" (UID: \"e590f1fe-f50e-4ca1-a7cf-ac07ae240dba\") "
Nov 02 12:46:56.294545 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:56.293259    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b5ce2e7-bb00-436e-bee3-2d87665168c3-kube-api-access-nnvf5\" (UniqueName: \"kubernetes.io/projected/0b5ce2e7-bb00-436e-bee3-2d87665168c3-kube-api-access-nnvf5\") pod \"0b5ce2e7-bb00-436e-bee3-2d87665168c3\" (UID: \"0b5ce2e7-bb00-436e-bee3-2d87665168c3\") "
Nov 02 12:46:56.294545 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:56.293327    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0d507435-7115-4950-93c5-da6c04d73874-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d507435-7115-4950-93c5-da6c04d73874-istio-podinfo\") pod \"0d507435-7115-4950-93c5-da6c04d73874\" (UID: \"0d507435-7115-4950-93c5-da6c04d73874\") "
Nov 02 12:46:56.294545 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:56.294200    4559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e590f1fe-f50e-4ca1-a7cf-ac07ae240dba-istiod-ca-cert\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:46:56.294545 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:56.294224    4559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e590f1fe-f50e-4ca1-a7cf-ac07ae240dba-credential-socket\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:46:56.294545 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:56.294234    4559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d507435-7115-4950-93c5-da6c04d73874-istio-data\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:46:56.294545 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:56.294246    4559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d507435-7115-4950-93c5-da6c04d73874-workload-socket\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:46:56.294545 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:56.294258    4559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d507435-7115-4950-93c5-da6c04d73874-istiod-ca-cert\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:46:56.294545 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:56.294267    4559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d507435-7115-4950-93c5-da6c04d73874-credential-socket\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:46:56.294545 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:56.294275    4559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d507435-7115-4950-93c5-da6c04d73874-workload-certs\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:46:56.294545 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:56.294287    4559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b5ce2e7-bb00-436e-bee3-2d87665168c3-istio-data\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:46:56.297256 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:56.293280    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e590f1fe-f50e-4ca1-a7cf-ac07ae240dba-workload-certs" pod "e590f1fe-f50e-4ca1-a7cf-ac07ae240dba" (UID: "e590f1fe-f50e-4ca1-a7cf-ac07ae240dba"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:46:56.297256 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:56.293572    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b5ce2e7-bb00-436e-bee3-2d87665168c3-istiod-ca-cert" pod "0b5ce2e7-bb00-436e-bee3-2d87665168c3" (UID: "0b5ce2e7-bb00-436e-bee3-2d87665168c3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:46:56.297256 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:56.293797    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e590f1fe-f50e-4ca1-a7cf-ac07ae240dba-istio-data" pod "e590f1fe-f50e-4ca1-a7cf-ac07ae240dba" (UID: "e590f1fe-f50e-4ca1-a7cf-ac07ae240dba"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:46:56.297256 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:56.296624    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e590f1fe-f50e-4ca1-a7cf-ac07ae240dba-istio-envoy" pod "e590f1fe-f50e-4ca1-a7cf-ac07ae240dba" (UID: "e590f1fe-f50e-4ca1-a7cf-ac07ae240dba"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:46:56.297256 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:56.297187    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b5ce2e7-bb00-436e-bee3-2d87665168c3-credential-socket" pod "0b5ce2e7-bb00-436e-bee3-2d87665168c3" (UID: "0b5ce2e7-bb00-436e-bee3-2d87665168c3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:46:56.297256 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:56.297206    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b5ce2e7-bb00-436e-bee3-2d87665168c3-workload-certs" pod "0b5ce2e7-bb00-436e-bee3-2d87665168c3" (UID: "0b5ce2e7-bb00-436e-bee3-2d87665168c3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:46:56.297256 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:56.296778    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d507435-7115-4950-93c5-da6c04d73874-kube-api-access-7zxfw" pod "0d507435-7115-4950-93c5-da6c04d73874" (UID: "0d507435-7115-4950-93c5-da6c04d73874"). InnerVolumeSpecName "kube-api-access-7zxfw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:46:56.297256 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:56.296787    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d507435-7115-4950-93c5-da6c04d73874-istio-envoy" pod "0d507435-7115-4950-93c5-da6c04d73874" (UID: "0d507435-7115-4950-93c5-da6c04d73874"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:46:56.297256 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:56.296980    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b5ce2e7-bb00-436e-bee3-2d87665168c3-istio-envoy" pod "0b5ce2e7-bb00-436e-bee3-2d87665168c3" (UID: "0b5ce2e7-bb00-436e-bee3-2d87665168c3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:46:56.297256 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:56.297237    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b5ce2e7-bb00-436e-bee3-2d87665168c3-workload-socket" pod "0b5ce2e7-bb00-436e-bee3-2d87665168c3" (UID: "0b5ce2e7-bb00-436e-bee3-2d87665168c3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:46:56.297256 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:56.296988    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e590f1fe-f50e-4ca1-a7cf-ac07ae240dba-workload-socket" pod "e590f1fe-f50e-4ca1-a7cf-ac07ae240dba" (UID: "e590f1fe-f50e-4ca1-a7cf-ac07ae240dba"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:46:56.297930 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:56.297031    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e590f1fe-f50e-4ca1-a7cf-ac07ae240dba-istio-token" pod "e590f1fe-f50e-4ca1-a7cf-ac07ae240dba" (UID: "e590f1fe-f50e-4ca1-a7cf-ac07ae240dba"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:46:56.297930 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:56.297047    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d507435-7115-4950-93c5-da6c04d73874-istio-token" pod "0d507435-7115-4950-93c5-da6c04d73874" (UID: "0d507435-7115-4950-93c5-da6c04d73874"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:46:56.297930 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:56.297062    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e590f1fe-f50e-4ca1-a7cf-ac07ae240dba-kube-api-access-rvbh4" pod "e590f1fe-f50e-4ca1-a7cf-ac07ae240dba" (UID: "e590f1fe-f50e-4ca1-a7cf-ac07ae240dba"). InnerVolumeSpecName "kube-api-access-rvbh4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:46:56.297930 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:56.297881    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0d507435-7115-4950-93c5-da6c04d73874-istio-podinfo" pod "0d507435-7115-4950-93c5-da6c04d73874" (UID: "0d507435-7115-4950-93c5-da6c04d73874"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 12:46:56.298689 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:56.298633    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b5ce2e7-bb00-436e-bee3-2d87665168c3-kube-api-access-nnvf5" pod "0b5ce2e7-bb00-436e-bee3-2d87665168c3" (UID: "0b5ce2e7-bb00-436e-bee3-2d87665168c3"). InnerVolumeSpecName "kube-api-access-nnvf5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:46:56.298919 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:56.298895    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b5ce2e7-bb00-436e-bee3-2d87665168c3-istio-token" pod "0b5ce2e7-bb00-436e-bee3-2d87665168c3" (UID: "0b5ce2e7-bb00-436e-bee3-2d87665168c3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:46:56.299188 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:56.299167    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0b5ce2e7-bb00-436e-bee3-2d87665168c3-istio-podinfo" pod "0b5ce2e7-bb00-436e-bee3-2d87665168c3" (UID: "0b5ce2e7-bb00-436e-bee3-2d87665168c3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 12:46:56.299353 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:56.299333    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e590f1fe-f50e-4ca1-a7cf-ac07ae240dba-istio-podinfo" pod "e590f1fe-f50e-4ca1-a7cf-ac07ae240dba" (UID: "e590f1fe-f50e-4ca1-a7cf-ac07ae240dba"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 12:46:56.396579 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:56.396493    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7zxfw\" (UniqueName: \"kubernetes.io/projected/0d507435-7115-4950-93c5-da6c04d73874-kube-api-access-7zxfw\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:46:56.396579 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:56.396534    4559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d507435-7115-4950-93c5-da6c04d73874-istio-envoy\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:46:56.396579 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:56.396547    4559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e590f1fe-f50e-4ca1-a7cf-ac07ae240dba-istio-data\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:46:56.396579 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:56.396561    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nnvf5\" (UniqueName: \"kubernetes.io/projected/0b5ce2e7-bb00-436e-bee3-2d87665168c3-kube-api-access-nnvf5\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:46:56.396579 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:56.396575    4559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d507435-7115-4950-93c5-da6c04d73874-istio-podinfo\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:46:56.396870 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:56.396588    4559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e590f1fe-f50e-4ca1-a7cf-ac07ae240dba-istio-token\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:46:56.396870 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:56.396601    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvbh4\" (UniqueName: \"kubernetes.io/projected/e590f1fe-f50e-4ca1-a7cf-ac07ae240dba-kube-api-access-rvbh4\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:46:56.396870 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:56.396615    4559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e590f1fe-f50e-4ca1-a7cf-ac07ae240dba-istio-envoy\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:46:56.396870 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:56.396628    4559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d507435-7115-4950-93c5-da6c04d73874-istio-token\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:46:56.396870 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:56.396658    4559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e590f1fe-f50e-4ca1-a7cf-ac07ae240dba-workload-certs\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:46:56.396870 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:56.396671    4559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b5ce2e7-bb00-436e-bee3-2d87665168c3-istiod-ca-cert\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:46:56.396870 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:56.396684    4559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b5ce2e7-bb00-436e-bee3-2d87665168c3-istio-token\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:46:56.396870 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:56.396695    4559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b5ce2e7-bb00-436e-bee3-2d87665168c3-istio-envoy\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:46:56.396870 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:56.396709    4559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e590f1fe-f50e-4ca1-a7cf-ac07ae240dba-workload-socket\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:46:56.396870 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:56.396720    4559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e590f1fe-f50e-4ca1-a7cf-ac07ae240dba-istio-podinfo\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:46:56.396870 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:56.396735    4559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b5ce2e7-bb00-436e-bee3-2d87665168c3-istio-podinfo\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:46:56.396870 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:56.396749    4559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b5ce2e7-bb00-436e-bee3-2d87665168c3-credential-socket\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:46:56.396870 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:56.396762    4559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b5ce2e7-bb00-436e-bee3-2d87665168c3-workload-certs\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:46:56.396870 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:56.396776    4559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b5ce2e7-bb00-436e-bee3-2d87665168c3-workload-socket\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:46:56.903538 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:56.903498    4559 generic.go:356] "Generic (PLEG): container finished" podID="0d507435-7115-4950-93c5-da6c04d73874" containerID="cfe4f960ff871c43450fc355da9c07406069824e196b0dd85d5b0af9a7300eb4" exitCode=0
Nov 02 12:46:56.903696 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:56.903560    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-gr5tx"
Nov 02 12:46:56.903696 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:56.903591    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-gr5tx" event={"ID":"0d507435-7115-4950-93c5-da6c04d73874","Type":"ContainerDied","Data":"cfe4f960ff871c43450fc355da9c07406069824e196b0dd85d5b0af9a7300eb4"}
Nov 02 12:46:56.903696 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:56.903681    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-gr5tx" event={"ID":"0d507435-7115-4950-93c5-da6c04d73874","Type":"ContainerDied","Data":"c1fbd53aa5b83a82139ffc946e01896498046df3c77d4e136ba2776d14ec7200"}
Nov 02 12:46:56.903959 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:56.903715    4559 scope.go:122] "RemoveContainer" containerID="cfe4f960ff871c43450fc355da9c07406069824e196b0dd85d5b0af9a7300eb4"
Nov 02 12:46:56.905340 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:56.905314    4559 generic.go:356] "Generic (PLEG): container finished" podID="e590f1fe-f50e-4ca1-a7cf-ac07ae240dba" containerID="49f1142f2c555ccb65a3759cf33dc628efcec7ec6c26d37107392a0e764c636d" exitCode=0
Nov 02 12:46:56.905446 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:56.905383    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-2hpjf"
Nov 02 12:46:56.905446 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:56.905407    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-2hpjf" event={"ID":"e590f1fe-f50e-4ca1-a7cf-ac07ae240dba","Type":"ContainerDied","Data":"49f1142f2c555ccb65a3759cf33dc628efcec7ec6c26d37107392a0e764c636d"}
Nov 02 12:46:56.905558 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:56.905458    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-2hpjf" event={"ID":"e590f1fe-f50e-4ca1-a7cf-ac07ae240dba","Type":"ContainerDied","Data":"e786d443cccac8e794025fa066438b0068c0675a0fb459b8fec8fc7e10cadd1c"}
Nov 02 12:46:56.912843 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:56.912822    4559 generic.go:356] "Generic (PLEG): container finished" podID="0b5ce2e7-bb00-436e-bee3-2d87665168c3" containerID="75dc888d7d589386f2c45149c17b2d94d74071e7184e670f201ecd78ab04e560" exitCode=0
Nov 02 12:46:56.912950 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:56.912894    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-tlgjz"
Nov 02 12:46:56.912950 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:56.912921    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-tlgjz" event={"ID":"0b5ce2e7-bb00-436e-bee3-2d87665168c3","Type":"ContainerDied","Data":"75dc888d7d589386f2c45149c17b2d94d74071e7184e670f201ecd78ab04e560"}
Nov 02 12:46:56.913070 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:56.912974    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-tlgjz" event={"ID":"0b5ce2e7-bb00-436e-bee3-2d87665168c3","Type":"ContainerDied","Data":"2544181f85828539114340665b59ee5f0a0c93d7ca77f20324ef1c5ec7b9eb2d"}
Nov 02 12:46:56.920041 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:56.920026    4559 scope.go:122] "RemoveContainer" containerID="cfe4f960ff871c43450fc355da9c07406069824e196b0dd85d5b0af9a7300eb4"
Nov 02 12:46:56.920351 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:46:56.920332    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cfe4f960ff871c43450fc355da9c07406069824e196b0dd85d5b0af9a7300eb4\": container with ID starting with cfe4f960ff871c43450fc355da9c07406069824e196b0dd85d5b0af9a7300eb4 not found: ID does not exist" containerID="cfe4f960ff871c43450fc355da9c07406069824e196b0dd85d5b0af9a7300eb4"
Nov 02 12:46:56.920407 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:56.920358    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cfe4f960ff871c43450fc355da9c07406069824e196b0dd85d5b0af9a7300eb4"} err="failed to get container status \"cfe4f960ff871c43450fc355da9c07406069824e196b0dd85d5b0af9a7300eb4\": rpc error: code = NotFound desc = could not find container \"cfe4f960ff871c43450fc355da9c07406069824e196b0dd85d5b0af9a7300eb4\": container with ID starting with cfe4f960ff871c43450fc355da9c07406069824e196b0dd85d5b0af9a7300eb4 not found: ID does not exist"
Nov 02 12:46:56.920407 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:56.920383    4559 scope.go:122] "RemoveContainer" containerID="49f1142f2c555ccb65a3759cf33dc628efcec7ec6c26d37107392a0e764c636d"
Nov 02 12:46:56.929755 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:56.929734    4559 scope.go:122] "RemoveContainer" containerID="49f1142f2c555ccb65a3759cf33dc628efcec7ec6c26d37107392a0e764c636d"
Nov 02 12:46:56.930023 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:46:56.930004    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49f1142f2c555ccb65a3759cf33dc628efcec7ec6c26d37107392a0e764c636d\": container with ID starting with 49f1142f2c555ccb65a3759cf33dc628efcec7ec6c26d37107392a0e764c636d not found: ID does not exist" containerID="49f1142f2c555ccb65a3759cf33dc628efcec7ec6c26d37107392a0e764c636d"
Nov 02 12:46:56.930084 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:56.930029    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49f1142f2c555ccb65a3759cf33dc628efcec7ec6c26d37107392a0e764c636d"} err="failed to get container status \"49f1142f2c555ccb65a3759cf33dc628efcec7ec6c26d37107392a0e764c636d\": rpc error: code = NotFound desc = could not find container \"49f1142f2c555ccb65a3759cf33dc628efcec7ec6c26d37107392a0e764c636d\": container with ID starting with 49f1142f2c555ccb65a3759cf33dc628efcec7ec6c26d37107392a0e764c636d not found: ID does not exist"
Nov 02 12:46:56.930084 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:56.930046    4559 scope.go:122] "RemoveContainer" containerID="75dc888d7d589386f2c45149c17b2d94d74071e7184e670f201ecd78ab04e560"
Nov 02 12:46:56.939221 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:56.939206    4559 scope.go:122] "RemoveContainer" containerID="75dc888d7d589386f2c45149c17b2d94d74071e7184e670f201ecd78ab04e560"
Nov 02 12:46:56.939473 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:46:56.939455    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75dc888d7d589386f2c45149c17b2d94d74071e7184e670f201ecd78ab04e560\": container with ID starting with 75dc888d7d589386f2c45149c17b2d94d74071e7184e670f201ecd78ab04e560 not found: ID does not exist" containerID="75dc888d7d589386f2c45149c17b2d94d74071e7184e670f201ecd78ab04e560"
Nov 02 12:46:56.939519 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:56.939479    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75dc888d7d589386f2c45149c17b2d94d74071e7184e670f201ecd78ab04e560"} err="failed to get container status \"75dc888d7d589386f2c45149c17b2d94d74071e7184e670f201ecd78ab04e560\": rpc error: code = NotFound desc = could not find container \"75dc888d7d589386f2c45149c17b2d94d74071e7184e670f201ecd78ab04e560\": container with ID starting with 75dc888d7d589386f2c45149c17b2d94d74071e7184e670f201ecd78ab04e560 not found: ID does not exist"
Nov 02 12:46:56.961860 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:56.961789    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-2hpjf"]
Nov 02 12:46:56.988698 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:56.977036    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-2hpjf"]
Nov 02 12:46:57.032003 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:57.031971    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-gr5tx"]
Nov 02 12:46:57.042268 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:57.042240    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-gr5tx"]
Nov 02 12:46:57.075220 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:57.075183    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-tlgjz"]
Nov 02 12:46:57.080341 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:57.080317    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-tlgjz"]
Nov 02 12:46:57.894427 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:57.894400    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b5ce2e7-bb00-436e-bee3-2d87665168c3" path="/var/lib/kubelet/pods/0b5ce2e7-bb00-436e-bee3-2d87665168c3/volumes"
Nov 02 12:46:57.894877 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:57.894862    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d507435-7115-4950-93c5-da6c04d73874" path="/var/lib/kubelet/pods/0d507435-7115-4950-93c5-da6c04d73874/volumes"
Nov 02 12:46:57.895295 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:57.895283    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e590f1fe-f50e-4ca1-a7cf-ac07ae240dba" path="/var/lib/kubelet/pods/e590f1fe-f50e-4ca1-a7cf-ac07ae240dba/volumes"
Nov 02 12:46:58.344368 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:58.344334    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-64thh"]
Nov 02 12:46:58.344606 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:58.344580    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-64thh" podUID="2a48a9d0-407c-4149-abe9-c675e2092854" containerName="istio-proxy" containerID="cri-o://e20f09afcc45a8e094c97b89927ccd8871b454591c60fb9df4705aa727954bbe" gracePeriod=30
Nov 02 12:46:58.356559 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:58.356512    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-nkmdl"]
Nov 02 12:46:58.356816 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:58.356788    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-nkmdl" podUID="13a4f88f-a38d-4fa4-ad71-8ec24dc429a5" containerName="istio-proxy" containerID="cri-o://9f943f38651bea528490600460e64b2ca8395880a34c6a115a54b6d40f9d8e09" gracePeriod=30
Nov 02 12:46:58.361291 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:58.361267    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-4xgzk"]
Nov 02 12:46:58.361450 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:46:58.361431    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-4xgzk" podUID="18cc5afa-245b-43f6-b92c-deec76232bd7" containerName="istio-proxy" containerID="cri-o://d56acd804fb207eaf32dce09b0f8e3b3579c7a87d8550a4b90b3cc1ee8b4b74c" gracePeriod=30
Nov 02 12:47:00.542662 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:00.542330    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"]
Nov 02 12:47:03.767255 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:03.767220    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-64thh"
Nov 02 12:47:03.860004 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:03.859971    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-nkmdl"
Nov 02 12:47:03.861995 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:03.861971    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-4xgzk"
Nov 02 12:47:03.920089 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:03.918423    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a48a9d0-407c-4149-abe9-c675e2092854-istiod-ca-cert" pod "2a48a9d0-407c-4149-abe9-c675e2092854" (UID: "2a48a9d0-407c-4149-abe9-c675e2092854"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:47:03.920089 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:03.918489    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a48a9d0-407c-4149-abe9-c675e2092854-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a48a9d0-407c-4149-abe9-c675e2092854-istiod-ca-cert\") pod \"2a48a9d0-407c-4149-abe9-c675e2092854\" (UID: \"2a48a9d0-407c-4149-abe9-c675e2092854\") "
Nov 02 12:47:03.920089 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:03.918590    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2a48a9d0-407c-4149-abe9-c675e2092854-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a48a9d0-407c-4149-abe9-c675e2092854-istio-podinfo\") pod \"2a48a9d0-407c-4149-abe9-c675e2092854\" (UID: \"2a48a9d0-407c-4149-abe9-c675e2092854\") "
Nov 02 12:47:03.920089 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:03.918690    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a48a9d0-407c-4149-abe9-c675e2092854-kube-api-access-mzzgb\" (UniqueName: \"kubernetes.io/projected/2a48a9d0-407c-4149-abe9-c675e2092854-kube-api-access-mzzgb\") pod \"2a48a9d0-407c-4149-abe9-c675e2092854\" (UID: \"2a48a9d0-407c-4149-abe9-c675e2092854\") "
Nov 02 12:47:03.920089 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:03.918727    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a48a9d0-407c-4149-abe9-c675e2092854-istio-token\" (UniqueName: \"kubernetes.io/projected/2a48a9d0-407c-4149-abe9-c675e2092854-istio-token\") pod \"2a48a9d0-407c-4149-abe9-c675e2092854\" (UID: \"2a48a9d0-407c-4149-abe9-c675e2092854\") "
Nov 02 12:47:03.920089 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:03.918785    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a48a9d0-407c-4149-abe9-c675e2092854-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a48a9d0-407c-4149-abe9-c675e2092854-istio-data\") pod \"2a48a9d0-407c-4149-abe9-c675e2092854\" (UID: \"2a48a9d0-407c-4149-abe9-c675e2092854\") "
Nov 02 12:47:03.920089 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:03.918944    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a48a9d0-407c-4149-abe9-c675e2092854-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a48a9d0-407c-4149-abe9-c675e2092854-credential-socket\") pod \"2a48a9d0-407c-4149-abe9-c675e2092854\" (UID: \"2a48a9d0-407c-4149-abe9-c675e2092854\") "
Nov 02 12:47:03.920089 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:03.919012    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a48a9d0-407c-4149-abe9-c675e2092854-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a48a9d0-407c-4149-abe9-c675e2092854-workload-certs\") pod \"2a48a9d0-407c-4149-abe9-c675e2092854\" (UID: \"2a48a9d0-407c-4149-abe9-c675e2092854\") "
Nov 02 12:47:03.920089 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:03.919146    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a48a9d0-407c-4149-abe9-c675e2092854-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a48a9d0-407c-4149-abe9-c675e2092854-istio-envoy\") pod \"2a48a9d0-407c-4149-abe9-c675e2092854\" (UID: \"2a48a9d0-407c-4149-abe9-c675e2092854\") "
Nov 02 12:47:03.920089 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:03.919294    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a48a9d0-407c-4149-abe9-c675e2092854-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a48a9d0-407c-4149-abe9-c675e2092854-workload-socket\") pod \"2a48a9d0-407c-4149-abe9-c675e2092854\" (UID: \"2a48a9d0-407c-4149-abe9-c675e2092854\") "
Nov 02 12:47:03.920735 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:03.920352    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a48a9d0-407c-4149-abe9-c675e2092854-workload-socket" pod "2a48a9d0-407c-4149-abe9-c675e2092854" (UID: "2a48a9d0-407c-4149-abe9-c675e2092854"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:47:03.921743 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:03.921713    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a48a9d0-407c-4149-abe9-c675e2092854-istio-data" pod "2a48a9d0-407c-4149-abe9-c675e2092854" (UID: "2a48a9d0-407c-4149-abe9-c675e2092854"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:47:03.922724 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:03.922698    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a48a9d0-407c-4149-abe9-c675e2092854-workload-certs" pod "2a48a9d0-407c-4149-abe9-c675e2092854" (UID: "2a48a9d0-407c-4149-abe9-c675e2092854"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:47:03.923011 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:03.922990    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a48a9d0-407c-4149-abe9-c675e2092854-credential-socket" pod "2a48a9d0-407c-4149-abe9-c675e2092854" (UID: "2a48a9d0-407c-4149-abe9-c675e2092854"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:47:03.923102 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:03.923038    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2a48a9d0-407c-4149-abe9-c675e2092854-istio-podinfo" pod "2a48a9d0-407c-4149-abe9-c675e2092854" (UID: "2a48a9d0-407c-4149-abe9-c675e2092854"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 12:47:03.924440 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:03.924417    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a48a9d0-407c-4149-abe9-c675e2092854-kube-api-access-mzzgb" pod "2a48a9d0-407c-4149-abe9-c675e2092854" (UID: "2a48a9d0-407c-4149-abe9-c675e2092854"). InnerVolumeSpecName "kube-api-access-mzzgb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:47:03.925316 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:03.925289    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a48a9d0-407c-4149-abe9-c675e2092854-istio-envoy" pod "2a48a9d0-407c-4149-abe9-c675e2092854" (UID: "2a48a9d0-407c-4149-abe9-c675e2092854"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:47:03.925768 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:03.925742    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a48a9d0-407c-4149-abe9-c675e2092854-istio-token" pod "2a48a9d0-407c-4149-abe9-c675e2092854" (UID: "2a48a9d0-407c-4149-abe9-c675e2092854"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:47:04.021422 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:04.021339    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13a4f88f-a38d-4fa4-ad71-8ec24dc429a5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13a4f88f-a38d-4fa4-ad71-8ec24dc429a5-istiod-ca-cert\") pod \"13a4f88f-a38d-4fa4-ad71-8ec24dc429a5\" (UID: \"13a4f88f-a38d-4fa4-ad71-8ec24dc429a5\") "
Nov 02 12:47:04.021422 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:04.021384    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18cc5afa-245b-43f6-b92c-deec76232bd7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18cc5afa-245b-43f6-b92c-deec76232bd7-workload-certs\") pod \"18cc5afa-245b-43f6-b92c-deec76232bd7\" (UID: \"18cc5afa-245b-43f6-b92c-deec76232bd7\") "
Nov 02 12:47:04.021422 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:04.021404    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13a4f88f-a38d-4fa4-ad71-8ec24dc429a5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13a4f88f-a38d-4fa4-ad71-8ec24dc429a5-istio-data\") pod \"13a4f88f-a38d-4fa4-ad71-8ec24dc429a5\" (UID: \"13a4f88f-a38d-4fa4-ad71-8ec24dc429a5\") "
Nov 02 12:47:04.021772 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:04.021428    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18cc5afa-245b-43f6-b92c-deec76232bd7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18cc5afa-245b-43f6-b92c-deec76232bd7-credential-socket\") pod \"18cc5afa-245b-43f6-b92c-deec76232bd7\" (UID: \"18cc5afa-245b-43f6-b92c-deec76232bd7\") "
Nov 02 12:47:04.021772 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:04.021449    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13a4f88f-a38d-4fa4-ad71-8ec24dc429a5-istio-token\" (UniqueName: \"kubernetes.io/projected/13a4f88f-a38d-4fa4-ad71-8ec24dc429a5-istio-token\") pod \"13a4f88f-a38d-4fa4-ad71-8ec24dc429a5\" (UID: \"13a4f88f-a38d-4fa4-ad71-8ec24dc429a5\") "
Nov 02 12:47:04.021772 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:04.021475    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18cc5afa-245b-43f6-b92c-deec76232bd7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18cc5afa-245b-43f6-b92c-deec76232bd7-istio-envoy\") pod \"18cc5afa-245b-43f6-b92c-deec76232bd7\" (UID: \"18cc5afa-245b-43f6-b92c-deec76232bd7\") "
Nov 02 12:47:04.021772 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:04.021520    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13a4f88f-a38d-4fa4-ad71-8ec24dc429a5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13a4f88f-a38d-4fa4-ad71-8ec24dc429a5-workload-certs\") pod \"13a4f88f-a38d-4fa4-ad71-8ec24dc429a5\" (UID: \"13a4f88f-a38d-4fa4-ad71-8ec24dc429a5\") "
Nov 02 12:47:04.021772 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:04.021575    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13a4f88f-a38d-4fa4-ad71-8ec24dc429a5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13a4f88f-a38d-4fa4-ad71-8ec24dc429a5-credential-socket\") pod \"13a4f88f-a38d-4fa4-ad71-8ec24dc429a5\" (UID: \"13a4f88f-a38d-4fa4-ad71-8ec24dc429a5\") "
Nov 02 12:47:04.021772 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:04.021593    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13a4f88f-a38d-4fa4-ad71-8ec24dc429a5-kube-api-access-tfqqx\" (UniqueName: \"kubernetes.io/projected/13a4f88f-a38d-4fa4-ad71-8ec24dc429a5-kube-api-access-tfqqx\") pod \"13a4f88f-a38d-4fa4-ad71-8ec24dc429a5\" (UID: \"13a4f88f-a38d-4fa4-ad71-8ec24dc429a5\") "
Nov 02 12:47:04.021772 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:04.021707    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/18cc5afa-245b-43f6-b92c-deec76232bd7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18cc5afa-245b-43f6-b92c-deec76232bd7-istio-podinfo\") pod \"18cc5afa-245b-43f6-b92c-deec76232bd7\" (UID: \"18cc5afa-245b-43f6-b92c-deec76232bd7\") "
Nov 02 12:47:04.021772 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:04.021743    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13a4f88f-a38d-4fa4-ad71-8ec24dc429a5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13a4f88f-a38d-4fa4-ad71-8ec24dc429a5-istio-envoy\") pod \"13a4f88f-a38d-4fa4-ad71-8ec24dc429a5\" (UID: \"13a4f88f-a38d-4fa4-ad71-8ec24dc429a5\") "
Nov 02 12:47:04.021772 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:04.021764    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/13a4f88f-a38d-4fa4-ad71-8ec24dc429a5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13a4f88f-a38d-4fa4-ad71-8ec24dc429a5-istio-podinfo\") pod \"13a4f88f-a38d-4fa4-ad71-8ec24dc429a5\" (UID: \"13a4f88f-a38d-4fa4-ad71-8ec24dc429a5\") "
Nov 02 12:47:04.022226 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:04.021808    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18cc5afa-245b-43f6-b92c-deec76232bd7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18cc5afa-245b-43f6-b92c-deec76232bd7-istiod-ca-cert\") pod \"18cc5afa-245b-43f6-b92c-deec76232bd7\" (UID: \"18cc5afa-245b-43f6-b92c-deec76232bd7\") "
Nov 02 12:47:04.022226 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:04.021851    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18cc5afa-245b-43f6-b92c-deec76232bd7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18cc5afa-245b-43f6-b92c-deec76232bd7-workload-socket\") pod \"18cc5afa-245b-43f6-b92c-deec76232bd7\" (UID: \"18cc5afa-245b-43f6-b92c-deec76232bd7\") "
Nov 02 12:47:04.022226 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:04.021894    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13a4f88f-a38d-4fa4-ad71-8ec24dc429a5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13a4f88f-a38d-4fa4-ad71-8ec24dc429a5-workload-socket\") pod \"13a4f88f-a38d-4fa4-ad71-8ec24dc429a5\" (UID: \"13a4f88f-a38d-4fa4-ad71-8ec24dc429a5\") "
Nov 02 12:47:04.022226 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:04.021941    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18cc5afa-245b-43f6-b92c-deec76232bd7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18cc5afa-245b-43f6-b92c-deec76232bd7-istio-data\") pod \"18cc5afa-245b-43f6-b92c-deec76232bd7\" (UID: \"18cc5afa-245b-43f6-b92c-deec76232bd7\") "
Nov 02 12:47:04.022226 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:04.021969    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18cc5afa-245b-43f6-b92c-deec76232bd7-istio-token\" (UniqueName: \"kubernetes.io/projected/18cc5afa-245b-43f6-b92c-deec76232bd7-istio-token\") pod \"18cc5afa-245b-43f6-b92c-deec76232bd7\" (UID: \"18cc5afa-245b-43f6-b92c-deec76232bd7\") "
Nov 02 12:47:04.022226 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:04.021997    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18cc5afa-245b-43f6-b92c-deec76232bd7-kube-api-access-wlghm\" (UniqueName: \"kubernetes.io/projected/18cc5afa-245b-43f6-b92c-deec76232bd7-kube-api-access-wlghm\") pod \"18cc5afa-245b-43f6-b92c-deec76232bd7\" (UID: \"18cc5afa-245b-43f6-b92c-deec76232bd7\") "
Nov 02 12:47:04.022537 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:04.022517    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mzzgb\" (UniqueName: \"kubernetes.io/projected/2a48a9d0-407c-4149-abe9-c675e2092854-kube-api-access-mzzgb\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:47:04.022537 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:04.022531    4559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a48a9d0-407c-4149-abe9-c675e2092854-istio-token\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:47:04.022664 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:04.022539    4559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a48a9d0-407c-4149-abe9-c675e2092854-istio-data\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:47:04.022664 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:04.022547    4559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a48a9d0-407c-4149-abe9-c675e2092854-credential-socket\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:47:04.022664 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:04.022555    4559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a48a9d0-407c-4149-abe9-c675e2092854-workload-certs\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:47:04.022664 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:04.022569    4559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a48a9d0-407c-4149-abe9-c675e2092854-istio-envoy\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:47:04.022664 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:04.022578    4559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a48a9d0-407c-4149-abe9-c675e2092854-workload-socket\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:47:04.022664 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:04.022586    4559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a48a9d0-407c-4149-abe9-c675e2092854-istiod-ca-cert\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:47:04.022664 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:04.022596    4559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a48a9d0-407c-4149-abe9-c675e2092854-istio-podinfo\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:47:04.023481 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:04.023278    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18cc5afa-245b-43f6-b92c-deec76232bd7-istiod-ca-cert" pod "18cc5afa-245b-43f6-b92c-deec76232bd7" (UID: "18cc5afa-245b-43f6-b92c-deec76232bd7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:47:04.025467 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:04.025246    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/18cc5afa-245b-43f6-b92c-deec76232bd7-istio-podinfo" pod "18cc5afa-245b-43f6-b92c-deec76232bd7" (UID: "18cc5afa-245b-43f6-b92c-deec76232bd7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 12:47:04.025582 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:04.025548    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13a4f88f-a38d-4fa4-ad71-8ec24dc429a5-kube-api-access-tfqqx" pod "13a4f88f-a38d-4fa4-ad71-8ec24dc429a5" (UID: "13a4f88f-a38d-4fa4-ad71-8ec24dc429a5"). InnerVolumeSpecName "kube-api-access-tfqqx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:47:04.025874 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:04.025854    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13a4f88f-a38d-4fa4-ad71-8ec24dc429a5-istiod-ca-cert" pod "13a4f88f-a38d-4fa4-ad71-8ec24dc429a5" (UID: "13a4f88f-a38d-4fa4-ad71-8ec24dc429a5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:47:04.026065 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:04.026048    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18cc5afa-245b-43f6-b92c-deec76232bd7-workload-certs" pod "18cc5afa-245b-43f6-b92c-deec76232bd7" (UID: "18cc5afa-245b-43f6-b92c-deec76232bd7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:47:04.026236 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:04.026219    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13a4f88f-a38d-4fa4-ad71-8ec24dc429a5-istio-data" pod "13a4f88f-a38d-4fa4-ad71-8ec24dc429a5" (UID: "13a4f88f-a38d-4fa4-ad71-8ec24dc429a5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:47:04.026384 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:04.026367    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18cc5afa-245b-43f6-b92c-deec76232bd7-credential-socket" pod "18cc5afa-245b-43f6-b92c-deec76232bd7" (UID: "18cc5afa-245b-43f6-b92c-deec76232bd7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:47:04.027624 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:04.027352    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13a4f88f-a38d-4fa4-ad71-8ec24dc429a5-istio-envoy" pod "13a4f88f-a38d-4fa4-ad71-8ec24dc429a5" (UID: "13a4f88f-a38d-4fa4-ad71-8ec24dc429a5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:47:04.027867 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:04.027844    4559 generic.go:356] "Generic (PLEG): container finished" podID="18cc5afa-245b-43f6-b92c-deec76232bd7" containerID="d56acd804fb207eaf32dce09b0f8e3b3579c7a87d8550a4b90b3cc1ee8b4b74c" exitCode=0
Nov 02 12:47:04.027961 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:04.027949    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-4xgzk"
Nov 02 12:47:04.028058 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:04.028036    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18cc5afa-245b-43f6-b92c-deec76232bd7-workload-socket" pod "18cc5afa-245b-43f6-b92c-deec76232bd7" (UID: "18cc5afa-245b-43f6-b92c-deec76232bd7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:47:04.028129 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:04.028100    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-4xgzk" event={"ID":"18cc5afa-245b-43f6-b92c-deec76232bd7","Type":"ContainerDied","Data":"d56acd804fb207eaf32dce09b0f8e3b3579c7a87d8550a4b90b3cc1ee8b4b74c"}
Nov 02 12:47:04.028188 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:04.028160    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-4xgzk" event={"ID":"18cc5afa-245b-43f6-b92c-deec76232bd7","Type":"ContainerDied","Data":"0dbc46ca2f98e28a6bafcdc807396519c6a46e5612e0108ec1586e09badb2378"}
Nov 02 12:47:04.028240 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:04.028192    4559 scope.go:122] "RemoveContainer" containerID="d56acd804fb207eaf32dce09b0f8e3b3579c7a87d8550a4b90b3cc1ee8b4b74c"
Nov 02 12:47:04.028579 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:04.028550    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13a4f88f-a38d-4fa4-ad71-8ec24dc429a5-workload-socket" pod "13a4f88f-a38d-4fa4-ad71-8ec24dc429a5" (UID: "13a4f88f-a38d-4fa4-ad71-8ec24dc429a5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:47:04.028717 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:04.028598    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18cc5afa-245b-43f6-b92c-deec76232bd7-istio-data" pod "18cc5afa-245b-43f6-b92c-deec76232bd7" (UID: "18cc5afa-245b-43f6-b92c-deec76232bd7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:47:04.028878 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:04.028853    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13a4f88f-a38d-4fa4-ad71-8ec24dc429a5-credential-socket" pod "13a4f88f-a38d-4fa4-ad71-8ec24dc429a5" (UID: "13a4f88f-a38d-4fa4-ad71-8ec24dc429a5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:47:04.030005 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:04.029978    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13a4f88f-a38d-4fa4-ad71-8ec24dc429a5-workload-certs" pod "13a4f88f-a38d-4fa4-ad71-8ec24dc429a5" (UID: "13a4f88f-a38d-4fa4-ad71-8ec24dc429a5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:47:04.031498 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:04.031177    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18cc5afa-245b-43f6-b92c-deec76232bd7-istio-envoy" pod "18cc5afa-245b-43f6-b92c-deec76232bd7" (UID: "18cc5afa-245b-43f6-b92c-deec76232bd7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:47:04.031498 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:04.031180    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/13a4f88f-a38d-4fa4-ad71-8ec24dc429a5-istio-podinfo" pod "13a4f88f-a38d-4fa4-ad71-8ec24dc429a5" (UID: "13a4f88f-a38d-4fa4-ad71-8ec24dc429a5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 12:47:04.031498 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:04.031226    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18cc5afa-245b-43f6-b92c-deec76232bd7-kube-api-access-wlghm" pod "18cc5afa-245b-43f6-b92c-deec76232bd7" (UID: "18cc5afa-245b-43f6-b92c-deec76232bd7"). InnerVolumeSpecName "kube-api-access-wlghm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:47:04.031498 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:04.031245    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13a4f88f-a38d-4fa4-ad71-8ec24dc429a5-istio-token" pod "13a4f88f-a38d-4fa4-ad71-8ec24dc429a5" (UID: "13a4f88f-a38d-4fa4-ad71-8ec24dc429a5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:47:04.032016 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:04.031992    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18cc5afa-245b-43f6-b92c-deec76232bd7-istio-token" pod "18cc5afa-245b-43f6-b92c-deec76232bd7" (UID: "18cc5afa-245b-43f6-b92c-deec76232bd7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:47:04.032313 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:04.032288    4559 generic.go:356] "Generic (PLEG): container finished" podID="13a4f88f-a38d-4fa4-ad71-8ec24dc429a5" containerID="9f943f38651bea528490600460e64b2ca8395880a34c6a115a54b6d40f9d8e09" exitCode=0
Nov 02 12:47:04.032409 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:04.032348    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-nkmdl"
Nov 02 12:47:04.032409 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:04.032374    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-nkmdl" event={"ID":"13a4f88f-a38d-4fa4-ad71-8ec24dc429a5","Type":"ContainerDied","Data":"9f943f38651bea528490600460e64b2ca8395880a34c6a115a54b6d40f9d8e09"}
Nov 02 12:47:04.032514 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:04.032428    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-nkmdl" event={"ID":"13a4f88f-a38d-4fa4-ad71-8ec24dc429a5","Type":"ContainerDied","Data":"742ebed4c38a928c0fd4e25a6d3c05388d6475fb293c5030e65c20803237df2b"}
Nov 02 12:47:04.042761 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:04.041948    4559 generic.go:356] "Generic (PLEG): container finished" podID="2a48a9d0-407c-4149-abe9-c675e2092854" containerID="e20f09afcc45a8e094c97b89927ccd8871b454591c60fb9df4705aa727954bbe" exitCode=0
Nov 02 12:47:04.042761 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:04.042013    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-64thh" event={"ID":"2a48a9d0-407c-4149-abe9-c675e2092854","Type":"ContainerDied","Data":"e20f09afcc45a8e094c97b89927ccd8871b454591c60fb9df4705aa727954bbe"}
Nov 02 12:47:04.042761 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:04.042051    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-64thh" event={"ID":"2a48a9d0-407c-4149-abe9-c675e2092854","Type":"ContainerDied","Data":"74eb292162364d97f724c58b67205f00418353e0610f1c8bcd631377ccba6423"}
Nov 02 12:47:04.042761 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:04.042119    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-64thh"
Nov 02 12:47:04.045802 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:04.045782    4559 scope.go:122] "RemoveContainer" containerID="d56acd804fb207eaf32dce09b0f8e3b3579c7a87d8550a4b90b3cc1ee8b4b74c"
Nov 02 12:47:04.046101 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:47:04.046076    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d56acd804fb207eaf32dce09b0f8e3b3579c7a87d8550a4b90b3cc1ee8b4b74c\": container with ID starting with d56acd804fb207eaf32dce09b0f8e3b3579c7a87d8550a4b90b3cc1ee8b4b74c not found: ID does not exist" containerID="d56acd804fb207eaf32dce09b0f8e3b3579c7a87d8550a4b90b3cc1ee8b4b74c"
Nov 02 12:47:04.046173 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:04.046111    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d56acd804fb207eaf32dce09b0f8e3b3579c7a87d8550a4b90b3cc1ee8b4b74c"} err="failed to get container status \"d56acd804fb207eaf32dce09b0f8e3b3579c7a87d8550a4b90b3cc1ee8b4b74c\": rpc error: code = NotFound desc = could not find container \"d56acd804fb207eaf32dce09b0f8e3b3579c7a87d8550a4b90b3cc1ee8b4b74c\": container with ID starting with d56acd804fb207eaf32dce09b0f8e3b3579c7a87d8550a4b90b3cc1ee8b4b74c not found: ID does not exist"
Nov 02 12:47:04.046173 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:04.046132    4559 scope.go:122] "RemoveContainer" containerID="9f943f38651bea528490600460e64b2ca8395880a34c6a115a54b6d40f9d8e09"
Nov 02 12:47:04.059635 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:04.059613    4559 scope.go:122] "RemoveContainer" containerID="9f943f38651bea528490600460e64b2ca8395880a34c6a115a54b6d40f9d8e09"
Nov 02 12:47:04.059986 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:47:04.059965    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f943f38651bea528490600460e64b2ca8395880a34c6a115a54b6d40f9d8e09\": container with ID starting with 9f943f38651bea528490600460e64b2ca8395880a34c6a115a54b6d40f9d8e09 not found: ID does not exist" containerID="9f943f38651bea528490600460e64b2ca8395880a34c6a115a54b6d40f9d8e09"
Nov 02 12:47:04.060075 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:04.059995    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f943f38651bea528490600460e64b2ca8395880a34c6a115a54b6d40f9d8e09"} err="failed to get container status \"9f943f38651bea528490600460e64b2ca8395880a34c6a115a54b6d40f9d8e09\": rpc error: code = NotFound desc = could not find container \"9f943f38651bea528490600460e64b2ca8395880a34c6a115a54b6d40f9d8e09\": container with ID starting with 9f943f38651bea528490600460e64b2ca8395880a34c6a115a54b6d40f9d8e09 not found: ID does not exist"
Nov 02 12:47:04.060075 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:04.060016    4559 scope.go:122] "RemoveContainer" containerID="e20f09afcc45a8e094c97b89927ccd8871b454591c60fb9df4705aa727954bbe"
Nov 02 12:47:04.069917 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:04.069896    4559 scope.go:122] "RemoveContainer" containerID="e20f09afcc45a8e094c97b89927ccd8871b454591c60fb9df4705aa727954bbe"
Nov 02 12:47:04.070255 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:47:04.070232    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e20f09afcc45a8e094c97b89927ccd8871b454591c60fb9df4705aa727954bbe\": container with ID starting with e20f09afcc45a8e094c97b89927ccd8871b454591c60fb9df4705aa727954bbe not found: ID does not exist" containerID="e20f09afcc45a8e094c97b89927ccd8871b454591c60fb9df4705aa727954bbe"
Nov 02 12:47:04.070336 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:04.070268    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e20f09afcc45a8e094c97b89927ccd8871b454591c60fb9df4705aa727954bbe"} err="failed to get container status \"e20f09afcc45a8e094c97b89927ccd8871b454591c60fb9df4705aa727954bbe\": rpc error: code = NotFound desc = could not find container \"e20f09afcc45a8e094c97b89927ccd8871b454591c60fb9df4705aa727954bbe\": container with ID starting with e20f09afcc45a8e094c97b89927ccd8871b454591c60fb9df4705aa727954bbe not found: ID does not exist"
Nov 02 12:47:04.123833 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:04.123801    4559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13a4f88f-a38d-4fa4-ad71-8ec24dc429a5-istiod-ca-cert\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:47:04.123833 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:04.123830    4559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18cc5afa-245b-43f6-b92c-deec76232bd7-workload-certs\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:47:04.123833 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:04.123841    4559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13a4f88f-a38d-4fa4-ad71-8ec24dc429a5-istio-data\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:47:04.124076 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:04.123849    4559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18cc5afa-245b-43f6-b92c-deec76232bd7-credential-socket\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:47:04.124076 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:04.123857    4559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13a4f88f-a38d-4fa4-ad71-8ec24dc429a5-istio-token\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:47:04.124076 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:04.123867    4559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18cc5afa-245b-43f6-b92c-deec76232bd7-istio-envoy\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:47:04.124076 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:04.123874    4559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13a4f88f-a38d-4fa4-ad71-8ec24dc429a5-workload-certs\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:47:04.124076 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:04.123882    4559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13a4f88f-a38d-4fa4-ad71-8ec24dc429a5-credential-socket\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:47:04.124076 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:04.123892    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tfqqx\" (UniqueName: \"kubernetes.io/projected/13a4f88f-a38d-4fa4-ad71-8ec24dc429a5-kube-api-access-tfqqx\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:47:04.124076 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:04.123901    4559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18cc5afa-245b-43f6-b92c-deec76232bd7-istio-podinfo\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:47:04.124076 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:04.123909    4559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13a4f88f-a38d-4fa4-ad71-8ec24dc429a5-istio-envoy\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:47:04.124076 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:04.123917    4559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13a4f88f-a38d-4fa4-ad71-8ec24dc429a5-istio-podinfo\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:47:04.124076 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:04.123925    4559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18cc5afa-245b-43f6-b92c-deec76232bd7-istiod-ca-cert\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:47:04.124076 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:04.123933    4559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18cc5afa-245b-43f6-b92c-deec76232bd7-workload-socket\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:47:04.124076 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:04.123941    4559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13a4f88f-a38d-4fa4-ad71-8ec24dc429a5-workload-socket\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:47:04.124076 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:04.123949    4559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18cc5afa-245b-43f6-b92c-deec76232bd7-istio-data\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:47:04.124076 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:04.123956    4559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18cc5afa-245b-43f6-b92c-deec76232bd7-istio-token\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:47:04.124076 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:04.123964    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wlghm\" (UniqueName: \"kubernetes.io/projected/18cc5afa-245b-43f6-b92c-deec76232bd7-kube-api-access-wlghm\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:47:04.141123 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:04.141090    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-nkmdl"]
Nov 02 12:47:04.145553 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:04.145530    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-nkmdl"]
Nov 02 12:47:04.182261 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:04.182224    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-64thh"]
Nov 02 12:47:04.210573 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:04.210537    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-64thh"]
Nov 02 12:47:04.409698 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:04.409597    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-4xgzk"]
Nov 02 12:47:04.412942 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:04.412912    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-4xgzk"]
Nov 02 12:47:05.453689 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:05.450768    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-078fd3-84495865c7-mqwt7"]
Nov 02 12:47:05.453689 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:05.451757    4559 cpu_manager.go:414] "containerMap: removing container" podUID="35f5e006-f020-4166-ab03-aedc1fbf2ce4" containerName="istio-proxy"
Nov 02 12:47:05.453689 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:05.451778    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35f5e006-f020-4166-ab03-aedc1fbf2ce4" containerName="istio-proxy"
Nov 02 12:47:05.453689 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:05.451796    4559 cpu_manager.go:414] "containerMap: removing container" podUID="e590f1fe-f50e-4ca1-a7cf-ac07ae240dba" containerName="istio-proxy"
Nov 02 12:47:05.453689 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:05.451808    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e590f1fe-f50e-4ca1-a7cf-ac07ae240dba" containerName="istio-proxy"
Nov 02 12:47:05.453689 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:05.451824    4559 cpu_manager.go:414] "containerMap: removing container" podUID="79ae08e3-f7c3-4e28-aabb-246f42f45b2e" containerName="istio-proxy"
Nov 02 12:47:05.453689 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:05.451831    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79ae08e3-f7c3-4e28-aabb-246f42f45b2e" containerName="istio-proxy"
Nov 02 12:47:05.453689 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:05.451849    4559 cpu_manager.go:414] "containerMap: removing container" podUID="0d507435-7115-4950-93c5-da6c04d73874" containerName="istio-proxy"
Nov 02 12:47:05.453689 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:05.451856    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d507435-7115-4950-93c5-da6c04d73874" containerName="istio-proxy"
Nov 02 12:47:05.453689 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:05.451891    4559 cpu_manager.go:414] "containerMap: removing container" podUID="18cc5afa-245b-43f6-b92c-deec76232bd7" containerName="istio-proxy"
Nov 02 12:47:05.453689 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:05.451900    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18cc5afa-245b-43f6-b92c-deec76232bd7" containerName="istio-proxy"
Nov 02 12:47:05.453689 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:05.451927    4559 cpu_manager.go:414] "containerMap: removing container" podUID="0b5ce2e7-bb00-436e-bee3-2d87665168c3" containerName="istio-proxy"
Nov 02 12:47:05.453689 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:05.451933    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b5ce2e7-bb00-436e-bee3-2d87665168c3" containerName="istio-proxy"
Nov 02 12:47:05.453689 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:05.451950    4559 cpu_manager.go:414] "containerMap: removing container" podUID="2a48a9d0-407c-4149-abe9-c675e2092854" containerName="istio-proxy"
Nov 02 12:47:05.453689 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:05.451957    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a48a9d0-407c-4149-abe9-c675e2092854" containerName="istio-proxy"
Nov 02 12:47:05.453689 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:05.451977    4559 cpu_manager.go:414] "containerMap: removing container" podUID="13a4f88f-a38d-4fa4-ad71-8ec24dc429a5" containerName="istio-proxy"
Nov 02 12:47:05.453689 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:05.451984    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13a4f88f-a38d-4fa4-ad71-8ec24dc429a5" containerName="istio-proxy"
Nov 02 12:47:05.453689 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:05.452129    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="18cc5afa-245b-43f6-b92c-deec76232bd7" containerName="istio-proxy"
Nov 02 12:47:05.453689 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:05.452148    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b5ce2e7-bb00-436e-bee3-2d87665168c3" containerName="istio-proxy"
Nov 02 12:47:05.453689 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:05.452157    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a48a9d0-407c-4149-abe9-c675e2092854" containerName="istio-proxy"
Nov 02 12:47:05.453689 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:05.452167    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="13a4f88f-a38d-4fa4-ad71-8ec24dc429a5" containerName="istio-proxy"
Nov 02 12:47:05.453689 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:05.452176    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="35f5e006-f020-4166-ab03-aedc1fbf2ce4" containerName="istio-proxy"
Nov 02 12:47:05.453689 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:05.452184    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="e590f1fe-f50e-4ca1-a7cf-ac07ae240dba" containerName="istio-proxy"
Nov 02 12:47:05.453689 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:05.452193    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="79ae08e3-f7c3-4e28-aabb-246f42f45b2e" containerName="istio-proxy"
Nov 02 12:47:05.453689 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:05.452201    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d507435-7115-4950-93c5-da6c04d73874" containerName="istio-proxy"
Nov 02 12:47:05.460733 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:05.459524    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-078fd3-84495865c7-mqwt7"
Nov 02 12:47:05.460733 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:05.459839    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-078fd3-84495865c7-mqwt7"]
Nov 02 12:47:05.467816 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:05.467296    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc3bedf6-b508-4297-9fe5-0cd87fe6dd1a-maas-api-tls\") pod \"maas-api-e2e-ipp-b-078fd3-84495865c7-mqwt7\" (UID: \"fc3bedf6-b508-4297-9fe5-0cd87fe6dd1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-078fd3-84495865c7-mqwt7"
Nov 02 12:47:05.467816 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:05.467372    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc3bedf6-b508-4297-9fe5-0cd87fe6dd1a-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-078fd3-84495865c7-mqwt7\" (UID: \"fc3bedf6-b508-4297-9fe5-0cd87fe6dd1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-078fd3-84495865c7-mqwt7"
Nov 02 12:47:05.467816 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:05.467546    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc3bedf6-b508-4297-9fe5-0cd87fe6dd1a-metrics-certs\") pod \"maas-api-e2e-ipp-b-078fd3-84495865c7-mqwt7\" (UID: \"fc3bedf6-b508-4297-9fe5-0cd87fe6dd1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-078fd3-84495865c7-mqwt7"
Nov 02 12:47:05.467816 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:05.467587    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rttf\" (UniqueName: \"kubernetes.io/projected/fc3bedf6-b508-4297-9fe5-0cd87fe6dd1a-kube-api-access-8rttf\") pod \"maas-api-e2e-ipp-b-078fd3-84495865c7-mqwt7\" (UID: \"fc3bedf6-b508-4297-9fe5-0cd87fe6dd1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-078fd3-84495865c7-mqwt7"
Nov 02 12:47:05.476091 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:05.476061    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-078fd3-serving-cert\""
Nov 02 12:47:05.641676 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:05.638701    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc3bedf6-b508-4297-9fe5-0cd87fe6dd1a-maas-api-tls\") pod \"maas-api-e2e-ipp-b-078fd3-84495865c7-mqwt7\" (UID: \"fc3bedf6-b508-4297-9fe5-0cd87fe6dd1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-078fd3-84495865c7-mqwt7"
Nov 02 12:47:05.641676 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:05.638809    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc3bedf6-b508-4297-9fe5-0cd87fe6dd1a-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-078fd3-84495865c7-mqwt7\" (UID: \"fc3bedf6-b508-4297-9fe5-0cd87fe6dd1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-078fd3-84495865c7-mqwt7"
Nov 02 12:47:05.641676 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:05.639031    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc3bedf6-b508-4297-9fe5-0cd87fe6dd1a-metrics-certs\") pod \"maas-api-e2e-ipp-b-078fd3-84495865c7-mqwt7\" (UID: \"fc3bedf6-b508-4297-9fe5-0cd87fe6dd1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-078fd3-84495865c7-mqwt7"
Nov 02 12:47:05.641676 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:05.639079    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rttf\" (UniqueName: \"kubernetes.io/projected/fc3bedf6-b508-4297-9fe5-0cd87fe6dd1a-kube-api-access-8rttf\") pod \"maas-api-e2e-ipp-b-078fd3-84495865c7-mqwt7\" (UID: \"fc3bedf6-b508-4297-9fe5-0cd87fe6dd1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-078fd3-84495865c7-mqwt7"
Nov 02 12:47:05.641676 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:05.640265    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc3bedf6-b508-4297-9fe5-0cd87fe6dd1a-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-078fd3-84495865c7-mqwt7\" (UID: \"fc3bedf6-b508-4297-9fe5-0cd87fe6dd1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-078fd3-84495865c7-mqwt7"
Nov 02 12:47:05.729707 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:05.727778    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc3bedf6-b508-4297-9fe5-0cd87fe6dd1a-metrics-certs\") pod \"maas-api-e2e-ipp-b-078fd3-84495865c7-mqwt7\" (UID: \"fc3bedf6-b508-4297-9fe5-0cd87fe6dd1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-078fd3-84495865c7-mqwt7"
Nov 02 12:47:05.782277 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:05.782215    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc3bedf6-b508-4297-9fe5-0cd87fe6dd1a-maas-api-tls\") pod \"maas-api-e2e-ipp-b-078fd3-84495865c7-mqwt7\" (UID: \"fc3bedf6-b508-4297-9fe5-0cd87fe6dd1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-078fd3-84495865c7-mqwt7"
Nov 02 12:47:05.807667 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:05.789701    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rttf\" (UniqueName: \"kubernetes.io/projected/fc3bedf6-b508-4297-9fe5-0cd87fe6dd1a-kube-api-access-8rttf\") pod \"maas-api-e2e-ipp-b-078fd3-84495865c7-mqwt7\" (UID: \"fc3bedf6-b508-4297-9fe5-0cd87fe6dd1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-078fd3-84495865c7-mqwt7"
Nov 02 12:47:05.947101 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:05.947066    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13a4f88f-a38d-4fa4-ad71-8ec24dc429a5" path="/var/lib/kubelet/pods/13a4f88f-a38d-4fa4-ad71-8ec24dc429a5/volumes"
Nov 02 12:47:05.947592 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:05.947573    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18cc5afa-245b-43f6-b92c-deec76232bd7" path="/var/lib/kubelet/pods/18cc5afa-245b-43f6-b92c-deec76232bd7/volumes"
Nov 02 12:47:05.948075 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:05.948058    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a48a9d0-407c-4149-abe9-c675e2092854" path="/var/lib/kubelet/pods/2a48a9d0-407c-4149-abe9-c675e2092854/volumes"
Nov 02 12:47:06.078573 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:06.078487    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-078fd3-84495865c7-mqwt7"
Nov 02 12:47:06.540224 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:06.540140    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-078fd3-84495865c7-mqwt7"]
Nov 02 12:47:07.444195 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:07.444142    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-078fd3-84495865c7-mqwt7" event={"ID":"fc3bedf6-b508-4297-9fe5-0cd87fe6dd1a","Type":"ContainerStarted","Data":"fb099379d1c403a597fd9221ab5636b49872a34339d197dadb4bb19a44f2a096"}
Nov 02 12:47:07.444400 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:07.444247    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-078fd3-84495865c7-mqwt7" event={"ID":"fc3bedf6-b508-4297-9fe5-0cd87fe6dd1a","Type":"ContainerStarted","Data":"4b59f29e45e9e684942295b8bc228c0ac1a9fb894c1fc54dc92af9729821f1d5"}
Nov 02 12:47:07.444400 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:07.444300    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-078fd3-84495865c7-mqwt7"
Nov 02 12:47:07.499664 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:07.496144    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-078fd3-84495865c7-mqwt7" podStartSLOduration=2.010426304 podStartE2EDuration="2.496123574s" totalImagesPullingTime="485.69727ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:47:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:47:07.495106016 +0000 UTC m=+1223.750505113" watchObservedRunningTime="2026-11-02 12:47:07.496123574 +0000 UTC m=+1223.751522671"
Nov 02 12:47:10.122912 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:10.122878    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-97de36-76c57bb9ff-9dv2z"]
Nov 02 12:47:13.456828 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:13.456760    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-078fd3-84495865c7-mqwt7"
Nov 02 12:47:14.498341 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:14.498236    4559 generic.go:356] "Generic (PLEG): container finished" podID="0f74d2ef-c796-4c91-b763-7593c16cfe71" containerID="c0efdec28f98e63e1d763aae8b0b3dfd8090e3c17e7641565e99d42b96efcd88" exitCode=0
Nov 02 12:47:14.498341 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:14.498310    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b2ec05-openshift-default-5f4c648f59-bldmv" event={"ID":"0f74d2ef-c796-4c91-b763-7593c16cfe71","Type":"ContainerDied","Data":"c0efdec28f98e63e1d763aae8b0b3dfd8090e3c17e7641565e99d42b96efcd88"}
Nov 02 12:47:14.498929 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:14.498779    4559 scope.go:122] "RemoveContainer" containerID="c0efdec28f98e63e1d763aae8b0b3dfd8090e3c17e7641565e99d42b96efcd88"
Nov 02 12:47:14.500502 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:14.500468    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:47:14.500612 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:14.500559    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:47:14.500612 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:14.500604    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:47:14.897688 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:14.897564    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-nsg75"]
Nov 02 12:47:14.897907 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:14.897831    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-nsg75" podUID="7f0dd625-e644-4b6a-b697-c7b66b68e6ca" containerName="istio-proxy" containerID="cri-o://54d254365c0ad15d60a825a2d5ee152c233fce1ea4d2e496e4b4bcc256a9dc2e" gracePeriod=30
Nov 02 12:47:14.918443 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:14.918409    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-rxdxm"]
Nov 02 12:47:14.918900 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:14.918870    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-rxdxm" podUID="c2989390-a19e-408b-ac9e-ec01df76bf12" containerName="istio-proxy" containerID="cri-o://3e3df72cf5e7359ef77f2f5048e63529635037abf749c2a0de01c73b4c1c00c8" gracePeriod=30
Nov 02 12:47:15.129668 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:15.123812    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-97de36-76c57bb9ff-9dv2z" podUID="7affa7ad-9c8c-4896-8ad8-2030fdc323fd" containerName="payload-pre-processing" containerID="cri-o://8ceae3a58addfbc12560a31a4a65029cf1e30cc051f3a1c187aec1947d38b3bb" gracePeriod=25
Nov 02 12:47:15.154569 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:47:15.150576    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqcmj" containerName="extractor"
Nov 02 12:47:15.212979 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:47:15.212949    4559 logging.go:55] [core] [Channel #1855 SubChannel #1856]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.116:50052", ServerName: "10.133.0.116:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.116:50052: connect: connection refused"
Nov 02 12:47:15.705284 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:15.528389    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b2ec05-openshift-default-5f4c648f59-bldmv" event={"ID":"0f74d2ef-c796-4c91-b763-7593c16cfe71","Type":"ContainerStarted","Data":"153e39ff35c98ce2c5fe699e58991b360731c0637f622ead6afed234f01d22b7"}
Nov 02 12:47:15.705284 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:15.531261    4559 generic.go:356] "Generic (PLEG): container finished" podID="7affa7ad-9c8c-4896-8ad8-2030fdc323fd" containerID="8ceae3a58addfbc12560a31a4a65029cf1e30cc051f3a1c187aec1947d38b3bb" exitCode=0
Nov 02 12:47:15.705284 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:15.531337    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-97de36-76c57bb9ff-9dv2z" event={"ID":"7affa7ad-9c8c-4896-8ad8-2030fdc323fd","Type":"ContainerDied","Data":"8ceae3a58addfbc12560a31a4a65029cf1e30cc051f3a1c187aec1947d38b3bb"}
Nov 02 12:47:15.705284 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:15.585281    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-97de36-76c57bb9ff-9dv2z"
Nov 02 12:47:15.724774 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:15.724696    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7affa7ad-9c8c-4896-8ad8-2030fdc323fd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7affa7ad-9c8c-4896-8ad8-2030fdc323fd-plugins-config-volume\") pod \"7affa7ad-9c8c-4896-8ad8-2030fdc323fd\" (UID: \"7affa7ad-9c8c-4896-8ad8-2030fdc323fd\") "
Nov 02 12:47:15.724774 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:15.724750    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7affa7ad-9c8c-4896-8ad8-2030fdc323fd-kube-api-access-ll7nf\" (UniqueName: \"kubernetes.io/projected/7affa7ad-9c8c-4896-8ad8-2030fdc323fd-kube-api-access-ll7nf\") pod \"7affa7ad-9c8c-4896-8ad8-2030fdc323fd\" (UID: \"7affa7ad-9c8c-4896-8ad8-2030fdc323fd\") "
Nov 02 12:47:15.725101 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:15.725075    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7affa7ad-9c8c-4896-8ad8-2030fdc323fd-plugins-config-volume" pod "7affa7ad-9c8c-4896-8ad8-2030fdc323fd" (UID: "7affa7ad-9c8c-4896-8ad8-2030fdc323fd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:47:15.725288 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:15.725270    4559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7affa7ad-9c8c-4896-8ad8-2030fdc323fd-plugins-config-volume\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:47:15.726846 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:15.726824    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7affa7ad-9c8c-4896-8ad8-2030fdc323fd-kube-api-access-ll7nf" pod "7affa7ad-9c8c-4896-8ad8-2030fdc323fd" (UID: "7affa7ad-9c8c-4896-8ad8-2030fdc323fd"). InnerVolumeSpecName "kube-api-access-ll7nf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:47:15.826665 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:15.826604    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ll7nf\" (UniqueName: \"kubernetes.io/projected/7affa7ad-9c8c-4896-8ad8-2030fdc323fd-kube-api-access-ll7nf\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:47:16.022039 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:16.021949    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-f7f44a-openshift-default-765dbcd8cf-t8fz5"]
Nov 02 12:47:16.024975 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:16.024948    4559 cpu_manager.go:414] "containerMap: removing container" podUID="7affa7ad-9c8c-4896-8ad8-2030fdc323fd" containerName="payload-pre-processing"
Nov 02 12:47:16.025165 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:16.025150    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7affa7ad-9c8c-4896-8ad8-2030fdc323fd" containerName="payload-pre-processing"
Nov 02 12:47:16.025425 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:16.025412    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="7affa7ad-9c8c-4896-8ad8-2030fdc323fd" containerName="payload-pre-processing"
Nov 02 12:47:16.029330 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:16.029104    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-f7f44a-openshift-default-765dbcd8cf-t8fz5"
Nov 02 12:47:16.051846 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:16.050753    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-f7f44a-openshift-default-765dbcd8cf-t8fz5"]
Nov 02 12:47:16.154669 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:16.153215    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-f7f44a-openshift-default-dockercfg-p9v6c\""
Nov 02 12:47:16.154669 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:16.154321    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4906067b-634f-4251-9953-e4c70616400b-istio-envoy\") pod \"e2e-api-b-f7f44a-openshift-default-765dbcd8cf-t8fz5\" (UID: \"4906067b-634f-4251-9953-e4c70616400b\") " pod="openshift-ingress/e2e-api-b-f7f44a-openshift-default-765dbcd8cf-t8fz5"
Nov 02 12:47:16.154669 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:16.154567    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7hft\" (UniqueName: \"kubernetes.io/projected/4906067b-634f-4251-9953-e4c70616400b-kube-api-access-q7hft\") pod \"e2e-api-b-f7f44a-openshift-default-765dbcd8cf-t8fz5\" (UID: \"4906067b-634f-4251-9953-e4c70616400b\") " pod="openshift-ingress/e2e-api-b-f7f44a-openshift-default-765dbcd8cf-t8fz5"
Nov 02 12:47:16.154669 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:16.154635    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4906067b-634f-4251-9953-e4c70616400b-credential-socket\") pod \"e2e-api-b-f7f44a-openshift-default-765dbcd8cf-t8fz5\" (UID: \"4906067b-634f-4251-9953-e4c70616400b\") " pod="openshift-ingress/e2e-api-b-f7f44a-openshift-default-765dbcd8cf-t8fz5"
Nov 02 12:47:16.155036 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:16.154694    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4906067b-634f-4251-9953-e4c70616400b-istio-podinfo\") pod \"e2e-api-b-f7f44a-openshift-default-765dbcd8cf-t8fz5\" (UID: \"4906067b-634f-4251-9953-e4c70616400b\") " pod="openshift-ingress/e2e-api-b-f7f44a-openshift-default-765dbcd8cf-t8fz5"
Nov 02 12:47:16.155036 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:16.154722    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4906067b-634f-4251-9953-e4c70616400b-workload-certs\") pod \"e2e-api-b-f7f44a-openshift-default-765dbcd8cf-t8fz5\" (UID: \"4906067b-634f-4251-9953-e4c70616400b\") " pod="openshift-ingress/e2e-api-b-f7f44a-openshift-default-765dbcd8cf-t8fz5"
Nov 02 12:47:16.155036 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:16.154773    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4906067b-634f-4251-9953-e4c70616400b-istio-data\") pod \"e2e-api-b-f7f44a-openshift-default-765dbcd8cf-t8fz5\" (UID: \"4906067b-634f-4251-9953-e4c70616400b\") " pod="openshift-ingress/e2e-api-b-f7f44a-openshift-default-765dbcd8cf-t8fz5"
Nov 02 12:47:16.155036 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:16.154884    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4906067b-634f-4251-9953-e4c70616400b-istiod-ca-cert\") pod \"e2e-api-b-f7f44a-openshift-default-765dbcd8cf-t8fz5\" (UID: \"4906067b-634f-4251-9953-e4c70616400b\") " pod="openshift-ingress/e2e-api-b-f7f44a-openshift-default-765dbcd8cf-t8fz5"
Nov 02 12:47:16.155036 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:16.154932    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4906067b-634f-4251-9953-e4c70616400b-istio-token\") pod \"e2e-api-b-f7f44a-openshift-default-765dbcd8cf-t8fz5\" (UID: \"4906067b-634f-4251-9953-e4c70616400b\") " pod="openshift-ingress/e2e-api-b-f7f44a-openshift-default-765dbcd8cf-t8fz5"
Nov 02 12:47:16.155036 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:16.155004    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4906067b-634f-4251-9953-e4c70616400b-workload-socket\") pod \"e2e-api-b-f7f44a-openshift-default-765dbcd8cf-t8fz5\" (UID: \"4906067b-634f-4251-9953-e4c70616400b\") " pod="openshift-ingress/e2e-api-b-f7f44a-openshift-default-765dbcd8cf-t8fz5"
Nov 02 12:47:16.216078 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:16.216028    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-97de36-76c57bb9ff-9dv2z" podUID="7affa7ad-9c8c-4896-8ad8-2030fdc323fd" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.116:50052\" within 1s: context deadline exceeded"
Nov 02 12:47:16.216452 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:47:16.216430    4559 logging.go:55] [core] [Channel #1855 SubChannel #1856]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.116:50052", ServerName: "10.133.0.116:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.116:50052: operation was canceled"
Nov 02 12:47:16.311739 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:16.307028    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4906067b-634f-4251-9953-e4c70616400b-istio-podinfo\") pod \"e2e-api-b-f7f44a-openshift-default-765dbcd8cf-t8fz5\" (UID: \"4906067b-634f-4251-9953-e4c70616400b\") " pod="openshift-ingress/e2e-api-b-f7f44a-openshift-default-765dbcd8cf-t8fz5"
Nov 02 12:47:16.311739 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:16.307081    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4906067b-634f-4251-9953-e4c70616400b-workload-certs\") pod \"e2e-api-b-f7f44a-openshift-default-765dbcd8cf-t8fz5\" (UID: \"4906067b-634f-4251-9953-e4c70616400b\") " pod="openshift-ingress/e2e-api-b-f7f44a-openshift-default-765dbcd8cf-t8fz5"
Nov 02 12:47:16.311739 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:16.307128    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4906067b-634f-4251-9953-e4c70616400b-istio-data\") pod \"e2e-api-b-f7f44a-openshift-default-765dbcd8cf-t8fz5\" (UID: \"4906067b-634f-4251-9953-e4c70616400b\") " pod="openshift-ingress/e2e-api-b-f7f44a-openshift-default-765dbcd8cf-t8fz5"
Nov 02 12:47:16.311739 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:16.307227    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4906067b-634f-4251-9953-e4c70616400b-istiod-ca-cert\") pod \"e2e-api-b-f7f44a-openshift-default-765dbcd8cf-t8fz5\" (UID: \"4906067b-634f-4251-9953-e4c70616400b\") " pod="openshift-ingress/e2e-api-b-f7f44a-openshift-default-765dbcd8cf-t8fz5"
Nov 02 12:47:16.311739 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:16.307271    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4906067b-634f-4251-9953-e4c70616400b-istio-token\") pod \"e2e-api-b-f7f44a-openshift-default-765dbcd8cf-t8fz5\" (UID: \"4906067b-634f-4251-9953-e4c70616400b\") " pod="openshift-ingress/e2e-api-b-f7f44a-openshift-default-765dbcd8cf-t8fz5"
Nov 02 12:47:16.311739 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:16.307333    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4906067b-634f-4251-9953-e4c70616400b-workload-socket\") pod \"e2e-api-b-f7f44a-openshift-default-765dbcd8cf-t8fz5\" (UID: \"4906067b-634f-4251-9953-e4c70616400b\") " pod="openshift-ingress/e2e-api-b-f7f44a-openshift-default-765dbcd8cf-t8fz5"
Nov 02 12:47:16.311739 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:16.307454    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4906067b-634f-4251-9953-e4c70616400b-istio-envoy\") pod \"e2e-api-b-f7f44a-openshift-default-765dbcd8cf-t8fz5\" (UID: \"4906067b-634f-4251-9953-e4c70616400b\") " pod="openshift-ingress/e2e-api-b-f7f44a-openshift-default-765dbcd8cf-t8fz5"
Nov 02 12:47:16.311739 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:16.307516    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7hft\" (UniqueName: \"kubernetes.io/projected/4906067b-634f-4251-9953-e4c70616400b-kube-api-access-q7hft\") pod \"e2e-api-b-f7f44a-openshift-default-765dbcd8cf-t8fz5\" (UID: \"4906067b-634f-4251-9953-e4c70616400b\") " pod="openshift-ingress/e2e-api-b-f7f44a-openshift-default-765dbcd8cf-t8fz5"
Nov 02 12:47:16.311739 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:16.307569    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4906067b-634f-4251-9953-e4c70616400b-credential-socket\") pod \"e2e-api-b-f7f44a-openshift-default-765dbcd8cf-t8fz5\" (UID: \"4906067b-634f-4251-9953-e4c70616400b\") " pod="openshift-ingress/e2e-api-b-f7f44a-openshift-default-765dbcd8cf-t8fz5"
Nov 02 12:47:16.332047 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:16.332009    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4906067b-634f-4251-9953-e4c70616400b-credential-socket\") pod \"e2e-api-b-f7f44a-openshift-default-765dbcd8cf-t8fz5\" (UID: \"4906067b-634f-4251-9953-e4c70616400b\") " pod="openshift-ingress/e2e-api-b-f7f44a-openshift-default-765dbcd8cf-t8fz5"
Nov 02 12:47:16.346670 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:16.333906    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4906067b-634f-4251-9953-e4c70616400b-workload-certs\") pod \"e2e-api-b-f7f44a-openshift-default-765dbcd8cf-t8fz5\" (UID: \"4906067b-634f-4251-9953-e4c70616400b\") " pod="openshift-ingress/e2e-api-b-f7f44a-openshift-default-765dbcd8cf-t8fz5"
Nov 02 12:47:16.347005 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:16.334899    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4906067b-634f-4251-9953-e4c70616400b-istio-data\") pod \"e2e-api-b-f7f44a-openshift-default-765dbcd8cf-t8fz5\" (UID: \"4906067b-634f-4251-9953-e4c70616400b\") " pod="openshift-ingress/e2e-api-b-f7f44a-openshift-default-765dbcd8cf-t8fz5"
Nov 02 12:47:16.347187 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:16.336555    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-b2ec05-openshift-default-5f4c648f59-bldmv"
Nov 02 12:47:16.347373 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:16.336735    4559 patch_prober.go:29] interesting pod/e2e-api-a-b2ec05-openshift-default-5f4c648f59-bldmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body=
Nov 02 12:47:16.347552 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:16.347514    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b2ec05-openshift-default-5f4c648f59-bldmv" podUID="0f74d2ef-c796-4c91-b763-7593c16cfe71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused"
Nov 02 12:47:16.348012 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:16.337532    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4906067b-634f-4251-9953-e4c70616400b-istiod-ca-cert\") pod \"e2e-api-b-f7f44a-openshift-default-765dbcd8cf-t8fz5\" (UID: \"4906067b-634f-4251-9953-e4c70616400b\") " pod="openshift-ingress/e2e-api-b-f7f44a-openshift-default-765dbcd8cf-t8fz5"
Nov 02 12:47:16.348165 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:16.336979    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4906067b-634f-4251-9953-e4c70616400b-workload-socket\") pod \"e2e-api-b-f7f44a-openshift-default-765dbcd8cf-t8fz5\" (UID: \"4906067b-634f-4251-9953-e4c70616400b\") " pod="openshift-ingress/e2e-api-b-f7f44a-openshift-default-765dbcd8cf-t8fz5"
Nov 02 12:47:16.399289 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:16.396394    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4906067b-634f-4251-9953-e4c70616400b-istio-token\") pod \"e2e-api-b-f7f44a-openshift-default-765dbcd8cf-t8fz5\" (UID: \"4906067b-634f-4251-9953-e4c70616400b\") " pod="openshift-ingress/e2e-api-b-f7f44a-openshift-default-765dbcd8cf-t8fz5"
Nov 02 12:47:16.435734 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:16.432619    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7hft\" (UniqueName: \"kubernetes.io/projected/4906067b-634f-4251-9953-e4c70616400b-kube-api-access-q7hft\") pod \"e2e-api-b-f7f44a-openshift-default-765dbcd8cf-t8fz5\" (UID: \"4906067b-634f-4251-9953-e4c70616400b\") " pod="openshift-ingress/e2e-api-b-f7f44a-openshift-default-765dbcd8cf-t8fz5"
Nov 02 12:47:16.435734 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:16.433177    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4906067b-634f-4251-9953-e4c70616400b-istio-podinfo\") pod \"e2e-api-b-f7f44a-openshift-default-765dbcd8cf-t8fz5\" (UID: \"4906067b-634f-4251-9953-e4c70616400b\") " pod="openshift-ingress/e2e-api-b-f7f44a-openshift-default-765dbcd8cf-t8fz5"
Nov 02 12:47:16.435734 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:16.433368    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4906067b-634f-4251-9953-e4c70616400b-istio-envoy\") pod \"e2e-api-b-f7f44a-openshift-default-765dbcd8cf-t8fz5\" (UID: \"4906067b-634f-4251-9953-e4c70616400b\") " pod="openshift-ingress/e2e-api-b-f7f44a-openshift-default-765dbcd8cf-t8fz5"
Nov 02 12:47:16.539705 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:16.539674    4559 generic.go:356] "Generic (PLEG): container finished" podID="5b55da6b-89cb-46d6-a7a0-3a09e7d7d17e" containerID="ef26164c25b358f05ae65310109400f62dc5283758f5989525cebc2709f8e391" exitCode=0
Nov 02 12:47:16.539895 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:16.539771    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-97de36-openshift-default-77899d747c-cq4bc" event={"ID":"5b55da6b-89cb-46d6-a7a0-3a09e7d7d17e","Type":"ContainerDied","Data":"ef26164c25b358f05ae65310109400f62dc5283758f5989525cebc2709f8e391"}
Nov 02 12:47:16.540249 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:16.540225    4559 scope.go:122] "RemoveContainer" containerID="ef26164c25b358f05ae65310109400f62dc5283758f5989525cebc2709f8e391"
Nov 02 12:47:16.541162 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:16.541148    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-97de36-76c57bb9ff-9dv2z"
Nov 02 12:47:16.541225 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:16.541179    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-97de36-76c57bb9ff-9dv2z" event={"ID":"7affa7ad-9c8c-4896-8ad8-2030fdc323fd","Type":"ContainerDied","Data":"e0758f725a5d37a91c2b5d148b78d06aecb7c933994ce134788db7b7baeedf7e"}
Nov 02 12:47:16.541278 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:16.541228    4559 scope.go:122] "RemoveContainer" containerID="8ceae3a58addfbc12560a31a4a65029cf1e30cc051f3a1c187aec1947d38b3bb"
Nov 02 12:47:16.542023 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:16.541991    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:47:16.542127 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:16.542068    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:47:16.542196 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:16.542124    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:47:16.666784 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:16.666762    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-97de36-76c57bb9ff-9dv2z"]
Nov 02 12:47:16.673819 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:16.673383    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-f7f44a-openshift-default-765dbcd8cf-t8fz5"
Nov 02 12:47:16.674011 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:16.673995    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-97de36-76c57bb9ff-9dv2z"]
Nov 02 12:47:17.283478 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:17.283047    4559 patch_prober.go:29] interesting pod/e2e-api-a-b2ec05-openshift-default-5f4c648f59-bldmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body=
Nov 02 12:47:17.283478 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:17.283108    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b2ec05-openshift-default-5f4c648f59-bldmv" podUID="0f74d2ef-c796-4c91-b763-7593c16cfe71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused"
Nov 02 12:47:17.506666 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:17.483953    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-f7f44a-openshift-default-765dbcd8cf-t8fz5"]
Nov 02 12:47:17.506666 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:17.495338    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:47:17.506666 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:17.495429    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:47:17.506666 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:17.495471    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:47:17.551207 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:17.551173    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f7f44a-openshift-default-765dbcd8cf-t8fz5" event={"ID":"4906067b-634f-4251-9953-e4c70616400b","Type":"ContainerStarted","Data":"e3bc190555f788f68a1c8a1e5f98e6e7fd27efe726b347942448ba8fe6b0ccca"}
Nov 02 12:47:17.556215 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:17.556190    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-97de36-openshift-default-77899d747c-cq4bc" event={"ID":"5b55da6b-89cb-46d6-a7a0-3a09e7d7d17e","Type":"ContainerStarted","Data":"0907660c6d6e2964677ccebb59dbaa23d4b900e33b7d413aa483136fa1e4a4d3"}
Nov 02 12:47:17.675292 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:17.675246    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-97de36-openshift-default-77899d747c-cq4bc"
Nov 02 12:47:17.676267 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:17.676239    4559 patch_prober.go:29] interesting pod/e2e-ipp-b-97de36-openshift-default-77899d747c-cq4bc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body=
Nov 02 12:47:17.676569 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:17.676499    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-97de36-openshift-default-77899d747c-cq4bc" podUID="5b55da6b-89cb-46d6-a7a0-3a09e7d7d17e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused"
Nov 02 12:47:17.944664 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:17.940495    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7affa7ad-9c8c-4896-8ad8-2030fdc323fd" path="/var/lib/kubelet/pods/7affa7ad-9c8c-4896-8ad8-2030fdc323fd/volumes"
Nov 02 12:47:18.268718 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:18.268057    4559 patch_prober.go:29] interesting pod/e2e-api-a-b2ec05-openshift-default-5f4c648f59-bldmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body=
Nov 02 12:47:18.268718 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:18.268110    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b2ec05-openshift-default-5f4c648f59-bldmv" podUID="0f74d2ef-c796-4c91-b763-7593c16cfe71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused"
Nov 02 12:47:18.566430 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:18.566334    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f7f44a-openshift-default-765dbcd8cf-t8fz5" event={"ID":"4906067b-634f-4251-9953-e4c70616400b","Type":"ContainerStarted","Data":"dc9e4f833b5068cfdc384b48c192cdc27e491ab61fbc76d4335a6ddfc342be46"}
Nov 02 12:47:18.640236 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:18.640178    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-f7f44a-openshift-default-765dbcd8cf-t8fz5" podStartSLOduration=3.640161202 podStartE2EDuration="3.640161202s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:47:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:47:18.615159863 +0000 UTC m=+1234.870558958" watchObservedRunningTime="2026-11-02 12:47:18.640161202 +0000 UTC m=+1234.895560371"
Nov 02 12:47:18.673649 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:18.673619    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-f7f44a-openshift-default-765dbcd8cf-t8fz5"
Nov 02 12:47:18.675847 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:18.675827    4559 patch_prober.go:29] interesting pod/e2e-ipp-b-97de36-openshift-default-77899d747c-cq4bc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body=
Nov 02 12:47:18.675911 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:18.675885    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-97de36-openshift-default-77899d747c-cq4bc" podUID="5b55da6b-89cb-46d6-a7a0-3a09e7d7d17e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused"
Nov 02 12:47:18.680143 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:18.680122    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-f7f44a-openshift-default-765dbcd8cf-t8fz5"
Nov 02 12:47:19.013574 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:19.013460    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-97de36-openshift-default-77899d747c-cq4bc"
Nov 02 12:47:19.269089 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:19.269009    4559 patch_prober.go:29] interesting pod/e2e-api-a-b2ec05-openshift-default-5f4c648f59-bldmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body=
Nov 02 12:47:19.269089 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:19.269073    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b2ec05-openshift-default-5f4c648f59-bldmv" podUID="0f74d2ef-c796-4c91-b763-7593c16cfe71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused"
Nov 02 12:47:19.580728 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:19.580605    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-f7f44a-openshift-default-765dbcd8cf-t8fz5"
Nov 02 12:47:19.581553 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:19.581527    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-f7f44a-openshift-default-765dbcd8cf-t8fz5"
Nov 02 12:47:19.675157 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:19.675127    4559 patch_prober.go:29] interesting pod/e2e-ipp-b-97de36-openshift-default-77899d747c-cq4bc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body=
Nov 02 12:47:19.675420 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:19.675393    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-97de36-openshift-default-77899d747c-cq4bc" podUID="5b55da6b-89cb-46d6-a7a0-3a09e7d7d17e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused"
Nov 02 12:47:20.268304 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:20.268271    4559 patch_prober.go:29] interesting pod/e2e-api-a-b2ec05-openshift-default-5f4c648f59-bldmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body=
Nov 02 12:47:20.268499 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:20.268469    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b2ec05-openshift-default-5f4c648f59-bldmv" podUID="0f74d2ef-c796-4c91-b763-7593c16cfe71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused"
Nov 02 12:47:20.321676 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:20.321175    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-nsg75"
Nov 02 12:47:20.367539 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:20.367506    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f0dd625-e644-4b6a-b697-c7b66b68e6ca-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f0dd625-e644-4b6a-b697-c7b66b68e6ca-workload-certs\") pod \"7f0dd625-e644-4b6a-b697-c7b66b68e6ca\" (UID: \"7f0dd625-e644-4b6a-b697-c7b66b68e6ca\") "
Nov 02 12:47:20.367772 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:20.367711    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f0dd625-e644-4b6a-b697-c7b66b68e6ca-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f0dd625-e644-4b6a-b697-c7b66b68e6ca-istio-envoy\") pod \"7f0dd625-e644-4b6a-b697-c7b66b68e6ca\" (UID: \"7f0dd625-e644-4b6a-b697-c7b66b68e6ca\") "
Nov 02 12:47:20.367865 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:20.367773    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f0dd625-e644-4b6a-b697-c7b66b68e6ca-istio-token\" (UniqueName: \"kubernetes.io/projected/7f0dd625-e644-4b6a-b697-c7b66b68e6ca-istio-token\") pod \"7f0dd625-e644-4b6a-b697-c7b66b68e6ca\" (UID: \"7f0dd625-e644-4b6a-b697-c7b66b68e6ca\") "
Nov 02 12:47:20.367865 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:20.367804    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f0dd625-e644-4b6a-b697-c7b66b68e6ca-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f0dd625-e644-4b6a-b697-c7b66b68e6ca-istio-data\") pod \"7f0dd625-e644-4b6a-b697-c7b66b68e6ca\" (UID: \"7f0dd625-e644-4b6a-b697-c7b66b68e6ca\") "
Nov 02 12:47:20.367865 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:20.367846    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f0dd625-e644-4b6a-b697-c7b66b68e6ca-kube-api-access-4sxgt\" (UniqueName: \"kubernetes.io/projected/7f0dd625-e644-4b6a-b697-c7b66b68e6ca-kube-api-access-4sxgt\") pod \"7f0dd625-e644-4b6a-b697-c7b66b68e6ca\" (UID: \"7f0dd625-e644-4b6a-b697-c7b66b68e6ca\") "
Nov 02 12:47:20.368037 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:20.367880    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f0dd625-e644-4b6a-b697-c7b66b68e6ca-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f0dd625-e644-4b6a-b697-c7b66b68e6ca-workload-socket\") pod \"7f0dd625-e644-4b6a-b697-c7b66b68e6ca\" (UID: \"7f0dd625-e644-4b6a-b697-c7b66b68e6ca\") "
Nov 02 12:47:20.368764 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:20.368483    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f0dd625-e644-4b6a-b697-c7b66b68e6ca-istio-data" pod "7f0dd625-e644-4b6a-b697-c7b66b68e6ca" (UID: "7f0dd625-e644-4b6a-b697-c7b66b68e6ca"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:47:20.368764 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:20.368491    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7f0dd625-e644-4b6a-b697-c7b66b68e6ca-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f0dd625-e644-4b6a-b697-c7b66b68e6ca-istio-podinfo\") pod \"7f0dd625-e644-4b6a-b697-c7b66b68e6ca\" (UID: \"7f0dd625-e644-4b6a-b697-c7b66b68e6ca\") "
Nov 02 12:47:20.368764 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:20.368614    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f0dd625-e644-4b6a-b697-c7b66b68e6ca-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f0dd625-e644-4b6a-b697-c7b66b68e6ca-credential-socket\") pod \"7f0dd625-e644-4b6a-b697-c7b66b68e6ca\" (UID: \"7f0dd625-e644-4b6a-b697-c7b66b68e6ca\") "
Nov 02 12:47:20.368764 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:20.368723    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f0dd625-e644-4b6a-b697-c7b66b68e6ca-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f0dd625-e644-4b6a-b697-c7b66b68e6ca-istiod-ca-cert\") pod \"7f0dd625-e644-4b6a-b697-c7b66b68e6ca\" (UID: \"7f0dd625-e644-4b6a-b697-c7b66b68e6ca\") "
Nov 02 12:47:20.369086 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:20.368944    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f0dd625-e644-4b6a-b697-c7b66b68e6ca-workload-socket" pod "7f0dd625-e644-4b6a-b697-c7b66b68e6ca" (UID: "7f0dd625-e644-4b6a-b697-c7b66b68e6ca"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:47:20.369154 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:20.369076    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f0dd625-e644-4b6a-b697-c7b66b68e6ca-credential-socket" pod "7f0dd625-e644-4b6a-b697-c7b66b68e6ca" (UID: "7f0dd625-e644-4b6a-b697-c7b66b68e6ca"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:47:20.369777 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:20.369300    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f0dd625-e644-4b6a-b697-c7b66b68e6ca-istiod-ca-cert" pod "7f0dd625-e644-4b6a-b697-c7b66b68e6ca" (UID: "7f0dd625-e644-4b6a-b697-c7b66b68e6ca"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:47:20.369777 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:20.369704    4559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f0dd625-e644-4b6a-b697-c7b66b68e6ca-istio-data\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:47:20.369777 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:20.369726    4559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f0dd625-e644-4b6a-b697-c7b66b68e6ca-workload-socket\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:47:20.369777 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:20.369741    4559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f0dd625-e644-4b6a-b697-c7b66b68e6ca-credential-socket\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:47:20.369777 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:20.369754    4559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f0dd625-e644-4b6a-b697-c7b66b68e6ca-istiod-ca-cert\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:47:20.370345 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:20.370323    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f0dd625-e644-4b6a-b697-c7b66b68e6ca-workload-certs" pod "7f0dd625-e644-4b6a-b697-c7b66b68e6ca" (UID: "7f0dd625-e644-4b6a-b697-c7b66b68e6ca"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:47:20.370810 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:20.370783    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f0dd625-e644-4b6a-b697-c7b66b68e6ca-istio-token" pod "7f0dd625-e644-4b6a-b697-c7b66b68e6ca" (UID: "7f0dd625-e644-4b6a-b697-c7b66b68e6ca"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:47:20.371830 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:20.371803    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7f0dd625-e644-4b6a-b697-c7b66b68e6ca-istio-podinfo" pod "7f0dd625-e644-4b6a-b697-c7b66b68e6ca" (UID: "7f0dd625-e644-4b6a-b697-c7b66b68e6ca"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 12:47:20.373145 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:20.373118    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f0dd625-e644-4b6a-b697-c7b66b68e6ca-istio-envoy" pod "7f0dd625-e644-4b6a-b697-c7b66b68e6ca" (UID: "7f0dd625-e644-4b6a-b697-c7b66b68e6ca"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:47:20.373305 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:20.373278    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f0dd625-e644-4b6a-b697-c7b66b68e6ca-kube-api-access-4sxgt" pod "7f0dd625-e644-4b6a-b697-c7b66b68e6ca" (UID: "7f0dd625-e644-4b6a-b697-c7b66b68e6ca"). InnerVolumeSpecName "kube-api-access-4sxgt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:47:20.381113 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:20.381053    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-rxdxm"
Nov 02 12:47:20.471239 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:20.471204    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2989390-a19e-408b-ac9e-ec01df76bf12-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2989390-a19e-408b-ac9e-ec01df76bf12-workload-certs\") pod \"c2989390-a19e-408b-ac9e-ec01df76bf12\" (UID: \"c2989390-a19e-408b-ac9e-ec01df76bf12\") "
Nov 02 12:47:20.471558 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:20.471530    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2989390-a19e-408b-ac9e-ec01df76bf12-workload-certs" pod "c2989390-a19e-408b-ac9e-ec01df76bf12" (UID: "c2989390-a19e-408b-ac9e-ec01df76bf12"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:47:20.471667 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:20.471622    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2989390-a19e-408b-ac9e-ec01df76bf12-workload-socket" pod "c2989390-a19e-408b-ac9e-ec01df76bf12" (UID: "c2989390-a19e-408b-ac9e-ec01df76bf12"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:47:20.471745 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:20.471306    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2989390-a19e-408b-ac9e-ec01df76bf12-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2989390-a19e-408b-ac9e-ec01df76bf12-workload-socket\") pod \"c2989390-a19e-408b-ac9e-ec01df76bf12\" (UID: \"c2989390-a19e-408b-ac9e-ec01df76bf12\") "
Nov 02 12:47:20.471806 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:20.471772    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2989390-a19e-408b-ac9e-ec01df76bf12-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2989390-a19e-408b-ac9e-ec01df76bf12-credential-socket\") pod \"c2989390-a19e-408b-ac9e-ec01df76bf12\" (UID: \"c2989390-a19e-408b-ac9e-ec01df76bf12\") "
Nov 02 12:47:20.472023 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:20.472000    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2989390-a19e-408b-ac9e-ec01df76bf12-credential-socket" pod "c2989390-a19e-408b-ac9e-ec01df76bf12" (UID: "c2989390-a19e-408b-ac9e-ec01df76bf12"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:47:20.472467 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:20.472445    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2989390-a19e-408b-ac9e-ec01df76bf12-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2989390-a19e-408b-ac9e-ec01df76bf12-istio-envoy\") pod \"c2989390-a19e-408b-ac9e-ec01df76bf12\" (UID: \"c2989390-a19e-408b-ac9e-ec01df76bf12\") "
Nov 02 12:47:20.472565 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:20.472522    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c2989390-a19e-408b-ac9e-ec01df76bf12-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2989390-a19e-408b-ac9e-ec01df76bf12-istio-podinfo\") pod \"c2989390-a19e-408b-ac9e-ec01df76bf12\" (UID: \"c2989390-a19e-408b-ac9e-ec01df76bf12\") "
Nov 02 12:47:20.473018 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:20.472992    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2989390-a19e-408b-ac9e-ec01df76bf12-istiod-ca-cert" pod "c2989390-a19e-408b-ac9e-ec01df76bf12" (UID: "c2989390-a19e-408b-ac9e-ec01df76bf12"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:47:20.473101 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:20.473089    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2989390-a19e-408b-ac9e-ec01df76bf12-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2989390-a19e-408b-ac9e-ec01df76bf12-istiod-ca-cert\") pod \"c2989390-a19e-408b-ac9e-ec01df76bf12\" (UID: \"c2989390-a19e-408b-ac9e-ec01df76bf12\") "
Nov 02 12:47:20.473216 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:20.473200    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2989390-a19e-408b-ac9e-ec01df76bf12-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2989390-a19e-408b-ac9e-ec01df76bf12-istio-data\") pod \"c2989390-a19e-408b-ac9e-ec01df76bf12\" (UID: \"c2989390-a19e-408b-ac9e-ec01df76bf12\") "
Nov 02 12:47:20.473288 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:20.473259    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2989390-a19e-408b-ac9e-ec01df76bf12-istio-token\" (UniqueName: \"kubernetes.io/projected/c2989390-a19e-408b-ac9e-ec01df76bf12-istio-token\") pod \"c2989390-a19e-408b-ac9e-ec01df76bf12\" (UID: \"c2989390-a19e-408b-ac9e-ec01df76bf12\") "
Nov 02 12:47:20.473575 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:20.473552    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2989390-a19e-408b-ac9e-ec01df76bf12-istio-data" pod "c2989390-a19e-408b-ac9e-ec01df76bf12" (UID: "c2989390-a19e-408b-ac9e-ec01df76bf12"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:47:20.473939 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:20.473916    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2989390-a19e-408b-ac9e-ec01df76bf12-kube-api-access-9gvgx\" (UniqueName: \"kubernetes.io/projected/c2989390-a19e-408b-ac9e-ec01df76bf12-kube-api-access-9gvgx\") pod \"c2989390-a19e-408b-ac9e-ec01df76bf12\" (UID: \"c2989390-a19e-408b-ac9e-ec01df76bf12\") "
Nov 02 12:47:20.474612 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:20.474586    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2989390-a19e-408b-ac9e-ec01df76bf12-istio-envoy" pod "c2989390-a19e-408b-ac9e-ec01df76bf12" (UID: "c2989390-a19e-408b-ac9e-ec01df76bf12"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:47:20.474883 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:20.474861    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c2989390-a19e-408b-ac9e-ec01df76bf12-istio-podinfo" pod "c2989390-a19e-408b-ac9e-ec01df76bf12" (UID: "c2989390-a19e-408b-ac9e-ec01df76bf12"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 12:47:20.475157 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:20.475142    4559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2989390-a19e-408b-ac9e-ec01df76bf12-istio-data\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:47:20.475232 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:20.475159    4559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f0dd625-e644-4b6a-b697-c7b66b68e6ca-workload-certs\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:47:20.475232 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:20.475168    4559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f0dd625-e644-4b6a-b697-c7b66b68e6ca-istio-envoy\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:47:20.475232 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:20.475177    4559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2989390-a19e-408b-ac9e-ec01df76bf12-workload-certs\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:47:20.475232 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:20.475185    4559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f0dd625-e644-4b6a-b697-c7b66b68e6ca-istio-token\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:47:20.475232 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:20.475193    4559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2989390-a19e-408b-ac9e-ec01df76bf12-workload-socket\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:47:20.475232 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:20.475204    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4sxgt\" (UniqueName: \"kubernetes.io/projected/7f0dd625-e644-4b6a-b697-c7b66b68e6ca-kube-api-access-4sxgt\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:47:20.475232 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:20.475214    4559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2989390-a19e-408b-ac9e-ec01df76bf12-credential-socket\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:47:20.475232 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:20.475223    4559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f0dd625-e644-4b6a-b697-c7b66b68e6ca-istio-podinfo\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:47:20.475232 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:20.475231    4559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2989390-a19e-408b-ac9e-ec01df76bf12-istio-envoy\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:47:20.475717 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:20.475240    4559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2989390-a19e-408b-ac9e-ec01df76bf12-istio-podinfo\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:47:20.475717 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:20.475249    4559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2989390-a19e-408b-ac9e-ec01df76bf12-istiod-ca-cert\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:47:20.476487 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:20.476459    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2989390-a19e-408b-ac9e-ec01df76bf12-kube-api-access-9gvgx" pod "c2989390-a19e-408b-ac9e-ec01df76bf12" (UID: "c2989390-a19e-408b-ac9e-ec01df76bf12"). InnerVolumeSpecName "kube-api-access-9gvgx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:47:20.477118 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:20.477083    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2989390-a19e-408b-ac9e-ec01df76bf12-istio-token" pod "c2989390-a19e-408b-ac9e-ec01df76bf12" (UID: "c2989390-a19e-408b-ac9e-ec01df76bf12"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:47:20.576621 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:20.576543    4559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2989390-a19e-408b-ac9e-ec01df76bf12-istio-token\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:47:20.576621 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:20.576572    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9gvgx\" (UniqueName: \"kubernetes.io/projected/c2989390-a19e-408b-ac9e-ec01df76bf12-kube-api-access-9gvgx\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:47:20.640449 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:20.640418    4559 generic.go:356] "Generic (PLEG): container finished" podID="c2989390-a19e-408b-ac9e-ec01df76bf12" containerID="3e3df72cf5e7359ef77f2f5048e63529635037abf749c2a0de01c73b4c1c00c8" exitCode=0
Nov 02 12:47:20.640942 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:20.640495    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-rxdxm"
Nov 02 12:47:20.641148 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:20.640516    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-rxdxm" event={"ID":"c2989390-a19e-408b-ac9e-ec01df76bf12","Type":"ContainerDied","Data":"3e3df72cf5e7359ef77f2f5048e63529635037abf749c2a0de01c73b4c1c00c8"}
Nov 02 12:47:20.641263 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:20.641201    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-rxdxm" event={"ID":"c2989390-a19e-408b-ac9e-ec01df76bf12","Type":"ContainerDied","Data":"32b8b7203ed9d02aeec1b4d4edd3c9fe6d27547f909ff1e04b792df4cfc5d3d1"}
Nov 02 12:47:20.641263 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:20.641244    4559 scope.go:122] "RemoveContainer" containerID="3e3df72cf5e7359ef77f2f5048e63529635037abf749c2a0de01c73b4c1c00c8"
Nov 02 12:47:20.643535 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:20.643512    4559 generic.go:356] "Generic (PLEG): container finished" podID="7f0dd625-e644-4b6a-b697-c7b66b68e6ca" containerID="54d254365c0ad15d60a825a2d5ee152c233fce1ea4d2e496e4b4bcc256a9dc2e" exitCode=0
Nov 02 12:47:20.644266 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:20.644251    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-nsg75"
Nov 02 12:47:20.644742 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:20.644717    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-nsg75" event={"ID":"7f0dd625-e644-4b6a-b697-c7b66b68e6ca","Type":"ContainerDied","Data":"54d254365c0ad15d60a825a2d5ee152c233fce1ea4d2e496e4b4bcc256a9dc2e"}
Nov 02 12:47:20.644807 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:20.644763    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-nsg75" event={"ID":"7f0dd625-e644-4b6a-b697-c7b66b68e6ca","Type":"ContainerDied","Data":"cca1564adb48724f6609b65d3d771b566f800c54113c74e5a44539b4df73258a"}
Nov 02 12:47:20.658035 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:20.658019    4559 scope.go:122] "RemoveContainer" containerID="0eafe970aeb286d1e6ba5e03bba870e417ca71634a3d1606425969ba2fad7d3e"
Nov 02 12:47:20.667368 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:20.667349    4559 scope.go:122] "RemoveContainer" containerID="3e3df72cf5e7359ef77f2f5048e63529635037abf749c2a0de01c73b4c1c00c8"
Nov 02 12:47:20.667618 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:47:20.667598    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e3df72cf5e7359ef77f2f5048e63529635037abf749c2a0de01c73b4c1c00c8\": container with ID starting with 3e3df72cf5e7359ef77f2f5048e63529635037abf749c2a0de01c73b4c1c00c8 not found: ID does not exist" containerID="3e3df72cf5e7359ef77f2f5048e63529635037abf749c2a0de01c73b4c1c00c8"
Nov 02 12:47:20.667726 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:20.667629    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e3df72cf5e7359ef77f2f5048e63529635037abf749c2a0de01c73b4c1c00c8"} err="failed to get container status \"3e3df72cf5e7359ef77f2f5048e63529635037abf749c2a0de01c73b4c1c00c8\": rpc error: code = NotFound desc = could not find container \"3e3df72cf5e7359ef77f2f5048e63529635037abf749c2a0de01c73b4c1c00c8\": container with ID starting with 3e3df72cf5e7359ef77f2f5048e63529635037abf749c2a0de01c73b4c1c00c8 not found: ID does not exist"
Nov 02 12:47:20.667726 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:20.667666    4559 scope.go:122] "RemoveContainer" containerID="0eafe970aeb286d1e6ba5e03bba870e417ca71634a3d1606425969ba2fad7d3e"
Nov 02 12:47:20.667906 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:47:20.667886    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0eafe970aeb286d1e6ba5e03bba870e417ca71634a3d1606425969ba2fad7d3e\": container with ID starting with 0eafe970aeb286d1e6ba5e03bba870e417ca71634a3d1606425969ba2fad7d3e not found: ID does not exist" containerID="0eafe970aeb286d1e6ba5e03bba870e417ca71634a3d1606425969ba2fad7d3e"
Nov 02 12:47:20.667957 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:20.667915    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0eafe970aeb286d1e6ba5e03bba870e417ca71634a3d1606425969ba2fad7d3e"} err="failed to get container status \"0eafe970aeb286d1e6ba5e03bba870e417ca71634a3d1606425969ba2fad7d3e\": rpc error: code = NotFound desc = could not find container \"0eafe970aeb286d1e6ba5e03bba870e417ca71634a3d1606425969ba2fad7d3e\": container with ID starting with 0eafe970aeb286d1e6ba5e03bba870e417ca71634a3d1606425969ba2fad7d3e not found: ID does not exist"
Nov 02 12:47:20.667957 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:20.667932    4559 scope.go:122] "RemoveContainer" containerID="54d254365c0ad15d60a825a2d5ee152c233fce1ea4d2e496e4b4bcc256a9dc2e"
Nov 02 12:47:20.675133 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:20.675113    4559 patch_prober.go:29] interesting pod/e2e-ipp-b-97de36-openshift-default-77899d747c-cq4bc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body=
Nov 02 12:47:20.675213 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:20.675162    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-97de36-openshift-default-77899d747c-cq4bc" podUID="5b55da6b-89cb-46d6-a7a0-3a09e7d7d17e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused"
Nov 02 12:47:20.677113 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:20.677095    4559 scope.go:122] "RemoveContainer" containerID="54d254365c0ad15d60a825a2d5ee152c233fce1ea4d2e496e4b4bcc256a9dc2e"
Nov 02 12:47:20.677378 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:47:20.677360    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54d254365c0ad15d60a825a2d5ee152c233fce1ea4d2e496e4b4bcc256a9dc2e\": container with ID starting with 54d254365c0ad15d60a825a2d5ee152c233fce1ea4d2e496e4b4bcc256a9dc2e not found: ID does not exist" containerID="54d254365c0ad15d60a825a2d5ee152c233fce1ea4d2e496e4b4bcc256a9dc2e"
Nov 02 12:47:20.677448 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:20.677386    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54d254365c0ad15d60a825a2d5ee152c233fce1ea4d2e496e4b4bcc256a9dc2e"} err="failed to get container status \"54d254365c0ad15d60a825a2d5ee152c233fce1ea4d2e496e4b4bcc256a9dc2e\": rpc error: code = NotFound desc = could not find container \"54d254365c0ad15d60a825a2d5ee152c233fce1ea4d2e496e4b4bcc256a9dc2e\": container with ID starting with 54d254365c0ad15d60a825a2d5ee152c233fce1ea4d2e496e4b4bcc256a9dc2e not found: ID does not exist"
Nov 02 12:47:20.738283 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:20.738208    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-nsg75"]
Nov 02 12:47:20.742489 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:20.742463    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-nsg75"]
Nov 02 12:47:20.776515 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:20.776486    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-rxdxm"]
Nov 02 12:47:20.821662 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:20.814953    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-rxdxm"]
Nov 02 12:47:21.268556 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:21.268480    4559 patch_prober.go:29] interesting pod/e2e-api-a-b2ec05-openshift-default-5f4c648f59-bldmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body=
Nov 02 12:47:21.268556 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:21.268548    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b2ec05-openshift-default-5f4c648f59-bldmv" podUID="0f74d2ef-c796-4c91-b763-7593c16cfe71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused"
Nov 02 12:47:21.675435 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:21.675402    4559 patch_prober.go:29] interesting pod/e2e-ipp-b-97de36-openshift-default-77899d747c-cq4bc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body=
Nov 02 12:47:21.675845 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:21.675459    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-97de36-openshift-default-77899d747c-cq4bc" podUID="5b55da6b-89cb-46d6-a7a0-3a09e7d7d17e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused"
Nov 02 12:47:21.946573 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:21.946491    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f0dd625-e644-4b6a-b697-c7b66b68e6ca" path="/var/lib/kubelet/pods/7f0dd625-e644-4b6a-b697-c7b66b68e6ca/volumes"
Nov 02 12:47:21.949152 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:21.949119    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2989390-a19e-408b-ac9e-ec01df76bf12" path="/var/lib/kubelet/pods/c2989390-a19e-408b-ac9e-ec01df76bf12/volumes"
Nov 02 12:47:22.131955 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:22.131924    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-b2ec05-openshift-default-5f4c648f59-bldmv"
Nov 02 12:47:22.273043 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:22.272965    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-b2ec05-openshift-default-5f4c648f59-bldmv"
Nov 02 12:47:22.661498 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:22.661421    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-b2ec05-openshift-default-5f4c648f59-bldmv"
Nov 02 12:47:22.675875 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:22.675834    4559 patch_prober.go:29] interesting pod/e2e-ipp-b-97de36-openshift-default-77899d747c-cq4bc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body=
Nov 02 12:47:22.676338 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:22.675889    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-97de36-openshift-default-77899d747c-cq4bc" podUID="5b55da6b-89cb-46d6-a7a0-3a09e7d7d17e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused"
Nov 02 12:47:23.270270 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:23.270233    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-f7f44a-bdf9b95cf-2dqpg"]
Nov 02 12:47:23.270883 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:23.270868    4559 cpu_manager.go:414] "containerMap: removing container" podUID="c2989390-a19e-408b-ac9e-ec01df76bf12" containerName="istio-proxy"
Nov 02 12:47:23.270946 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:23.270884    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2989390-a19e-408b-ac9e-ec01df76bf12" containerName="istio-proxy"
Nov 02 12:47:23.270946 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:23.270904    4559 cpu_manager.go:414] "containerMap: removing container" podUID="7f0dd625-e644-4b6a-b697-c7b66b68e6ca" containerName="istio-proxy"
Nov 02 12:47:23.270946 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:23.270909    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f0dd625-e644-4b6a-b697-c7b66b68e6ca" containerName="istio-proxy"
Nov 02 12:47:23.271044 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:23.271022    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2989390-a19e-408b-ac9e-ec01df76bf12" containerName="istio-proxy"
Nov 02 12:47:23.271044 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:23.271033    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2989390-a19e-408b-ac9e-ec01df76bf12" containerName="istio-proxy"
Nov 02 12:47:23.271044 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:23.271042    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f0dd625-e644-4b6a-b697-c7b66b68e6ca" containerName="istio-proxy"
Nov 02 12:47:23.275970 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:23.275953    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f7f44a-bdf9b95cf-2dqpg"
Nov 02 12:47:23.297832 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:23.297800    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-f7f44a-dockercfg-6wf6d\""
Nov 02 12:47:23.305329 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:23.305303    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-f7f44a\""
Nov 02 12:47:23.310778 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:23.310754    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-f7f44a-bdf9b95cf-2dqpg"]
Nov 02 12:47:23.411696 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:23.409024    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84f69d99-5503-471f-95c2-a01256f2a337-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-f7f44a-bdf9b95cf-2dqpg\" (UID: \"84f69d99-5503-471f-95c2-a01256f2a337\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-f7f44a-bdf9b95cf-2dqpg"
Nov 02 12:47:23.411696 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:23.409213    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzrd4\" (UniqueName: \"kubernetes.io/projected/84f69d99-5503-471f-95c2-a01256f2a337-kube-api-access-pzrd4\") pod \"payload-pre-processing-e2e-api-b-f7f44a-bdf9b95cf-2dqpg\" (UID: \"84f69d99-5503-471f-95c2-a01256f2a337\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-f7f44a-bdf9b95cf-2dqpg"
Nov 02 12:47:23.565709 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:23.557809    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84f69d99-5503-471f-95c2-a01256f2a337-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-f7f44a-bdf9b95cf-2dqpg\" (UID: \"84f69d99-5503-471f-95c2-a01256f2a337\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-f7f44a-bdf9b95cf-2dqpg"
Nov 02 12:47:23.565709 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:23.558094    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzrd4\" (UniqueName: \"kubernetes.io/projected/84f69d99-5503-471f-95c2-a01256f2a337-kube-api-access-pzrd4\") pod \"payload-pre-processing-e2e-api-b-f7f44a-bdf9b95cf-2dqpg\" (UID: \"84f69d99-5503-471f-95c2-a01256f2a337\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-f7f44a-bdf9b95cf-2dqpg"
Nov 02 12:47:23.565709 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:23.559929    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84f69d99-5503-471f-95c2-a01256f2a337-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-f7f44a-bdf9b95cf-2dqpg\" (UID: \"84f69d99-5503-471f-95c2-a01256f2a337\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-f7f44a-bdf9b95cf-2dqpg"
Nov 02 12:47:23.587947 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:23.587917    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzrd4\" (UniqueName: \"kubernetes.io/projected/84f69d99-5503-471f-95c2-a01256f2a337-kube-api-access-pzrd4\") pod \"payload-pre-processing-e2e-api-b-f7f44a-bdf9b95cf-2dqpg\" (UID: \"84f69d99-5503-471f-95c2-a01256f2a337\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-f7f44a-bdf9b95cf-2dqpg"
Nov 02 12:47:23.612172 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:23.603961    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f7f44a-bdf9b95cf-2dqpg"
Nov 02 12:47:23.619151 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:23.619012    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f7f44a-584b988b97-tzb42"]
Nov 02 12:47:23.620423 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:23.620330    4559 cpu_manager.go:414] "containerMap: removing container" podUID="c2989390-a19e-408b-ac9e-ec01df76bf12" containerName="istio-proxy"
Nov 02 12:47:23.620423 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:23.620356    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2989390-a19e-408b-ac9e-ec01df76bf12" containerName="istio-proxy"
Nov 02 12:47:23.640603 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:23.640545    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f7f44a-584b988b97-tzb42"]
Nov 02 12:47:23.640937 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:23.640919    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-f7f44a-584b988b97-tzb42"
Nov 02 12:47:23.799679 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:23.691363    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vp59b\" (UniqueName: \"kubernetes.io/projected/e5bd0c86-030e-45be-882f-69cde6c34bac-kube-api-access-vp59b\") pod \"payload-processing-e2e-api-b-f7f44a-584b988b97-tzb42\" (UID: \"e5bd0c86-030e-45be-882f-69cde6c34bac\") " pod="openshift-ingress/payload-processing-e2e-api-b-f7f44a-584b988b97-tzb42"
Nov 02 12:47:23.852237 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:23.852136    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5bd0c86-030e-45be-882f-69cde6c34bac-plugins-config-volume\") pod \"payload-processing-e2e-api-b-f7f44a-584b988b97-tzb42\" (UID: \"e5bd0c86-030e-45be-882f-69cde6c34bac\") " pod="openshift-ingress/payload-processing-e2e-api-b-f7f44a-584b988b97-tzb42"
Nov 02 12:47:23.956839 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:23.954555    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vp59b\" (UniqueName: \"kubernetes.io/projected/e5bd0c86-030e-45be-882f-69cde6c34bac-kube-api-access-vp59b\") pod \"payload-processing-e2e-api-b-f7f44a-584b988b97-tzb42\" (UID: \"e5bd0c86-030e-45be-882f-69cde6c34bac\") " pod="openshift-ingress/payload-processing-e2e-api-b-f7f44a-584b988b97-tzb42"
Nov 02 12:47:23.956839 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:23.954775    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5bd0c86-030e-45be-882f-69cde6c34bac-plugins-config-volume\") pod \"payload-processing-e2e-api-b-f7f44a-584b988b97-tzb42\" (UID: \"e5bd0c86-030e-45be-882f-69cde6c34bac\") " pod="openshift-ingress/payload-processing-e2e-api-b-f7f44a-584b988b97-tzb42"
Nov 02 12:47:23.956839 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:23.955929    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5bd0c86-030e-45be-882f-69cde6c34bac-plugins-config-volume\") pod \"payload-processing-e2e-api-b-f7f44a-584b988b97-tzb42\" (UID: \"e5bd0c86-030e-45be-882f-69cde6c34bac\") " pod="openshift-ingress/payload-processing-e2e-api-b-f7f44a-584b988b97-tzb42"
Nov 02 12:47:24.023062 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:24.022047    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vp59b\" (UniqueName: \"kubernetes.io/projected/e5bd0c86-030e-45be-882f-69cde6c34bac-kube-api-access-vp59b\") pod \"payload-processing-e2e-api-b-f7f44a-584b988b97-tzb42\" (UID: \"e5bd0c86-030e-45be-882f-69cde6c34bac\") " pod="openshift-ingress/payload-processing-e2e-api-b-f7f44a-584b988b97-tzb42"
Nov 02 12:47:24.093126 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:24.093092    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-97de36-openshift-default-77899d747c-cq4bc"
Nov 02 12:47:24.095784 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:24.095750    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-97de36-openshift-default-77899d747c-cq4bc"
Nov 02 12:47:24.211204 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:24.211171    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-f7f44a-bdf9b95cf-2dqpg"]
Nov 02 12:47:24.211699 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:47:24.211669    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84f69d99_5503_471f_95c2_a01256f2a337.slice/crio-7fdcab877b08a834e6d0c4525de1d742cb37a412df1d51176c0ea621cfe5798c WatchSource:0}: Error finding container 7fdcab877b08a834e6d0c4525de1d742cb37a412df1d51176c0ea621cfe5798c: Status 404 returned error can't find the container with id 7fdcab877b08a834e6d0c4525de1d742cb37a412df1d51176c0ea621cfe5798c
Nov 02 12:47:24.218391 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:24.218373    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-f7f44a-584b988b97-tzb42"
Nov 02 12:47:24.542817 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:47:24.542594    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5bd0c86_030e_45be_882f_69cde6c34bac.slice/crio-5e3d2389d8277092c548e755678d91d96fcaf2993adfa52b088880add3788fcc WatchSource:0}: Error finding container 5e3d2389d8277092c548e755678d91d96fcaf2993adfa52b088880add3788fcc: Status 404 returned error can't find the container with id 5e3d2389d8277092c548e755678d91d96fcaf2993adfa52b088880add3788fcc
Nov 02 12:47:24.542929 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:24.542906    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f7f44a-584b988b97-tzb42"]
Nov 02 12:47:24.904106 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:24.904025    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f7f44a-bdf9b95cf-2dqpg" event={"ID":"84f69d99-5503-471f-95c2-a01256f2a337","Type":"ContainerStarted","Data":"5d4c1eaebbc1e90b236fbbcc6127b6372bd7c0a38da48cd41a29adba35695096"}
Nov 02 12:47:24.904106 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:24.904069    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f7f44a-bdf9b95cf-2dqpg" event={"ID":"84f69d99-5503-471f-95c2-a01256f2a337","Type":"ContainerStarted","Data":"7fdcab877b08a834e6d0c4525de1d742cb37a412df1d51176c0ea621cfe5798c"}
Nov 02 12:47:24.904582 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:24.904123    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f7f44a-bdf9b95cf-2dqpg"
Nov 02 12:47:24.905780 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:24.905751    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-f7f44a-584b988b97-tzb42" event={"ID":"e5bd0c86-030e-45be-882f-69cde6c34bac","Type":"ContainerStarted","Data":"bf05045d118f0f82364704b2763594f15cfc326c55fef9507b6a4e98b0c9fbe0"}
Nov 02 12:47:24.905907 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:24.905800    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-f7f44a-584b988b97-tzb42" event={"ID":"e5bd0c86-030e-45be-882f-69cde6c34bac","Type":"ContainerStarted","Data":"5e3d2389d8277092c548e755678d91d96fcaf2993adfa52b088880add3788fcc"}
Nov 02 12:47:24.905986 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:24.905969    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-f7f44a-584b988b97-tzb42"
Nov 02 12:47:24.954713 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:24.954666    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f7f44a-bdf9b95cf-2dqpg" podStartSLOduration=1.954631969 podStartE2EDuration="1.954631969s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:47:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:47:24.951494377 +0000 UTC m=+1241.206893475" watchObservedRunningTime="2026-11-02 12:47:24.954631969 +0000 UTC m=+1241.210031081"
Nov 02 12:47:24.999886 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:24.999843    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-f7f44a-584b988b97-tzb42" podStartSLOduration=1.999831367 podStartE2EDuration="1.999831367s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:47:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:47:24.997041886 +0000 UTC m=+1241.252440982" watchObservedRunningTime="2026-11-02 12:47:24.999831367 +0000 UTC m=+1241.255230461"
Nov 02 12:47:29.534819 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:29.534785    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-729108-5c8649cbff-9xhdl"]
Nov 02 12:47:29.590447 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:29.590415    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-729108-6df46f4ff9-qw5p5"]
Nov 02 12:47:32.181433 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:32.181402    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-97de36-59d99bc75f-hbhc5"]
Nov 02 12:47:32.236712 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:32.227827    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-97de36-59d99bc75f-hbhc5"]
Nov 02 12:47:32.236712 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:32.227971    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-97de36-59d99bc75f-hbhc5"
Nov 02 12:47:32.270934 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:32.265737    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-97de36-744bd9758d-jhcpn"]
Nov 02 12:47:32.276657 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:32.275934    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-97de36\""
Nov 02 12:47:32.276657 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:32.276035    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-97de36-dockercfg-4blv8\""
Nov 02 12:47:32.281359 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:32.277690    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-97de36-744bd9758d-jhcpn"
Nov 02 12:47:32.283799 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:32.283062    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/160b5200-de01-4a04-a42a-6504f7bf5685-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-97de36-744bd9758d-jhcpn\" (UID: \"160b5200-de01-4a04-a42a-6504f7bf5685\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-97de36-744bd9758d-jhcpn"
Nov 02 12:47:32.283799 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:32.283105    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tswgg\" (UniqueName: \"kubernetes.io/projected/160b5200-de01-4a04-a42a-6504f7bf5685-kube-api-access-tswgg\") pod \"payload-processing-e2e-ipp-b-97de36-744bd9758d-jhcpn\" (UID: \"160b5200-de01-4a04-a42a-6504f7bf5685\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-97de36-744bd9758d-jhcpn"
Nov 02 12:47:32.283799 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:32.283217    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9b2a7c1-69df-4109-b678-147aad2d044e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-97de36-59d99bc75f-hbhc5\" (UID: \"d9b2a7c1-69df-4109-b678-147aad2d044e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-97de36-59d99bc75f-hbhc5"
Nov 02 12:47:32.283799 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:32.283267    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4qpb\" (UniqueName: \"kubernetes.io/projected/d9b2a7c1-69df-4109-b678-147aad2d044e-kube-api-access-f4qpb\") pod \"payload-pre-processing-e2e-ipp-b-97de36-59d99bc75f-hbhc5\" (UID: \"d9b2a7c1-69df-4109-b678-147aad2d044e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-97de36-59d99bc75f-hbhc5"
Nov 02 12:47:32.309390 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:32.309364    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-97de36-744bd9758d-jhcpn"]
Nov 02 12:47:32.384463 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:32.384423    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/160b5200-de01-4a04-a42a-6504f7bf5685-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-97de36-744bd9758d-jhcpn\" (UID: \"160b5200-de01-4a04-a42a-6504f7bf5685\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-97de36-744bd9758d-jhcpn"
Nov 02 12:47:32.384625 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:32.384478    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tswgg\" (UniqueName: \"kubernetes.io/projected/160b5200-de01-4a04-a42a-6504f7bf5685-kube-api-access-tswgg\") pod \"payload-processing-e2e-ipp-b-97de36-744bd9758d-jhcpn\" (UID: \"160b5200-de01-4a04-a42a-6504f7bf5685\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-97de36-744bd9758d-jhcpn"
Nov 02 12:47:32.384625 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:32.384583    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9b2a7c1-69df-4109-b678-147aad2d044e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-97de36-59d99bc75f-hbhc5\" (UID: \"d9b2a7c1-69df-4109-b678-147aad2d044e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-97de36-59d99bc75f-hbhc5"
Nov 02 12:47:32.384783 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:32.384651    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4qpb\" (UniqueName: \"kubernetes.io/projected/d9b2a7c1-69df-4109-b678-147aad2d044e-kube-api-access-f4qpb\") pod \"payload-pre-processing-e2e-ipp-b-97de36-59d99bc75f-hbhc5\" (UID: \"d9b2a7c1-69df-4109-b678-147aad2d044e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-97de36-59d99bc75f-hbhc5"
Nov 02 12:47:32.385439 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:32.385407    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/160b5200-de01-4a04-a42a-6504f7bf5685-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-97de36-744bd9758d-jhcpn\" (UID: \"160b5200-de01-4a04-a42a-6504f7bf5685\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-97de36-744bd9758d-jhcpn"
Nov 02 12:47:32.385549 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:32.385500    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9b2a7c1-69df-4109-b678-147aad2d044e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-97de36-59d99bc75f-hbhc5\" (UID: \"d9b2a7c1-69df-4109-b678-147aad2d044e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-97de36-59d99bc75f-hbhc5"
Nov 02 12:47:32.409808 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:32.409773    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4qpb\" (UniqueName: \"kubernetes.io/projected/d9b2a7c1-69df-4109-b678-147aad2d044e-kube-api-access-f4qpb\") pod \"payload-pre-processing-e2e-ipp-b-97de36-59d99bc75f-hbhc5\" (UID: \"d9b2a7c1-69df-4109-b678-147aad2d044e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-97de36-59d99bc75f-hbhc5"
Nov 02 12:47:32.410442 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:32.410417    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tswgg\" (UniqueName: \"kubernetes.io/projected/160b5200-de01-4a04-a42a-6504f7bf5685-kube-api-access-tswgg\") pod \"payload-processing-e2e-ipp-b-97de36-744bd9758d-jhcpn\" (UID: \"160b5200-de01-4a04-a42a-6504f7bf5685\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-97de36-744bd9758d-jhcpn"
Nov 02 12:47:32.602486 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:32.602401    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-97de36-59d99bc75f-hbhc5"
Nov 02 12:47:32.612249 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:32.612221    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-97de36-744bd9758d-jhcpn"
Nov 02 12:47:32.939941 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:47:32.939912    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod160b5200_de01_4a04_a42a_6504f7bf5685.slice/crio-80d50a2d8da3f70d89e1c1439cd550e4e20246bfe55db1a0c0bf4b07033ed071 WatchSource:0}: Error finding container 80d50a2d8da3f70d89e1c1439cd550e4e20246bfe55db1a0c0bf4b07033ed071: Status 404 returned error can't find the container with id 80d50a2d8da3f70d89e1c1439cd550e4e20246bfe55db1a0c0bf4b07033ed071
Nov 02 12:47:32.941479 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:32.941451    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-97de36-744bd9758d-jhcpn"]
Nov 02 12:47:32.947721 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:47:32.947691    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9b2a7c1_69df_4109_b678_147aad2d044e.slice/crio-6be106cc9aae7655637c95c6587260495df791135ba60804b40c41fc82e91f6a WatchSource:0}: Error finding container 6be106cc9aae7655637c95c6587260495df791135ba60804b40c41fc82e91f6a: Status 404 returned error can't find the container with id 6be106cc9aae7655637c95c6587260495df791135ba60804b40c41fc82e91f6a
Nov 02 12:47:32.950258 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:32.950237    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-97de36-59d99bc75f-hbhc5"]
Nov 02 12:47:32.966877 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:32.966851    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-97de36-59d99bc75f-hbhc5" event={"ID":"d9b2a7c1-69df-4109-b678-147aad2d044e","Type":"ContainerStarted","Data":"6be106cc9aae7655637c95c6587260495df791135ba60804b40c41fc82e91f6a"}
Nov 02 12:47:32.968288 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:32.968265    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-97de36-744bd9758d-jhcpn" event={"ID":"160b5200-de01-4a04-a42a-6504f7bf5685","Type":"ContainerStarted","Data":"80d50a2d8da3f70d89e1c1439cd550e4e20246bfe55db1a0c0bf4b07033ed071"}
Nov 02 12:47:33.976022 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:33.975988    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-97de36-59d99bc75f-hbhc5" event={"ID":"d9b2a7c1-69df-4109-b678-147aad2d044e","Type":"ContainerStarted","Data":"f12160d94647216e3b92ab1edaf4b30fd590d9c6b59f20dce904202acd860d67"}
Nov 02 12:47:33.976525 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:33.976054    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-97de36-59d99bc75f-hbhc5"
Nov 02 12:47:33.977627 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:33.977586    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-97de36-744bd9758d-jhcpn" event={"ID":"160b5200-de01-4a04-a42a-6504f7bf5685","Type":"ContainerStarted","Data":"87d0fe472fd54350d7f0e63db0115d302569608f9b561904b7b734814f6aad30"}
Nov 02 12:47:33.977755 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:33.977658    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-97de36-744bd9758d-jhcpn"
Nov 02 12:47:34.037008 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:34.036955    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-97de36-59d99bc75f-hbhc5" podStartSLOduration=2.036940795 podStartE2EDuration="2.036940795s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:47:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:47:34.036132844 +0000 UTC m=+1250.291531965" watchObservedRunningTime="2026-11-02 12:47:34.036940795 +0000 UTC m=+1250.292339889"
Nov 02 12:47:34.106973 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:34.106907    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-97de36-744bd9758d-jhcpn" podStartSLOduration=2.106886183 podStartE2EDuration="2.106886183s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:47:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:47:34.105960099 +0000 UTC m=+1250.361359196" watchObservedRunningTime="2026-11-02 12:47:34.106886183 +0000 UTC m=+1250.362285281"
Nov 02 12:47:34.536665 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:34.536187    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-729108-5c8649cbff-9xhdl" podUID="0ed81343-a77a-4401-bc65-21c804cfbead" containerName="payload-processing" containerID="cri-o://09d862a8ca9b796daf03cf836eedc91dba22752c8e9be1e80d18c3c05d05eecb" gracePeriod=25
Nov 02 12:47:34.597663 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:34.596716    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-729108-6df46f4ff9-qw5p5" podUID="b98cdf4a-b4e0-46f3-b791-5cc63707a366" containerName="payload-pre-processing" containerID="cri-o://7bdffcb5acd75f5193bc9ab46b5343c6b9855224d00bd74d5ee35881af05ae7f" gracePeriod=25
Nov 02 12:47:34.798232 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:47:34.798144    4559 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb98cdf4a_b4e0_46f3_b791_5cc63707a366.slice/crio-conmon-7bdffcb5acd75f5193bc9ab46b5343c6b9855224d00bd74d5ee35881af05ae7f.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb98cdf4a_b4e0_46f3_b791_5cc63707a366.slice/crio-7bdffcb5acd75f5193bc9ab46b5343c6b9855224d00bd74d5ee35881af05ae7f.scope\": RecentStats: unable to find data in memory cache]"
Nov 02 12:47:34.938877 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:34.938847    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-729108-5c8649cbff-9xhdl"
Nov 02 12:47:34.971325 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:34.971301    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-729108-6df46f4ff9-qw5p5"
Nov 02 12:47:34.992969 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:34.992937    4559 generic.go:356] "Generic (PLEG): container finished" podID="0ed81343-a77a-4401-bc65-21c804cfbead" containerID="09d862a8ca9b796daf03cf836eedc91dba22752c8e9be1e80d18c3c05d05eecb" exitCode=0
Nov 02 12:47:34.993397 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:34.992980    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-729108-5c8649cbff-9xhdl"
Nov 02 12:47:34.993397 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:34.993020    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-729108-5c8649cbff-9xhdl" event={"ID":"0ed81343-a77a-4401-bc65-21c804cfbead","Type":"ContainerDied","Data":"09d862a8ca9b796daf03cf836eedc91dba22752c8e9be1e80d18c3c05d05eecb"}
Nov 02 12:47:34.993397 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:34.993111    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-729108-5c8649cbff-9xhdl" event={"ID":"0ed81343-a77a-4401-bc65-21c804cfbead","Type":"ContainerDied","Data":"c8fb1940d6c7564239c13cce1b67cf59eebf409f954f005b3a503e2164476aaa"}
Nov 02 12:47:34.993397 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:34.993145    4559 scope.go:122] "RemoveContainer" containerID="09d862a8ca9b796daf03cf836eedc91dba22752c8e9be1e80d18c3c05d05eecb"
Nov 02 12:47:34.994508 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:34.994482    4559 generic.go:356] "Generic (PLEG): container finished" podID="b98cdf4a-b4e0-46f3-b791-5cc63707a366" containerID="7bdffcb5acd75f5193bc9ab46b5343c6b9855224d00bd74d5ee35881af05ae7f" exitCode=0
Nov 02 12:47:34.994628 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:34.994567    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-729108-6df46f4ff9-qw5p5" event={"ID":"b98cdf4a-b4e0-46f3-b791-5cc63707a366","Type":"ContainerDied","Data":"7bdffcb5acd75f5193bc9ab46b5343c6b9855224d00bd74d5ee35881af05ae7f"}
Nov 02 12:47:34.994628 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:34.994614    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-729108-6df46f4ff9-qw5p5" event={"ID":"b98cdf4a-b4e0-46f3-b791-5cc63707a366","Type":"ContainerDied","Data":"b21e0524a87c05807f664334e92d80960f1221fa5cae609edd0c637cccd11322"}
Nov 02 12:47:34.995910 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:34.995892    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-729108-6df46f4ff9-qw5p5"
Nov 02 12:47:35.101897 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:35.093073    4559 scope.go:122] "RemoveContainer" containerID="09d862a8ca9b796daf03cf836eedc91dba22752c8e9be1e80d18c3c05d05eecb"
Nov 02 12:47:35.101897 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:47:35.099504    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09d862a8ca9b796daf03cf836eedc91dba22752c8e9be1e80d18c3c05d05eecb\": container with ID starting with 09d862a8ca9b796daf03cf836eedc91dba22752c8e9be1e80d18c3c05d05eecb not found: ID does not exist" containerID="09d862a8ca9b796daf03cf836eedc91dba22752c8e9be1e80d18c3c05d05eecb"
Nov 02 12:47:35.101897 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:35.099573    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09d862a8ca9b796daf03cf836eedc91dba22752c8e9be1e80d18c3c05d05eecb"} err="failed to get container status \"09d862a8ca9b796daf03cf836eedc91dba22752c8e9be1e80d18c3c05d05eecb\": rpc error: code = NotFound desc = could not find container \"09d862a8ca9b796daf03cf836eedc91dba22752c8e9be1e80d18c3c05d05eecb\": container with ID starting with 09d862a8ca9b796daf03cf836eedc91dba22752c8e9be1e80d18c3c05d05eecb not found: ID does not exist"
Nov 02 12:47:35.101897 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:35.099602    4559 scope.go:122] "RemoveContainer" containerID="7bdffcb5acd75f5193bc9ab46b5343c6b9855224d00bd74d5ee35881af05ae7f"
Nov 02 12:47:35.111398 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:35.110934    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ed81343-a77a-4401-bc65-21c804cfbead-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ed81343-a77a-4401-bc65-21c804cfbead-plugins-config-volume\") pod \"0ed81343-a77a-4401-bc65-21c804cfbead\" (UID: \"0ed81343-a77a-4401-bc65-21c804cfbead\") "
Nov 02 12:47:35.111398 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:35.111027    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b98cdf4a-b4e0-46f3-b791-5cc63707a366-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b98cdf4a-b4e0-46f3-b791-5cc63707a366-plugins-config-volume\") pod \"b98cdf4a-b4e0-46f3-b791-5cc63707a366\" (UID: \"b98cdf4a-b4e0-46f3-b791-5cc63707a366\") "
Nov 02 12:47:35.111398 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:35.111055    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b98cdf4a-b4e0-46f3-b791-5cc63707a366-kube-api-access-8jf2c\" (UniqueName: \"kubernetes.io/projected/b98cdf4a-b4e0-46f3-b791-5cc63707a366-kube-api-access-8jf2c\") pod \"b98cdf4a-b4e0-46f3-b791-5cc63707a366\" (UID: \"b98cdf4a-b4e0-46f3-b791-5cc63707a366\") "
Nov 02 12:47:35.111398 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:35.111118    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ed81343-a77a-4401-bc65-21c804cfbead-kube-api-access-j85d2\" (UniqueName: \"kubernetes.io/projected/0ed81343-a77a-4401-bc65-21c804cfbead-kube-api-access-j85d2\") pod \"0ed81343-a77a-4401-bc65-21c804cfbead\" (UID: \"0ed81343-a77a-4401-bc65-21c804cfbead\") "
Nov 02 12:47:35.112697 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:35.112658    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b98cdf4a-b4e0-46f3-b791-5cc63707a366-plugins-config-volume" pod "b98cdf4a-b4e0-46f3-b791-5cc63707a366" (UID: "b98cdf4a-b4e0-46f3-b791-5cc63707a366"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:47:35.113053 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:35.113033    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ed81343-a77a-4401-bc65-21c804cfbead-plugins-config-volume" pod "0ed81343-a77a-4401-bc65-21c804cfbead" (UID: "0ed81343-a77a-4401-bc65-21c804cfbead"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:47:35.114786 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:35.114763    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b98cdf4a-b4e0-46f3-b791-5cc63707a366-kube-api-access-8jf2c" pod "b98cdf4a-b4e0-46f3-b791-5cc63707a366" (UID: "b98cdf4a-b4e0-46f3-b791-5cc63707a366"). InnerVolumeSpecName "kube-api-access-8jf2c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:47:35.115885 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:35.115852    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ed81343-a77a-4401-bc65-21c804cfbead-kube-api-access-j85d2" pod "0ed81343-a77a-4401-bc65-21c804cfbead" (UID: "0ed81343-a77a-4401-bc65-21c804cfbead"). InnerVolumeSpecName "kube-api-access-j85d2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:47:35.123356 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:35.123338    4559 scope.go:122] "RemoveContainer" containerID="7bdffcb5acd75f5193bc9ab46b5343c6b9855224d00bd74d5ee35881af05ae7f"
Nov 02 12:47:35.123681 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:47:35.123655    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7bdffcb5acd75f5193bc9ab46b5343c6b9855224d00bd74d5ee35881af05ae7f\": container with ID starting with 7bdffcb5acd75f5193bc9ab46b5343c6b9855224d00bd74d5ee35881af05ae7f not found: ID does not exist" containerID="7bdffcb5acd75f5193bc9ab46b5343c6b9855224d00bd74d5ee35881af05ae7f"
Nov 02 12:47:35.123765 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:35.123692    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7bdffcb5acd75f5193bc9ab46b5343c6b9855224d00bd74d5ee35881af05ae7f"} err="failed to get container status \"7bdffcb5acd75f5193bc9ab46b5343c6b9855224d00bd74d5ee35881af05ae7f\": rpc error: code = NotFound desc = could not find container \"7bdffcb5acd75f5193bc9ab46b5343c6b9855224d00bd74d5ee35881af05ae7f\": container with ID starting with 7bdffcb5acd75f5193bc9ab46b5343c6b9855224d00bd74d5ee35881af05ae7f not found: ID does not exist"
Nov 02 12:47:35.213294 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:35.213261    4559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b98cdf4a-b4e0-46f3-b791-5cc63707a366-plugins-config-volume\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:47:35.213294 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:35.213287    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8jf2c\" (UniqueName: \"kubernetes.io/projected/b98cdf4a-b4e0-46f3-b791-5cc63707a366-kube-api-access-8jf2c\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:47:35.213446 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:35.213302    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j85d2\" (UniqueName: \"kubernetes.io/projected/0ed81343-a77a-4401-bc65-21c804cfbead-kube-api-access-j85d2\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:47:35.213446 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:35.213315    4559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ed81343-a77a-4401-bc65-21c804cfbead-plugins-config-volume\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:47:35.342321 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:35.342295    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-729108-5c8649cbff-9xhdl"]
Nov 02 12:47:35.346773 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:35.346750    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-729108-5c8649cbff-9xhdl"]
Nov 02 12:47:35.382800 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:35.382745    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-729108-6df46f4ff9-qw5p5"]
Nov 02 12:47:35.386538 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:35.386520    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-729108-6df46f4ff9-qw5p5"]
Nov 02 12:47:35.894115 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:35.894085    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ed81343-a77a-4401-bc65-21c804cfbead" path="/var/lib/kubelet/pods/0ed81343-a77a-4401-bc65-21c804cfbead/volumes"
Nov 02 12:47:35.894536 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:35.894523    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b98cdf4a-b4e0-46f3-b791-5cc63707a366" path="/var/lib/kubelet/pods/b98cdf4a-b4e0-46f3-b791-5cc63707a366/volumes"
Nov 02 12:47:35.916218 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:35.916194    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f7f44a-bdf9b95cf-2dqpg"
Nov 02 12:47:35.916341 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:35.916250    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-f7f44a-584b988b97-tzb42"
Nov 02 12:47:43.157631 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:43.157588    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"]
Nov 02 12:47:44.997730 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:44.997706    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-97de36-59d99bc75f-hbhc5"
Nov 02 12:47:44.998085 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:44.997780    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-97de36-744bd9758d-jhcpn"
Nov 02 12:47:45.112903 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:47:45.112862    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqcmj" containerName="extractor"
Nov 02 12:47:45.285242 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:45.285151    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-77cf70-99dbd47d7-4rbjg"]
Nov 02 12:47:45.285416 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:45.285389    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-77cf70-99dbd47d7-4rbjg" podUID="630a0e10-db9e-42e4-80a9-2268163c591a" containerName="maas-api" containerID="cri-o://ae229b6d4b81f3b0dc2c8a60559d000be699e7bed4e6aa1fd9f81132da2e5b88" gracePeriod=30
Nov 02 12:47:45.437037 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:45.437007    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-77cf70-8686479b7b-mcd6f"]
Nov 02 12:47:45.437282 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:45.437180    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-77cf70-8686479b7b-mcd6f" podUID="6ee04a57-e5e9-48fa-a5a8-c89bcf917616" containerName="payload-processing" containerID="cri-o://dd9e4200d5332ef941b46088cb0070d1c6d40ee4933dd60bedd4e88737c2e1a8" gracePeriod=30
Nov 02 12:47:45.457782 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:45.457754    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-77cf70-86cf7b694c-45888"]
Nov 02 12:47:45.458012 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:45.457990    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-77cf70-86cf7b694c-45888" podUID="bc18257f-4549-4cf1-9fe5-227496db7c80" containerName="payload-pre-processing" containerID="cri-o://a1d8ea82c378d3fbb5c3c1ca0a57a8200213b14ba9a1d88cf8b784efe0cd4252" gracePeriod=30
Nov 02 12:47:45.604501 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:45.604470    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-77cf70-99dbd47d7-4rbjg"
Nov 02 12:47:45.700824 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:45.700784    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/630a0e10-db9e-42e4-80a9-2268163c591a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/630a0e10-db9e-42e4-80a9-2268163c591a-maas-api-tls\") pod \"630a0e10-db9e-42e4-80a9-2268163c591a\" (UID: \"630a0e10-db9e-42e4-80a9-2268163c591a\") "
Nov 02 12:47:45.700999 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:45.700858    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/630a0e10-db9e-42e4-80a9-2268163c591a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/630a0e10-db9e-42e4-80a9-2268163c591a-metrics-certs\") pod \"630a0e10-db9e-42e4-80a9-2268163c591a\" (UID: \"630a0e10-db9e-42e4-80a9-2268163c591a\") "
Nov 02 12:47:45.700999 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:45.700922    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/630a0e10-db9e-42e4-80a9-2268163c591a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/630a0e10-db9e-42e4-80a9-2268163c591a-openshift-service-ca\") pod \"630a0e10-db9e-42e4-80a9-2268163c591a\" (UID: \"630a0e10-db9e-42e4-80a9-2268163c591a\") "
Nov 02 12:47:45.701123 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:45.701086    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/630a0e10-db9e-42e4-80a9-2268163c591a-kube-api-access-2v4ll\" (UniqueName: \"kubernetes.io/projected/630a0e10-db9e-42e4-80a9-2268163c591a-kube-api-access-2v4ll\") pod \"630a0e10-db9e-42e4-80a9-2268163c591a\" (UID: \"630a0e10-db9e-42e4-80a9-2268163c591a\") "
Nov 02 12:47:45.702255 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:45.702225    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/630a0e10-db9e-42e4-80a9-2268163c591a-openshift-service-ca" pod "630a0e10-db9e-42e4-80a9-2268163c591a" (UID: "630a0e10-db9e-42e4-80a9-2268163c591a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:47:45.709046 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:45.709020    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/630a0e10-db9e-42e4-80a9-2268163c591a-kube-api-access-2v4ll" pod "630a0e10-db9e-42e4-80a9-2268163c591a" (UID: "630a0e10-db9e-42e4-80a9-2268163c591a"). InnerVolumeSpecName "kube-api-access-2v4ll". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:47:45.713957 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:45.713934    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/630a0e10-db9e-42e4-80a9-2268163c591a-metrics-certs" pod "630a0e10-db9e-42e4-80a9-2268163c591a" (UID: "630a0e10-db9e-42e4-80a9-2268163c591a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:47:45.715274 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:45.715250    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/630a0e10-db9e-42e4-80a9-2268163c591a-maas-api-tls" pod "630a0e10-db9e-42e4-80a9-2268163c591a" (UID: "630a0e10-db9e-42e4-80a9-2268163c591a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:47:45.748063 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:45.748036    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-77cf70-8686479b7b-mcd6f"
Nov 02 12:47:45.883707 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:45.882890    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/630a0e10-db9e-42e4-80a9-2268163c591a-maas-api-tls\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:47:45.883707 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:45.882925    4559 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/630a0e10-db9e-42e4-80a9-2268163c591a-metrics-certs\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:47:45.883707 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:45.882938    4559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/630a0e10-db9e-42e4-80a9-2268163c591a-openshift-service-ca\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:47:45.883707 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:45.882951    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2v4ll\" (UniqueName: \"kubernetes.io/projected/630a0e10-db9e-42e4-80a9-2268163c591a-kube-api-access-2v4ll\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:47:45.993669 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:45.991053    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ee04a57-e5e9-48fa-a5a8-c89bcf917616-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ee04a57-e5e9-48fa-a5a8-c89bcf917616-plugins-config-volume\") pod \"6ee04a57-e5e9-48fa-a5a8-c89bcf917616\" (UID: \"6ee04a57-e5e9-48fa-a5a8-c89bcf917616\") "
Nov 02 12:47:45.993669 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:45.991166    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ee04a57-e5e9-48fa-a5a8-c89bcf917616-kube-api-access-8gchq\" (UniqueName: \"kubernetes.io/projected/6ee04a57-e5e9-48fa-a5a8-c89bcf917616-kube-api-access-8gchq\") pod \"6ee04a57-e5e9-48fa-a5a8-c89bcf917616\" (UID: \"6ee04a57-e5e9-48fa-a5a8-c89bcf917616\") "
Nov 02 12:47:46.004735 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:46.001727    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ee04a57-e5e9-48fa-a5a8-c89bcf917616-plugins-config-volume" pod "6ee04a57-e5e9-48fa-a5a8-c89bcf917616" (UID: "6ee04a57-e5e9-48fa-a5a8-c89bcf917616"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:47:46.018837 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:46.018805    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ee04a57-e5e9-48fa-a5a8-c89bcf917616-kube-api-access-8gchq" pod "6ee04a57-e5e9-48fa-a5a8-c89bcf917616" (UID: "6ee04a57-e5e9-48fa-a5a8-c89bcf917616"). InnerVolumeSpecName "kube-api-access-8gchq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:47:46.026678 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:46.026490    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-77cf70-86cf7b694c-45888"
Nov 02 12:47:46.091385 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:46.091348    4559 generic.go:356] "Generic (PLEG): container finished" podID="6ee04a57-e5e9-48fa-a5a8-c89bcf917616" containerID="dd9e4200d5332ef941b46088cb0070d1c6d40ee4933dd60bedd4e88737c2e1a8" exitCode=0
Nov 02 12:47:46.091385 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:46.091390    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-77cf70-8686479b7b-mcd6f"
Nov 02 12:47:46.091676 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:46.091439    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-77cf70-8686479b7b-mcd6f" event={"ID":"6ee04a57-e5e9-48fa-a5a8-c89bcf917616","Type":"ContainerDied","Data":"dd9e4200d5332ef941b46088cb0070d1c6d40ee4933dd60bedd4e88737c2e1a8"}
Nov 02 12:47:46.091676 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:46.091501    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-77cf70-8686479b7b-mcd6f" event={"ID":"6ee04a57-e5e9-48fa-a5a8-c89bcf917616","Type":"ContainerDied","Data":"7bca123cb8bca1e9f4a93957626fe9d8256fd1cca29846f150ef7a1cfc974722"}
Nov 02 12:47:46.091676 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:46.091532    4559 scope.go:122] "RemoveContainer" containerID="dd9e4200d5332ef941b46088cb0070d1c6d40ee4933dd60bedd4e88737c2e1a8"
Nov 02 12:47:46.092932 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:46.092904    4559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ee04a57-e5e9-48fa-a5a8-c89bcf917616-plugins-config-volume\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:47:46.093068 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:46.092935    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8gchq\" (UniqueName: \"kubernetes.io/projected/6ee04a57-e5e9-48fa-a5a8-c89bcf917616-kube-api-access-8gchq\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:47:46.094938 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:46.094913    4559 generic.go:356] "Generic (PLEG): container finished" podID="630a0e10-db9e-42e4-80a9-2268163c591a" containerID="ae229b6d4b81f3b0dc2c8a60559d000be699e7bed4e6aa1fd9f81132da2e5b88" exitCode=0
Nov 02 12:47:46.095040 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:46.095028    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-77cf70-99dbd47d7-4rbjg" event={"ID":"630a0e10-db9e-42e4-80a9-2268163c591a","Type":"ContainerDied","Data":"ae229b6d4b81f3b0dc2c8a60559d000be699e7bed4e6aa1fd9f81132da2e5b88"}
Nov 02 12:47:46.095178 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:46.095071    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-77cf70-99dbd47d7-4rbjg" event={"ID":"630a0e10-db9e-42e4-80a9-2268163c591a","Type":"ContainerDied","Data":"0b8a4bdda3cd79f3ce2120ab753b4cd8895e44543a60811116c4259a1674eb66"}
Nov 02 12:47:46.095178 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:46.095158    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-77cf70-99dbd47d7-4rbjg"
Nov 02 12:47:46.097401 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:46.097378    4559 generic.go:356] "Generic (PLEG): container finished" podID="bc18257f-4549-4cf1-9fe5-227496db7c80" containerID="a1d8ea82c378d3fbb5c3c1ca0a57a8200213b14ba9a1d88cf8b784efe0cd4252" exitCode=0
Nov 02 12:47:46.098543 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:46.098226    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-77cf70-86cf7b694c-45888"
Nov 02 12:47:46.098717 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:46.098244    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-77cf70-86cf7b694c-45888" event={"ID":"bc18257f-4549-4cf1-9fe5-227496db7c80","Type":"ContainerDied","Data":"a1d8ea82c378d3fbb5c3c1ca0a57a8200213b14ba9a1d88cf8b784efe0cd4252"}
Nov 02 12:47:46.098822 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:46.098745    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-77cf70-86cf7b694c-45888" event={"ID":"bc18257f-4549-4cf1-9fe5-227496db7c80","Type":"ContainerDied","Data":"103a1295884f274165fd4fe01692e934a624a19dfa3a9d5a3bcb4ea36116a3ea"}
Nov 02 12:47:46.106622 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:46.106604    4559 scope.go:122] "RemoveContainer" containerID="dd9e4200d5332ef941b46088cb0070d1c6d40ee4933dd60bedd4e88737c2e1a8"
Nov 02 12:47:46.106924 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:47:46.106904    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd9e4200d5332ef941b46088cb0070d1c6d40ee4933dd60bedd4e88737c2e1a8\": container with ID starting with dd9e4200d5332ef941b46088cb0070d1c6d40ee4933dd60bedd4e88737c2e1a8 not found: ID does not exist" containerID="dd9e4200d5332ef941b46088cb0070d1c6d40ee4933dd60bedd4e88737c2e1a8"
Nov 02 12:47:46.107001 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:46.106931    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd9e4200d5332ef941b46088cb0070d1c6d40ee4933dd60bedd4e88737c2e1a8"} err="failed to get container status \"dd9e4200d5332ef941b46088cb0070d1c6d40ee4933dd60bedd4e88737c2e1a8\": rpc error: code = NotFound desc = could not find container \"dd9e4200d5332ef941b46088cb0070d1c6d40ee4933dd60bedd4e88737c2e1a8\": container with ID starting with dd9e4200d5332ef941b46088cb0070d1c6d40ee4933dd60bedd4e88737c2e1a8 not found: ID does not exist"
Nov 02 12:47:46.107001 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:46.106948    4559 scope.go:122] "RemoveContainer" containerID="ae229b6d4b81f3b0dc2c8a60559d000be699e7bed4e6aa1fd9f81132da2e5b88"
Nov 02 12:47:46.122999 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:46.122921    4559 scope.go:122] "RemoveContainer" containerID="ae229b6d4b81f3b0dc2c8a60559d000be699e7bed4e6aa1fd9f81132da2e5b88"
Nov 02 12:47:46.123395 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:47:46.123274    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae229b6d4b81f3b0dc2c8a60559d000be699e7bed4e6aa1fd9f81132da2e5b88\": container with ID starting with ae229b6d4b81f3b0dc2c8a60559d000be699e7bed4e6aa1fd9f81132da2e5b88 not found: ID does not exist" containerID="ae229b6d4b81f3b0dc2c8a60559d000be699e7bed4e6aa1fd9f81132da2e5b88"
Nov 02 12:47:46.123395 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:46.123307    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae229b6d4b81f3b0dc2c8a60559d000be699e7bed4e6aa1fd9f81132da2e5b88"} err="failed to get container status \"ae229b6d4b81f3b0dc2c8a60559d000be699e7bed4e6aa1fd9f81132da2e5b88\": rpc error: code = NotFound desc = could not find container \"ae229b6d4b81f3b0dc2c8a60559d000be699e7bed4e6aa1fd9f81132da2e5b88\": container with ID starting with ae229b6d4b81f3b0dc2c8a60559d000be699e7bed4e6aa1fd9f81132da2e5b88 not found: ID does not exist"
Nov 02 12:47:46.123395 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:46.123326    4559 scope.go:122] "RemoveContainer" containerID="a1d8ea82c378d3fbb5c3c1ca0a57a8200213b14ba9a1d88cf8b784efe0cd4252"
Nov 02 12:47:46.124536 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:46.124511    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f7f44a-84dc986db4-q8cww"]
Nov 02 12:47:46.125452 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:46.125428    4559 cpu_manager.go:414] "containerMap: removing container" podUID="b98cdf4a-b4e0-46f3-b791-5cc63707a366" containerName="payload-pre-processing"
Nov 02 12:47:46.125554 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:46.125454    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b98cdf4a-b4e0-46f3-b791-5cc63707a366" containerName="payload-pre-processing"
Nov 02 12:47:46.125554 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:46.125475    4559 cpu_manager.go:414] "containerMap: removing container" podUID="bc18257f-4549-4cf1-9fe5-227496db7c80" containerName="payload-pre-processing"
Nov 02 12:47:46.125554 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:46.125483    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc18257f-4549-4cf1-9fe5-227496db7c80" containerName="payload-pre-processing"
Nov 02 12:47:46.125554 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:46.125506    4559 cpu_manager.go:414] "containerMap: removing container" podUID="630a0e10-db9e-42e4-80a9-2268163c591a" containerName="maas-api"
Nov 02 12:47:46.125554 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:46.125513    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="630a0e10-db9e-42e4-80a9-2268163c591a" containerName="maas-api"
Nov 02 12:47:46.125863 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:46.125558    4559 cpu_manager.go:414] "containerMap: removing container" podUID="6ee04a57-e5e9-48fa-a5a8-c89bcf917616" containerName="payload-processing"
Nov 02 12:47:46.125863 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:46.125566    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ee04a57-e5e9-48fa-a5a8-c89bcf917616" containerName="payload-processing"
Nov 02 12:47:46.125863 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:46.125575    4559 cpu_manager.go:414] "containerMap: removing container" podUID="0ed81343-a77a-4401-bc65-21c804cfbead" containerName="payload-processing"
Nov 02 12:47:46.125863 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:46.125582    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ed81343-a77a-4401-bc65-21c804cfbead" containerName="payload-processing"
Nov 02 12:47:46.125863 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:46.125787    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="b98cdf4a-b4e0-46f3-b791-5cc63707a366" containerName="payload-pre-processing"
Nov 02 12:47:46.125863 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:46.125804    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc18257f-4549-4cf1-9fe5-227496db7c80" containerName="payload-pre-processing"
Nov 02 12:47:46.125863 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:46.125815    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="630a0e10-db9e-42e4-80a9-2268163c591a" containerName="maas-api"
Nov 02 12:47:46.125863 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:46.125833    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ee04a57-e5e9-48fa-a5a8-c89bcf917616" containerName="payload-processing"
Nov 02 12:47:46.125863 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:46.125842    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ed81343-a77a-4401-bc65-21c804cfbead" containerName="payload-processing"
Nov 02 12:47:46.131108 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:46.131086    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f7f44a-84dc986db4-q8cww"
Nov 02 12:47:46.134259 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:46.134243    4559 scope.go:122] "RemoveContainer" containerID="a1d8ea82c378d3fbb5c3c1ca0a57a8200213b14ba9a1d88cf8b784efe0cd4252"
Nov 02 12:47:46.134517 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:47:46.134501    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1d8ea82c378d3fbb5c3c1ca0a57a8200213b14ba9a1d88cf8b784efe0cd4252\": container with ID starting with a1d8ea82c378d3fbb5c3c1ca0a57a8200213b14ba9a1d88cf8b784efe0cd4252 not found: ID does not exist" containerID="a1d8ea82c378d3fbb5c3c1ca0a57a8200213b14ba9a1d88cf8b784efe0cd4252"
Nov 02 12:47:46.134583 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:46.134521    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1d8ea82c378d3fbb5c3c1ca0a57a8200213b14ba9a1d88cf8b784efe0cd4252"} err="failed to get container status \"a1d8ea82c378d3fbb5c3c1ca0a57a8200213b14ba9a1d88cf8b784efe0cd4252\": rpc error: code = NotFound desc = could not find container \"a1d8ea82c378d3fbb5c3c1ca0a57a8200213b14ba9a1d88cf8b784efe0cd4252\": container with ID starting with a1d8ea82c378d3fbb5c3c1ca0a57a8200213b14ba9a1d88cf8b784efe0cd4252 not found: ID does not exist"
Nov 02 12:47:46.166874 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:46.166841    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f7f44a-84dc986db4-q8cww"]
Nov 02 12:47:46.167055 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:46.167039    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-f7f44a-serving-cert\""
Nov 02 12:47:46.186084 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:46.186056    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-77cf70-8686479b7b-mcd6f"]
Nov 02 12:47:46.194201 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:46.194175    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc18257f-4549-4cf1-9fe5-227496db7c80-kube-api-access-dj6qf\" (UniqueName: \"kubernetes.io/projected/bc18257f-4549-4cf1-9fe5-227496db7c80-kube-api-access-dj6qf\") pod \"bc18257f-4549-4cf1-9fe5-227496db7c80\" (UID: \"bc18257f-4549-4cf1-9fe5-227496db7c80\") "
Nov 02 12:47:46.194329 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:46.194242    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc18257f-4549-4cf1-9fe5-227496db7c80-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc18257f-4549-4cf1-9fe5-227496db7c80-plugins-config-volume\") pod \"bc18257f-4549-4cf1-9fe5-227496db7c80\" (UID: \"bc18257f-4549-4cf1-9fe5-227496db7c80\") "
Nov 02 12:47:46.194628 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:46.194608    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc18257f-4549-4cf1-9fe5-227496db7c80-plugins-config-volume" pod "bc18257f-4549-4cf1-9fe5-227496db7c80" (UID: "bc18257f-4549-4cf1-9fe5-227496db7c80"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:47:46.194992 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:46.194970    4559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc18257f-4549-4cf1-9fe5-227496db7c80-plugins-config-volume\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:47:46.196475 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:46.196453    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc18257f-4549-4cf1-9fe5-227496db7c80-kube-api-access-dj6qf" pod "bc18257f-4549-4cf1-9fe5-227496db7c80" (UID: "bc18257f-4549-4cf1-9fe5-227496db7c80"). InnerVolumeSpecName "kube-api-access-dj6qf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:47:46.205765 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:46.205743    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-77cf70-8686479b7b-mcd6f"]
Nov 02 12:47:46.276194 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:46.276170    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-77cf70-99dbd47d7-4rbjg"]
Nov 02 12:47:46.293406 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:46.293375    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-77cf70-99dbd47d7-4rbjg"]
Nov 02 12:47:46.295859 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:46.295837    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/61ae8a44-679a-4df1-abdd-3e5decbd2403-maas-api-tls\") pod \"maas-api-e2e-api-b-f7f44a-84dc986db4-q8cww\" (UID: \"61ae8a44-679a-4df1-abdd-3e5decbd2403\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f7f44a-84dc986db4-q8cww"
Nov 02 12:47:46.295969 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:46.295953    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/61ae8a44-679a-4df1-abdd-3e5decbd2403-openshift-service-ca\") pod \"maas-api-e2e-api-b-f7f44a-84dc986db4-q8cww\" (UID: \"61ae8a44-679a-4df1-abdd-3e5decbd2403\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f7f44a-84dc986db4-q8cww"
Nov 02 12:47:46.296073 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:46.296060    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/61ae8a44-679a-4df1-abdd-3e5decbd2403-metrics-certs\") pod \"maas-api-e2e-api-b-f7f44a-84dc986db4-q8cww\" (UID: \"61ae8a44-679a-4df1-abdd-3e5decbd2403\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f7f44a-84dc986db4-q8cww"
Nov 02 12:47:46.296118 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:46.296082    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kq7zj\" (UniqueName: \"kubernetes.io/projected/61ae8a44-679a-4df1-abdd-3e5decbd2403-kube-api-access-kq7zj\") pod \"maas-api-e2e-api-b-f7f44a-84dc986db4-q8cww\" (UID: \"61ae8a44-679a-4df1-abdd-3e5decbd2403\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f7f44a-84dc986db4-q8cww"
Nov 02 12:47:46.296169 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:46.296132    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dj6qf\" (UniqueName: \"kubernetes.io/projected/bc18257f-4549-4cf1-9fe5-227496db7c80-kube-api-access-dj6qf\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:47:46.397930 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:46.397865    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/61ae8a44-679a-4df1-abdd-3e5decbd2403-maas-api-tls\") pod \"maas-api-e2e-api-b-f7f44a-84dc986db4-q8cww\" (UID: \"61ae8a44-679a-4df1-abdd-3e5decbd2403\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f7f44a-84dc986db4-q8cww"
Nov 02 12:47:46.398058 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:46.397982    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/61ae8a44-679a-4df1-abdd-3e5decbd2403-openshift-service-ca\") pod \"maas-api-e2e-api-b-f7f44a-84dc986db4-q8cww\" (UID: \"61ae8a44-679a-4df1-abdd-3e5decbd2403\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f7f44a-84dc986db4-q8cww"
Nov 02 12:47:46.398125 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:46.398108    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/61ae8a44-679a-4df1-abdd-3e5decbd2403-metrics-certs\") pod \"maas-api-e2e-api-b-f7f44a-84dc986db4-q8cww\" (UID: \"61ae8a44-679a-4df1-abdd-3e5decbd2403\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f7f44a-84dc986db4-q8cww"
Nov 02 12:47:46.398160 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:46.398140    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kq7zj\" (UniqueName: \"kubernetes.io/projected/61ae8a44-679a-4df1-abdd-3e5decbd2403-kube-api-access-kq7zj\") pod \"maas-api-e2e-api-b-f7f44a-84dc986db4-q8cww\" (UID: \"61ae8a44-679a-4df1-abdd-3e5decbd2403\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f7f44a-84dc986db4-q8cww"
Nov 02 12:47:46.398671 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:46.398620    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/61ae8a44-679a-4df1-abdd-3e5decbd2403-openshift-service-ca\") pod \"maas-api-e2e-api-b-f7f44a-84dc986db4-q8cww\" (UID: \"61ae8a44-679a-4df1-abdd-3e5decbd2403\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f7f44a-84dc986db4-q8cww"
Nov 02 12:47:46.400387 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:46.400355    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/61ae8a44-679a-4df1-abdd-3e5decbd2403-metrics-certs\") pod \"maas-api-e2e-api-b-f7f44a-84dc986db4-q8cww\" (UID: \"61ae8a44-679a-4df1-abdd-3e5decbd2403\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f7f44a-84dc986db4-q8cww"
Nov 02 12:47:46.400472 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:46.400410    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/61ae8a44-679a-4df1-abdd-3e5decbd2403-maas-api-tls\") pod \"maas-api-e2e-api-b-f7f44a-84dc986db4-q8cww\" (UID: \"61ae8a44-679a-4df1-abdd-3e5decbd2403\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f7f44a-84dc986db4-q8cww"
Nov 02 12:47:46.434108 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:46.433172    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kq7zj\" (UniqueName: \"kubernetes.io/projected/61ae8a44-679a-4df1-abdd-3e5decbd2403-kube-api-access-kq7zj\") pod \"maas-api-e2e-api-b-f7f44a-84dc986db4-q8cww\" (UID: \"61ae8a44-679a-4df1-abdd-3e5decbd2403\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f7f44a-84dc986db4-q8cww"
Nov 02 12:47:46.446731 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:46.445992    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f7f44a-84dc986db4-q8cww"
Nov 02 12:47:46.572748 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:46.572717    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-77cf70-86cf7b694c-45888"]
Nov 02 12:47:46.576445 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:46.576417    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-77cf70-86cf7b694c-45888"]
Nov 02 12:47:46.857621 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:47:46.857588    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61ae8a44_679a_4df1_abdd_3e5decbd2403.slice/crio-372861f1d00c598b8593cff8094609103dc185c1a0d046621a454ee28003e9d5 WatchSource:0}: Error finding container 372861f1d00c598b8593cff8094609103dc185c1a0d046621a454ee28003e9d5: Status 404 returned error can't find the container with id 372861f1d00c598b8593cff8094609103dc185c1a0d046621a454ee28003e9d5
Nov 02 12:47:46.859072 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:46.859051    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f7f44a-84dc986db4-q8cww"]
Nov 02 12:47:47.163927 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:47.163810    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f7f44a-84dc986db4-q8cww" event={"ID":"61ae8a44-679a-4df1-abdd-3e5decbd2403","Type":"ContainerStarted","Data":"372861f1d00c598b8593cff8094609103dc185c1a0d046621a454ee28003e9d5"}
Nov 02 12:47:47.895007 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:47.894977    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="630a0e10-db9e-42e4-80a9-2268163c591a" path="/var/lib/kubelet/pods/630a0e10-db9e-42e4-80a9-2268163c591a/volumes"
Nov 02 12:47:47.895377 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:47.895364    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ee04a57-e5e9-48fa-a5a8-c89bcf917616" path="/var/lib/kubelet/pods/6ee04a57-e5e9-48fa-a5a8-c89bcf917616/volumes"
Nov 02 12:47:47.895739 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:47.895726    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc18257f-4549-4cf1-9fe5-227496db7c80" path="/var/lib/kubelet/pods/bc18257f-4549-4cf1-9fe5-227496db7c80/volumes"
Nov 02 12:47:48.172204 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:48.172167    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f7f44a-84dc986db4-q8cww" event={"ID":"61ae8a44-679a-4df1-abdd-3e5decbd2403","Type":"ContainerStarted","Data":"5ff6fb71b2273cf2ce0f4f10009dec40aeece72ffeaedf2238900850bd0b3227"}
Nov 02 12:47:48.172600 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:48.172328    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f7f44a-84dc986db4-q8cww"
Nov 02 12:47:48.217142 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:48.217091    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f7f44a-84dc986db4-q8cww" podStartSLOduration=1.828278351 podStartE2EDuration="2.217076932s" totalImagesPullingTime="388.798581ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:47:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:47:48.215996199 +0000 UTC m=+1264.471395293" watchObservedRunningTime="2026-11-02 12:47:48.217076932 +0000 UTC m=+1264.472476074"
Nov 02 12:47:50.106663 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:50.105664    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-729108-7f68cf4d79-8clxs"]
Nov 02 12:47:50.135660 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:50.128854    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-729108-7f68cf4d79-8clxs"
Nov 02 12:47:50.135660 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:50.131867    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-729108-7f68cf4d79-8clxs"]
Nov 02 12:47:50.176978 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:50.176943    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-729108-dockercfg-wfnrf\""
Nov 02 12:47:50.177327 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:50.177307    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-729108\""
Nov 02 12:47:50.222690 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:50.212503    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxj96\" (UniqueName: \"kubernetes.io/projected/e3df8de3-6c80-4695-8f07-0ec0873a7b9f-kube-api-access-kxj96\") pod \"payload-pre-processing-e2e-ipp-a-729108-7f68cf4d79-8clxs\" (UID: \"e3df8de3-6c80-4695-8f07-0ec0873a7b9f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-729108-7f68cf4d79-8clxs"
Nov 02 12:47:50.222690 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:50.212661    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3df8de3-6c80-4695-8f07-0ec0873a7b9f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-729108-7f68cf4d79-8clxs\" (UID: \"e3df8de3-6c80-4695-8f07-0ec0873a7b9f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-729108-7f68cf4d79-8clxs"
Nov 02 12:47:50.258028 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:50.257373    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-729108-6898dcb756-ksksg"]
Nov 02 12:47:50.262617 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:50.262588    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-729108-6898dcb756-ksksg"
Nov 02 12:47:50.280325 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:50.280283    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-729108-6898dcb756-ksksg"]
Nov 02 12:47:50.313784 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:50.313750    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxj96\" (UniqueName: \"kubernetes.io/projected/e3df8de3-6c80-4695-8f07-0ec0873a7b9f-kube-api-access-kxj96\") pod \"payload-pre-processing-e2e-ipp-a-729108-7f68cf4d79-8clxs\" (UID: \"e3df8de3-6c80-4695-8f07-0ec0873a7b9f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-729108-7f68cf4d79-8clxs"
Nov 02 12:47:50.313947 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:50.313821    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3df8de3-6c80-4695-8f07-0ec0873a7b9f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-729108-7f68cf4d79-8clxs\" (UID: \"e3df8de3-6c80-4695-8f07-0ec0873a7b9f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-729108-7f68cf4d79-8clxs"
Nov 02 12:47:50.314564 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:50.314542    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3df8de3-6c80-4695-8f07-0ec0873a7b9f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-729108-7f68cf4d79-8clxs\" (UID: \"e3df8de3-6c80-4695-8f07-0ec0873a7b9f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-729108-7f68cf4d79-8clxs"
Nov 02 12:47:50.337797 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:50.337772    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxj96\" (UniqueName: \"kubernetes.io/projected/e3df8de3-6c80-4695-8f07-0ec0873a7b9f-kube-api-access-kxj96\") pod \"payload-pre-processing-e2e-ipp-a-729108-7f68cf4d79-8clxs\" (UID: \"e3df8de3-6c80-4695-8f07-0ec0873a7b9f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-729108-7f68cf4d79-8clxs"
Nov 02 12:47:50.415875 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:50.415796    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52cf3408-1bb5-49aa-9a7e-2b70df75ccf7-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-729108-6898dcb756-ksksg\" (UID: \"52cf3408-1bb5-49aa-9a7e-2b70df75ccf7\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-729108-6898dcb756-ksksg"
Nov 02 12:47:50.416006 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:50.415918    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fcht\" (UniqueName: \"kubernetes.io/projected/52cf3408-1bb5-49aa-9a7e-2b70df75ccf7-kube-api-access-6fcht\") pod \"payload-processing-e2e-ipp-a-729108-6898dcb756-ksksg\" (UID: \"52cf3408-1bb5-49aa-9a7e-2b70df75ccf7\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-729108-6898dcb756-ksksg"
Nov 02 12:47:50.507299 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:50.507269    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-729108-7f68cf4d79-8clxs"
Nov 02 12:47:50.517046 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:50.517011    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52cf3408-1bb5-49aa-9a7e-2b70df75ccf7-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-729108-6898dcb756-ksksg\" (UID: \"52cf3408-1bb5-49aa-9a7e-2b70df75ccf7\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-729108-6898dcb756-ksksg"
Nov 02 12:47:50.517181 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:50.517118    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fcht\" (UniqueName: \"kubernetes.io/projected/52cf3408-1bb5-49aa-9a7e-2b70df75ccf7-kube-api-access-6fcht\") pod \"payload-processing-e2e-ipp-a-729108-6898dcb756-ksksg\" (UID: \"52cf3408-1bb5-49aa-9a7e-2b70df75ccf7\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-729108-6898dcb756-ksksg"
Nov 02 12:47:50.517665 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:50.517624    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52cf3408-1bb5-49aa-9a7e-2b70df75ccf7-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-729108-6898dcb756-ksksg\" (UID: \"52cf3408-1bb5-49aa-9a7e-2b70df75ccf7\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-729108-6898dcb756-ksksg"
Nov 02 12:47:50.541310 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:50.541283    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fcht\" (UniqueName: \"kubernetes.io/projected/52cf3408-1bb5-49aa-9a7e-2b70df75ccf7-kube-api-access-6fcht\") pod \"payload-processing-e2e-ipp-a-729108-6898dcb756-ksksg\" (UID: \"52cf3408-1bb5-49aa-9a7e-2b70df75ccf7\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-729108-6898dcb756-ksksg"
Nov 02 12:47:50.592938 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:50.592893    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-729108-6898dcb756-ksksg"
Nov 02 12:47:50.850902 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:50.842371    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-729108-7f68cf4d79-8clxs"]
Nov 02 12:47:50.862243 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:47:50.862096    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3df8de3_6c80_4695_8f07_0ec0873a7b9f.slice/crio-c2b054e027eeb4f3f64d8defabd6cef85fcffeaff00bc46984f61587ec3b22ef WatchSource:0}: Error finding container c2b054e027eeb4f3f64d8defabd6cef85fcffeaff00bc46984f61587ec3b22ef: Status 404 returned error can't find the container with id c2b054e027eeb4f3f64d8defabd6cef85fcffeaff00bc46984f61587ec3b22ef
Nov 02 12:47:50.920966 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:47:50.920941    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52cf3408_1bb5_49aa_9a7e_2b70df75ccf7.slice/crio-ec947d6ee9da31c323a9c2538e4788b65eae8001063fcc5e8bec5c4fbdb8ce66 WatchSource:0}: Error finding container ec947d6ee9da31c323a9c2538e4788b65eae8001063fcc5e8bec5c4fbdb8ce66: Status 404 returned error can't find the container with id ec947d6ee9da31c323a9c2538e4788b65eae8001063fcc5e8bec5c4fbdb8ce66
Nov 02 12:47:50.921051 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:50.921007    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-729108-6898dcb756-ksksg"]
Nov 02 12:47:51.229297 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:51.229212    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-729108-6898dcb756-ksksg" event={"ID":"52cf3408-1bb5-49aa-9a7e-2b70df75ccf7","Type":"ContainerStarted","Data":"aa8210b26914c5e0c1210e5ef099bb12595e70eba8ae998884b5ccc654a3005f"}
Nov 02 12:47:51.229297 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:51.229282    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-729108-6898dcb756-ksksg"
Nov 02 12:47:51.229831 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:51.229308    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-729108-6898dcb756-ksksg" event={"ID":"52cf3408-1bb5-49aa-9a7e-2b70df75ccf7","Type":"ContainerStarted","Data":"ec947d6ee9da31c323a9c2538e4788b65eae8001063fcc5e8bec5c4fbdb8ce66"}
Nov 02 12:47:51.231107 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:51.231083    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-729108-7f68cf4d79-8clxs" event={"ID":"e3df8de3-6c80-4695-8f07-0ec0873a7b9f","Type":"ContainerStarted","Data":"ebb88c03701127796f08227a9123f1afe36b16c9401ad355c735721390214c54"}
Nov 02 12:47:51.231214 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:51.231122    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-729108-7f68cf4d79-8clxs" event={"ID":"e3df8de3-6c80-4695-8f07-0ec0873a7b9f","Type":"ContainerStarted","Data":"c2b054e027eeb4f3f64d8defabd6cef85fcffeaff00bc46984f61587ec3b22ef"}
Nov 02 12:47:51.231214 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:51.231169    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-729108-7f68cf4d79-8clxs"
Nov 02 12:47:51.296592 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:51.296545    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-729108-6898dcb756-ksksg" podStartSLOduration=1.296531342 podStartE2EDuration="1.296531342s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:47:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:47:51.295149373 +0000 UTC m=+1267.550548479" watchObservedRunningTime="2026-11-02 12:47:51.296531342 +0000 UTC m=+1267.551930437"
Nov 02 12:47:51.399026 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:51.398968    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-729108-7f68cf4d79-8clxs" podStartSLOduration=1.398949214 podStartE2EDuration="1.398949214s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:47:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:47:51.394120926 +0000 UTC m=+1267.649520024" watchObservedRunningTime="2026-11-02 12:47:51.398949214 +0000 UTC m=+1267.654348311"
Nov 02 12:47:53.285585 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:53.285553    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-77cf70-openshift-default-d8f7564b9-rftzg"]
Nov 02 12:47:53.286076 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:53.285864    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-77cf70-openshift-default-d8f7564b9-rftzg" podUID="60be7b9e-07a0-4cc6-99d1-1278b6bad791" containerName="istio-proxy" containerID="cri-o://2c0e95211f34c0ba2ab371f9f6e7b430bf725216b1e73f988eb5a2ae1327fed1" gracePeriod=30
Nov 02 12:47:54.184320 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:54.184292    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f7f44a-84dc986db4-q8cww"
Nov 02 12:47:58.580428 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:58.580402    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-77cf70-openshift-default-d8f7564b9-rftzg"
Nov 02 12:47:58.631819 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:58.631787    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60be7b9e-07a0-4cc6-99d1-1278b6bad791-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60be7b9e-07a0-4cc6-99d1-1278b6bad791-istiod-ca-cert\") pod \"60be7b9e-07a0-4cc6-99d1-1278b6bad791\" (UID: \"60be7b9e-07a0-4cc6-99d1-1278b6bad791\") "
Nov 02 12:47:58.632017 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:58.631985    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60be7b9e-07a0-4cc6-99d1-1278b6bad791-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60be7b9e-07a0-4cc6-99d1-1278b6bad791-credential-socket\") pod \"60be7b9e-07a0-4cc6-99d1-1278b6bad791\" (UID: \"60be7b9e-07a0-4cc6-99d1-1278b6bad791\") "
Nov 02 12:47:58.632096 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:58.632020    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60be7b9e-07a0-4cc6-99d1-1278b6bad791-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60be7b9e-07a0-4cc6-99d1-1278b6bad791-istio-data\") pod \"60be7b9e-07a0-4cc6-99d1-1278b6bad791\" (UID: \"60be7b9e-07a0-4cc6-99d1-1278b6bad791\") "
Nov 02 12:47:58.632154 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:58.632110    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/60be7b9e-07a0-4cc6-99d1-1278b6bad791-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60be7b9e-07a0-4cc6-99d1-1278b6bad791-istio-podinfo\") pod \"60be7b9e-07a0-4cc6-99d1-1278b6bad791\" (UID: \"60be7b9e-07a0-4cc6-99d1-1278b6bad791\") "
Nov 02 12:47:58.632207 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:58.632178    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60be7b9e-07a0-4cc6-99d1-1278b6bad791-istiod-ca-cert" pod "60be7b9e-07a0-4cc6-99d1-1278b6bad791" (UID: "60be7b9e-07a0-4cc6-99d1-1278b6bad791"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:47:58.632207 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:58.632186    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60be7b9e-07a0-4cc6-99d1-1278b6bad791-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60be7b9e-07a0-4cc6-99d1-1278b6bad791-istio-envoy\") pod \"60be7b9e-07a0-4cc6-99d1-1278b6bad791\" (UID: \"60be7b9e-07a0-4cc6-99d1-1278b6bad791\") "
Nov 02 12:47:58.632349 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:58.632322    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60be7b9e-07a0-4cc6-99d1-1278b6bad791-credential-socket" pod "60be7b9e-07a0-4cc6-99d1-1278b6bad791" (UID: "60be7b9e-07a0-4cc6-99d1-1278b6bad791"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:47:58.632410 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:58.632347    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60be7b9e-07a0-4cc6-99d1-1278b6bad791-istio-data" pod "60be7b9e-07a0-4cc6-99d1-1278b6bad791" (UID: "60be7b9e-07a0-4cc6-99d1-1278b6bad791"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:47:58.632410 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:58.632280    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60be7b9e-07a0-4cc6-99d1-1278b6bad791-kube-api-access-xfgdl\" (UniqueName: \"kubernetes.io/projected/60be7b9e-07a0-4cc6-99d1-1278b6bad791-kube-api-access-xfgdl\") pod \"60be7b9e-07a0-4cc6-99d1-1278b6bad791\" (UID: \"60be7b9e-07a0-4cc6-99d1-1278b6bad791\") "
Nov 02 12:47:58.632508 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:58.632423    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60be7b9e-07a0-4cc6-99d1-1278b6bad791-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60be7b9e-07a0-4cc6-99d1-1278b6bad791-workload-certs\") pod \"60be7b9e-07a0-4cc6-99d1-1278b6bad791\" (UID: \"60be7b9e-07a0-4cc6-99d1-1278b6bad791\") "
Nov 02 12:47:58.632508 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:58.632464    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60be7b9e-07a0-4cc6-99d1-1278b6bad791-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60be7b9e-07a0-4cc6-99d1-1278b6bad791-workload-socket\") pod \"60be7b9e-07a0-4cc6-99d1-1278b6bad791\" (UID: \"60be7b9e-07a0-4cc6-99d1-1278b6bad791\") "
Nov 02 12:47:58.632615 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:58.632511    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60be7b9e-07a0-4cc6-99d1-1278b6bad791-istio-token\" (UniqueName: \"kubernetes.io/projected/60be7b9e-07a0-4cc6-99d1-1278b6bad791-istio-token\") pod \"60be7b9e-07a0-4cc6-99d1-1278b6bad791\" (UID: \"60be7b9e-07a0-4cc6-99d1-1278b6bad791\") "
Nov 02 12:47:58.632820 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:58.632793    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60be7b9e-07a0-4cc6-99d1-1278b6bad791-workload-certs" pod "60be7b9e-07a0-4cc6-99d1-1278b6bad791" (UID: "60be7b9e-07a0-4cc6-99d1-1278b6bad791"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:47:58.633028 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:58.633004    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60be7b9e-07a0-4cc6-99d1-1278b6bad791-workload-socket" pod "60be7b9e-07a0-4cc6-99d1-1278b6bad791" (UID: "60be7b9e-07a0-4cc6-99d1-1278b6bad791"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:47:58.633271 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:58.633250    4559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60be7b9e-07a0-4cc6-99d1-1278b6bad791-credential-socket\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:47:58.633394 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:58.633284    4559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60be7b9e-07a0-4cc6-99d1-1278b6bad791-istio-data\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:47:58.633394 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:58.633299    4559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60be7b9e-07a0-4cc6-99d1-1278b6bad791-workload-certs\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:47:58.633394 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:58.633311    4559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60be7b9e-07a0-4cc6-99d1-1278b6bad791-workload-socket\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:47:58.633394 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:58.633325    4559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60be7b9e-07a0-4cc6-99d1-1278b6bad791-istiod-ca-cert\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:47:58.634971 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:58.634923    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60be7b9e-07a0-4cc6-99d1-1278b6bad791-istio-envoy" pod "60be7b9e-07a0-4cc6-99d1-1278b6bad791" (UID: "60be7b9e-07a0-4cc6-99d1-1278b6bad791"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:47:58.635116 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:58.635088    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/60be7b9e-07a0-4cc6-99d1-1278b6bad791-istio-podinfo" pod "60be7b9e-07a0-4cc6-99d1-1278b6bad791" (UID: "60be7b9e-07a0-4cc6-99d1-1278b6bad791"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 12:47:58.635190 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:58.635136    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60be7b9e-07a0-4cc6-99d1-1278b6bad791-kube-api-access-xfgdl" pod "60be7b9e-07a0-4cc6-99d1-1278b6bad791" (UID: "60be7b9e-07a0-4cc6-99d1-1278b6bad791"). InnerVolumeSpecName "kube-api-access-xfgdl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:47:58.635254 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:58.635236    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60be7b9e-07a0-4cc6-99d1-1278b6bad791-istio-token" pod "60be7b9e-07a0-4cc6-99d1-1278b6bad791" (UID: "60be7b9e-07a0-4cc6-99d1-1278b6bad791"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:47:58.734332 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:58.734300    4559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60be7b9e-07a0-4cc6-99d1-1278b6bad791-istio-podinfo\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:47:58.734332 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:58.734329    4559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60be7b9e-07a0-4cc6-99d1-1278b6bad791-istio-envoy\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:47:58.734332 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:58.734340    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xfgdl\" (UniqueName: \"kubernetes.io/projected/60be7b9e-07a0-4cc6-99d1-1278b6bad791-kube-api-access-xfgdl\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:47:58.734575 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:58.734352    4559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60be7b9e-07a0-4cc6-99d1-1278b6bad791-istio-token\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:47:59.351259 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:59.351211    4559 generic.go:356] "Generic (PLEG): container finished" podID="60be7b9e-07a0-4cc6-99d1-1278b6bad791" containerID="2c0e95211f34c0ba2ab371f9f6e7b430bf725216b1e73f988eb5a2ae1327fed1" exitCode=0
Nov 02 12:47:59.351805 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:59.351569    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-77cf70-openshift-default-d8f7564b9-rftzg"
Nov 02 12:47:59.358676 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:59.351592    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-77cf70-openshift-default-d8f7564b9-rftzg" event={"ID":"60be7b9e-07a0-4cc6-99d1-1278b6bad791","Type":"ContainerDied","Data":"2c0e95211f34c0ba2ab371f9f6e7b430bf725216b1e73f988eb5a2ae1327fed1"}
Nov 02 12:47:59.358676 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:59.358064    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-77cf70-openshift-default-d8f7564b9-rftzg" event={"ID":"60be7b9e-07a0-4cc6-99d1-1278b6bad791","Type":"ContainerDied","Data":"0503d6d411b77548038f4b74adec0711b53127292d7180cdcef710a29606b729"}
Nov 02 12:47:59.358676 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:59.358106    4559 scope.go:122] "RemoveContainer" containerID="2c0e95211f34c0ba2ab371f9f6e7b430bf725216b1e73f988eb5a2ae1327fed1"
Nov 02 12:47:59.382218 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:59.382195    4559 scope.go:122] "RemoveContainer" containerID="fd000b435ffe806cb1b71a659d652a87fc1284ebe313cb5576983e842e971154"
Nov 02 12:47:59.421862 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:59.421809    4559 scope.go:122] "RemoveContainer" containerID="2c0e95211f34c0ba2ab371f9f6e7b430bf725216b1e73f988eb5a2ae1327fed1"
Nov 02 12:47:59.422579 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:47:59.422553    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c0e95211f34c0ba2ab371f9f6e7b430bf725216b1e73f988eb5a2ae1327fed1\": container with ID starting with 2c0e95211f34c0ba2ab371f9f6e7b430bf725216b1e73f988eb5a2ae1327fed1 not found: ID does not exist" containerID="2c0e95211f34c0ba2ab371f9f6e7b430bf725216b1e73f988eb5a2ae1327fed1"
Nov 02 12:47:59.422757 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:59.422584    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c0e95211f34c0ba2ab371f9f6e7b430bf725216b1e73f988eb5a2ae1327fed1"} err="failed to get container status \"2c0e95211f34c0ba2ab371f9f6e7b430bf725216b1e73f988eb5a2ae1327fed1\": rpc error: code = NotFound desc = could not find container \"2c0e95211f34c0ba2ab371f9f6e7b430bf725216b1e73f988eb5a2ae1327fed1\": container with ID starting with 2c0e95211f34c0ba2ab371f9f6e7b430bf725216b1e73f988eb5a2ae1327fed1 not found: ID does not exist"
Nov 02 12:47:59.422757 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:59.422603    4559 scope.go:122] "RemoveContainer" containerID="fd000b435ffe806cb1b71a659d652a87fc1284ebe313cb5576983e842e971154"
Nov 02 12:47:59.423435 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:47:59.423403    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd000b435ffe806cb1b71a659d652a87fc1284ebe313cb5576983e842e971154\": container with ID starting with fd000b435ffe806cb1b71a659d652a87fc1284ebe313cb5576983e842e971154 not found: ID does not exist" containerID="fd000b435ffe806cb1b71a659d652a87fc1284ebe313cb5576983e842e971154"
Nov 02 12:47:59.423543 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:59.423438    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd000b435ffe806cb1b71a659d652a87fc1284ebe313cb5576983e842e971154"} err="failed to get container status \"fd000b435ffe806cb1b71a659d652a87fc1284ebe313cb5576983e842e971154\": rpc error: code = NotFound desc = could not find container \"fd000b435ffe806cb1b71a659d652a87fc1284ebe313cb5576983e842e971154\": container with ID starting with fd000b435ffe806cb1b71a659d652a87fc1284ebe313cb5576983e842e971154 not found: ID does not exist"
Nov 02 12:47:59.433516 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:59.433490    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-77cf70-openshift-default-d8f7564b9-rftzg"]
Nov 02 12:47:59.438243 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:59.438223    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-77cf70-openshift-default-d8f7564b9-rftzg"]
Nov 02 12:47:59.914102 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:47:59.914065    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60be7b9e-07a0-4cc6-99d1-1278b6bad791" path="/var/lib/kubelet/pods/60be7b9e-07a0-4cc6-99d1-1278b6bad791/volumes"
Nov 02 12:48:01.344757 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:01.344713    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-078fd3-7b66f65b99-7cclg"]
Nov 02 12:48:01.345724 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:01.345686    4559 cpu_manager.go:414] "containerMap: removing container" podUID="60be7b9e-07a0-4cc6-99d1-1278b6bad791" containerName="istio-proxy"
Nov 02 12:48:01.345724 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:01.345724    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60be7b9e-07a0-4cc6-99d1-1278b6bad791" containerName="istio-proxy"
Nov 02 12:48:01.346008 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:01.345748    4559 cpu_manager.go:414] "containerMap: removing container" podUID="60be7b9e-07a0-4cc6-99d1-1278b6bad791" containerName="istio-proxy"
Nov 02 12:48:01.346008 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:01.345763    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60be7b9e-07a0-4cc6-99d1-1278b6bad791" containerName="istio-proxy"
Nov 02 12:48:01.346008 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:01.345954    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="60be7b9e-07a0-4cc6-99d1-1278b6bad791" containerName="istio-proxy"
Nov 02 12:48:01.346008 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:01.345977    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="60be7b9e-07a0-4cc6-99d1-1278b6bad791" containerName="istio-proxy"
Nov 02 12:48:01.444304 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:01.425958    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-078fd3-7b66f65b99-7cclg"
Nov 02 12:48:01.464673 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:01.463519    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-078fd3-7b66f65b99-7cclg"]
Nov 02 12:48:01.474461 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:01.474430    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-078fd3\""
Nov 02 12:48:01.475451 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:01.475429    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-078fd3-dockercfg-x7bfh\""
Nov 02 12:48:01.505854 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:01.503694    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-078fd3-6ff794864d-8sxnf"]
Nov 02 12:48:01.516836 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:01.515953    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-078fd3-6ff794864d-8sxnf"]
Nov 02 12:48:01.516836 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:01.516094    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-078fd3-6ff794864d-8sxnf"
Nov 02 12:48:01.546186 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:01.546159    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/daae67e4-da6e-4eb7-b044-8e795f3b3a29-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-078fd3-7b66f65b99-7cclg\" (UID: \"daae67e4-da6e-4eb7-b044-8e795f3b3a29\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-078fd3-7b66f65b99-7cclg"
Nov 02 12:48:01.546424 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:01.546405    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dltqs\" (UniqueName: \"kubernetes.io/projected/daae67e4-da6e-4eb7-b044-8e795f3b3a29-kube-api-access-dltqs\") pod \"payload-pre-processing-e2e-ipp-b-078fd3-7b66f65b99-7cclg\" (UID: \"daae67e4-da6e-4eb7-b044-8e795f3b3a29\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-078fd3-7b66f65b99-7cclg"
Nov 02 12:48:01.647902 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:01.647818    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fdb54d38-ce90-43af-9ba9-da98f13d788a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-078fd3-6ff794864d-8sxnf\" (UID: \"fdb54d38-ce90-43af-9ba9-da98f13d788a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-078fd3-6ff794864d-8sxnf"
Nov 02 12:48:01.648036 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:01.647934    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chr2x\" (UniqueName: \"kubernetes.io/projected/fdb54d38-ce90-43af-9ba9-da98f13d788a-kube-api-access-chr2x\") pod \"payload-processing-e2e-ipp-b-078fd3-6ff794864d-8sxnf\" (UID: \"fdb54d38-ce90-43af-9ba9-da98f13d788a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-078fd3-6ff794864d-8sxnf"
Nov 02 12:48:01.648036 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:01.647992    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dltqs\" (UniqueName: \"kubernetes.io/projected/daae67e4-da6e-4eb7-b044-8e795f3b3a29-kube-api-access-dltqs\") pod \"payload-pre-processing-e2e-ipp-b-078fd3-7b66f65b99-7cclg\" (UID: \"daae67e4-da6e-4eb7-b044-8e795f3b3a29\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-078fd3-7b66f65b99-7cclg"
Nov 02 12:48:01.648134 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:01.648113    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/daae67e4-da6e-4eb7-b044-8e795f3b3a29-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-078fd3-7b66f65b99-7cclg\" (UID: \"daae67e4-da6e-4eb7-b044-8e795f3b3a29\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-078fd3-7b66f65b99-7cclg"
Nov 02 12:48:01.648826 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:01.648809    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/daae67e4-da6e-4eb7-b044-8e795f3b3a29-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-078fd3-7b66f65b99-7cclg\" (UID: \"daae67e4-da6e-4eb7-b044-8e795f3b3a29\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-078fd3-7b66f65b99-7cclg"
Nov 02 12:48:01.671798 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:01.671768    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dltqs\" (UniqueName: \"kubernetes.io/projected/daae67e4-da6e-4eb7-b044-8e795f3b3a29-kube-api-access-dltqs\") pod \"payload-pre-processing-e2e-ipp-b-078fd3-7b66f65b99-7cclg\" (UID: \"daae67e4-da6e-4eb7-b044-8e795f3b3a29\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-078fd3-7b66f65b99-7cclg"
Nov 02 12:48:01.749849 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:01.749801    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fdb54d38-ce90-43af-9ba9-da98f13d788a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-078fd3-6ff794864d-8sxnf\" (UID: \"fdb54d38-ce90-43af-9ba9-da98f13d788a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-078fd3-6ff794864d-8sxnf"
Nov 02 12:48:01.750013 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:01.749904    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chr2x\" (UniqueName: \"kubernetes.io/projected/fdb54d38-ce90-43af-9ba9-da98f13d788a-kube-api-access-chr2x\") pod \"payload-processing-e2e-ipp-b-078fd3-6ff794864d-8sxnf\" (UID: \"fdb54d38-ce90-43af-9ba9-da98f13d788a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-078fd3-6ff794864d-8sxnf"
Nov 02 12:48:01.750739 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:01.750712    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fdb54d38-ce90-43af-9ba9-da98f13d788a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-078fd3-6ff794864d-8sxnf\" (UID: \"fdb54d38-ce90-43af-9ba9-da98f13d788a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-078fd3-6ff794864d-8sxnf"
Nov 02 12:48:01.774324 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:01.774296    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-chr2x\" (UniqueName: \"kubernetes.io/projected/fdb54d38-ce90-43af-9ba9-da98f13d788a-kube-api-access-chr2x\") pod \"payload-processing-e2e-ipp-b-078fd3-6ff794864d-8sxnf\" (UID: \"fdb54d38-ce90-43af-9ba9-da98f13d788a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-078fd3-6ff794864d-8sxnf"
Nov 02 12:48:01.817114 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:01.817090    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-078fd3-7b66f65b99-7cclg"
Nov 02 12:48:01.834432 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:01.834406    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-078fd3-6ff794864d-8sxnf"
Nov 02 12:48:02.122830 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:02.122784    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-ee509d-5568559575-g8848"]
Nov 02 12:48:02.129522 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:02.128730    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ee509d-5568559575-g8848"
Nov 02 12:48:02.131702 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:48:02.131624    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddaae67e4_da6e_4eb7_b044_8e795f3b3a29.slice/crio-98d353529eae44951f56953d6a795ac056d831aa9097d8f0fb3ae4e2526814b8 WatchSource:0}: Error finding container 98d353529eae44951f56953d6a795ac056d831aa9097d8f0fb3ae4e2526814b8: Status 404 returned error can't find the container with id 98d353529eae44951f56953d6a795ac056d831aa9097d8f0fb3ae4e2526814b8
Nov 02 12:48:02.132051 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:02.132033    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-078fd3-7b66f65b99-7cclg"]
Nov 02 12:48:02.139679 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:02.139656    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-ee509d-5568559575-g8848"]
Nov 02 12:48:02.144565 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:02.144543    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-ee509d\""
Nov 02 12:48:02.144727 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:02.144599    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-ee509d-dockercfg-dc74l\""
Nov 02 12:48:02.302297 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:02.268523    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bn9f\" (UniqueName: \"kubernetes.io/projected/14d49a01-92ac-4a12-9c40-82dcb6dc0bfd-kube-api-access-6bn9f\") pod \"payload-pre-processing-e2e-praxis-a-ee509d-5568559575-g8848\" (UID: \"14d49a01-92ac-4a12-9c40-82dcb6dc0bfd\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ee509d-5568559575-g8848"
Nov 02 12:48:02.302297 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:02.295003    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14d49a01-92ac-4a12-9c40-82dcb6dc0bfd-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-ee509d-5568559575-g8848\" (UID: \"14d49a01-92ac-4a12-9c40-82dcb6dc0bfd\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ee509d-5568559575-g8848"
Nov 02 12:48:02.302297 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:02.298444    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-729108-6898dcb756-ksksg"
Nov 02 12:48:02.302297 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:02.301025    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-729108-7f68cf4d79-8clxs"
Nov 02 12:48:02.387068 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:02.386980    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-078fd3-6ff794864d-8sxnf"]
Nov 02 12:48:02.414671 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:02.400977    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bn9f\" (UniqueName: \"kubernetes.io/projected/14d49a01-92ac-4a12-9c40-82dcb6dc0bfd-kube-api-access-6bn9f\") pod \"payload-pre-processing-e2e-praxis-a-ee509d-5568559575-g8848\" (UID: \"14d49a01-92ac-4a12-9c40-82dcb6dc0bfd\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ee509d-5568559575-g8848"
Nov 02 12:48:02.414671 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:02.401213    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14d49a01-92ac-4a12-9c40-82dcb6dc0bfd-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-ee509d-5568559575-g8848\" (UID: \"14d49a01-92ac-4a12-9c40-82dcb6dc0bfd\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ee509d-5568559575-g8848"
Nov 02 12:48:02.414671 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:02.402328    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14d49a01-92ac-4a12-9c40-82dcb6dc0bfd-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-ee509d-5568559575-g8848\" (UID: \"14d49a01-92ac-4a12-9c40-82dcb6dc0bfd\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ee509d-5568559575-g8848"
Nov 02 12:48:02.455290 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:02.436804    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bn9f\" (UniqueName: \"kubernetes.io/projected/14d49a01-92ac-4a12-9c40-82dcb6dc0bfd-kube-api-access-6bn9f\") pod \"payload-pre-processing-e2e-praxis-a-ee509d-5568559575-g8848\" (UID: \"14d49a01-92ac-4a12-9c40-82dcb6dc0bfd\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ee509d-5568559575-g8848"
Nov 02 12:48:02.455290 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:02.454982    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ee509d-5568559575-g8848"
Nov 02 12:48:02.508994 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:02.507878    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-ee509d-6d46494d8c-q2qwp"]
Nov 02 12:48:02.571964 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:02.571930    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-ee509d-6d46494d8c-q2qwp"]
Nov 02 12:48:02.572278 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:02.572261    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-ee509d-6d46494d8c-q2qwp"
Nov 02 12:48:02.624247 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:02.624216    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12ce9360-ddff-4f72-acbb-bf9659a5c0a3-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-ee509d-6d46494d8c-q2qwp\" (UID: \"12ce9360-ddff-4f72-acbb-bf9659a5c0a3\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-ee509d-6d46494d8c-q2qwp"
Nov 02 12:48:02.630006 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:02.629938    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrxb8\" (UniqueName: \"kubernetes.io/projected/12ce9360-ddff-4f72-acbb-bf9659a5c0a3-kube-api-access-zrxb8\") pod \"payload-processing-e2e-praxis-a-ee509d-6d46494d8c-q2qwp\" (UID: \"12ce9360-ddff-4f72-acbb-bf9659a5c0a3\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-ee509d-6d46494d8c-q2qwp"
Nov 02 12:48:02.650203 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:02.649940    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-078fd3-6ff794864d-8sxnf" event={"ID":"fdb54d38-ce90-43af-9ba9-da98f13d788a","Type":"ContainerStarted","Data":"9fde782efa1dc7ed325cc5270fabc462cc35fe9438d7ccbb0c380343fc6ae929"}
Nov 02 12:48:02.653366 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:02.652396    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-078fd3-7b66f65b99-7cclg" event={"ID":"daae67e4-da6e-4eb7-b044-8e795f3b3a29","Type":"ContainerStarted","Data":"98d353529eae44951f56953d6a795ac056d831aa9097d8f0fb3ae4e2526814b8"}
Nov 02 12:48:02.653622 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:02.653593    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-078fd3-7b66f65b99-7cclg"
Nov 02 12:48:02.728658 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:02.723340    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-078fd3-7b66f65b99-7cclg" podStartSLOduration=1.723320094 podStartE2EDuration="1.723320094s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:48:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:48:02.702933489 +0000 UTC m=+1278.958332584" watchObservedRunningTime="2026-11-02 12:48:02.723320094 +0000 UTC m=+1278.978719190"
Nov 02 12:48:02.739434 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:02.737501    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12ce9360-ddff-4f72-acbb-bf9659a5c0a3-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-ee509d-6d46494d8c-q2qwp\" (UID: \"12ce9360-ddff-4f72-acbb-bf9659a5c0a3\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-ee509d-6d46494d8c-q2qwp"
Nov 02 12:48:02.739434 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:02.737611    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrxb8\" (UniqueName: \"kubernetes.io/projected/12ce9360-ddff-4f72-acbb-bf9659a5c0a3-kube-api-access-zrxb8\") pod \"payload-processing-e2e-praxis-a-ee509d-6d46494d8c-q2qwp\" (UID: \"12ce9360-ddff-4f72-acbb-bf9659a5c0a3\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-ee509d-6d46494d8c-q2qwp"
Nov 02 12:48:02.739434 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:02.738987    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12ce9360-ddff-4f72-acbb-bf9659a5c0a3-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-ee509d-6d46494d8c-q2qwp\" (UID: \"12ce9360-ddff-4f72-acbb-bf9659a5c0a3\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-ee509d-6d46494d8c-q2qwp"
Nov 02 12:48:02.765216 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:02.763408    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrxb8\" (UniqueName: \"kubernetes.io/projected/12ce9360-ddff-4f72-acbb-bf9659a5c0a3-kube-api-access-zrxb8\") pod \"payload-processing-e2e-praxis-a-ee509d-6d46494d8c-q2qwp\" (UID: \"12ce9360-ddff-4f72-acbb-bf9659a5c0a3\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-ee509d-6d46494d8c-q2qwp"
Nov 02 12:48:02.952010 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:02.951971    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-ee509d-6d46494d8c-q2qwp"
Nov 02 12:48:03.022922 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:03.022891    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-ee509d-5568559575-g8848"]
Nov 02 12:48:03.320394 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:03.320356    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-ee509d-6d46494d8c-q2qwp"]
Nov 02 12:48:03.321611 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:48:03.321572    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12ce9360_ddff_4f72_acbb_bf9659a5c0a3.slice/crio-129dbacc09e08f55314750f2b0df21c748c591ab3860fbfab611f398e0b20f81 WatchSource:0}: Error finding container 129dbacc09e08f55314750f2b0df21c748c591ab3860fbfab611f398e0b20f81: Status 404 returned error can't find the container with id 129dbacc09e08f55314750f2b0df21c748c591ab3860fbfab611f398e0b20f81
Nov 02 12:48:03.661095 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:03.660995    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-078fd3-6ff794864d-8sxnf" event={"ID":"fdb54d38-ce90-43af-9ba9-da98f13d788a","Type":"ContainerStarted","Data":"86a042f51b922205612de60dcee71cbbb75bfedc0f875e4c0534877f742f6102"}
Nov 02 12:48:03.661095 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:03.661063    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-078fd3-6ff794864d-8sxnf"
Nov 02 12:48:03.662723 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:03.662698    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ee509d-5568559575-g8848" event={"ID":"14d49a01-92ac-4a12-9c40-82dcb6dc0bfd","Type":"ContainerStarted","Data":"9bd2879074940650dcbe3ab5ac388e44726ff6a87845877eb219ae810ff57854"}
Nov 02 12:48:03.662858 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:03.662745    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ee509d-5568559575-g8848" event={"ID":"14d49a01-92ac-4a12-9c40-82dcb6dc0bfd","Type":"ContainerStarted","Data":"22bcaef227f301233a964199f9a25a78a335c531e541c2f78a8e4eae8bfe0cf8"}
Nov 02 12:48:03.662858 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:03.662773    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ee509d-5568559575-g8848"
Nov 02 12:48:03.664383 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:03.664359    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-ee509d-6d46494d8c-q2qwp" event={"ID":"12ce9360-ddff-4f72-acbb-bf9659a5c0a3","Type":"ContainerStarted","Data":"2d279267b8f4d636a8680c93e7a7eabbe24e3c562437c16fd56e64dbcbe85aad"}
Nov 02 12:48:03.664497 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:03.664408    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-ee509d-6d46494d8c-q2qwp" event={"ID":"12ce9360-ddff-4f72-acbb-bf9659a5c0a3","Type":"ContainerStarted","Data":"129dbacc09e08f55314750f2b0df21c748c591ab3860fbfab611f398e0b20f81"}
Nov 02 12:48:03.664497 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:03.664439    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-ee509d-6d46494d8c-q2qwp"
Nov 02 12:48:03.665995 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:03.665973    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-078fd3-7b66f65b99-7cclg" event={"ID":"daae67e4-da6e-4eb7-b044-8e795f3b3a29","Type":"ContainerStarted","Data":"661fc7bb28f4703f0c7e60c9427a1cf8d7af837fa6aeab3dd419f713cea87371"}
Nov 02 12:48:03.715151 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:03.715094    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-078fd3-6ff794864d-8sxnf" podStartSLOduration=2.715078898 podStartE2EDuration="2.715078898s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:48:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:48:03.709987105 +0000 UTC m=+1279.965386213" watchObservedRunningTime="2026-11-02 12:48:03.715078898 +0000 UTC m=+1279.970477995"
Nov 02 12:48:03.752750 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:03.752698    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ee509d-5568559575-g8848" podStartSLOduration=1.752684122 podStartE2EDuration="1.752684122s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:48:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:48:03.752076199 +0000 UTC m=+1280.007475291" watchObservedRunningTime="2026-11-02 12:48:03.752684122 +0000 UTC m=+1280.008083217"
Nov 02 12:48:03.813665 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:03.812048    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-ee509d-6d46494d8c-q2qwp" podStartSLOduration=1.812025671 podStartE2EDuration="1.812025671s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:48:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:48:03.799017101 +0000 UTC m=+1280.054416209" watchObservedRunningTime="2026-11-02 12:48:03.812025671 +0000 UTC m=+1280.067424769"
Nov 02 12:48:08.774728 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:08.774192    4559 generic.go:356] "Generic (PLEG): container finished" podID="4906067b-634f-4251-9953-e4c70616400b" containerID="dc9e4f833b5068cfdc384b48c192cdc27e491ab61fbc76d4335a6ddfc342be46" exitCode=0
Nov 02 12:48:08.774728 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:08.774332    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f7f44a-openshift-default-765dbcd8cf-t8fz5" event={"ID":"4906067b-634f-4251-9953-e4c70616400b","Type":"ContainerDied","Data":"dc9e4f833b5068cfdc384b48c192cdc27e491ab61fbc76d4335a6ddfc342be46"}
Nov 02 12:48:08.775283 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:08.774884    4559 scope.go:122] "RemoveContainer" containerID="dc9e4f833b5068cfdc384b48c192cdc27e491ab61fbc76d4335a6ddfc342be46"
Nov 02 12:48:08.779217 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:08.777295    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:48:08.779217 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:08.777391    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:48:08.779217 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:08.777448    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:48:09.782575 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:09.782542    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f7f44a-openshift-default-765dbcd8cf-t8fz5" event={"ID":"4906067b-634f-4251-9953-e4c70616400b","Type":"ContainerStarted","Data":"b3c98a866162e106ba9faa3937f33559acd6ad622d7778dbb5ba2265d5c4b042"}
Nov 02 12:48:10.673943 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:10.673909    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-f7f44a-openshift-default-765dbcd8cf-t8fz5"
Nov 02 12:48:10.674110 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:10.674049    4559 patch_prober.go:29] interesting pod/e2e-api-b-f7f44a-openshift-default-765dbcd8cf-t8fz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body=
Nov 02 12:48:10.674110 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:10.674093    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f7f44a-openshift-default-765dbcd8cf-t8fz5" podUID="4906067b-634f-4251-9953-e4c70616400b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused"
Nov 02 12:48:11.673984 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:11.673945    4559 patch_prober.go:29] interesting pod/e2e-api-b-f7f44a-openshift-default-765dbcd8cf-t8fz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body=
Nov 02 12:48:11.674398 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:11.673997    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f7f44a-openshift-default-765dbcd8cf-t8fz5" podUID="4906067b-634f-4251-9953-e4c70616400b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused"
Nov 02 12:48:12.673797 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:12.673753    4559 patch_prober.go:29] interesting pod/e2e-api-b-f7f44a-openshift-default-765dbcd8cf-t8fz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body=
Nov 02 12:48:12.674028 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:12.673817    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f7f44a-openshift-default-765dbcd8cf-t8fz5" podUID="4906067b-634f-4251-9953-e4c70616400b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused"
Nov 02 12:48:12.975509 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:12.975422    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-299e74-7c7d6bb879-bkw2f"]
Nov 02 12:48:12.980443 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:12.980421    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-299e74-7c7d6bb879-bkw2f"
Nov 02 12:48:13.015489 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:13.015453    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-299e74-7c7d6bb879-bkw2f"]
Nov 02 12:48:13.019298 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:13.019268    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-299e74-dockercfg-v9428\""
Nov 02 12:48:13.019575 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:13.019557    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-299e74\""
Nov 02 12:48:13.088668 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:13.080701    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/545d586e-9e3a-452a-8b0b-2775d5e49317-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-299e74-7c7d6bb879-bkw2f\" (UID: \"545d586e-9e3a-452a-8b0b-2775d5e49317\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-299e74-7c7d6bb879-bkw2f"
Nov 02 12:48:13.103755 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:13.103707    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpbtc\" (UniqueName: \"kubernetes.io/projected/545d586e-9e3a-452a-8b0b-2775d5e49317-kube-api-access-hpbtc\") pod \"payload-pre-processing-e2e-ipp-b-299e74-7c7d6bb879-bkw2f\" (UID: \"545d586e-9e3a-452a-8b0b-2775d5e49317\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-299e74-7c7d6bb879-bkw2f"
Nov 02 12:48:13.192528 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:13.192486    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-299e74-865787d6ff-c8lt4"]
Nov 02 12:48:13.241725 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:13.229786    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/545d586e-9e3a-452a-8b0b-2775d5e49317-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-299e74-7c7d6bb879-bkw2f\" (UID: \"545d586e-9e3a-452a-8b0b-2775d5e49317\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-299e74-7c7d6bb879-bkw2f"
Nov 02 12:48:13.241725 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:13.229993    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpbtc\" (UniqueName: \"kubernetes.io/projected/545d586e-9e3a-452a-8b0b-2775d5e49317-kube-api-access-hpbtc\") pod \"payload-pre-processing-e2e-ipp-b-299e74-7c7d6bb879-bkw2f\" (UID: \"545d586e-9e3a-452a-8b0b-2775d5e49317\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-299e74-7c7d6bb879-bkw2f"
Nov 02 12:48:13.241725 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:13.230538    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-299e74-865787d6ff-c8lt4"
Nov 02 12:48:13.263668 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:13.252205    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/545d586e-9e3a-452a-8b0b-2775d5e49317-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-299e74-7c7d6bb879-bkw2f\" (UID: \"545d586e-9e3a-452a-8b0b-2775d5e49317\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-299e74-7c7d6bb879-bkw2f"
Nov 02 12:48:13.344665 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:13.336467    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9x8sp\" (UniqueName: \"kubernetes.io/projected/17f700b6-f498-497e-a292-3ae243f65858-kube-api-access-9x8sp\") pod \"payload-processing-e2e-ipp-b-299e74-865787d6ff-c8lt4\" (UID: \"17f700b6-f498-497e-a292-3ae243f65858\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-299e74-865787d6ff-c8lt4"
Nov 02 12:48:13.344665 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:13.337432    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17f700b6-f498-497e-a292-3ae243f65858-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-299e74-865787d6ff-c8lt4\" (UID: \"17f700b6-f498-497e-a292-3ae243f65858\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-299e74-865787d6ff-c8lt4"
Nov 02 12:48:13.370180 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:13.370140    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-299e74-865787d6ff-c8lt4"]
Nov 02 12:48:13.378685 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:13.375963    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpbtc\" (UniqueName: \"kubernetes.io/projected/545d586e-9e3a-452a-8b0b-2775d5e49317-kube-api-access-hpbtc\") pod \"payload-pre-processing-e2e-ipp-b-299e74-7c7d6bb879-bkw2f\" (UID: \"545d586e-9e3a-452a-8b0b-2775d5e49317\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-299e74-7c7d6bb879-bkw2f"
Nov 02 12:48:13.448727 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:13.448689    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17f700b6-f498-497e-a292-3ae243f65858-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-299e74-865787d6ff-c8lt4\" (UID: \"17f700b6-f498-497e-a292-3ae243f65858\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-299e74-865787d6ff-c8lt4"
Nov 02 12:48:13.449160 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:13.449131    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9x8sp\" (UniqueName: \"kubernetes.io/projected/17f700b6-f498-497e-a292-3ae243f65858-kube-api-access-9x8sp\") pod \"payload-processing-e2e-ipp-b-299e74-865787d6ff-c8lt4\" (UID: \"17f700b6-f498-497e-a292-3ae243f65858\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-299e74-865787d6ff-c8lt4"
Nov 02 12:48:13.449940 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:13.449899    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17f700b6-f498-497e-a292-3ae243f65858-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-299e74-865787d6ff-c8lt4\" (UID: \"17f700b6-f498-497e-a292-3ae243f65858\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-299e74-865787d6ff-c8lt4"
Nov 02 12:48:13.473552 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:13.473522    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9x8sp\" (UniqueName: \"kubernetes.io/projected/17f700b6-f498-497e-a292-3ae243f65858-kube-api-access-9x8sp\") pod \"payload-processing-e2e-ipp-b-299e74-865787d6ff-c8lt4\" (UID: \"17f700b6-f498-497e-a292-3ae243f65858\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-299e74-865787d6ff-c8lt4"
Nov 02 12:48:13.506713 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:13.501905    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-299e74-7c7d6bb879-bkw2f"
Nov 02 12:48:13.673886 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:13.673834    4559 patch_prober.go:29] interesting pod/e2e-api-b-f7f44a-openshift-default-765dbcd8cf-t8fz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body=
Nov 02 12:48:13.674129 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:13.673910    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f7f44a-openshift-default-765dbcd8cf-t8fz5" podUID="4906067b-634f-4251-9953-e4c70616400b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused"
Nov 02 12:48:13.743494 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:13.743458    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-299e74-865787d6ff-c8lt4"
Nov 02 12:48:13.821848 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:48:13.821818    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod545d586e_9e3a_452a_8b0b_2775d5e49317.slice/crio-965387834e53617ca8764304592c70bd6f6d27c7f7547e53414c43056dee9c5b WatchSource:0}: Error finding container 965387834e53617ca8764304592c70bd6f6d27c7f7547e53414c43056dee9c5b: Status 404 returned error can't find the container with id 965387834e53617ca8764304592c70bd6f6d27c7f7547e53414c43056dee9c5b
Nov 02 12:48:13.824347 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:13.824320    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-299e74-7c7d6bb879-bkw2f"]
Nov 02 12:48:13.841224 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:13.841176    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-299e74-7c7d6bb879-bkw2f" event={"ID":"545d586e-9e3a-452a-8b0b-2775d5e49317","Type":"ContainerStarted","Data":"965387834e53617ca8764304592c70bd6f6d27c7f7547e53414c43056dee9c5b"}
Nov 02 12:48:14.074218 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:48:14.074190    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17f700b6_f498_497e_a292_3ae243f65858.slice/crio-3f1fae93371e027e01f710c957d8b31843ec4cd6f3827ce658eb6178c954c1c8 WatchSource:0}: Error finding container 3f1fae93371e027e01f710c957d8b31843ec4cd6f3827ce658eb6178c954c1c8: Status 404 returned error can't find the container with id 3f1fae93371e027e01f710c957d8b31843ec4cd6f3827ce658eb6178c954c1c8
Nov 02 12:48:14.074380 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:14.074357    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-299e74-865787d6ff-c8lt4"]
Nov 02 12:48:14.683666 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:14.682839    4559 patch_prober.go:29] interesting pod/e2e-api-b-f7f44a-openshift-default-765dbcd8cf-t8fz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body=
Nov 02 12:48:14.683666 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:14.682896    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f7f44a-openshift-default-765dbcd8cf-t8fz5" podUID="4906067b-634f-4251-9953-e4c70616400b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused"
Nov 02 12:48:14.718666 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:14.711920    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-078fd3-7b66f65b99-7cclg"
Nov 02 12:48:14.731720 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:14.721023    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-078fd3-6ff794864d-8sxnf"
Nov 02 12:48:14.731720 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:14.721791    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-ee509d-6d46494d8c-q2qwp"
Nov 02 12:48:14.743707 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:14.742844    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ee509d-5568559575-g8848"
Nov 02 12:48:14.856749 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:14.856714    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-299e74-7c7d6bb879-bkw2f" event={"ID":"545d586e-9e3a-452a-8b0b-2775d5e49317","Type":"ContainerStarted","Data":"9c33c19e71901fcb230c538093285f9890a97fa4e8117e2f7f96da7ddb163ba5"}
Nov 02 12:48:14.857066 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:14.857032    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-299e74-7c7d6bb879-bkw2f"
Nov 02 12:48:14.859063 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:14.859035    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-299e74-865787d6ff-c8lt4" event={"ID":"17f700b6-f498-497e-a292-3ae243f65858","Type":"ContainerStarted","Data":"2b171615ed74f01ddd8f91a6543e7981cae93a7e5de9d5368e648c7701cfcfeb"}
Nov 02 12:48:14.859186 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:14.859088    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-299e74-865787d6ff-c8lt4" event={"ID":"17f700b6-f498-497e-a292-3ae243f65858","Type":"ContainerStarted","Data":"3f1fae93371e027e01f710c957d8b31843ec4cd6f3827ce658eb6178c954c1c8"}
Nov 02 12:48:14.859186 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:14.859117    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-299e74-865787d6ff-c8lt4"
Nov 02 12:48:14.968023 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:14.967931    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-299e74-865787d6ff-c8lt4" podStartSLOduration=1.967916754 podStartE2EDuration="1.967916754s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:48:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:48:14.956824832 +0000 UTC m=+1291.212223937" watchObservedRunningTime="2026-11-02 12:48:14.967916754 +0000 UTC m=+1291.223315849"
Nov 02 12:48:14.999913 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:14.999862    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-299e74-7c7d6bb879-bkw2f" podStartSLOduration=2.999848053 podStartE2EDuration="2.999848053s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:48:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:48:14.997604306 +0000 UTC m=+1291.253003383" watchObservedRunningTime="2026-11-02 12:48:14.999848053 +0000 UTC m=+1291.255247147"
Nov 02 12:48:15.087700 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:48:15.087664    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqcmj" containerName="extractor"
Nov 02 12:48:15.679020 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:15.678989    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-f7f44a-openshift-default-765dbcd8cf-t8fz5"
Nov 02 12:48:15.679291 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:15.679258    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-f7f44a-openshift-default-765dbcd8cf-t8fz5"
Nov 02 12:48:15.679906 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:15.679892    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-f7f44a-openshift-default-765dbcd8cf-t8fz5"
Nov 02 12:48:20.169457 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:20.169424    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"]
Nov 02 12:48:23.687920 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:23.687887    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-b2ec05-57ff8d5965-9k4gl"]
Nov 02 12:48:25.869789 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:25.869703    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-299e74-865787d6ff-c8lt4"
Nov 02 12:48:25.869789 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:25.869787    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-299e74-7c7d6bb879-bkw2f"
Nov 02 12:48:28.506300 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:28.506266    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-59060a-cfb6d4595-zt7rh"]
Nov 02 12:48:28.506799 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:28.506528    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-59060a-cfb6d4595-zt7rh" podUID="2265e1d5-bae9-495b-b51d-b3e638c128ea" containerName="maas-api" containerID="cri-o://5249048539e9edfb82c9a8ba02fbe9f17fcb0a1ab3ec77bd34fef810bb779e5e" gracePeriod=30
Nov 02 12:48:28.689554 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:28.688871    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-b2ec05-57ff8d5965-9k4gl" podUID="d3f2ed27-59d4-4eba-933a-bea46a18d385" containerName="payload-processing" containerID="cri-o://e16362647b3e3af3aa3f13e13a6de2dd8b285ccb9ee155661df99d2a919d1d3a" gracePeriod=25
Nov 02 12:48:28.830713 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:28.830688    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-59060a-cfb6d4595-zt7rh"
Nov 02 12:48:28.944672 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:28.944608    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2265e1d5-bae9-495b-b51d-b3e638c128ea-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2265e1d5-bae9-495b-b51d-b3e638c128ea-maas-api-tls\") pod \"2265e1d5-bae9-495b-b51d-b3e638c128ea\" (UID: \"2265e1d5-bae9-495b-b51d-b3e638c128ea\") "
Nov 02 12:48:28.944859 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:28.944747    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2265e1d5-bae9-495b-b51d-b3e638c128ea-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2265e1d5-bae9-495b-b51d-b3e638c128ea-metrics-certs\") pod \"2265e1d5-bae9-495b-b51d-b3e638c128ea\" (UID: \"2265e1d5-bae9-495b-b51d-b3e638c128ea\") "
Nov 02 12:48:28.944938 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:28.944874    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2265e1d5-bae9-495b-b51d-b3e638c128ea-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2265e1d5-bae9-495b-b51d-b3e638c128ea-openshift-service-ca\") pod \"2265e1d5-bae9-495b-b51d-b3e638c128ea\" (UID: \"2265e1d5-bae9-495b-b51d-b3e638c128ea\") "
Nov 02 12:48:28.945230 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:28.945205    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2265e1d5-bae9-495b-b51d-b3e638c128ea-openshift-service-ca" pod "2265e1d5-bae9-495b-b51d-b3e638c128ea" (UID: "2265e1d5-bae9-495b-b51d-b3e638c128ea"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:48:28.945419 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:28.945317    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2265e1d5-bae9-495b-b51d-b3e638c128ea-kube-api-access-t9wkt\" (UniqueName: \"kubernetes.io/projected/2265e1d5-bae9-495b-b51d-b3e638c128ea-kube-api-access-t9wkt\") pod \"2265e1d5-bae9-495b-b51d-b3e638c128ea\" (UID: \"2265e1d5-bae9-495b-b51d-b3e638c128ea\") "
Nov 02 12:48:28.945921 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:28.945900    4559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2265e1d5-bae9-495b-b51d-b3e638c128ea-openshift-service-ca\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:48:28.947031 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:28.947007    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2265e1d5-bae9-495b-b51d-b3e638c128ea-maas-api-tls" pod "2265e1d5-bae9-495b-b51d-b3e638c128ea" (UID: "2265e1d5-bae9-495b-b51d-b3e638c128ea"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:48:28.947338 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:28.947317    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2265e1d5-bae9-495b-b51d-b3e638c128ea-metrics-certs" pod "2265e1d5-bae9-495b-b51d-b3e638c128ea" (UID: "2265e1d5-bae9-495b-b51d-b3e638c128ea"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:48:28.947539 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:28.947517    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2265e1d5-bae9-495b-b51d-b3e638c128ea-kube-api-access-t9wkt" pod "2265e1d5-bae9-495b-b51d-b3e638c128ea" (UID: "2265e1d5-bae9-495b-b51d-b3e638c128ea"). InnerVolumeSpecName "kube-api-access-t9wkt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:48:28.968120 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:28.968069    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-b2ec05-57ff8d5965-9k4gl"
Nov 02 12:48:28.973382 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:28.973348    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-b2ec05-5c56f74f78-jnq6q"]
Nov 02 12:48:28.973715 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:28.973591    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b2ec05-5c56f74f78-jnq6q" podUID="00c5fac7-35bf-426e-b9f2-d1900ac09c83" containerName="maas-api" containerID="cri-o://d7454aba9ed7cd2c3350eed1b4ef1b967228ed7daa8540a741e2202013290ca9" gracePeriod=30
Nov 02 12:48:28.982037 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:28.982011    4559 generic.go:356] "Generic (PLEG): container finished" podID="2265e1d5-bae9-495b-b51d-b3e638c128ea" containerID="5249048539e9edfb82c9a8ba02fbe9f17fcb0a1ab3ec77bd34fef810bb779e5e" exitCode=0
Nov 02 12:48:28.982294 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:28.982165    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-59060a-cfb6d4595-zt7rh" event={"ID":"2265e1d5-bae9-495b-b51d-b3e638c128ea","Type":"ContainerDied","Data":"5249048539e9edfb82c9a8ba02fbe9f17fcb0a1ab3ec77bd34fef810bb779e5e"}
Nov 02 12:48:28.982294 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:28.982217    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-59060a-cfb6d4595-zt7rh" event={"ID":"2265e1d5-bae9-495b-b51d-b3e638c128ea","Type":"ContainerDied","Data":"517988dc30077f2e8c3da04ce6b11c47e5a554f60a6dfe125c52a5b14fbe2028"}
Nov 02 12:48:28.982294 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:28.982244    4559 scope.go:122] "RemoveContainer" containerID="5249048539e9edfb82c9a8ba02fbe9f17fcb0a1ab3ec77bd34fef810bb779e5e"
Nov 02 12:48:28.982475 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:28.982423    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-59060a-cfb6d4595-zt7rh"
Nov 02 12:48:28.989381 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:28.989351    4559 generic.go:356] "Generic (PLEG): container finished" podID="d3f2ed27-59d4-4eba-933a-bea46a18d385" containerID="e16362647b3e3af3aa3f13e13a6de2dd8b285ccb9ee155661df99d2a919d1d3a" exitCode=0
Nov 02 12:48:28.989528 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:28.989513    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-b2ec05-57ff8d5965-9k4gl" event={"ID":"d3f2ed27-59d4-4eba-933a-bea46a18d385","Type":"ContainerDied","Data":"e16362647b3e3af3aa3f13e13a6de2dd8b285ccb9ee155661df99d2a919d1d3a"}
Nov 02 12:48:28.989601 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:28.989560    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-b2ec05-57ff8d5965-9k4gl" event={"ID":"d3f2ed27-59d4-4eba-933a-bea46a18d385","Type":"ContainerDied","Data":"acacf2ee9417de7eb9635d20d10f94a6d7eec49b9bae14202415cdc221781a60"}
Nov 02 12:48:28.989716 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:28.989679    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-b2ec05-57ff8d5965-9k4gl"
Nov 02 12:48:29.048423 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:29.047193    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3f2ed27-59d4-4eba-933a-bea46a18d385-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3f2ed27-59d4-4eba-933a-bea46a18d385-plugins-config-volume\") pod \"d3f2ed27-59d4-4eba-933a-bea46a18d385\" (UID: \"d3f2ed27-59d4-4eba-933a-bea46a18d385\") "
Nov 02 12:48:29.048423 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:29.047380    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3f2ed27-59d4-4eba-933a-bea46a18d385-kube-api-access-8tsmg\" (UniqueName: \"kubernetes.io/projected/d3f2ed27-59d4-4eba-933a-bea46a18d385-kube-api-access-8tsmg\") pod \"d3f2ed27-59d4-4eba-933a-bea46a18d385\" (UID: \"d3f2ed27-59d4-4eba-933a-bea46a18d385\") "
Nov 02 12:48:29.048423 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:29.048073    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t9wkt\" (UniqueName: \"kubernetes.io/projected/2265e1d5-bae9-495b-b51d-b3e638c128ea-kube-api-access-t9wkt\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:48:29.048423 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:29.048094    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2265e1d5-bae9-495b-b51d-b3e638c128ea-maas-api-tls\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:48:29.048423 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:29.048113    4559 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2265e1d5-bae9-495b-b51d-b3e638c128ea-metrics-certs\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:48:29.049401 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:29.049370    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3f2ed27-59d4-4eba-933a-bea46a18d385-plugins-config-volume" pod "d3f2ed27-59d4-4eba-933a-bea46a18d385" (UID: "d3f2ed27-59d4-4eba-933a-bea46a18d385"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:48:29.051233 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:29.051208    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3f2ed27-59d4-4eba-933a-bea46a18d385-kube-api-access-8tsmg" pod "d3f2ed27-59d4-4eba-933a-bea46a18d385" (UID: "d3f2ed27-59d4-4eba-933a-bea46a18d385"). InnerVolumeSpecName "kube-api-access-8tsmg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:48:29.092670 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:29.092631    4559 scope.go:122] "RemoveContainer" containerID="5249048539e9edfb82c9a8ba02fbe9f17fcb0a1ab3ec77bd34fef810bb779e5e"
Nov 02 12:48:29.093036 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:48:29.093014    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5249048539e9edfb82c9a8ba02fbe9f17fcb0a1ab3ec77bd34fef810bb779e5e\": container with ID starting with 5249048539e9edfb82c9a8ba02fbe9f17fcb0a1ab3ec77bd34fef810bb779e5e not found: ID does not exist" containerID="5249048539e9edfb82c9a8ba02fbe9f17fcb0a1ab3ec77bd34fef810bb779e5e"
Nov 02 12:48:29.093126 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:29.093046    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5249048539e9edfb82c9a8ba02fbe9f17fcb0a1ab3ec77bd34fef810bb779e5e"} err="failed to get container status \"5249048539e9edfb82c9a8ba02fbe9f17fcb0a1ab3ec77bd34fef810bb779e5e\": rpc error: code = NotFound desc = could not find container \"5249048539e9edfb82c9a8ba02fbe9f17fcb0a1ab3ec77bd34fef810bb779e5e\": container with ID starting with 5249048539e9edfb82c9a8ba02fbe9f17fcb0a1ab3ec77bd34fef810bb779e5e not found: ID does not exist"
Nov 02 12:48:29.093126 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:29.093063    4559 scope.go:122] "RemoveContainer" containerID="e16362647b3e3af3aa3f13e13a6de2dd8b285ccb9ee155661df99d2a919d1d3a"
Nov 02 12:48:29.105234 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:29.105209    4559 scope.go:122] "RemoveContainer" containerID="e16362647b3e3af3aa3f13e13a6de2dd8b285ccb9ee155661df99d2a919d1d3a"
Nov 02 12:48:29.105620 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:48:29.105584    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e16362647b3e3af3aa3f13e13a6de2dd8b285ccb9ee155661df99d2a919d1d3a\": container with ID starting with e16362647b3e3af3aa3f13e13a6de2dd8b285ccb9ee155661df99d2a919d1d3a not found: ID does not exist" containerID="e16362647b3e3af3aa3f13e13a6de2dd8b285ccb9ee155661df99d2a919d1d3a"
Nov 02 12:48:29.105748 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:29.105630    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e16362647b3e3af3aa3f13e13a6de2dd8b285ccb9ee155661df99d2a919d1d3a"} err="failed to get container status \"e16362647b3e3af3aa3f13e13a6de2dd8b285ccb9ee155661df99d2a919d1d3a\": rpc error: code = NotFound desc = could not find container \"e16362647b3e3af3aa3f13e13a6de2dd8b285ccb9ee155661df99d2a919d1d3a\": container with ID starting with e16362647b3e3af3aa3f13e13a6de2dd8b285ccb9ee155661df99d2a919d1d3a not found: ID does not exist"
Nov 02 12:48:29.135099 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:29.135064    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-59060a-cfb6d4595-zt7rh"]
Nov 02 12:48:29.145685 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:29.145653    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-59060a-cfb6d4595-zt7rh"]
Nov 02 12:48:29.149975 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:29.149949    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8tsmg\" (UniqueName: \"kubernetes.io/projected/d3f2ed27-59d4-4eba-933a-bea46a18d385-kube-api-access-8tsmg\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:48:29.150108 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:29.149980    4559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3f2ed27-59d4-4eba-933a-bea46a18d385-plugins-config-volume\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:48:29.406079 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:29.405953    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b2ec05-5c56f74f78-jnq6q"
Nov 02 12:48:29.429677 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:29.425740    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-b2ec05-57ff8d5965-9k4gl"]
Nov 02 12:48:29.452679 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:29.446558    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-b2ec05-57ff8d5965-9k4gl"]
Nov 02 12:48:29.512599 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:29.512029    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00c5fac7-35bf-426e-b9f2-d1900ac09c83-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/00c5fac7-35bf-426e-b9f2-d1900ac09c83-openshift-service-ca\") pod \"00c5fac7-35bf-426e-b9f2-d1900ac09c83\" (UID: \"00c5fac7-35bf-426e-b9f2-d1900ac09c83\") "
Nov 02 12:48:29.512599 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:29.512096    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/00c5fac7-35bf-426e-b9f2-d1900ac09c83-metrics-certs\" (UniqueName: \"kubernetes.io/secret/00c5fac7-35bf-426e-b9f2-d1900ac09c83-metrics-certs\") pod \"00c5fac7-35bf-426e-b9f2-d1900ac09c83\" (UID: \"00c5fac7-35bf-426e-b9f2-d1900ac09c83\") "
Nov 02 12:48:29.516187 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:29.512776    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00c5fac7-35bf-426e-b9f2-d1900ac09c83-openshift-service-ca" pod "00c5fac7-35bf-426e-b9f2-d1900ac09c83" (UID: "00c5fac7-35bf-426e-b9f2-d1900ac09c83"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:48:29.516187 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:29.512943    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00c5fac7-35bf-426e-b9f2-d1900ac09c83-kube-api-access-2kzgk\" (UniqueName: \"kubernetes.io/projected/00c5fac7-35bf-426e-b9f2-d1900ac09c83-kube-api-access-2kzgk\") pod \"00c5fac7-35bf-426e-b9f2-d1900ac09c83\" (UID: \"00c5fac7-35bf-426e-b9f2-d1900ac09c83\") "
Nov 02 12:48:29.516187 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:29.513207    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/00c5fac7-35bf-426e-b9f2-d1900ac09c83-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00c5fac7-35bf-426e-b9f2-d1900ac09c83-maas-api-tls\") pod \"00c5fac7-35bf-426e-b9f2-d1900ac09c83\" (UID: \"00c5fac7-35bf-426e-b9f2-d1900ac09c83\") "
Nov 02 12:48:29.516187 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:29.514254    4559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/00c5fac7-35bf-426e-b9f2-d1900ac09c83-openshift-service-ca\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:48:29.523702 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:29.523632    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00c5fac7-35bf-426e-b9f2-d1900ac09c83-maas-api-tls" pod "00c5fac7-35bf-426e-b9f2-d1900ac09c83" (UID: "00c5fac7-35bf-426e-b9f2-d1900ac09c83"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:48:29.541673 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:29.541624    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00c5fac7-35bf-426e-b9f2-d1900ac09c83-kube-api-access-2kzgk" pod "00c5fac7-35bf-426e-b9f2-d1900ac09c83" (UID: "00c5fac7-35bf-426e-b9f2-d1900ac09c83"). InnerVolumeSpecName "kube-api-access-2kzgk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:48:29.627391 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:29.616852    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2kzgk\" (UniqueName: \"kubernetes.io/projected/00c5fac7-35bf-426e-b9f2-d1900ac09c83-kube-api-access-2kzgk\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:48:29.627391 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:29.616893    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00c5fac7-35bf-426e-b9f2-d1900ac09c83-maas-api-tls\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:48:29.714364 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:29.711808    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00c5fac7-35bf-426e-b9f2-d1900ac09c83-metrics-certs" pod "00c5fac7-35bf-426e-b9f2-d1900ac09c83" (UID: "00c5fac7-35bf-426e-b9f2-d1900ac09c83"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:48:29.723543 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:29.723494    4559 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/00c5fac7-35bf-426e-b9f2-d1900ac09c83-metrics-certs\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:48:29.947600 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:29.946286    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2265e1d5-bae9-495b-b51d-b3e638c128ea" path="/var/lib/kubelet/pods/2265e1d5-bae9-495b-b51d-b3e638c128ea/volumes"
Nov 02 12:48:29.947600 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:29.947229    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3f2ed27-59d4-4eba-933a-bea46a18d385" path="/var/lib/kubelet/pods/d3f2ed27-59d4-4eba-933a-bea46a18d385/volumes"
Nov 02 12:48:29.999211 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:29.999181    4559 generic.go:356] "Generic (PLEG): container finished" podID="00c5fac7-35bf-426e-b9f2-d1900ac09c83" containerID="d7454aba9ed7cd2c3350eed1b4ef1b967228ed7daa8540a741e2202013290ca9" exitCode=0
Nov 02 12:48:29.999386 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:29.999234    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b2ec05-5c56f74f78-jnq6q"
Nov 02 12:48:29.999386 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:29.999255    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b2ec05-5c56f74f78-jnq6q" event={"ID":"00c5fac7-35bf-426e-b9f2-d1900ac09c83","Type":"ContainerDied","Data":"d7454aba9ed7cd2c3350eed1b4ef1b967228ed7daa8540a741e2202013290ca9"}
Nov 02 12:48:29.999386 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:29.999307    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b2ec05-5c56f74f78-jnq6q" event={"ID":"00c5fac7-35bf-426e-b9f2-d1900ac09c83","Type":"ContainerDied","Data":"2b0ca30455ee6582b6533ae8d0ff3b8ab101e32937e6b1a1dc8789f8099ecb36"}
Nov 02 12:48:29.999386 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:29.999330    4559 scope.go:122] "RemoveContainer" containerID="d7454aba9ed7cd2c3350eed1b4ef1b967228ed7daa8540a741e2202013290ca9"
Nov 02 12:48:30.010234 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:30.010213    4559 scope.go:122] "RemoveContainer" containerID="d7454aba9ed7cd2c3350eed1b4ef1b967228ed7daa8540a741e2202013290ca9"
Nov 02 12:48:30.010508 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:48:30.010489    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7454aba9ed7cd2c3350eed1b4ef1b967228ed7daa8540a741e2202013290ca9\": container with ID starting with d7454aba9ed7cd2c3350eed1b4ef1b967228ed7daa8540a741e2202013290ca9 not found: ID does not exist" containerID="d7454aba9ed7cd2c3350eed1b4ef1b967228ed7daa8540a741e2202013290ca9"
Nov 02 12:48:30.010596 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:30.010515    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7454aba9ed7cd2c3350eed1b4ef1b967228ed7daa8540a741e2202013290ca9"} err="failed to get container status \"d7454aba9ed7cd2c3350eed1b4ef1b967228ed7daa8540a741e2202013290ca9\": rpc error: code = NotFound desc = could not find container \"d7454aba9ed7cd2c3350eed1b4ef1b967228ed7daa8540a741e2202013290ca9\": container with ID starting with d7454aba9ed7cd2c3350eed1b4ef1b967228ed7daa8540a741e2202013290ca9 not found: ID does not exist"
Nov 02 12:48:30.057251 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:30.057223    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-b2ec05-5c56f74f78-jnq6q"]
Nov 02 12:48:30.067332 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:30.067307    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-b2ec05-5c56f74f78-jnq6q"]
Nov 02 12:48:31.894168 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:31.894134    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00c5fac7-35bf-426e-b9f2-d1900ac09c83" path="/var/lib/kubelet/pods/00c5fac7-35bf-426e-b9f2-d1900ac09c83/volumes"
Nov 02 12:48:32.244464 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:32.244422    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-7s2lq"]
Nov 02 12:48:32.244685 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:32.244661    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-7s2lq" podUID="0ac9ccd9-92db-4bc2-8900-772272b4cc80" containerName="istio-proxy" containerID="cri-o://34968672bdf04df79e66a09d897e7c6a67902c459dabd4f3f5870ef9a23542ef" gracePeriod=30
Nov 02 12:48:34.475827 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:34.475789    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-59060a-openshift-default-8666cdf765-hfz77"]
Nov 02 12:48:34.476301 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:34.476030    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-59060a-openshift-default-8666cdf765-hfz77" podUID="9ed1a149-3a01-4af1-9b01-e059ff121bed" containerName="istio-proxy" containerID="cri-o://73d0524d7341a61dfeeda451f1a2a72043fc9e3d3f045d0d0fe49a0b6eae043d" gracePeriod=30
Nov 02 12:48:35.313581 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:35.312997    4559 patch_prober.go:29] interesting pod/e2e-ipp-a-59060a-openshift-default-8666cdf765-hfz77 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 12:48:35.313581 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:35.313065    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-59060a-openshift-default-8666cdf765-hfz77" podUID="9ed1a149-3a01-4af1-9b01-e059ff121bed" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 12:48:35.586620 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:35.586531    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-b2ec05-openshift-default-5f4c648f59-bldmv"]
Nov 02 12:48:35.587328 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:35.586838    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-b2ec05-openshift-default-5f4c648f59-bldmv" podUID="0f74d2ef-c796-4c91-b763-7593c16cfe71" containerName="istio-proxy" containerID="cri-o://153e39ff35c98ce2c5fe699e58991b360731c0637f622ead6afed234f01d22b7" gracePeriod=30
Nov 02 12:48:37.544235 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:37.544203    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-7s2lq"
Nov 02 12:48:37.574960 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:37.574921    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ac9ccd9-92db-4bc2-8900-772272b4cc80-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ac9ccd9-92db-4bc2-8900-772272b4cc80-istio-envoy\") pod \"0ac9ccd9-92db-4bc2-8900-772272b4cc80\" (UID: \"0ac9ccd9-92db-4bc2-8900-772272b4cc80\") "
Nov 02 12:48:37.575122 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:37.575033    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ac9ccd9-92db-4bc2-8900-772272b4cc80-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ac9ccd9-92db-4bc2-8900-772272b4cc80-credential-socket\") pod \"0ac9ccd9-92db-4bc2-8900-772272b4cc80\" (UID: \"0ac9ccd9-92db-4bc2-8900-772272b4cc80\") "
Nov 02 12:48:37.575122 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:37.575066    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ac9ccd9-92db-4bc2-8900-772272b4cc80-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ac9ccd9-92db-4bc2-8900-772272b4cc80-workload-socket\") pod \"0ac9ccd9-92db-4bc2-8900-772272b4cc80\" (UID: \"0ac9ccd9-92db-4bc2-8900-772272b4cc80\") "
Nov 02 12:48:37.575243 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:37.575130    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ac9ccd9-92db-4bc2-8900-772272b4cc80-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ac9ccd9-92db-4bc2-8900-772272b4cc80-istio-data\") pod \"0ac9ccd9-92db-4bc2-8900-772272b4cc80\" (UID: \"0ac9ccd9-92db-4bc2-8900-772272b4cc80\") "
Nov 02 12:48:37.575243 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:37.575155    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ac9ccd9-92db-4bc2-8900-772272b4cc80-kube-api-access-dwrqx\" (UniqueName: \"kubernetes.io/projected/0ac9ccd9-92db-4bc2-8900-772272b4cc80-kube-api-access-dwrqx\") pod \"0ac9ccd9-92db-4bc2-8900-772272b4cc80\" (UID: \"0ac9ccd9-92db-4bc2-8900-772272b4cc80\") "
Nov 02 12:48:37.575243 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:37.575231    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ac9ccd9-92db-4bc2-8900-772272b4cc80-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ac9ccd9-92db-4bc2-8900-772272b4cc80-istiod-ca-cert\") pod \"0ac9ccd9-92db-4bc2-8900-772272b4cc80\" (UID: \"0ac9ccd9-92db-4bc2-8900-772272b4cc80\") "
Nov 02 12:48:37.575470 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:37.575294    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ac9ccd9-92db-4bc2-8900-772272b4cc80-istio-token\" (UniqueName: \"kubernetes.io/projected/0ac9ccd9-92db-4bc2-8900-772272b4cc80-istio-token\") pod \"0ac9ccd9-92db-4bc2-8900-772272b4cc80\" (UID: \"0ac9ccd9-92db-4bc2-8900-772272b4cc80\") "
Nov 02 12:48:37.575470 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:37.575329    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ac9ccd9-92db-4bc2-8900-772272b4cc80-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ac9ccd9-92db-4bc2-8900-772272b4cc80-workload-certs\") pod \"0ac9ccd9-92db-4bc2-8900-772272b4cc80\" (UID: \"0ac9ccd9-92db-4bc2-8900-772272b4cc80\") "
Nov 02 12:48:37.575470 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:37.575411    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ac9ccd9-92db-4bc2-8900-772272b4cc80-credential-socket" pod "0ac9ccd9-92db-4bc2-8900-772272b4cc80" (UID: "0ac9ccd9-92db-4bc2-8900-772272b4cc80"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:48:37.575470 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:37.575457    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0ac9ccd9-92db-4bc2-8900-772272b4cc80-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ac9ccd9-92db-4bc2-8900-772272b4cc80-istio-podinfo\") pod \"0ac9ccd9-92db-4bc2-8900-772272b4cc80\" (UID: \"0ac9ccd9-92db-4bc2-8900-772272b4cc80\") "
Nov 02 12:48:37.575672 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:37.575437    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ac9ccd9-92db-4bc2-8900-772272b4cc80-workload-socket" pod "0ac9ccd9-92db-4bc2-8900-772272b4cc80" (UID: "0ac9ccd9-92db-4bc2-8900-772272b4cc80"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:48:37.575672 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:37.575474    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ac9ccd9-92db-4bc2-8900-772272b4cc80-istio-data" pod "0ac9ccd9-92db-4bc2-8900-772272b4cc80" (UID: "0ac9ccd9-92db-4bc2-8900-772272b4cc80"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:48:37.575672 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:37.575613    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ac9ccd9-92db-4bc2-8900-772272b4cc80-workload-certs" pod "0ac9ccd9-92db-4bc2-8900-772272b4cc80" (UID: "0ac9ccd9-92db-4bc2-8900-772272b4cc80"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:48:37.576011 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:37.575984    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ac9ccd9-92db-4bc2-8900-772272b4cc80-istiod-ca-cert" pod "0ac9ccd9-92db-4bc2-8900-772272b4cc80" (UID: "0ac9ccd9-92db-4bc2-8900-772272b4cc80"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:48:37.576118 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:37.576065    4559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ac9ccd9-92db-4bc2-8900-772272b4cc80-credential-socket\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:48:37.576118 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:37.576086    4559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ac9ccd9-92db-4bc2-8900-772272b4cc80-workload-socket\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:48:37.576118 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:37.576102    4559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ac9ccd9-92db-4bc2-8900-772272b4cc80-istio-data\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:48:37.576118 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:37.576116    4559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ac9ccd9-92db-4bc2-8900-772272b4cc80-workload-certs\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:48:37.577378 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:37.577348    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ac9ccd9-92db-4bc2-8900-772272b4cc80-istio-envoy" pod "0ac9ccd9-92db-4bc2-8900-772272b4cc80" (UID: "0ac9ccd9-92db-4bc2-8900-772272b4cc80"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:48:37.577557 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:37.577532    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ac9ccd9-92db-4bc2-8900-772272b4cc80-istio-token" pod "0ac9ccd9-92db-4bc2-8900-772272b4cc80" (UID: "0ac9ccd9-92db-4bc2-8900-772272b4cc80"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:48:37.577773 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:37.577751    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0ac9ccd9-92db-4bc2-8900-772272b4cc80-istio-podinfo" pod "0ac9ccd9-92db-4bc2-8900-772272b4cc80" (UID: "0ac9ccd9-92db-4bc2-8900-772272b4cc80"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 12:48:37.578064 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:37.578046    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ac9ccd9-92db-4bc2-8900-772272b4cc80-kube-api-access-dwrqx" pod "0ac9ccd9-92db-4bc2-8900-772272b4cc80" (UID: "0ac9ccd9-92db-4bc2-8900-772272b4cc80"). InnerVolumeSpecName "kube-api-access-dwrqx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:48:37.663479 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:37.663442    4559 patch_prober.go:29] interesting pod/e2e-api-a-b2ec05-openshift-default-5f4c648f59-bldmv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 12:48:37.663704 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:37.663510    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-b2ec05-openshift-default-5f4c648f59-bldmv" podUID="0f74d2ef-c796-4c91-b763-7593c16cfe71" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 12:48:37.678054 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:37.677972    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dwrqx\" (UniqueName: \"kubernetes.io/projected/0ac9ccd9-92db-4bc2-8900-772272b4cc80-kube-api-access-dwrqx\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:48:37.678054 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:37.678008    4559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ac9ccd9-92db-4bc2-8900-772272b4cc80-istiod-ca-cert\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:48:37.678054 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:37.678019    4559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ac9ccd9-92db-4bc2-8900-772272b4cc80-istio-token\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:48:37.678054 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:37.678027    4559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ac9ccd9-92db-4bc2-8900-772272b4cc80-istio-podinfo\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:48:37.678054 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:37.678036    4559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ac9ccd9-92db-4bc2-8900-772272b4cc80-istio-envoy\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:48:38.090294 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:38.090253    4559 generic.go:356] "Generic (PLEG): container finished" podID="0ac9ccd9-92db-4bc2-8900-772272b4cc80" containerID="34968672bdf04df79e66a09d897e7c6a67902c459dabd4f3f5870ef9a23542ef" exitCode=0
Nov 02 12:48:38.090294 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:38.090295    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-7s2lq"
Nov 02 12:48:38.090486 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:38.090304    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-7s2lq" event={"ID":"0ac9ccd9-92db-4bc2-8900-772272b4cc80","Type":"ContainerDied","Data":"34968672bdf04df79e66a09d897e7c6a67902c459dabd4f3f5870ef9a23542ef"}
Nov 02 12:48:38.090486 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:38.090362    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-7s2lq" event={"ID":"0ac9ccd9-92db-4bc2-8900-772272b4cc80","Type":"ContainerDied","Data":"92a67199357b6c5c526afeb52729c4c2411d137376c2fbc9b03123dceb922aca"}
Nov 02 12:48:38.090486 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:38.090392    4559 scope.go:122] "RemoveContainer" containerID="34968672bdf04df79e66a09d897e7c6a67902c459dabd4f3f5870ef9a23542ef"
Nov 02 12:48:38.102026 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:38.102001    4559 scope.go:122] "RemoveContainer" containerID="34968672bdf04df79e66a09d897e7c6a67902c459dabd4f3f5870ef9a23542ef"
Nov 02 12:48:38.102386 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:48:38.102357    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34968672bdf04df79e66a09d897e7c6a67902c459dabd4f3f5870ef9a23542ef\": container with ID starting with 34968672bdf04df79e66a09d897e7c6a67902c459dabd4f3f5870ef9a23542ef not found: ID does not exist" containerID="34968672bdf04df79e66a09d897e7c6a67902c459dabd4f3f5870ef9a23542ef"
Nov 02 12:48:38.102520 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:38.102390    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34968672bdf04df79e66a09d897e7c6a67902c459dabd4f3f5870ef9a23542ef"} err="failed to get container status \"34968672bdf04df79e66a09d897e7c6a67902c459dabd4f3f5870ef9a23542ef\": rpc error: code = NotFound desc = could not find container \"34968672bdf04df79e66a09d897e7c6a67902c459dabd4f3f5870ef9a23542ef\": container with ID starting with 34968672bdf04df79e66a09d897e7c6a67902c459dabd4f3f5870ef9a23542ef not found: ID does not exist"
Nov 02 12:48:38.146021 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:38.145992    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-7s2lq"]
Nov 02 12:48:38.158597 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:38.158532    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-7s2lq"]
Nov 02 12:48:39.776475 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:39.776446    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-59060a-openshift-default-8666cdf765-hfz77"
Nov 02 12:48:39.893921 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:39.893884    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ac9ccd9-92db-4bc2-8900-772272b4cc80" path="/var/lib/kubelet/pods/0ac9ccd9-92db-4bc2-8900-772272b4cc80/volumes"
Nov 02 12:48:39.913519 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:39.913491    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ed1a149-3a01-4af1-9b01-e059ff121bed-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ed1a149-3a01-4af1-9b01-e059ff121bed-istio-data\") pod \"9ed1a149-3a01-4af1-9b01-e059ff121bed\" (UID: \"9ed1a149-3a01-4af1-9b01-e059ff121bed\") "
Nov 02 12:48:39.913752 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:39.913577    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ed1a149-3a01-4af1-9b01-e059ff121bed-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ed1a149-3a01-4af1-9b01-e059ff121bed-workload-certs\") pod \"9ed1a149-3a01-4af1-9b01-e059ff121bed\" (UID: \"9ed1a149-3a01-4af1-9b01-e059ff121bed\") "
Nov 02 12:48:39.913752 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:39.913624    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ed1a149-3a01-4af1-9b01-e059ff121bed-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ed1a149-3a01-4af1-9b01-e059ff121bed-workload-socket\") pod \"9ed1a149-3a01-4af1-9b01-e059ff121bed\" (UID: \"9ed1a149-3a01-4af1-9b01-e059ff121bed\") "
Nov 02 12:48:39.913752 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:39.913705    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9ed1a149-3a01-4af1-9b01-e059ff121bed-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ed1a149-3a01-4af1-9b01-e059ff121bed-istio-podinfo\") pod \"9ed1a149-3a01-4af1-9b01-e059ff121bed\" (UID: \"9ed1a149-3a01-4af1-9b01-e059ff121bed\") "
Nov 02 12:48:39.913752 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:39.913746    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ed1a149-3a01-4af1-9b01-e059ff121bed-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ed1a149-3a01-4af1-9b01-e059ff121bed-istio-envoy\") pod \"9ed1a149-3a01-4af1-9b01-e059ff121bed\" (UID: \"9ed1a149-3a01-4af1-9b01-e059ff121bed\") "
Nov 02 12:48:39.913969 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:39.913800    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ed1a149-3a01-4af1-9b01-e059ff121bed-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ed1a149-3a01-4af1-9b01-e059ff121bed-credential-socket\") pod \"9ed1a149-3a01-4af1-9b01-e059ff121bed\" (UID: \"9ed1a149-3a01-4af1-9b01-e059ff121bed\") "
Nov 02 12:48:39.913969 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:39.913821    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ed1a149-3a01-4af1-9b01-e059ff121bed-workload-certs" pod "9ed1a149-3a01-4af1-9b01-e059ff121bed" (UID: "9ed1a149-3a01-4af1-9b01-e059ff121bed"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:48:39.913969 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:39.913944    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ed1a149-3a01-4af1-9b01-e059ff121bed-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ed1a149-3a01-4af1-9b01-e059ff121bed-istiod-ca-cert\") pod \"9ed1a149-3a01-4af1-9b01-e059ff121bed\" (UID: \"9ed1a149-3a01-4af1-9b01-e059ff121bed\") "
Nov 02 12:48:39.914122 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:39.913970    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ed1a149-3a01-4af1-9b01-e059ff121bed-workload-socket" pod "9ed1a149-3a01-4af1-9b01-e059ff121bed" (UID: "9ed1a149-3a01-4af1-9b01-e059ff121bed"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:48:39.914122 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:39.913989    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ed1a149-3a01-4af1-9b01-e059ff121bed-istio-token\" (UniqueName: \"kubernetes.io/projected/9ed1a149-3a01-4af1-9b01-e059ff121bed-istio-token\") pod \"9ed1a149-3a01-4af1-9b01-e059ff121bed\" (UID: \"9ed1a149-3a01-4af1-9b01-e059ff121bed\") "
Nov 02 12:48:39.914122 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:39.914102    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ed1a149-3a01-4af1-9b01-e059ff121bed-kube-api-access-tr8md\" (UniqueName: \"kubernetes.io/projected/9ed1a149-3a01-4af1-9b01-e059ff121bed-kube-api-access-tr8md\") pod \"9ed1a149-3a01-4af1-9b01-e059ff121bed\" (UID: \"9ed1a149-3a01-4af1-9b01-e059ff121bed\") "
Nov 02 12:48:39.914277 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:39.914200    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ed1a149-3a01-4af1-9b01-e059ff121bed-credential-socket" pod "9ed1a149-3a01-4af1-9b01-e059ff121bed" (UID: "9ed1a149-3a01-4af1-9b01-e059ff121bed"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:48:39.914277 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:39.914254    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ed1a149-3a01-4af1-9b01-e059ff121bed-istio-data" pod "9ed1a149-3a01-4af1-9b01-e059ff121bed" (UID: "9ed1a149-3a01-4af1-9b01-e059ff121bed"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:48:39.914626 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:39.914457    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ed1a149-3a01-4af1-9b01-e059ff121bed-istiod-ca-cert" pod "9ed1a149-3a01-4af1-9b01-e059ff121bed" (UID: "9ed1a149-3a01-4af1-9b01-e059ff121bed"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:48:39.914812 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:39.914721    4559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ed1a149-3a01-4af1-9b01-e059ff121bed-istiod-ca-cert\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:48:39.914812 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:39.914744    4559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ed1a149-3a01-4af1-9b01-e059ff121bed-istio-data\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:48:39.914812 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:39.914757    4559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ed1a149-3a01-4af1-9b01-e059ff121bed-workload-certs\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:48:39.914812 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:39.914769    4559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ed1a149-3a01-4af1-9b01-e059ff121bed-workload-socket\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:48:39.914812 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:39.914782    4559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ed1a149-3a01-4af1-9b01-e059ff121bed-credential-socket\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:48:39.916871 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:39.916841    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ed1a149-3a01-4af1-9b01-e059ff121bed-istio-token" pod "9ed1a149-3a01-4af1-9b01-e059ff121bed" (UID: "9ed1a149-3a01-4af1-9b01-e059ff121bed"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:48:39.917357 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:39.917331    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9ed1a149-3a01-4af1-9b01-e059ff121bed-istio-podinfo" pod "9ed1a149-3a01-4af1-9b01-e059ff121bed" (UID: "9ed1a149-3a01-4af1-9b01-e059ff121bed"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 12:48:39.917583 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:39.917553    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ed1a149-3a01-4af1-9b01-e059ff121bed-istio-envoy" pod "9ed1a149-3a01-4af1-9b01-e059ff121bed" (UID: "9ed1a149-3a01-4af1-9b01-e059ff121bed"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:48:39.918153 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:39.918130    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ed1a149-3a01-4af1-9b01-e059ff121bed-kube-api-access-tr8md" pod "9ed1a149-3a01-4af1-9b01-e059ff121bed" (UID: "9ed1a149-3a01-4af1-9b01-e059ff121bed"). InnerVolumeSpecName "kube-api-access-tr8md". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:48:40.016532 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:40.016452    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tr8md\" (UniqueName: \"kubernetes.io/projected/9ed1a149-3a01-4af1-9b01-e059ff121bed-kube-api-access-tr8md\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:48:40.016532 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:40.016476    4559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ed1a149-3a01-4af1-9b01-e059ff121bed-istio-podinfo\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:48:40.016532 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:40.016488    4559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ed1a149-3a01-4af1-9b01-e059ff121bed-istio-envoy\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:48:40.016532 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:40.016500    4559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ed1a149-3a01-4af1-9b01-e059ff121bed-istio-token\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:48:40.116578 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:40.116540    4559 generic.go:356] "Generic (PLEG): container finished" podID="9ed1a149-3a01-4af1-9b01-e059ff121bed" containerID="73d0524d7341a61dfeeda451f1a2a72043fc9e3d3f045d0d0fe49a0b6eae043d" exitCode=0
Nov 02 12:48:40.116758 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:40.116581    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-59060a-openshift-default-8666cdf765-hfz77" event={"ID":"9ed1a149-3a01-4af1-9b01-e059ff121bed","Type":"ContainerDied","Data":"73d0524d7341a61dfeeda451f1a2a72043fc9e3d3f045d0d0fe49a0b6eae043d"}
Nov 02 12:48:40.116758 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:40.116618    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-59060a-openshift-default-8666cdf765-hfz77"
Nov 02 12:48:40.116758 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:40.116666    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-59060a-openshift-default-8666cdf765-hfz77" event={"ID":"9ed1a149-3a01-4af1-9b01-e059ff121bed","Type":"ContainerDied","Data":"2616f33b24e8ea4555d9d4444d083d2f499bb0221abb9070afbb38284e9fd340"}
Nov 02 12:48:40.116758 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:40.116700    4559 scope.go:122] "RemoveContainer" containerID="73d0524d7341a61dfeeda451f1a2a72043fc9e3d3f045d0d0fe49a0b6eae043d"
Nov 02 12:48:40.127823 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:40.127807    4559 scope.go:122] "RemoveContainer" containerID="53df5ae504ad7e4d8df77fb7760dd73b569c12cb28a56f5a0cb1bb2f5402e97e"
Nov 02 12:48:40.136959 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:40.136942    4559 scope.go:122] "RemoveContainer" containerID="73d0524d7341a61dfeeda451f1a2a72043fc9e3d3f045d0d0fe49a0b6eae043d"
Nov 02 12:48:40.137203 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:48:40.137185    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73d0524d7341a61dfeeda451f1a2a72043fc9e3d3f045d0d0fe49a0b6eae043d\": container with ID starting with 73d0524d7341a61dfeeda451f1a2a72043fc9e3d3f045d0d0fe49a0b6eae043d not found: ID does not exist" containerID="73d0524d7341a61dfeeda451f1a2a72043fc9e3d3f045d0d0fe49a0b6eae043d"
Nov 02 12:48:40.137253 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:40.137211    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73d0524d7341a61dfeeda451f1a2a72043fc9e3d3f045d0d0fe49a0b6eae043d"} err="failed to get container status \"73d0524d7341a61dfeeda451f1a2a72043fc9e3d3f045d0d0fe49a0b6eae043d\": rpc error: code = NotFound desc = could not find container \"73d0524d7341a61dfeeda451f1a2a72043fc9e3d3f045d0d0fe49a0b6eae043d\": container with ID starting with 73d0524d7341a61dfeeda451f1a2a72043fc9e3d3f045d0d0fe49a0b6eae043d not found: ID does not exist"
Nov 02 12:48:40.137253 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:40.137227    4559 scope.go:122] "RemoveContainer" containerID="53df5ae504ad7e4d8df77fb7760dd73b569c12cb28a56f5a0cb1bb2f5402e97e"
Nov 02 12:48:40.137441 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:48:40.137425    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53df5ae504ad7e4d8df77fb7760dd73b569c12cb28a56f5a0cb1bb2f5402e97e\": container with ID starting with 53df5ae504ad7e4d8df77fb7760dd73b569c12cb28a56f5a0cb1bb2f5402e97e not found: ID does not exist" containerID="53df5ae504ad7e4d8df77fb7760dd73b569c12cb28a56f5a0cb1bb2f5402e97e"
Nov 02 12:48:40.137486 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:40.137443    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53df5ae504ad7e4d8df77fb7760dd73b569c12cb28a56f5a0cb1bb2f5402e97e"} err="failed to get container status \"53df5ae504ad7e4d8df77fb7760dd73b569c12cb28a56f5a0cb1bb2f5402e97e\": rpc error: code = NotFound desc = could not find container \"53df5ae504ad7e4d8df77fb7760dd73b569c12cb28a56f5a0cb1bb2f5402e97e\": container with ID starting with 53df5ae504ad7e4d8df77fb7760dd73b569c12cb28a56f5a0cb1bb2f5402e97e not found: ID does not exist"
Nov 02 12:48:40.171521 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:40.171489    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-59060a-openshift-default-8666cdf765-hfz77"]
Nov 02 12:48:40.184404 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:40.184381    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-59060a-openshift-default-8666cdf765-hfz77"]
Nov 02 12:48:40.874419 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:40.874393    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-b2ec05-openshift-default-5f4c648f59-bldmv"
Nov 02 12:48:40.927904 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:40.927870    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f74d2ef-c796-4c91-b763-7593c16cfe71-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f74d2ef-c796-4c91-b763-7593c16cfe71-istio-envoy\") pod \"0f74d2ef-c796-4c91-b763-7593c16cfe71\" (UID: \"0f74d2ef-c796-4c91-b763-7593c16cfe71\") "
Nov 02 12:48:40.928067 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:40.927992    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f74d2ef-c796-4c91-b763-7593c16cfe71-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f74d2ef-c796-4c91-b763-7593c16cfe71-istio-data\") pod \"0f74d2ef-c796-4c91-b763-7593c16cfe71\" (UID: \"0f74d2ef-c796-4c91-b763-7593c16cfe71\") "
Nov 02 12:48:40.928067 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:40.928030    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f74d2ef-c796-4c91-b763-7593c16cfe71-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f74d2ef-c796-4c91-b763-7593c16cfe71-workload-socket\") pod \"0f74d2ef-c796-4c91-b763-7593c16cfe71\" (UID: \"0f74d2ef-c796-4c91-b763-7593c16cfe71\") "
Nov 02 12:48:40.928140 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:40.928068    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f74d2ef-c796-4c91-b763-7593c16cfe71-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f74d2ef-c796-4c91-b763-7593c16cfe71-credential-socket\") pod \"0f74d2ef-c796-4c91-b763-7593c16cfe71\" (UID: \"0f74d2ef-c796-4c91-b763-7593c16cfe71\") "
Nov 02 12:48:40.928140 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:40.928100    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f74d2ef-c796-4c91-b763-7593c16cfe71-kube-api-access-wwv9f\" (UniqueName: \"kubernetes.io/projected/0f74d2ef-c796-4c91-b763-7593c16cfe71-kube-api-access-wwv9f\") pod \"0f74d2ef-c796-4c91-b763-7593c16cfe71\" (UID: \"0f74d2ef-c796-4c91-b763-7593c16cfe71\") "
Nov 02 12:48:40.928239 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:40.928224    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f74d2ef-c796-4c91-b763-7593c16cfe71-istio-token\" (UniqueName: \"kubernetes.io/projected/0f74d2ef-c796-4c91-b763-7593c16cfe71-istio-token\") pod \"0f74d2ef-c796-4c91-b763-7593c16cfe71\" (UID: \"0f74d2ef-c796-4c91-b763-7593c16cfe71\") "
Nov 02 12:48:40.928312 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:40.928267    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f74d2ef-c796-4c91-b763-7593c16cfe71-istio-data" pod "0f74d2ef-c796-4c91-b763-7593c16cfe71" (UID: "0f74d2ef-c796-4c91-b763-7593c16cfe71"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:48:40.928370 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:40.928289    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0f74d2ef-c796-4c91-b763-7593c16cfe71-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f74d2ef-c796-4c91-b763-7593c16cfe71-istio-podinfo\") pod \"0f74d2ef-c796-4c91-b763-7593c16cfe71\" (UID: \"0f74d2ef-c796-4c91-b763-7593c16cfe71\") "
Nov 02 12:48:40.928370 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:40.928322    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f74d2ef-c796-4c91-b763-7593c16cfe71-workload-socket" pod "0f74d2ef-c796-4c91-b763-7593c16cfe71" (UID: "0f74d2ef-c796-4c91-b763-7593c16cfe71"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:48:40.928476 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:40.928376    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f74d2ef-c796-4c91-b763-7593c16cfe71-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f74d2ef-c796-4c91-b763-7593c16cfe71-workload-certs\") pod \"0f74d2ef-c796-4c91-b763-7593c16cfe71\" (UID: \"0f74d2ef-c796-4c91-b763-7593c16cfe71\") "
Nov 02 12:48:40.928476 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:40.928402    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f74d2ef-c796-4c91-b763-7593c16cfe71-credential-socket" pod "0f74d2ef-c796-4c91-b763-7593c16cfe71" (UID: "0f74d2ef-c796-4c91-b763-7593c16cfe71"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:48:40.928476 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:40.928463    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f74d2ef-c796-4c91-b763-7593c16cfe71-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f74d2ef-c796-4c91-b763-7593c16cfe71-istiod-ca-cert\") pod \"0f74d2ef-c796-4c91-b763-7593c16cfe71\" (UID: \"0f74d2ef-c796-4c91-b763-7593c16cfe71\") "
Nov 02 12:48:40.928679 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:40.928620    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f74d2ef-c796-4c91-b763-7593c16cfe71-workload-certs" pod "0f74d2ef-c796-4c91-b763-7593c16cfe71" (UID: "0f74d2ef-c796-4c91-b763-7593c16cfe71"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:48:40.928870 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:40.928846    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f74d2ef-c796-4c91-b763-7593c16cfe71-istiod-ca-cert" pod "0f74d2ef-c796-4c91-b763-7593c16cfe71" (UID: "0f74d2ef-c796-4c91-b763-7593c16cfe71"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:48:40.929135 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:40.929118    4559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f74d2ef-c796-4c91-b763-7593c16cfe71-istio-data\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:48:40.929214 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:40.929136    4559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f74d2ef-c796-4c91-b763-7593c16cfe71-workload-socket\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:48:40.929214 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:40.929148    4559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f74d2ef-c796-4c91-b763-7593c16cfe71-credential-socket\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:48:40.929214 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:40.929158    4559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f74d2ef-c796-4c91-b763-7593c16cfe71-workload-certs\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:48:40.929214 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:40.929166    4559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f74d2ef-c796-4c91-b763-7593c16cfe71-istiod-ca-cert\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:48:40.930441 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:40.930416    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f74d2ef-c796-4c91-b763-7593c16cfe71-istio-token" pod "0f74d2ef-c796-4c91-b763-7593c16cfe71" (UID: "0f74d2ef-c796-4c91-b763-7593c16cfe71"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:48:40.930441 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:40.930427    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f74d2ef-c796-4c91-b763-7593c16cfe71-kube-api-access-wwv9f" pod "0f74d2ef-c796-4c91-b763-7593c16cfe71" (UID: "0f74d2ef-c796-4c91-b763-7593c16cfe71"). InnerVolumeSpecName "kube-api-access-wwv9f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:48:40.930600 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:40.930581    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0f74d2ef-c796-4c91-b763-7593c16cfe71-istio-podinfo" pod "0f74d2ef-c796-4c91-b763-7593c16cfe71" (UID: "0f74d2ef-c796-4c91-b763-7593c16cfe71"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 12:48:40.930655 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:40.930599    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f74d2ef-c796-4c91-b763-7593c16cfe71-istio-envoy" pod "0f74d2ef-c796-4c91-b763-7593c16cfe71" (UID: "0f74d2ef-c796-4c91-b763-7593c16cfe71"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:48:41.031088 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:41.031056    4559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f74d2ef-c796-4c91-b763-7593c16cfe71-istio-envoy\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:48:41.031226 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:41.031161    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wwv9f\" (UniqueName: \"kubernetes.io/projected/0f74d2ef-c796-4c91-b763-7593c16cfe71-kube-api-access-wwv9f\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:48:41.031319 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:41.031296    4559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f74d2ef-c796-4c91-b763-7593c16cfe71-istio-token\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:48:41.031354 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:41.031326    4559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f74d2ef-c796-4c91-b763-7593c16cfe71-istio-podinfo\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:48:41.124313 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:41.124224    4559 generic.go:356] "Generic (PLEG): container finished" podID="0f74d2ef-c796-4c91-b763-7593c16cfe71" containerID="153e39ff35c98ce2c5fe699e58991b360731c0637f622ead6afed234f01d22b7" exitCode=0
Nov 02 12:48:41.124313 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:41.124266    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-b2ec05-openshift-default-5f4c648f59-bldmv"
Nov 02 12:48:41.124529 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:41.124314    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b2ec05-openshift-default-5f4c648f59-bldmv" event={"ID":"0f74d2ef-c796-4c91-b763-7593c16cfe71","Type":"ContainerDied","Data":"153e39ff35c98ce2c5fe699e58991b360731c0637f622ead6afed234f01d22b7"}
Nov 02 12:48:41.124529 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:41.124371    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b2ec05-openshift-default-5f4c648f59-bldmv" event={"ID":"0f74d2ef-c796-4c91-b763-7593c16cfe71","Type":"ContainerDied","Data":"bca71ad2adc636aeec68fd95e82bb351112d33b4381596fe9e51c64698c99e98"}
Nov 02 12:48:41.124529 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:41.124403    4559 scope.go:122] "RemoveContainer" containerID="153e39ff35c98ce2c5fe699e58991b360731c0637f622ead6afed234f01d22b7"
Nov 02 12:48:41.135482 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:41.135461    4559 scope.go:122] "RemoveContainer" containerID="c0efdec28f98e63e1d763aae8b0b3dfd8090e3c17e7641565e99d42b96efcd88"
Nov 02 12:48:41.145200 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:41.145183    4559 scope.go:122] "RemoveContainer" containerID="153e39ff35c98ce2c5fe699e58991b360731c0637f622ead6afed234f01d22b7"
Nov 02 12:48:41.145479 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:48:41.145457    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"153e39ff35c98ce2c5fe699e58991b360731c0637f622ead6afed234f01d22b7\": container with ID starting with 153e39ff35c98ce2c5fe699e58991b360731c0637f622ead6afed234f01d22b7 not found: ID does not exist" containerID="153e39ff35c98ce2c5fe699e58991b360731c0637f622ead6afed234f01d22b7"
Nov 02 12:48:41.145532 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:41.145488    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"153e39ff35c98ce2c5fe699e58991b360731c0637f622ead6afed234f01d22b7"} err="failed to get container status \"153e39ff35c98ce2c5fe699e58991b360731c0637f622ead6afed234f01d22b7\": rpc error: code = NotFound desc = could not find container \"153e39ff35c98ce2c5fe699e58991b360731c0637f622ead6afed234f01d22b7\": container with ID starting with 153e39ff35c98ce2c5fe699e58991b360731c0637f622ead6afed234f01d22b7 not found: ID does not exist"
Nov 02 12:48:41.145532 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:41.145512    4559 scope.go:122] "RemoveContainer" containerID="c0efdec28f98e63e1d763aae8b0b3dfd8090e3c17e7641565e99d42b96efcd88"
Nov 02 12:48:41.145752 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:48:41.145730    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0efdec28f98e63e1d763aae8b0b3dfd8090e3c17e7641565e99d42b96efcd88\": container with ID starting with c0efdec28f98e63e1d763aae8b0b3dfd8090e3c17e7641565e99d42b96efcd88 not found: ID does not exist" containerID="c0efdec28f98e63e1d763aae8b0b3dfd8090e3c17e7641565e99d42b96efcd88"
Nov 02 12:48:41.145806 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:41.145759    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0efdec28f98e63e1d763aae8b0b3dfd8090e3c17e7641565e99d42b96efcd88"} err="failed to get container status \"c0efdec28f98e63e1d763aae8b0b3dfd8090e3c17e7641565e99d42b96efcd88\": rpc error: code = NotFound desc = could not find container \"c0efdec28f98e63e1d763aae8b0b3dfd8090e3c17e7641565e99d42b96efcd88\": container with ID starting with c0efdec28f98e63e1d763aae8b0b3dfd8090e3c17e7641565e99d42b96efcd88 not found: ID does not exist"
Nov 02 12:48:41.182001 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:41.181972    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-b2ec05-openshift-default-5f4c648f59-bldmv"]
Nov 02 12:48:41.194952 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:41.194930    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-b2ec05-openshift-default-5f4c648f59-bldmv"]
Nov 02 12:48:41.933705 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:41.916198    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f74d2ef-c796-4c91-b763-7593c16cfe71" path="/var/lib/kubelet/pods/0f74d2ef-c796-4c91-b763-7593c16cfe71/volumes"
Nov 02 12:48:41.933705 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:41.917041    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ed1a149-3a01-4af1-9b01-e059ff121bed" path="/var/lib/kubelet/pods/9ed1a149-3a01-4af1-9b01-e059ff121bed/volumes"
Nov 02 12:48:42.183925 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:42.183839    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f7f44a-584b988b97-tzb42"]
Nov 02 12:48:42.219990 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:42.219955    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-f7f44a-bdf9b95cf-2dqpg"]
Nov 02 12:48:45.088326 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:48:45.088292    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqcmj" containerName="extractor"
Nov 02 12:48:47.184334 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:47.184256    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-f7f44a-584b988b97-tzb42" podUID="e5bd0c86-030e-45be-882f-69cde6c34bac" containerName="payload-processing" containerID="cri-o://bf05045d118f0f82364704b2763594f15cfc326c55fef9507b6a4e98b0c9fbe0" gracePeriod=25
Nov 02 12:48:47.220474 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:47.220417    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f7f44a-bdf9b95cf-2dqpg" podUID="84f69d99-5503-471f-95c2-a01256f2a337" containerName="payload-pre-processing" containerID="cri-o://5d4c1eaebbc1e90b236fbbcc6127b6372bd7c0a38da48cd41a29adba35695096" gracePeriod=25
Nov 02 12:48:47.508318 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:47.508285    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-f7f44a-584b988b97-tzb42"
Nov 02 12:48:47.539663 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:47.539616    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f7f44a-bdf9b95cf-2dqpg"
Nov 02 12:48:47.630500 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:47.630467    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84f69d99-5503-471f-95c2-a01256f2a337-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84f69d99-5503-471f-95c2-a01256f2a337-plugins-config-volume\") pod \"84f69d99-5503-471f-95c2-a01256f2a337\" (UID: \"84f69d99-5503-471f-95c2-a01256f2a337\") "
Nov 02 12:48:47.630704 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:47.630510    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5bd0c86-030e-45be-882f-69cde6c34bac-kube-api-access-vp59b\" (UniqueName: \"kubernetes.io/projected/e5bd0c86-030e-45be-882f-69cde6c34bac-kube-api-access-vp59b\") pod \"e5bd0c86-030e-45be-882f-69cde6c34bac\" (UID: \"e5bd0c86-030e-45be-882f-69cde6c34bac\") "
Nov 02 12:48:47.630791 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:47.630709    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84f69d99-5503-471f-95c2-a01256f2a337-kube-api-access-pzrd4\" (UniqueName: \"kubernetes.io/projected/84f69d99-5503-471f-95c2-a01256f2a337-kube-api-access-pzrd4\") pod \"84f69d99-5503-471f-95c2-a01256f2a337\" (UID: \"84f69d99-5503-471f-95c2-a01256f2a337\") "
Nov 02 12:48:47.630912 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:47.630878    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84f69d99-5503-471f-95c2-a01256f2a337-plugins-config-volume" pod "84f69d99-5503-471f-95c2-a01256f2a337" (UID: "84f69d99-5503-471f-95c2-a01256f2a337"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:48:47.630912 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:47.630898    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5bd0c86-030e-45be-882f-69cde6c34bac-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5bd0c86-030e-45be-882f-69cde6c34bac-plugins-config-volume\") pod \"e5bd0c86-030e-45be-882f-69cde6c34bac\" (UID: \"e5bd0c86-030e-45be-882f-69cde6c34bac\") "
Nov 02 12:48:47.631326 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:47.631296    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5bd0c86-030e-45be-882f-69cde6c34bac-plugins-config-volume" pod "e5bd0c86-030e-45be-882f-69cde6c34bac" (UID: "e5bd0c86-030e-45be-882f-69cde6c34bac"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:48:47.631326 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:47.631306    4559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84f69d99-5503-471f-95c2-a01256f2a337-plugins-config-volume\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:48:47.632854 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:47.632832    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5bd0c86-030e-45be-882f-69cde6c34bac-kube-api-access-vp59b" pod "e5bd0c86-030e-45be-882f-69cde6c34bac" (UID: "e5bd0c86-030e-45be-882f-69cde6c34bac"). InnerVolumeSpecName "kube-api-access-vp59b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:48:47.632960 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:47.632862    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84f69d99-5503-471f-95c2-a01256f2a337-kube-api-access-pzrd4" pod "84f69d99-5503-471f-95c2-a01256f2a337" (UID: "84f69d99-5503-471f-95c2-a01256f2a337"). InnerVolumeSpecName "kube-api-access-pzrd4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:48:47.732306 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:47.732226    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pzrd4\" (UniqueName: \"kubernetes.io/projected/84f69d99-5503-471f-95c2-a01256f2a337-kube-api-access-pzrd4\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:48:47.732306 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:47.732253    4559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5bd0c86-030e-45be-882f-69cde6c34bac-plugins-config-volume\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:48:47.732306 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:47.732262    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vp59b\" (UniqueName: \"kubernetes.io/projected/e5bd0c86-030e-45be-882f-69cde6c34bac-kube-api-access-vp59b\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:48:47.750851 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:47.750816    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-97de36-58bfcc8d45-xpkdt"]
Nov 02 12:48:47.755677 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:47.751020    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-97de36-58bfcc8d45-xpkdt" podUID="b68f0680-158b-4a2e-970c-44e2d0604262" containerName="maas-api" containerID="cri-o://f7b6d4d0cda80a464f7a2135db1a04491b0639220c5a3484526a72cb8d6da7b5" gracePeriod=30
Nov 02 12:48:47.947143 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:47.947109    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-97de36-744bd9758d-jhcpn"]
Nov 02 12:48:47.947382 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:47.947356    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-97de36-744bd9758d-jhcpn" podUID="160b5200-de01-4a04-a42a-6504f7bf5685" containerName="payload-processing" containerID="cri-o://87d0fe472fd54350d7f0e63db0115d302569608f9b561904b7b734814f6aad30" gracePeriod=30
Nov 02 12:48:47.987963 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:47.987885    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-97de36-59d99bc75f-hbhc5"]
Nov 02 12:48:47.988356 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:47.988319    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-97de36-59d99bc75f-hbhc5" podUID="d9b2a7c1-69df-4109-b678-147aad2d044e" containerName="payload-pre-processing" containerID="cri-o://f12160d94647216e3b92ab1edaf4b30fd590d9c6b59f20dce904202acd860d67" gracePeriod=30
Nov 02 12:48:48.102150 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:48.102129    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-97de36-58bfcc8d45-xpkdt"
Nov 02 12:48:48.187110 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:48.187075    4559 generic.go:356] "Generic (PLEG): container finished" podID="84f69d99-5503-471f-95c2-a01256f2a337" containerID="5d4c1eaebbc1e90b236fbbcc6127b6372bd7c0a38da48cd41a29adba35695096" exitCode=0
Nov 02 12:48:48.187501 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:48.187124    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f7f44a-bdf9b95cf-2dqpg"
Nov 02 12:48:48.187501 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:48.187184    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f7f44a-bdf9b95cf-2dqpg" event={"ID":"84f69d99-5503-471f-95c2-a01256f2a337","Type":"ContainerDied","Data":"5d4c1eaebbc1e90b236fbbcc6127b6372bd7c0a38da48cd41a29adba35695096"}
Nov 02 12:48:48.187501 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:48.187224    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f7f44a-bdf9b95cf-2dqpg" event={"ID":"84f69d99-5503-471f-95c2-a01256f2a337","Type":"ContainerDied","Data":"7fdcab877b08a834e6d0c4525de1d742cb37a412df1d51176c0ea621cfe5798c"}
Nov 02 12:48:48.187501 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:48.187253    4559 scope.go:122] "RemoveContainer" containerID="5d4c1eaebbc1e90b236fbbcc6127b6372bd7c0a38da48cd41a29adba35695096"
Nov 02 12:48:48.189128 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:48.189102    4559 generic.go:356] "Generic (PLEG): container finished" podID="e5bd0c86-030e-45be-882f-69cde6c34bac" containerID="bf05045d118f0f82364704b2763594f15cfc326c55fef9507b6a4e98b0c9fbe0" exitCode=0
Nov 02 12:48:48.189226 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:48.189173    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-f7f44a-584b988b97-tzb42"
Nov 02 12:48:48.189286 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:48.189242    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-f7f44a-584b988b97-tzb42" event={"ID":"e5bd0c86-030e-45be-882f-69cde6c34bac","Type":"ContainerDied","Data":"bf05045d118f0f82364704b2763594f15cfc326c55fef9507b6a4e98b0c9fbe0"}
Nov 02 12:48:48.189343 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:48.189290    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-f7f44a-584b988b97-tzb42" event={"ID":"e5bd0c86-030e-45be-882f-69cde6c34bac","Type":"ContainerDied","Data":"5e3d2389d8277092c548e755678d91d96fcaf2993adfa52b088880add3788fcc"}
Nov 02 12:48:48.191224 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:48.191186    4559 generic.go:356] "Generic (PLEG): container finished" podID="d9b2a7c1-69df-4109-b678-147aad2d044e" containerID="f12160d94647216e3b92ab1edaf4b30fd590d9c6b59f20dce904202acd860d67" exitCode=0
Nov 02 12:48:48.191348 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:48.191330    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-97de36-59d99bc75f-hbhc5" event={"ID":"d9b2a7c1-69df-4109-b678-147aad2d044e","Type":"ContainerDied","Data":"f12160d94647216e3b92ab1edaf4b30fd590d9c6b59f20dce904202acd860d67"}
Nov 02 12:48:48.192874 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:48.192852    4559 generic.go:356] "Generic (PLEG): container finished" podID="160b5200-de01-4a04-a42a-6504f7bf5685" containerID="87d0fe472fd54350d7f0e63db0115d302569608f9b561904b7b734814f6aad30" exitCode=0
Nov 02 12:48:48.192970 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:48.192955    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-97de36-744bd9758d-jhcpn" event={"ID":"160b5200-de01-4a04-a42a-6504f7bf5685","Type":"ContainerDied","Data":"87d0fe472fd54350d7f0e63db0115d302569608f9b561904b7b734814f6aad30"}
Nov 02 12:48:48.195544 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:48.194463    4559 generic.go:356] "Generic (PLEG): container finished" podID="b68f0680-158b-4a2e-970c-44e2d0604262" containerID="f7b6d4d0cda80a464f7a2135db1a04491b0639220c5a3484526a72cb8d6da7b5" exitCode=0
Nov 02 12:48:48.195544 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:48.194536    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-97de36-58bfcc8d45-xpkdt" event={"ID":"b68f0680-158b-4a2e-970c-44e2d0604262","Type":"ContainerDied","Data":"f7b6d4d0cda80a464f7a2135db1a04491b0639220c5a3484526a72cb8d6da7b5"}
Nov 02 12:48:48.195544 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:48.194573    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-97de36-58bfcc8d45-xpkdt" event={"ID":"b68f0680-158b-4a2e-970c-44e2d0604262","Type":"ContainerDied","Data":"7c1bb3badaf94c64a30f2274a5cab68ea5d7796d4fceb28f0d6c6d41fbe2fd7f"}
Nov 02 12:48:48.195544 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:48.194716    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-97de36-58bfcc8d45-xpkdt"
Nov 02 12:48:48.203316 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:48.203296    4559 scope.go:122] "RemoveContainer" containerID="5d4c1eaebbc1e90b236fbbcc6127b6372bd7c0a38da48cd41a29adba35695096"
Nov 02 12:48:48.206290 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:48:48.206265    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d4c1eaebbc1e90b236fbbcc6127b6372bd7c0a38da48cd41a29adba35695096\": container with ID starting with 5d4c1eaebbc1e90b236fbbcc6127b6372bd7c0a38da48cd41a29adba35695096 not found: ID does not exist" containerID="5d4c1eaebbc1e90b236fbbcc6127b6372bd7c0a38da48cd41a29adba35695096"
Nov 02 12:48:48.206368 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:48.206302    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d4c1eaebbc1e90b236fbbcc6127b6372bd7c0a38da48cd41a29adba35695096"} err="failed to get container status \"5d4c1eaebbc1e90b236fbbcc6127b6372bd7c0a38da48cd41a29adba35695096\": rpc error: code = NotFound desc = could not find container \"5d4c1eaebbc1e90b236fbbcc6127b6372bd7c0a38da48cd41a29adba35695096\": container with ID starting with 5d4c1eaebbc1e90b236fbbcc6127b6372bd7c0a38da48cd41a29adba35695096 not found: ID does not exist"
Nov 02 12:48:48.206368 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:48.206323    4559 scope.go:122] "RemoveContainer" containerID="bf05045d118f0f82364704b2763594f15cfc326c55fef9507b6a4e98b0c9fbe0"
Nov 02 12:48:48.222794 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:48.222758    4559 scope.go:122] "RemoveContainer" containerID="bf05045d118f0f82364704b2763594f15cfc326c55fef9507b6a4e98b0c9fbe0"
Nov 02 12:48:48.223106 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:48:48.223086    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf05045d118f0f82364704b2763594f15cfc326c55fef9507b6a4e98b0c9fbe0\": container with ID starting with bf05045d118f0f82364704b2763594f15cfc326c55fef9507b6a4e98b0c9fbe0 not found: ID does not exist" containerID="bf05045d118f0f82364704b2763594f15cfc326c55fef9507b6a4e98b0c9fbe0"
Nov 02 12:48:48.223186 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:48.223115    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf05045d118f0f82364704b2763594f15cfc326c55fef9507b6a4e98b0c9fbe0"} err="failed to get container status \"bf05045d118f0f82364704b2763594f15cfc326c55fef9507b6a4e98b0c9fbe0\": rpc error: code = NotFound desc = could not find container \"bf05045d118f0f82364704b2763594f15cfc326c55fef9507b6a4e98b0c9fbe0\": container with ID starting with bf05045d118f0f82364704b2763594f15cfc326c55fef9507b6a4e98b0c9fbe0 not found: ID does not exist"
Nov 02 12:48:48.223186 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:48.223131    4559 scope.go:122] "RemoveContainer" containerID="f7b6d4d0cda80a464f7a2135db1a04491b0639220c5a3484526a72cb8d6da7b5"
Nov 02 12:48:48.239047 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:48.237796    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b68f0680-158b-4a2e-970c-44e2d0604262-kube-api-access-cmtnb\" (UniqueName: \"kubernetes.io/projected/b68f0680-158b-4a2e-970c-44e2d0604262-kube-api-access-cmtnb\") pod \"b68f0680-158b-4a2e-970c-44e2d0604262\" (UID: \"b68f0680-158b-4a2e-970c-44e2d0604262\") "
Nov 02 12:48:48.239047 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:48.237926    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b68f0680-158b-4a2e-970c-44e2d0604262-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b68f0680-158b-4a2e-970c-44e2d0604262-metrics-certs\") pod \"b68f0680-158b-4a2e-970c-44e2d0604262\" (UID: \"b68f0680-158b-4a2e-970c-44e2d0604262\") "
Nov 02 12:48:48.239047 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:48.238160    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b68f0680-158b-4a2e-970c-44e2d0604262-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b68f0680-158b-4a2e-970c-44e2d0604262-openshift-service-ca\") pod \"b68f0680-158b-4a2e-970c-44e2d0604262\" (UID: \"b68f0680-158b-4a2e-970c-44e2d0604262\") "
Nov 02 12:48:48.239047 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:48.238204    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b68f0680-158b-4a2e-970c-44e2d0604262-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b68f0680-158b-4a2e-970c-44e2d0604262-maas-api-tls\") pod \"b68f0680-158b-4a2e-970c-44e2d0604262\" (UID: \"b68f0680-158b-4a2e-970c-44e2d0604262\") "
Nov 02 12:48:48.239363 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:48.239340    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b68f0680-158b-4a2e-970c-44e2d0604262-openshift-service-ca" pod "b68f0680-158b-4a2e-970c-44e2d0604262" (UID: "b68f0680-158b-4a2e-970c-44e2d0604262"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:48:48.256169 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:48.256106    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b68f0680-158b-4a2e-970c-44e2d0604262-metrics-certs" pod "b68f0680-158b-4a2e-970c-44e2d0604262" (UID: "b68f0680-158b-4a2e-970c-44e2d0604262"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:48:48.256169 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:48.256130    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b68f0680-158b-4a2e-970c-44e2d0604262-maas-api-tls" pod "b68f0680-158b-4a2e-970c-44e2d0604262" (UID: "b68f0680-158b-4a2e-970c-44e2d0604262"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:48:48.256169 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:48.256142    4559 scope.go:122] "RemoveContainer" containerID="f7b6d4d0cda80a464f7a2135db1a04491b0639220c5a3484526a72cb8d6da7b5"
Nov 02 12:48:48.256169 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:48.256132    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b68f0680-158b-4a2e-970c-44e2d0604262-kube-api-access-cmtnb" pod "b68f0680-158b-4a2e-970c-44e2d0604262" (UID: "b68f0680-158b-4a2e-970c-44e2d0604262"). InnerVolumeSpecName "kube-api-access-cmtnb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:48:48.256566 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:48:48.256513    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7b6d4d0cda80a464f7a2135db1a04491b0639220c5a3484526a72cb8d6da7b5\": container with ID starting with f7b6d4d0cda80a464f7a2135db1a04491b0639220c5a3484526a72cb8d6da7b5 not found: ID does not exist" containerID="f7b6d4d0cda80a464f7a2135db1a04491b0639220c5a3484526a72cb8d6da7b5"
Nov 02 12:48:48.256566 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:48.256543    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7b6d4d0cda80a464f7a2135db1a04491b0639220c5a3484526a72cb8d6da7b5"} err="failed to get container status \"f7b6d4d0cda80a464f7a2135db1a04491b0639220c5a3484526a72cb8d6da7b5\": rpc error: code = NotFound desc = could not find container \"f7b6d4d0cda80a464f7a2135db1a04491b0639220c5a3484526a72cb8d6da7b5\": container with ID starting with f7b6d4d0cda80a464f7a2135db1a04491b0639220c5a3484526a72cb8d6da7b5 not found: ID does not exist"
Nov 02 12:48:48.370664 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:48.349302    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cmtnb\" (UniqueName: \"kubernetes.io/projected/b68f0680-158b-4a2e-970c-44e2d0604262-kube-api-access-cmtnb\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:48:48.370664 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:48.349342    4559 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b68f0680-158b-4a2e-970c-44e2d0604262-metrics-certs\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:48:48.370664 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:48.349357    4559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b68f0680-158b-4a2e-970c-44e2d0604262-openshift-service-ca\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:48:48.370664 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:48.349373    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b68f0680-158b-4a2e-970c-44e2d0604262-maas-api-tls\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:48:48.370664 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:48.350394    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-f7f44a-bdf9b95cf-2dqpg"]
Nov 02 12:48:48.417664 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:48.401659    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-f7f44a-bdf9b95cf-2dqpg"]
Nov 02 12:48:48.417664 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:48.407320    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f7f44a-584b988b97-tzb42"]
Nov 02 12:48:48.417664 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:48.412319    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f7f44a-584b988b97-tzb42"]
Nov 02 12:48:48.509552 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:48.509524    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-97de36-59d99bc75f-hbhc5"
Nov 02 12:48:48.512648 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:48.512614    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-97de36-744bd9758d-jhcpn"
Nov 02 12:48:48.602326 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:48.602290    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-97de36-58bfcc8d45-xpkdt"]
Nov 02 12:48:48.605367 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:48.605343    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-97de36-58bfcc8d45-xpkdt"]
Nov 02 12:48:48.673483 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:48.673443    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/160b5200-de01-4a04-a42a-6504f7bf5685-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/160b5200-de01-4a04-a42a-6504f7bf5685-plugins-config-volume\") pod \"160b5200-de01-4a04-a42a-6504f7bf5685\" (UID: \"160b5200-de01-4a04-a42a-6504f7bf5685\") "
Nov 02 12:48:48.673676 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:48.673556    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9b2a7c1-69df-4109-b678-147aad2d044e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9b2a7c1-69df-4109-b678-147aad2d044e-plugins-config-volume\") pod \"d9b2a7c1-69df-4109-b678-147aad2d044e\" (UID: \"d9b2a7c1-69df-4109-b678-147aad2d044e\") "
Nov 02 12:48:48.673722 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:48.673691    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9b2a7c1-69df-4109-b678-147aad2d044e-kube-api-access-f4qpb\" (UniqueName: \"kubernetes.io/projected/d9b2a7c1-69df-4109-b678-147aad2d044e-kube-api-access-f4qpb\") pod \"d9b2a7c1-69df-4109-b678-147aad2d044e\" (UID: \"d9b2a7c1-69df-4109-b678-147aad2d044e\") "
Nov 02 12:48:48.673873 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:48.673857    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/160b5200-de01-4a04-a42a-6504f7bf5685-kube-api-access-tswgg\" (UniqueName: \"kubernetes.io/projected/160b5200-de01-4a04-a42a-6504f7bf5685-kube-api-access-tswgg\") pod \"160b5200-de01-4a04-a42a-6504f7bf5685\" (UID: \"160b5200-de01-4a04-a42a-6504f7bf5685\") "
Nov 02 12:48:48.699966 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:48.699873    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9b2a7c1-69df-4109-b678-147aad2d044e-plugins-config-volume" pod "d9b2a7c1-69df-4109-b678-147aad2d044e" (UID: "d9b2a7c1-69df-4109-b678-147aad2d044e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:48:48.705375 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:48.701859    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/160b5200-de01-4a04-a42a-6504f7bf5685-plugins-config-volume" pod "160b5200-de01-4a04-a42a-6504f7bf5685" (UID: "160b5200-de01-4a04-a42a-6504f7bf5685"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:48:48.705960 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:48.705928    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9b2a7c1-69df-4109-b678-147aad2d044e-kube-api-access-f4qpb" pod "d9b2a7c1-69df-4109-b678-147aad2d044e" (UID: "d9b2a7c1-69df-4109-b678-147aad2d044e"). InnerVolumeSpecName "kube-api-access-f4qpb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:48:48.722899 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:48.722858    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/160b5200-de01-4a04-a42a-6504f7bf5685-kube-api-access-tswgg" pod "160b5200-de01-4a04-a42a-6504f7bf5685" (UID: "160b5200-de01-4a04-a42a-6504f7bf5685"). InnerVolumeSpecName "kube-api-access-tswgg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:48:48.775402 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:48.775319    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tswgg\" (UniqueName: \"kubernetes.io/projected/160b5200-de01-4a04-a42a-6504f7bf5685-kube-api-access-tswgg\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:48:48.775402 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:48.775350    4559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/160b5200-de01-4a04-a42a-6504f7bf5685-plugins-config-volume\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:48:48.775402 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:48.775360    4559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9b2a7c1-69df-4109-b678-147aad2d044e-plugins-config-volume\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:48:48.775402 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:48.775369    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f4qpb\" (UniqueName: \"kubernetes.io/projected/d9b2a7c1-69df-4109-b678-147aad2d044e-kube-api-access-f4qpb\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:48:49.204311 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:49.204280    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-97de36-59d99bc75f-hbhc5"
Nov 02 12:48:49.204311 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:49.204299    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-97de36-59d99bc75f-hbhc5" event={"ID":"d9b2a7c1-69df-4109-b678-147aad2d044e","Type":"ContainerDied","Data":"6be106cc9aae7655637c95c6587260495df791135ba60804b40c41fc82e91f6a"}
Nov 02 12:48:49.204815 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:49.204361    4559 scope.go:122] "RemoveContainer" containerID="f12160d94647216e3b92ab1edaf4b30fd590d9c6b59f20dce904202acd860d67"
Nov 02 12:48:49.205980 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:49.205960    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-97de36-744bd9758d-jhcpn"
Nov 02 12:48:49.206212 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:49.205989    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-97de36-744bd9758d-jhcpn" event={"ID":"160b5200-de01-4a04-a42a-6504f7bf5685","Type":"ContainerDied","Data":"80d50a2d8da3f70d89e1c1439cd550e4e20246bfe55db1a0c0bf4b07033ed071"}
Nov 02 12:48:49.229238 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:49.220990    4559 scope.go:122] "RemoveContainer" containerID="87d0fe472fd54350d7f0e63db0115d302569608f9b561904b7b734814f6aad30"
Nov 02 12:48:49.275716 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:49.275669    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-97de36-744bd9758d-jhcpn"]
Nov 02 12:48:49.304204 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:49.304162    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-97de36-744bd9758d-jhcpn"]
Nov 02 12:48:49.369876 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:49.369844    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-97de36-59d99bc75f-hbhc5"]
Nov 02 12:48:49.383695 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:49.383662    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-97de36-59d99bc75f-hbhc5"]
Nov 02 12:48:49.894329 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:49.894294    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="160b5200-de01-4a04-a42a-6504f7bf5685" path="/var/lib/kubelet/pods/160b5200-de01-4a04-a42a-6504f7bf5685/volumes"
Nov 02 12:48:49.894758 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:49.894743    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84f69d99-5503-471f-95c2-a01256f2a337" path="/var/lib/kubelet/pods/84f69d99-5503-471f-95c2-a01256f2a337/volumes"
Nov 02 12:48:49.895160 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:49.895144    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b68f0680-158b-4a2e-970c-44e2d0604262" path="/var/lib/kubelet/pods/b68f0680-158b-4a2e-970c-44e2d0604262/volumes"
Nov 02 12:48:49.895922 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:49.895839    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9b2a7c1-69df-4109-b678-147aad2d044e" path="/var/lib/kubelet/pods/d9b2a7c1-69df-4109-b678-147aad2d044e/volumes"
Nov 02 12:48:49.896381 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:49.896364    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5bd0c86-030e-45be-882f-69cde6c34bac" path="/var/lib/kubelet/pods/e5bd0c86-030e-45be-882f-69cde6c34bac/volumes"
Nov 02 12:48:50.086279 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:50.086241    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-078fd3-84495865c7-mqwt7"]
Nov 02 12:48:50.086495 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:50.086469    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-078fd3-84495865c7-mqwt7" podUID="fc3bedf6-b508-4297-9fe5-0cd87fe6dd1a" containerName="maas-api" containerID="cri-o://fb099379d1c403a597fd9221ab5636b49872a34339d197dadb4bb19a44f2a096" gracePeriod=30
Nov 02 12:48:50.221891 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:50.221862    4559 generic.go:356] "Generic (PLEG): container finished" podID="fc3bedf6-b508-4297-9fe5-0cd87fe6dd1a" containerID="fb099379d1c403a597fd9221ab5636b49872a34339d197dadb4bb19a44f2a096" exitCode=0
Nov 02 12:48:50.222361 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:50.221947    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-078fd3-84495865c7-mqwt7" event={"ID":"fc3bedf6-b508-4297-9fe5-0cd87fe6dd1a","Type":"ContainerDied","Data":"fb099379d1c403a597fd9221ab5636b49872a34339d197dadb4bb19a44f2a096"}
Nov 02 12:48:50.251285 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:50.251255    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-078fd3-6ff794864d-8sxnf"]
Nov 02 12:48:50.251477 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:50.251453    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-078fd3-6ff794864d-8sxnf" podUID="fdb54d38-ce90-43af-9ba9-da98f13d788a" containerName="payload-processing" containerID="cri-o://86a042f51b922205612de60dcee71cbbb75bfedc0f875e4c0534877f742f6102" gracePeriod=30
Nov 02 12:48:50.290956 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:50.281221    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-078fd3-7b66f65b99-7cclg"]
Nov 02 12:48:50.290956 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:50.281480    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-078fd3-7b66f65b99-7cclg" podUID="daae67e4-da6e-4eb7-b044-8e795f3b3a29" containerName="payload-pre-processing" containerID="cri-o://661fc7bb28f4703f0c7e60c9427a1cf8d7af837fa6aeab3dd419f713cea87371" gracePeriod=30
Nov 02 12:48:50.388317 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:50.388277    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-97de36-openshift-default-77899d747c-cq4bc"]
Nov 02 12:48:50.388572 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:50.388542    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-97de36-openshift-default-77899d747c-cq4bc" podUID="5b55da6b-89cb-46d6-a7a0-3a09e7d7d17e" containerName="istio-proxy" containerID="cri-o://0907660c6d6e2964677ccebb59dbaa23d4b900e33b7d413aa483136fa1e4a4d3" gracePeriod=30
Nov 02 12:48:50.447606 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:50.447575    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-078fd3-84495865c7-mqwt7"
Nov 02 12:48:50.588746 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:50.588719    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-078fd3-6ff794864d-8sxnf"
Nov 02 12:48:50.600444 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:50.600416    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fc3bedf6-b508-4297-9fe5-0cd87fe6dd1a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc3bedf6-b508-4297-9fe5-0cd87fe6dd1a-maas-api-tls\") pod \"fc3bedf6-b508-4297-9fe5-0cd87fe6dd1a\" (UID: \"fc3bedf6-b508-4297-9fe5-0cd87fe6dd1a\") "
Nov 02 12:48:50.600578 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:50.600560    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc3bedf6-b508-4297-9fe5-0cd87fe6dd1a-kube-api-access-8rttf\" (UniqueName: \"kubernetes.io/projected/fc3bedf6-b508-4297-9fe5-0cd87fe6dd1a-kube-api-access-8rttf\") pod \"fc3bedf6-b508-4297-9fe5-0cd87fe6dd1a\" (UID: \"fc3bedf6-b508-4297-9fe5-0cd87fe6dd1a\") "
Nov 02 12:48:50.600680 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:50.600599    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fc3bedf6-b508-4297-9fe5-0cd87fe6dd1a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc3bedf6-b508-4297-9fe5-0cd87fe6dd1a-metrics-certs\") pod \"fc3bedf6-b508-4297-9fe5-0cd87fe6dd1a\" (UID: \"fc3bedf6-b508-4297-9fe5-0cd87fe6dd1a\") "
Nov 02 12:48:50.600736 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:50.600715    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc3bedf6-b508-4297-9fe5-0cd87fe6dd1a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc3bedf6-b508-4297-9fe5-0cd87fe6dd1a-openshift-service-ca\") pod \"fc3bedf6-b508-4297-9fe5-0cd87fe6dd1a\" (UID: \"fc3bedf6-b508-4297-9fe5-0cd87fe6dd1a\") "
Nov 02 12:48:50.601296 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:50.601267    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc3bedf6-b508-4297-9fe5-0cd87fe6dd1a-openshift-service-ca" pod "fc3bedf6-b508-4297-9fe5-0cd87fe6dd1a" (UID: "fc3bedf6-b508-4297-9fe5-0cd87fe6dd1a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:48:50.601449 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:50.601417    4559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc3bedf6-b508-4297-9fe5-0cd87fe6dd1a-openshift-service-ca\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:48:50.602814 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:50.602787    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc3bedf6-b508-4297-9fe5-0cd87fe6dd1a-kube-api-access-8rttf" pod "fc3bedf6-b508-4297-9fe5-0cd87fe6dd1a" (UID: "fc3bedf6-b508-4297-9fe5-0cd87fe6dd1a"). InnerVolumeSpecName "kube-api-access-8rttf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:48:50.602997 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:50.602978    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc3bedf6-b508-4297-9fe5-0cd87fe6dd1a-metrics-certs" pod "fc3bedf6-b508-4297-9fe5-0cd87fe6dd1a" (UID: "fc3bedf6-b508-4297-9fe5-0cd87fe6dd1a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:48:50.603079 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:50.603038    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc3bedf6-b508-4297-9fe5-0cd87fe6dd1a-maas-api-tls" pod "fc3bedf6-b508-4297-9fe5-0cd87fe6dd1a" (UID: "fc3bedf6-b508-4297-9fe5-0cd87fe6dd1a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:48:50.652664 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:50.643901    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-078fd3-7b66f65b99-7cclg"
Nov 02 12:48:50.799713 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:50.783977    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/daae67e4-da6e-4eb7-b044-8e795f3b3a29-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/daae67e4-da6e-4eb7-b044-8e795f3b3a29-plugins-config-volume\") pod \"daae67e4-da6e-4eb7-b044-8e795f3b3a29\" (UID: \"daae67e4-da6e-4eb7-b044-8e795f3b3a29\") "
Nov 02 12:48:50.799713 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:50.784070    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fdb54d38-ce90-43af-9ba9-da98f13d788a-kube-api-access-chr2x\" (UniqueName: \"kubernetes.io/projected/fdb54d38-ce90-43af-9ba9-da98f13d788a-kube-api-access-chr2x\") pod \"fdb54d38-ce90-43af-9ba9-da98f13d788a\" (UID: \"fdb54d38-ce90-43af-9ba9-da98f13d788a\") "
Nov 02 12:48:50.799713 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:50.784231    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/daae67e4-da6e-4eb7-b044-8e795f3b3a29-kube-api-access-dltqs\" (UniqueName: \"kubernetes.io/projected/daae67e4-da6e-4eb7-b044-8e795f3b3a29-kube-api-access-dltqs\") pod \"daae67e4-da6e-4eb7-b044-8e795f3b3a29\" (UID: \"daae67e4-da6e-4eb7-b044-8e795f3b3a29\") "
Nov 02 12:48:50.799713 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:50.784325    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fdb54d38-ce90-43af-9ba9-da98f13d788a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fdb54d38-ce90-43af-9ba9-da98f13d788a-plugins-config-volume\") pod \"fdb54d38-ce90-43af-9ba9-da98f13d788a\" (UID: \"fdb54d38-ce90-43af-9ba9-da98f13d788a\") "
Nov 02 12:48:50.799713 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:50.784878    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc3bedf6-b508-4297-9fe5-0cd87fe6dd1a-maas-api-tls\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:48:50.799713 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:50.784899    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8rttf\" (UniqueName: \"kubernetes.io/projected/fc3bedf6-b508-4297-9fe5-0cd87fe6dd1a-kube-api-access-8rttf\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:48:50.799713 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:50.784933    4559 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc3bedf6-b508-4297-9fe5-0cd87fe6dd1a-metrics-certs\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:48:50.799713 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:50.785329    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdb54d38-ce90-43af-9ba9-da98f13d788a-plugins-config-volume" pod "fdb54d38-ce90-43af-9ba9-da98f13d788a" (UID: "fdb54d38-ce90-43af-9ba9-da98f13d788a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:48:50.799713 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:50.788469    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/daae67e4-da6e-4eb7-b044-8e795f3b3a29-kube-api-access-dltqs" pod "daae67e4-da6e-4eb7-b044-8e795f3b3a29" (UID: "daae67e4-da6e-4eb7-b044-8e795f3b3a29"). InnerVolumeSpecName "kube-api-access-dltqs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:48:50.799713 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:50.788577    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/daae67e4-da6e-4eb7-b044-8e795f3b3a29-plugins-config-volume" pod "daae67e4-da6e-4eb7-b044-8e795f3b3a29" (UID: "daae67e4-da6e-4eb7-b044-8e795f3b3a29"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:48:50.829165 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:50.828664    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdb54d38-ce90-43af-9ba9-da98f13d788a-kube-api-access-chr2x" pod "fdb54d38-ce90-43af-9ba9-da98f13d788a" (UID: "fdb54d38-ce90-43af-9ba9-da98f13d788a"). InnerVolumeSpecName "kube-api-access-chr2x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:48:50.886620 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:50.886587    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-chr2x\" (UniqueName: \"kubernetes.io/projected/fdb54d38-ce90-43af-9ba9-da98f13d788a-kube-api-access-chr2x\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:48:50.886620 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:50.886613    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dltqs\" (UniqueName: \"kubernetes.io/projected/daae67e4-da6e-4eb7-b044-8e795f3b3a29-kube-api-access-dltqs\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:48:50.886620 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:50.886623    4559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fdb54d38-ce90-43af-9ba9-da98f13d788a-plugins-config-volume\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:48:50.886878 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:50.886632    4559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/daae67e4-da6e-4eb7-b044-8e795f3b3a29-plugins-config-volume\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:48:51.232002 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:51.231966    4559 generic.go:356] "Generic (PLEG): container finished" podID="daae67e4-da6e-4eb7-b044-8e795f3b3a29" containerID="661fc7bb28f4703f0c7e60c9427a1cf8d7af837fa6aeab3dd419f713cea87371" exitCode=0
Nov 02 12:48:51.232459 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:51.232009    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-078fd3-7b66f65b99-7cclg"
Nov 02 12:48:51.232459 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:51.232046    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-078fd3-7b66f65b99-7cclg" event={"ID":"daae67e4-da6e-4eb7-b044-8e795f3b3a29","Type":"ContainerDied","Data":"661fc7bb28f4703f0c7e60c9427a1cf8d7af837fa6aeab3dd419f713cea87371"}
Nov 02 12:48:51.232459 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:51.232101    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-078fd3-7b66f65b99-7cclg" event={"ID":"daae67e4-da6e-4eb7-b044-8e795f3b3a29","Type":"ContainerDied","Data":"98d353529eae44951f56953d6a795ac056d831aa9097d8f0fb3ae4e2526814b8"}
Nov 02 12:48:51.232459 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:51.232127    4559 scope.go:122] "RemoveContainer" containerID="661fc7bb28f4703f0c7e60c9427a1cf8d7af837fa6aeab3dd419f713cea87371"
Nov 02 12:48:51.233785 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:51.233761    4559 generic.go:356] "Generic (PLEG): container finished" podID="fdb54d38-ce90-43af-9ba9-da98f13d788a" containerID="86a042f51b922205612de60dcee71cbbb75bfedc0f875e4c0534877f742f6102" exitCode=0
Nov 02 12:48:51.233925 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:51.233815    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-078fd3-6ff794864d-8sxnf"
Nov 02 12:48:51.233925 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:51.233832    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-078fd3-6ff794864d-8sxnf" event={"ID":"fdb54d38-ce90-43af-9ba9-da98f13d788a","Type":"ContainerDied","Data":"86a042f51b922205612de60dcee71cbbb75bfedc0f875e4c0534877f742f6102"}
Nov 02 12:48:51.233925 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:51.233870    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-078fd3-6ff794864d-8sxnf" event={"ID":"fdb54d38-ce90-43af-9ba9-da98f13d788a","Type":"ContainerDied","Data":"9fde782efa1dc7ed325cc5270fabc462cc35fe9438d7ccbb0c380343fc6ae929"}
Nov 02 12:48:51.235348 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:51.235323    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-078fd3-84495865c7-mqwt7" event={"ID":"fc3bedf6-b508-4297-9fe5-0cd87fe6dd1a","Type":"ContainerDied","Data":"4b59f29e45e9e684942295b8bc228c0ac1a9fb894c1fc54dc92af9729821f1d5"}
Nov 02 12:48:51.235438 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:51.235339    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-078fd3-84495865c7-mqwt7"
Nov 02 12:48:51.244608 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:51.244564    4559 scope.go:122] "RemoveContainer" containerID="661fc7bb28f4703f0c7e60c9427a1cf8d7af837fa6aeab3dd419f713cea87371"
Nov 02 12:48:51.244947 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:48:51.244919    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"661fc7bb28f4703f0c7e60c9427a1cf8d7af837fa6aeab3dd419f713cea87371\": container with ID starting with 661fc7bb28f4703f0c7e60c9427a1cf8d7af837fa6aeab3dd419f713cea87371 not found: ID does not exist" containerID="661fc7bb28f4703f0c7e60c9427a1cf8d7af837fa6aeab3dd419f713cea87371"
Nov 02 12:48:51.245026 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:51.244958    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"661fc7bb28f4703f0c7e60c9427a1cf8d7af837fa6aeab3dd419f713cea87371"} err="failed to get container status \"661fc7bb28f4703f0c7e60c9427a1cf8d7af837fa6aeab3dd419f713cea87371\": rpc error: code = NotFound desc = could not find container \"661fc7bb28f4703f0c7e60c9427a1cf8d7af837fa6aeab3dd419f713cea87371\": container with ID starting with 661fc7bb28f4703f0c7e60c9427a1cf8d7af837fa6aeab3dd419f713cea87371 not found: ID does not exist"
Nov 02 12:48:51.245026 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:51.244982    4559 scope.go:122] "RemoveContainer" containerID="86a042f51b922205612de60dcee71cbbb75bfedc0f875e4c0534877f742f6102"
Nov 02 12:48:51.264868 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:51.264676    4559 scope.go:122] "RemoveContainer" containerID="86a042f51b922205612de60dcee71cbbb75bfedc0f875e4c0534877f742f6102"
Nov 02 12:48:51.265079 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:48:51.265058    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86a042f51b922205612de60dcee71cbbb75bfedc0f875e4c0534877f742f6102\": container with ID starting with 86a042f51b922205612de60dcee71cbbb75bfedc0f875e4c0534877f742f6102 not found: ID does not exist" containerID="86a042f51b922205612de60dcee71cbbb75bfedc0f875e4c0534877f742f6102"
Nov 02 12:48:51.265169 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:51.265087    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86a042f51b922205612de60dcee71cbbb75bfedc0f875e4c0534877f742f6102"} err="failed to get container status \"86a042f51b922205612de60dcee71cbbb75bfedc0f875e4c0534877f742f6102\": rpc error: code = NotFound desc = could not find container \"86a042f51b922205612de60dcee71cbbb75bfedc0f875e4c0534877f742f6102\": container with ID starting with 86a042f51b922205612de60dcee71cbbb75bfedc0f875e4c0534877f742f6102 not found: ID does not exist"
Nov 02 12:48:51.265169 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:51.265106    4559 scope.go:122] "RemoveContainer" containerID="fb099379d1c403a597fd9221ab5636b49872a34339d197dadb4bb19a44f2a096"
Nov 02 12:48:51.287142 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:51.287118    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-078fd3-6ff794864d-8sxnf"]
Nov 02 12:48:51.299361 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:51.299335    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-078fd3-6ff794864d-8sxnf"]
Nov 02 12:48:51.334098 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:51.334074    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-078fd3-7b66f65b99-7cclg"]
Nov 02 12:48:51.339089 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:51.339069    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-078fd3-7b66f65b99-7cclg"]
Nov 02 12:48:51.373120 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:51.373092    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-078fd3-84495865c7-mqwt7"]
Nov 02 12:48:51.377915 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:51.377892    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-078fd3-84495865c7-mqwt7"]
Nov 02 12:48:51.894077 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:51.894044    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="daae67e4-da6e-4eb7-b044-8e795f3b3a29" path="/var/lib/kubelet/pods/daae67e4-da6e-4eb7-b044-8e795f3b3a29/volumes"
Nov 02 12:48:51.894500 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:51.894486    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc3bedf6-b508-4297-9fe5-0cd87fe6dd1a" path="/var/lib/kubelet/pods/fc3bedf6-b508-4297-9fe5-0cd87fe6dd1a/volumes"
Nov 02 12:48:51.894910 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:51.894895    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fdb54d38-ce90-43af-9ba9-da98f13d788a" path="/var/lib/kubelet/pods/fdb54d38-ce90-43af-9ba9-da98f13d788a/volumes"
Nov 02 12:48:52.368400 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:52.368365    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-b271f0-openshift-default-5544549cdf-q9c6j"]
Nov 02 12:48:52.368995 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:52.368968    4559 cpu_manager.go:414] "containerMap: removing container" podUID="0f74d2ef-c796-4c91-b763-7593c16cfe71" containerName="istio-proxy"
Nov 02 12:48:52.368995 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:52.368982    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f74d2ef-c796-4c91-b763-7593c16cfe71" containerName="istio-proxy"
Nov 02 12:48:52.368995 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:52.368997    4559 cpu_manager.go:414] "containerMap: removing container" podUID="e5bd0c86-030e-45be-882f-69cde6c34bac" containerName="payload-processing"
Nov 02 12:48:52.369131 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:52.369002    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5bd0c86-030e-45be-882f-69cde6c34bac" containerName="payload-processing"
Nov 02 12:48:52.369131 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:52.369012    4559 cpu_manager.go:414] "containerMap: removing container" podUID="daae67e4-da6e-4eb7-b044-8e795f3b3a29" containerName="payload-pre-processing"
Nov 02 12:48:52.369131 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:52.369017    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="daae67e4-da6e-4eb7-b044-8e795f3b3a29" containerName="payload-pre-processing"
Nov 02 12:48:52.369131 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:52.369027    4559 cpu_manager.go:414] "containerMap: removing container" podUID="fc3bedf6-b508-4297-9fe5-0cd87fe6dd1a" containerName="maas-api"
Nov 02 12:48:52.369131 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:52.369032    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc3bedf6-b508-4297-9fe5-0cd87fe6dd1a" containerName="maas-api"
Nov 02 12:48:52.369131 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:52.369044    4559 cpu_manager.go:414] "containerMap: removing container" podUID="0ac9ccd9-92db-4bc2-8900-772272b4cc80" containerName="istio-proxy"
Nov 02 12:48:52.369131 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:52.369049    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ac9ccd9-92db-4bc2-8900-772272b4cc80" containerName="istio-proxy"
Nov 02 12:48:52.369131 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:52.369055    4559 cpu_manager.go:414] "containerMap: removing container" podUID="fdb54d38-ce90-43af-9ba9-da98f13d788a" containerName="payload-processing"
Nov 02 12:48:52.369131 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:52.369060    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fdb54d38-ce90-43af-9ba9-da98f13d788a" containerName="payload-processing"
Nov 02 12:48:52.369131 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:52.369076    4559 cpu_manager.go:414] "containerMap: removing container" podUID="2265e1d5-bae9-495b-b51d-b3e638c128ea" containerName="maas-api"
Nov 02 12:48:52.369131 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:52.369080    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2265e1d5-bae9-495b-b51d-b3e638c128ea" containerName="maas-api"
Nov 02 12:48:52.369131 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:52.369087    4559 cpu_manager.go:414] "containerMap: removing container" podUID="84f69d99-5503-471f-95c2-a01256f2a337" containerName="payload-pre-processing"
Nov 02 12:48:52.369131 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:52.369091    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84f69d99-5503-471f-95c2-a01256f2a337" containerName="payload-pre-processing"
Nov 02 12:48:52.369131 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:52.369098    4559 cpu_manager.go:414] "containerMap: removing container" podUID="d3f2ed27-59d4-4eba-933a-bea46a18d385" containerName="payload-processing"
Nov 02 12:48:52.369131 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:52.369102    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3f2ed27-59d4-4eba-933a-bea46a18d385" containerName="payload-processing"
Nov 02 12:48:52.369131 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:52.369110    4559 cpu_manager.go:414] "containerMap: removing container" podUID="b68f0680-158b-4a2e-970c-44e2d0604262" containerName="maas-api"
Nov 02 12:48:52.369131 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:52.369114    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b68f0680-158b-4a2e-970c-44e2d0604262" containerName="maas-api"
Nov 02 12:48:52.369131 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:52.369120    4559 cpu_manager.go:414] "containerMap: removing container" podUID="9ed1a149-3a01-4af1-9b01-e059ff121bed" containerName="istio-proxy"
Nov 02 12:48:52.369131 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:52.369125    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ed1a149-3a01-4af1-9b01-e059ff121bed" containerName="istio-proxy"
Nov 02 12:48:52.369131 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:52.369131    4559 cpu_manager.go:414] "containerMap: removing container" podUID="160b5200-de01-4a04-a42a-6504f7bf5685" containerName="payload-processing"
Nov 02 12:48:52.369131 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:52.369136    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="160b5200-de01-4a04-a42a-6504f7bf5685" containerName="payload-processing"
Nov 02 12:48:52.369131 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:52.369142    4559 cpu_manager.go:414] "containerMap: removing container" podUID="9ed1a149-3a01-4af1-9b01-e059ff121bed" containerName="istio-proxy"
Nov 02 12:48:52.370008 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:52.369147    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ed1a149-3a01-4af1-9b01-e059ff121bed" containerName="istio-proxy"
Nov 02 12:48:52.370008 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:52.369154    4559 cpu_manager.go:414] "containerMap: removing container" podUID="00c5fac7-35bf-426e-b9f2-d1900ac09c83" containerName="maas-api"
Nov 02 12:48:52.370008 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:52.369159    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00c5fac7-35bf-426e-b9f2-d1900ac09c83" containerName="maas-api"
Nov 02 12:48:52.370008 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:52.369175    4559 cpu_manager.go:414] "containerMap: removing container" podUID="d9b2a7c1-69df-4109-b678-147aad2d044e" containerName="payload-pre-processing"
Nov 02 12:48:52.370008 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:52.369179    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9b2a7c1-69df-4109-b678-147aad2d044e" containerName="payload-pre-processing"
Nov 02 12:48:52.370008 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:52.369261    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="2265e1d5-bae9-495b-b51d-b3e638c128ea" containerName="maas-api"
Nov 02 12:48:52.370008 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:52.369268    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="84f69d99-5503-471f-95c2-a01256f2a337" containerName="payload-pre-processing"
Nov 02 12:48:52.370008 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:52.369274    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3f2ed27-59d4-4eba-933a-bea46a18d385" containerName="payload-processing"
Nov 02 12:48:52.370008 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:52.369280    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="b68f0680-158b-4a2e-970c-44e2d0604262" containerName="maas-api"
Nov 02 12:48:52.370008 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:52.369285    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ed1a149-3a01-4af1-9b01-e059ff121bed" containerName="istio-proxy"
Nov 02 12:48:52.370008 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:52.369290    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="160b5200-de01-4a04-a42a-6504f7bf5685" containerName="payload-processing"
Nov 02 12:48:52.370008 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:52.369295    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ed1a149-3a01-4af1-9b01-e059ff121bed" containerName="istio-proxy"
Nov 02 12:48:52.370008 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:52.369300    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="00c5fac7-35bf-426e-b9f2-d1900ac09c83" containerName="maas-api"
Nov 02 12:48:52.370008 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:52.369308    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f74d2ef-c796-4c91-b763-7593c16cfe71" containerName="istio-proxy"
Nov 02 12:48:52.370008 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:52.369314    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9b2a7c1-69df-4109-b678-147aad2d044e" containerName="payload-pre-processing"
Nov 02 12:48:52.370008 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:52.369319    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f74d2ef-c796-4c91-b763-7593c16cfe71" containerName="istio-proxy"
Nov 02 12:48:52.370008 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:52.369324    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5bd0c86-030e-45be-882f-69cde6c34bac" containerName="payload-processing"
Nov 02 12:48:52.370008 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:52.369329    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="daae67e4-da6e-4eb7-b044-8e795f3b3a29" containerName="payload-pre-processing"
Nov 02 12:48:52.370008 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:52.369335    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc3bedf6-b508-4297-9fe5-0cd87fe6dd1a" containerName="maas-api"
Nov 02 12:48:52.370008 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:52.369341    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ac9ccd9-92db-4bc2-8900-772272b4cc80" containerName="istio-proxy"
Nov 02 12:48:52.370008 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:52.369346    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="fdb54d38-ce90-43af-9ba9-da98f13d788a" containerName="payload-processing"
Nov 02 12:48:52.374664 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:52.374627    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-b271f0-openshift-default-5544549cdf-q9c6j"
Nov 02 12:48:52.393197 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:52.393176    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-b271f0-openshift-default-dockercfg-9kb9p\""
Nov 02 12:48:52.409071 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:52.409040    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-b271f0-openshift-default-5544549cdf-q9c6j"]
Nov 02 12:48:52.415204 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:52.415174    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htbqv\" (UniqueName: \"kubernetes.io/projected/e5435e4b-b8a8-45b9-a90a-bca4b5a65506-kube-api-access-htbqv\") pod \"e2e-praxis-a-b271f0-openshift-default-5544549cdf-q9c6j\" (UID: \"e5435e4b-b8a8-45b9-a90a-bca4b5a65506\") " pod="openshift-ingress/e2e-praxis-a-b271f0-openshift-default-5544549cdf-q9c6j"
Nov 02 12:48:52.415342 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:52.415232    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5435e4b-b8a8-45b9-a90a-bca4b5a65506-istio-envoy\") pod \"e2e-praxis-a-b271f0-openshift-default-5544549cdf-q9c6j\" (UID: \"e5435e4b-b8a8-45b9-a90a-bca4b5a65506\") " pod="openshift-ingress/e2e-praxis-a-b271f0-openshift-default-5544549cdf-q9c6j"
Nov 02 12:48:52.415420 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:52.415350    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5435e4b-b8a8-45b9-a90a-bca4b5a65506-istio-podinfo\") pod \"e2e-praxis-a-b271f0-openshift-default-5544549cdf-q9c6j\" (UID: \"e5435e4b-b8a8-45b9-a90a-bca4b5a65506\") " pod="openshift-ingress/e2e-praxis-a-b271f0-openshift-default-5544549cdf-q9c6j"
Nov 02 12:48:52.415420 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:52.415375    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5435e4b-b8a8-45b9-a90a-bca4b5a65506-istio-data\") pod \"e2e-praxis-a-b271f0-openshift-default-5544549cdf-q9c6j\" (UID: \"e5435e4b-b8a8-45b9-a90a-bca4b5a65506\") " pod="openshift-ingress/e2e-praxis-a-b271f0-openshift-default-5544549cdf-q9c6j"
Nov 02 12:48:52.415537 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:52.415432    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5435e4b-b8a8-45b9-a90a-bca4b5a65506-credential-socket\") pod \"e2e-praxis-a-b271f0-openshift-default-5544549cdf-q9c6j\" (UID: \"e5435e4b-b8a8-45b9-a90a-bca4b5a65506\") " pod="openshift-ingress/e2e-praxis-a-b271f0-openshift-default-5544549cdf-q9c6j"
Nov 02 12:48:52.415537 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:52.415452    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5435e4b-b8a8-45b9-a90a-bca4b5a65506-workload-socket\") pod \"e2e-praxis-a-b271f0-openshift-default-5544549cdf-q9c6j\" (UID: \"e5435e4b-b8a8-45b9-a90a-bca4b5a65506\") " pod="openshift-ingress/e2e-praxis-a-b271f0-openshift-default-5544549cdf-q9c6j"
Nov 02 12:48:52.415537 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:52.415475    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5435e4b-b8a8-45b9-a90a-bca4b5a65506-workload-certs\") pod \"e2e-praxis-a-b271f0-openshift-default-5544549cdf-q9c6j\" (UID: \"e5435e4b-b8a8-45b9-a90a-bca4b5a65506\") " pod="openshift-ingress/e2e-praxis-a-b271f0-openshift-default-5544549cdf-q9c6j"
Nov 02 12:48:52.415537 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:52.415519    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5435e4b-b8a8-45b9-a90a-bca4b5a65506-istio-token\") pod \"e2e-praxis-a-b271f0-openshift-default-5544549cdf-q9c6j\" (UID: \"e5435e4b-b8a8-45b9-a90a-bca4b5a65506\") " pod="openshift-ingress/e2e-praxis-a-b271f0-openshift-default-5544549cdf-q9c6j"
Nov 02 12:48:52.415796 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:52.415605    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5435e4b-b8a8-45b9-a90a-bca4b5a65506-istiod-ca-cert\") pod \"e2e-praxis-a-b271f0-openshift-default-5544549cdf-q9c6j\" (UID: \"e5435e4b-b8a8-45b9-a90a-bca4b5a65506\") " pod="openshift-ingress/e2e-praxis-a-b271f0-openshift-default-5544549cdf-q9c6j"
Nov 02 12:48:52.516539 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:52.516513    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5435e4b-b8a8-45b9-a90a-bca4b5a65506-istio-envoy\") pod \"e2e-praxis-a-b271f0-openshift-default-5544549cdf-q9c6j\" (UID: \"e5435e4b-b8a8-45b9-a90a-bca4b5a65506\") " pod="openshift-ingress/e2e-praxis-a-b271f0-openshift-default-5544549cdf-q9c6j"
Nov 02 12:48:52.516733 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:52.516571    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5435e4b-b8a8-45b9-a90a-bca4b5a65506-istio-podinfo\") pod \"e2e-praxis-a-b271f0-openshift-default-5544549cdf-q9c6j\" (UID: \"e5435e4b-b8a8-45b9-a90a-bca4b5a65506\") " pod="openshift-ingress/e2e-praxis-a-b271f0-openshift-default-5544549cdf-q9c6j"
Nov 02 12:48:52.516733 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:52.516592    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5435e4b-b8a8-45b9-a90a-bca4b5a65506-istio-data\") pod \"e2e-praxis-a-b271f0-openshift-default-5544549cdf-q9c6j\" (UID: \"e5435e4b-b8a8-45b9-a90a-bca4b5a65506\") " pod="openshift-ingress/e2e-praxis-a-b271f0-openshift-default-5544549cdf-q9c6j"
Nov 02 12:48:52.516733 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:52.516629    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5435e4b-b8a8-45b9-a90a-bca4b5a65506-credential-socket\") pod \"e2e-praxis-a-b271f0-openshift-default-5544549cdf-q9c6j\" (UID: \"e5435e4b-b8a8-45b9-a90a-bca4b5a65506\") " pod="openshift-ingress/e2e-praxis-a-b271f0-openshift-default-5544549cdf-q9c6j"
Nov 02 12:48:52.516733 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:52.516662    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5435e4b-b8a8-45b9-a90a-bca4b5a65506-workload-socket\") pod \"e2e-praxis-a-b271f0-openshift-default-5544549cdf-q9c6j\" (UID: \"e5435e4b-b8a8-45b9-a90a-bca4b5a65506\") " pod="openshift-ingress/e2e-praxis-a-b271f0-openshift-default-5544549cdf-q9c6j"
Nov 02 12:48:52.516733 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:52.516678    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5435e4b-b8a8-45b9-a90a-bca4b5a65506-workload-certs\") pod \"e2e-praxis-a-b271f0-openshift-default-5544549cdf-q9c6j\" (UID: \"e5435e4b-b8a8-45b9-a90a-bca4b5a65506\") " pod="openshift-ingress/e2e-praxis-a-b271f0-openshift-default-5544549cdf-q9c6j"
Nov 02 12:48:52.516733 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:52.516733    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5435e4b-b8a8-45b9-a90a-bca4b5a65506-istio-token\") pod \"e2e-praxis-a-b271f0-openshift-default-5544549cdf-q9c6j\" (UID: \"e5435e4b-b8a8-45b9-a90a-bca4b5a65506\") " pod="openshift-ingress/e2e-praxis-a-b271f0-openshift-default-5544549cdf-q9c6j"
Nov 02 12:48:52.517063 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:52.516827    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5435e4b-b8a8-45b9-a90a-bca4b5a65506-istiod-ca-cert\") pod \"e2e-praxis-a-b271f0-openshift-default-5544549cdf-q9c6j\" (UID: \"e5435e4b-b8a8-45b9-a90a-bca4b5a65506\") " pod="openshift-ingress/e2e-praxis-a-b271f0-openshift-default-5544549cdf-q9c6j"
Nov 02 12:48:52.517063 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:52.516862    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htbqv\" (UniqueName: \"kubernetes.io/projected/e5435e4b-b8a8-45b9-a90a-bca4b5a65506-kube-api-access-htbqv\") pod \"e2e-praxis-a-b271f0-openshift-default-5544549cdf-q9c6j\" (UID: \"e5435e4b-b8a8-45b9-a90a-bca4b5a65506\") " pod="openshift-ingress/e2e-praxis-a-b271f0-openshift-default-5544549cdf-q9c6j"
Nov 02 12:48:52.517178 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:52.517002    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5435e4b-b8a8-45b9-a90a-bca4b5a65506-istio-data\") pod \"e2e-praxis-a-b271f0-openshift-default-5544549cdf-q9c6j\" (UID: \"e5435e4b-b8a8-45b9-a90a-bca4b5a65506\") " pod="openshift-ingress/e2e-praxis-a-b271f0-openshift-default-5544549cdf-q9c6j"
Nov 02 12:48:52.517178 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:52.517090    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5435e4b-b8a8-45b9-a90a-bca4b5a65506-credential-socket\") pod \"e2e-praxis-a-b271f0-openshift-default-5544549cdf-q9c6j\" (UID: \"e5435e4b-b8a8-45b9-a90a-bca4b5a65506\") " pod="openshift-ingress/e2e-praxis-a-b271f0-openshift-default-5544549cdf-q9c6j"
Nov 02 12:48:52.517178 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:52.517146    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5435e4b-b8a8-45b9-a90a-bca4b5a65506-workload-socket\") pod \"e2e-praxis-a-b271f0-openshift-default-5544549cdf-q9c6j\" (UID: \"e5435e4b-b8a8-45b9-a90a-bca4b5a65506\") " pod="openshift-ingress/e2e-praxis-a-b271f0-openshift-default-5544549cdf-q9c6j"
Nov 02 12:48:52.517416 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:52.517387    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5435e4b-b8a8-45b9-a90a-bca4b5a65506-workload-certs\") pod \"e2e-praxis-a-b271f0-openshift-default-5544549cdf-q9c6j\" (UID: \"e5435e4b-b8a8-45b9-a90a-bca4b5a65506\") " pod="openshift-ingress/e2e-praxis-a-b271f0-openshift-default-5544549cdf-q9c6j"
Nov 02 12:48:52.517580 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:52.517555    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5435e4b-b8a8-45b9-a90a-bca4b5a65506-istiod-ca-cert\") pod \"e2e-praxis-a-b271f0-openshift-default-5544549cdf-q9c6j\" (UID: \"e5435e4b-b8a8-45b9-a90a-bca4b5a65506\") " pod="openshift-ingress/e2e-praxis-a-b271f0-openshift-default-5544549cdf-q9c6j"
Nov 02 12:48:52.519441 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:52.519415    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5435e4b-b8a8-45b9-a90a-bca4b5a65506-istio-envoy\") pod \"e2e-praxis-a-b271f0-openshift-default-5544549cdf-q9c6j\" (UID: \"e5435e4b-b8a8-45b9-a90a-bca4b5a65506\") " pod="openshift-ingress/e2e-praxis-a-b271f0-openshift-default-5544549cdf-q9c6j"
Nov 02 12:48:52.519657 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:52.519625    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5435e4b-b8a8-45b9-a90a-bca4b5a65506-istio-podinfo\") pod \"e2e-praxis-a-b271f0-openshift-default-5544549cdf-q9c6j\" (UID: \"e5435e4b-b8a8-45b9-a90a-bca4b5a65506\") " pod="openshift-ingress/e2e-praxis-a-b271f0-openshift-default-5544549cdf-q9c6j"
Nov 02 12:48:52.541402 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:52.541372    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5435e4b-b8a8-45b9-a90a-bca4b5a65506-istio-token\") pod \"e2e-praxis-a-b271f0-openshift-default-5544549cdf-q9c6j\" (UID: \"e5435e4b-b8a8-45b9-a90a-bca4b5a65506\") " pod="openshift-ingress/e2e-praxis-a-b271f0-openshift-default-5544549cdf-q9c6j"
Nov 02 12:48:52.541513 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:52.541498    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-htbqv\" (UniqueName: \"kubernetes.io/projected/e5435e4b-b8a8-45b9-a90a-bca4b5a65506-kube-api-access-htbqv\") pod \"e2e-praxis-a-b271f0-openshift-default-5544549cdf-q9c6j\" (UID: \"e5435e4b-b8a8-45b9-a90a-bca4b5a65506\") " pod="openshift-ingress/e2e-praxis-a-b271f0-openshift-default-5544549cdf-q9c6j"
Nov 02 12:48:52.693207 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:52.692903    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-b271f0-openshift-default-5544549cdf-q9c6j"
Nov 02 12:48:53.086897 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:48:53.086850    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5435e4b_b8a8_45b9_a90a_bca4b5a65506.slice/crio-e58195cf96c3b3195e8ca0b8f8e7fa291484081a895f1b316b37dc5f9d435950 WatchSource:0}: Error finding container e58195cf96c3b3195e8ca0b8f8e7fa291484081a895f1b316b37dc5f9d435950: Status 404 returned error can't find the container with id e58195cf96c3b3195e8ca0b8f8e7fa291484081a895f1b316b37dc5f9d435950
Nov 02 12:48:53.093150 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:53.091825    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:48:53.093150 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:53.091923    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:48:53.093150 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:53.091980    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:48:53.104423 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:53.104380    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-b271f0-openshift-default-5544549cdf-q9c6j"]
Nov 02 12:48:53.261997 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:53.261965    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-b271f0-openshift-default-5544549cdf-q9c6j" event={"ID":"e5435e4b-b8a8-45b9-a90a-bca4b5a65506","Type":"ContainerStarted","Data":"b79c65999f4b4d8153087bcfb6f1e7caf141c10c574e6be52b7157b40c272036"}
Nov 02 12:48:53.262174 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:53.262025    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-b271f0-openshift-default-5544549cdf-q9c6j" event={"ID":"e5435e4b-b8a8-45b9-a90a-bca4b5a65506","Type":"ContainerStarted","Data":"e58195cf96c3b3195e8ca0b8f8e7fa291484081a895f1b316b37dc5f9d435950"}
Nov 02 12:48:53.311977 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:53.311920    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-b271f0-openshift-default-5544549cdf-q9c6j" podStartSLOduration=1.311901031 podStartE2EDuration="1.311901031s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:48:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:48:53.309515615 +0000 UTC m=+1329.564914711" watchObservedRunningTime="2026-11-02 12:48:53.311901031 +0000 UTC m=+1329.567300127"
Nov 02 12:48:53.693701 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:53.693668    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-b271f0-openshift-default-5544549cdf-q9c6j"
Nov 02 12:48:54.096028 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:54.095950    4559 patch_prober.go:29] interesting pod/e2e-ipp-b-97de36-openshift-default-77899d747c-cq4bc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 12:48:54.096028 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:54.096007    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-97de36-openshift-default-77899d747c-cq4bc" podUID="5b55da6b-89cb-46d6-a7a0-3a09e7d7d17e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 12:48:54.700509 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:54.700477    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-b271f0-openshift-default-5544549cdf-q9c6j"
Nov 02 12:48:55.273965 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:55.273935    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-b271f0-openshift-default-5544549cdf-q9c6j"
Nov 02 12:48:55.274787 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:55.274767    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-b271f0-openshift-default-5544549cdf-q9c6j"
Nov 02 12:48:55.689704 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:55.689677    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-97de36-openshift-default-77899d747c-cq4bc"
Nov 02 12:48:55.795375 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:55.795328    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b55da6b-89cb-46d6-a7a0-3a09e7d7d17e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b55da6b-89cb-46d6-a7a0-3a09e7d7d17e-istio-data\") pod \"5b55da6b-89cb-46d6-a7a0-3a09e7d7d17e\" (UID: \"5b55da6b-89cb-46d6-a7a0-3a09e7d7d17e\") "
Nov 02 12:48:55.795921 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:55.795410    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b55da6b-89cb-46d6-a7a0-3a09e7d7d17e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b55da6b-89cb-46d6-a7a0-3a09e7d7d17e-credential-socket\") pod \"5b55da6b-89cb-46d6-a7a0-3a09e7d7d17e\" (UID: \"5b55da6b-89cb-46d6-a7a0-3a09e7d7d17e\") "
Nov 02 12:48:55.795921 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:55.795473    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b55da6b-89cb-46d6-a7a0-3a09e7d7d17e-kube-api-access-h8wrg\" (UniqueName: \"kubernetes.io/projected/5b55da6b-89cb-46d6-a7a0-3a09e7d7d17e-kube-api-access-h8wrg\") pod \"5b55da6b-89cb-46d6-a7a0-3a09e7d7d17e\" (UID: \"5b55da6b-89cb-46d6-a7a0-3a09e7d7d17e\") "
Nov 02 12:48:55.795921 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:55.795501    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5b55da6b-89cb-46d6-a7a0-3a09e7d7d17e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b55da6b-89cb-46d6-a7a0-3a09e7d7d17e-istio-podinfo\") pod \"5b55da6b-89cb-46d6-a7a0-3a09e7d7d17e\" (UID: \"5b55da6b-89cb-46d6-a7a0-3a09e7d7d17e\") "
Nov 02 12:48:55.795921 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:55.795538    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b55da6b-89cb-46d6-a7a0-3a09e7d7d17e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b55da6b-89cb-46d6-a7a0-3a09e7d7d17e-workload-certs\") pod \"5b55da6b-89cb-46d6-a7a0-3a09e7d7d17e\" (UID: \"5b55da6b-89cb-46d6-a7a0-3a09e7d7d17e\") "
Nov 02 12:48:55.795921 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:55.795593    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b55da6b-89cb-46d6-a7a0-3a09e7d7d17e-istio-token\" (UniqueName: \"kubernetes.io/projected/5b55da6b-89cb-46d6-a7a0-3a09e7d7d17e-istio-token\") pod \"5b55da6b-89cb-46d6-a7a0-3a09e7d7d17e\" (UID: \"5b55da6b-89cb-46d6-a7a0-3a09e7d7d17e\") "
Nov 02 12:48:55.795921 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:55.795699    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b55da6b-89cb-46d6-a7a0-3a09e7d7d17e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b55da6b-89cb-46d6-a7a0-3a09e7d7d17e-istiod-ca-cert\") pod \"5b55da6b-89cb-46d6-a7a0-3a09e7d7d17e\" (UID: \"5b55da6b-89cb-46d6-a7a0-3a09e7d7d17e\") "
Nov 02 12:48:55.795921 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:55.795738    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b55da6b-89cb-46d6-a7a0-3a09e7d7d17e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b55da6b-89cb-46d6-a7a0-3a09e7d7d17e-istio-envoy\") pod \"5b55da6b-89cb-46d6-a7a0-3a09e7d7d17e\" (UID: \"5b55da6b-89cb-46d6-a7a0-3a09e7d7d17e\") "
Nov 02 12:48:55.795921 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:55.795831    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b55da6b-89cb-46d6-a7a0-3a09e7d7d17e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b55da6b-89cb-46d6-a7a0-3a09e7d7d17e-workload-socket\") pod \"5b55da6b-89cb-46d6-a7a0-3a09e7d7d17e\" (UID: \"5b55da6b-89cb-46d6-a7a0-3a09e7d7d17e\") "
Nov 02 12:48:55.795921 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:55.795832    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b55da6b-89cb-46d6-a7a0-3a09e7d7d17e-istio-data" pod "5b55da6b-89cb-46d6-a7a0-3a09e7d7d17e" (UID: "5b55da6b-89cb-46d6-a7a0-3a09e7d7d17e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:48:55.795921 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:55.795848    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b55da6b-89cb-46d6-a7a0-3a09e7d7d17e-credential-socket" pod "5b55da6b-89cb-46d6-a7a0-3a09e7d7d17e" (UID: "5b55da6b-89cb-46d6-a7a0-3a09e7d7d17e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:48:55.796406 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:55.796385    4559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b55da6b-89cb-46d6-a7a0-3a09e7d7d17e-istio-data\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:48:55.796483 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:55.796414    4559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b55da6b-89cb-46d6-a7a0-3a09e7d7d17e-credential-socket\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:48:55.796975 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:55.796934    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b55da6b-89cb-46d6-a7a0-3a09e7d7d17e-istiod-ca-cert" pod "5b55da6b-89cb-46d6-a7a0-3a09e7d7d17e" (UID: "5b55da6b-89cb-46d6-a7a0-3a09e7d7d17e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:48:55.797242 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:55.797201    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b55da6b-89cb-46d6-a7a0-3a09e7d7d17e-workload-socket" pod "5b55da6b-89cb-46d6-a7a0-3a09e7d7d17e" (UID: "5b55da6b-89cb-46d6-a7a0-3a09e7d7d17e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:48:55.799144 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:55.799118    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b55da6b-89cb-46d6-a7a0-3a09e7d7d17e-workload-certs" pod "5b55da6b-89cb-46d6-a7a0-3a09e7d7d17e" (UID: "5b55da6b-89cb-46d6-a7a0-3a09e7d7d17e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:48:55.799349 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:55.799321    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b55da6b-89cb-46d6-a7a0-3a09e7d7d17e-kube-api-access-h8wrg" pod "5b55da6b-89cb-46d6-a7a0-3a09e7d7d17e" (UID: "5b55da6b-89cb-46d6-a7a0-3a09e7d7d17e"). InnerVolumeSpecName "kube-api-access-h8wrg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:48:55.804972 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:55.804878    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5b55da6b-89cb-46d6-a7a0-3a09e7d7d17e-istio-podinfo" pod "5b55da6b-89cb-46d6-a7a0-3a09e7d7d17e" (UID: "5b55da6b-89cb-46d6-a7a0-3a09e7d7d17e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 12:48:55.804972 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:55.804917    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b55da6b-89cb-46d6-a7a0-3a09e7d7d17e-istio-envoy" pod "5b55da6b-89cb-46d6-a7a0-3a09e7d7d17e" (UID: "5b55da6b-89cb-46d6-a7a0-3a09e7d7d17e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:48:55.804972 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:55.804924    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b55da6b-89cb-46d6-a7a0-3a09e7d7d17e-istio-token" pod "5b55da6b-89cb-46d6-a7a0-3a09e7d7d17e" (UID: "5b55da6b-89cb-46d6-a7a0-3a09e7d7d17e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:48:55.897320 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:55.897249    4559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b55da6b-89cb-46d6-a7a0-3a09e7d7d17e-istio-envoy\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:48:55.897320 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:55.897277    4559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b55da6b-89cb-46d6-a7a0-3a09e7d7d17e-workload-socket\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:48:55.897320 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:55.897289    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h8wrg\" (UniqueName: \"kubernetes.io/projected/5b55da6b-89cb-46d6-a7a0-3a09e7d7d17e-kube-api-access-h8wrg\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:48:55.897320 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:55.897300    4559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b55da6b-89cb-46d6-a7a0-3a09e7d7d17e-istio-podinfo\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:48:55.897320 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:55.897308    4559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b55da6b-89cb-46d6-a7a0-3a09e7d7d17e-workload-certs\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:48:55.897320 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:55.897317    4559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b55da6b-89cb-46d6-a7a0-3a09e7d7d17e-istio-token\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:48:55.897320 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:55.897324    4559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b55da6b-89cb-46d6-a7a0-3a09e7d7d17e-istiod-ca-cert\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:48:56.284718 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:56.284630    4559 generic.go:356] "Generic (PLEG): container finished" podID="5b55da6b-89cb-46d6-a7a0-3a09e7d7d17e" containerID="0907660c6d6e2964677ccebb59dbaa23d4b900e33b7d413aa483136fa1e4a4d3" exitCode=0
Nov 02 12:48:56.284895 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:56.284808    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-97de36-openshift-default-77899d747c-cq4bc"
Nov 02 12:48:56.284895 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:56.284827    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-97de36-openshift-default-77899d747c-cq4bc" event={"ID":"5b55da6b-89cb-46d6-a7a0-3a09e7d7d17e","Type":"ContainerDied","Data":"0907660c6d6e2964677ccebb59dbaa23d4b900e33b7d413aa483136fa1e4a4d3"}
Nov 02 12:48:56.284895 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:56.284892    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-97de36-openshift-default-77899d747c-cq4bc" event={"ID":"5b55da6b-89cb-46d6-a7a0-3a09e7d7d17e","Type":"ContainerDied","Data":"c7f8060de2a53e3b8849723a538a4b650d1214f1c2774e5ab7fdf317842f501e"}
Nov 02 12:48:56.285086 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:56.284922    4559 scope.go:122] "RemoveContainer" containerID="0907660c6d6e2964677ccebb59dbaa23d4b900e33b7d413aa483136fa1e4a4d3"
Nov 02 12:48:56.294981 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:56.294952    4559 scope.go:122] "RemoveContainer" containerID="ef26164c25b358f05ae65310109400f62dc5283758f5989525cebc2709f8e391"
Nov 02 12:48:56.304068 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:56.304052    4559 scope.go:122] "RemoveContainer" containerID="0907660c6d6e2964677ccebb59dbaa23d4b900e33b7d413aa483136fa1e4a4d3"
Nov 02 12:48:56.304342 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:48:56.304322    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0907660c6d6e2964677ccebb59dbaa23d4b900e33b7d413aa483136fa1e4a4d3\": container with ID starting with 0907660c6d6e2964677ccebb59dbaa23d4b900e33b7d413aa483136fa1e4a4d3 not found: ID does not exist" containerID="0907660c6d6e2964677ccebb59dbaa23d4b900e33b7d413aa483136fa1e4a4d3"
Nov 02 12:48:56.304421 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:56.304347    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0907660c6d6e2964677ccebb59dbaa23d4b900e33b7d413aa483136fa1e4a4d3"} err="failed to get container status \"0907660c6d6e2964677ccebb59dbaa23d4b900e33b7d413aa483136fa1e4a4d3\": rpc error: code = NotFound desc = could not find container \"0907660c6d6e2964677ccebb59dbaa23d4b900e33b7d413aa483136fa1e4a4d3\": container with ID starting with 0907660c6d6e2964677ccebb59dbaa23d4b900e33b7d413aa483136fa1e4a4d3 not found: ID does not exist"
Nov 02 12:48:56.304421 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:56.304363    4559 scope.go:122] "RemoveContainer" containerID="ef26164c25b358f05ae65310109400f62dc5283758f5989525cebc2709f8e391"
Nov 02 12:48:56.304651 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:48:56.304618    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef26164c25b358f05ae65310109400f62dc5283758f5989525cebc2709f8e391\": container with ID starting with ef26164c25b358f05ae65310109400f62dc5283758f5989525cebc2709f8e391 not found: ID does not exist" containerID="ef26164c25b358f05ae65310109400f62dc5283758f5989525cebc2709f8e391"
Nov 02 12:48:56.304703 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:56.304657    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef26164c25b358f05ae65310109400f62dc5283758f5989525cebc2709f8e391"} err="failed to get container status \"ef26164c25b358f05ae65310109400f62dc5283758f5989525cebc2709f8e391\": rpc error: code = NotFound desc = could not find container \"ef26164c25b358f05ae65310109400f62dc5283758f5989525cebc2709f8e391\": container with ID starting with ef26164c25b358f05ae65310109400f62dc5283758f5989525cebc2709f8e391 not found: ID does not exist"
Nov 02 12:48:56.359057 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:56.359027    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-97de36-openshift-default-77899d747c-cq4bc"]
Nov 02 12:48:56.372275 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:56.372250    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-97de36-openshift-default-77899d747c-cq4bc"]
Nov 02 12:48:57.492764 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:57.492729    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-e93a4b-openshift-default-7f8c56bf9c-4pjxc"]
Nov 02 12:48:57.493543 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:57.493524    4559 cpu_manager.go:414] "containerMap: removing container" podUID="0f74d2ef-c796-4c91-b763-7593c16cfe71" containerName="istio-proxy"
Nov 02 12:48:57.493675 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:57.493545    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f74d2ef-c796-4c91-b763-7593c16cfe71" containerName="istio-proxy"
Nov 02 12:48:57.493675 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:57.493609    4559 cpu_manager.go:414] "containerMap: removing container" podUID="5b55da6b-89cb-46d6-a7a0-3a09e7d7d17e" containerName="istio-proxy"
Nov 02 12:48:57.493675 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:57.493625    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b55da6b-89cb-46d6-a7a0-3a09e7d7d17e" containerName="istio-proxy"
Nov 02 12:48:57.494130 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:57.494109    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b55da6b-89cb-46d6-a7a0-3a09e7d7d17e" containerName="istio-proxy"
Nov 02 12:48:57.499045 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:57.499021    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-e93a4b-openshift-default-7f8c56bf9c-4pjxc"
Nov 02 12:48:57.519589 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:57.519560    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/624a0494-692e-4864-810e-42ba5306a5f0-workload-socket\") pod \"e2e-praxis-b-e93a4b-openshift-default-7f8c56bf9c-4pjxc\" (UID: \"624a0494-692e-4864-810e-42ba5306a5f0\") " pod="openshift-ingress/e2e-praxis-b-e93a4b-openshift-default-7f8c56bf9c-4pjxc"
Nov 02 12:48:57.519730 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:57.519598    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/624a0494-692e-4864-810e-42ba5306a5f0-istio-data\") pod \"e2e-praxis-b-e93a4b-openshift-default-7f8c56bf9c-4pjxc\" (UID: \"624a0494-692e-4864-810e-42ba5306a5f0\") " pod="openshift-ingress/e2e-praxis-b-e93a4b-openshift-default-7f8c56bf9c-4pjxc"
Nov 02 12:48:57.519730 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:57.519634    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/624a0494-692e-4864-810e-42ba5306a5f0-istiod-ca-cert\") pod \"e2e-praxis-b-e93a4b-openshift-default-7f8c56bf9c-4pjxc\" (UID: \"624a0494-692e-4864-810e-42ba5306a5f0\") " pod="openshift-ingress/e2e-praxis-b-e93a4b-openshift-default-7f8c56bf9c-4pjxc"
Nov 02 12:48:57.519840 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:57.519820    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/624a0494-692e-4864-810e-42ba5306a5f0-istio-envoy\") pod \"e2e-praxis-b-e93a4b-openshift-default-7f8c56bf9c-4pjxc\" (UID: \"624a0494-692e-4864-810e-42ba5306a5f0\") " pod="openshift-ingress/e2e-praxis-b-e93a4b-openshift-default-7f8c56bf9c-4pjxc"
Nov 02 12:48:57.519879 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:57.519853    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/624a0494-692e-4864-810e-42ba5306a5f0-credential-socket\") pod \"e2e-praxis-b-e93a4b-openshift-default-7f8c56bf9c-4pjxc\" (UID: \"624a0494-692e-4864-810e-42ba5306a5f0\") " pod="openshift-ingress/e2e-praxis-b-e93a4b-openshift-default-7f8c56bf9c-4pjxc"
Nov 02 12:48:57.519948 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:57.519926    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/624a0494-692e-4864-810e-42ba5306a5f0-istio-token\") pod \"e2e-praxis-b-e93a4b-openshift-default-7f8c56bf9c-4pjxc\" (UID: \"624a0494-692e-4864-810e-42ba5306a5f0\") " pod="openshift-ingress/e2e-praxis-b-e93a4b-openshift-default-7f8c56bf9c-4pjxc"
Nov 02 12:48:57.520017 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:57.520003    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/624a0494-692e-4864-810e-42ba5306a5f0-workload-certs\") pod \"e2e-praxis-b-e93a4b-openshift-default-7f8c56bf9c-4pjxc\" (UID: \"624a0494-692e-4864-810e-42ba5306a5f0\") " pod="openshift-ingress/e2e-praxis-b-e93a4b-openshift-default-7f8c56bf9c-4pjxc"
Nov 02 12:48:57.520064 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:57.520024    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jgtn\" (UniqueName: \"kubernetes.io/projected/624a0494-692e-4864-810e-42ba5306a5f0-kube-api-access-8jgtn\") pod \"e2e-praxis-b-e93a4b-openshift-default-7f8c56bf9c-4pjxc\" (UID: \"624a0494-692e-4864-810e-42ba5306a5f0\") " pod="openshift-ingress/e2e-praxis-b-e93a4b-openshift-default-7f8c56bf9c-4pjxc"
Nov 02 12:48:57.520064 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:57.520044    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/624a0494-692e-4864-810e-42ba5306a5f0-istio-podinfo\") pod \"e2e-praxis-b-e93a4b-openshift-default-7f8c56bf9c-4pjxc\" (UID: \"624a0494-692e-4864-810e-42ba5306a5f0\") " pod="openshift-ingress/e2e-praxis-b-e93a4b-openshift-default-7f8c56bf9c-4pjxc"
Nov 02 12:48:57.526235 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:57.526207    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-e93a4b-openshift-default-7f8c56bf9c-4pjxc"]
Nov 02 12:48:57.536430 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:57.536398    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-e93a4b-openshift-default-dockercfg-gxkdp\""
Nov 02 12:48:57.621411 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:57.621376    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/624a0494-692e-4864-810e-42ba5306a5f0-istio-envoy\") pod \"e2e-praxis-b-e93a4b-openshift-default-7f8c56bf9c-4pjxc\" (UID: \"624a0494-692e-4864-810e-42ba5306a5f0\") " pod="openshift-ingress/e2e-praxis-b-e93a4b-openshift-default-7f8c56bf9c-4pjxc"
Nov 02 12:48:57.621411 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:57.621415    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/624a0494-692e-4864-810e-42ba5306a5f0-credential-socket\") pod \"e2e-praxis-b-e93a4b-openshift-default-7f8c56bf9c-4pjxc\" (UID: \"624a0494-692e-4864-810e-42ba5306a5f0\") " pod="openshift-ingress/e2e-praxis-b-e93a4b-openshift-default-7f8c56bf9c-4pjxc"
Nov 02 12:48:57.621731 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:57.621443    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/624a0494-692e-4864-810e-42ba5306a5f0-istio-token\") pod \"e2e-praxis-b-e93a4b-openshift-default-7f8c56bf9c-4pjxc\" (UID: \"624a0494-692e-4864-810e-42ba5306a5f0\") " pod="openshift-ingress/e2e-praxis-b-e93a4b-openshift-default-7f8c56bf9c-4pjxc"
Nov 02 12:48:57.621731 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:57.621478    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/624a0494-692e-4864-810e-42ba5306a5f0-workload-certs\") pod \"e2e-praxis-b-e93a4b-openshift-default-7f8c56bf9c-4pjxc\" (UID: \"624a0494-692e-4864-810e-42ba5306a5f0\") " pod="openshift-ingress/e2e-praxis-b-e93a4b-openshift-default-7f8c56bf9c-4pjxc"
Nov 02 12:48:57.621731 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:57.621494    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jgtn\" (UniqueName: \"kubernetes.io/projected/624a0494-692e-4864-810e-42ba5306a5f0-kube-api-access-8jgtn\") pod \"e2e-praxis-b-e93a4b-openshift-default-7f8c56bf9c-4pjxc\" (UID: \"624a0494-692e-4864-810e-42ba5306a5f0\") " pod="openshift-ingress/e2e-praxis-b-e93a4b-openshift-default-7f8c56bf9c-4pjxc"
Nov 02 12:48:57.621731 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:57.621511    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/624a0494-692e-4864-810e-42ba5306a5f0-istio-podinfo\") pod \"e2e-praxis-b-e93a4b-openshift-default-7f8c56bf9c-4pjxc\" (UID: \"624a0494-692e-4864-810e-42ba5306a5f0\") " pod="openshift-ingress/e2e-praxis-b-e93a4b-openshift-default-7f8c56bf9c-4pjxc"
Nov 02 12:48:57.621731 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:57.621621    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/624a0494-692e-4864-810e-42ba5306a5f0-workload-socket\") pod \"e2e-praxis-b-e93a4b-openshift-default-7f8c56bf9c-4pjxc\" (UID: \"624a0494-692e-4864-810e-42ba5306a5f0\") " pod="openshift-ingress/e2e-praxis-b-e93a4b-openshift-default-7f8c56bf9c-4pjxc"
Nov 02 12:48:57.621731 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:57.621681    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/624a0494-692e-4864-810e-42ba5306a5f0-istio-data\") pod \"e2e-praxis-b-e93a4b-openshift-default-7f8c56bf9c-4pjxc\" (UID: \"624a0494-692e-4864-810e-42ba5306a5f0\") " pod="openshift-ingress/e2e-praxis-b-e93a4b-openshift-default-7f8c56bf9c-4pjxc"
Nov 02 12:48:57.622030 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:57.621739    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/624a0494-692e-4864-810e-42ba5306a5f0-istiod-ca-cert\") pod \"e2e-praxis-b-e93a4b-openshift-default-7f8c56bf9c-4pjxc\" (UID: \"624a0494-692e-4864-810e-42ba5306a5f0\") " pod="openshift-ingress/e2e-praxis-b-e93a4b-openshift-default-7f8c56bf9c-4pjxc"
Nov 02 12:48:57.622030 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:57.621951    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/624a0494-692e-4864-810e-42ba5306a5f0-workload-certs\") pod \"e2e-praxis-b-e93a4b-openshift-default-7f8c56bf9c-4pjxc\" (UID: \"624a0494-692e-4864-810e-42ba5306a5f0\") " pod="openshift-ingress/e2e-praxis-b-e93a4b-openshift-default-7f8c56bf9c-4pjxc"
Nov 02 12:48:57.622030 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:57.621997    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/624a0494-692e-4864-810e-42ba5306a5f0-credential-socket\") pod \"e2e-praxis-b-e93a4b-openshift-default-7f8c56bf9c-4pjxc\" (UID: \"624a0494-692e-4864-810e-42ba5306a5f0\") " pod="openshift-ingress/e2e-praxis-b-e93a4b-openshift-default-7f8c56bf9c-4pjxc"
Nov 02 12:48:57.622170 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:57.622074    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/624a0494-692e-4864-810e-42ba5306a5f0-workload-socket\") pod \"e2e-praxis-b-e93a4b-openshift-default-7f8c56bf9c-4pjxc\" (UID: \"624a0494-692e-4864-810e-42ba5306a5f0\") " pod="openshift-ingress/e2e-praxis-b-e93a4b-openshift-default-7f8c56bf9c-4pjxc"
Nov 02 12:48:57.622170 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:57.622100    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/624a0494-692e-4864-810e-42ba5306a5f0-istio-data\") pod \"e2e-praxis-b-e93a4b-openshift-default-7f8c56bf9c-4pjxc\" (UID: \"624a0494-692e-4864-810e-42ba5306a5f0\") " pod="openshift-ingress/e2e-praxis-b-e93a4b-openshift-default-7f8c56bf9c-4pjxc"
Nov 02 12:48:57.622470 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:57.622444    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/624a0494-692e-4864-810e-42ba5306a5f0-istiod-ca-cert\") pod \"e2e-praxis-b-e93a4b-openshift-default-7f8c56bf9c-4pjxc\" (UID: \"624a0494-692e-4864-810e-42ba5306a5f0\") " pod="openshift-ingress/e2e-praxis-b-e93a4b-openshift-default-7f8c56bf9c-4pjxc"
Nov 02 12:48:57.623850 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:57.623821    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/624a0494-692e-4864-810e-42ba5306a5f0-istio-envoy\") pod \"e2e-praxis-b-e93a4b-openshift-default-7f8c56bf9c-4pjxc\" (UID: \"624a0494-692e-4864-810e-42ba5306a5f0\") " pod="openshift-ingress/e2e-praxis-b-e93a4b-openshift-default-7f8c56bf9c-4pjxc"
Nov 02 12:48:57.624075 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:57.624057    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/624a0494-692e-4864-810e-42ba5306a5f0-istio-podinfo\") pod \"e2e-praxis-b-e93a4b-openshift-default-7f8c56bf9c-4pjxc\" (UID: \"624a0494-692e-4864-810e-42ba5306a5f0\") " pod="openshift-ingress/e2e-praxis-b-e93a4b-openshift-default-7f8c56bf9c-4pjxc"
Nov 02 12:48:57.657410 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:57.657374    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/624a0494-692e-4864-810e-42ba5306a5f0-istio-token\") pod \"e2e-praxis-b-e93a4b-openshift-default-7f8c56bf9c-4pjxc\" (UID: \"624a0494-692e-4864-810e-42ba5306a5f0\") " pod="openshift-ingress/e2e-praxis-b-e93a4b-openshift-default-7f8c56bf9c-4pjxc"
Nov 02 12:48:57.657673 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:57.657653    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jgtn\" (UniqueName: \"kubernetes.io/projected/624a0494-692e-4864-810e-42ba5306a5f0-kube-api-access-8jgtn\") pod \"e2e-praxis-b-e93a4b-openshift-default-7f8c56bf9c-4pjxc\" (UID: \"624a0494-692e-4864-810e-42ba5306a5f0\") " pod="openshift-ingress/e2e-praxis-b-e93a4b-openshift-default-7f8c56bf9c-4pjxc"
Nov 02 12:48:57.721790 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:57.721758    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-b271f0-547b5469dc-42kll"]
Nov 02 12:48:57.722558 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:57.722537    4559 cpu_manager.go:414] "containerMap: removing container" podUID="5b55da6b-89cb-46d6-a7a0-3a09e7d7d17e" containerName="istio-proxy"
Nov 02 12:48:57.722558 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:57.722557    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b55da6b-89cb-46d6-a7a0-3a09e7d7d17e" containerName="istio-proxy"
Nov 02 12:48:57.722774 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:57.722684    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b55da6b-89cb-46d6-a7a0-3a09e7d7d17e" containerName="istio-proxy"
Nov 02 12:48:57.727323 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:57.727299    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b271f0-547b5469dc-42kll"
Nov 02 12:48:57.743819 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:57.743762    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-b271f0-dockercfg-xxbb5\""
Nov 02 12:48:57.744052 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:57.744031    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-b271f0\""
Nov 02 12:48:57.745706 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:57.745682    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-b271f0-547b5469dc-42kll"]
Nov 02 12:48:57.819664 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:57.817544    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-e93a4b-openshift-default-7f8c56bf9c-4pjxc"
Nov 02 12:48:57.873348 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:57.873309    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wplfm\" (UniqueName: \"kubernetes.io/projected/32309c5a-744b-4225-b1c1-1b92704e4c2a-kube-api-access-wplfm\") pod \"payload-pre-processing-e2e-praxis-a-b271f0-547b5469dc-42kll\" (UID: \"32309c5a-744b-4225-b1c1-1b92704e4c2a\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b271f0-547b5469dc-42kll"
Nov 02 12:48:57.873840 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:57.873814    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32309c5a-744b-4225-b1c1-1b92704e4c2a-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-b271f0-547b5469dc-42kll\" (UID: \"32309c5a-744b-4225-b1c1-1b92704e4c2a\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b271f0-547b5469dc-42kll"
Nov 02 12:48:57.973758 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:57.973722    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b55da6b-89cb-46d6-a7a0-3a09e7d7d17e" path="/var/lib/kubelet/pods/5b55da6b-89cb-46d6-a7a0-3a09e7d7d17e/volumes"
Nov 02 12:48:57.974462 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:57.974422    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-b271f0-6966d4c887-jthwp"]
Nov 02 12:48:57.977350 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:57.977323    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wplfm\" (UniqueName: \"kubernetes.io/projected/32309c5a-744b-4225-b1c1-1b92704e4c2a-kube-api-access-wplfm\") pod \"payload-pre-processing-e2e-praxis-a-b271f0-547b5469dc-42kll\" (UID: \"32309c5a-744b-4225-b1c1-1b92704e4c2a\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b271f0-547b5469dc-42kll"
Nov 02 12:48:57.977956 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:57.977931    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32309c5a-744b-4225-b1c1-1b92704e4c2a-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-b271f0-547b5469dc-42kll\" (UID: \"32309c5a-744b-4225-b1c1-1b92704e4c2a\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b271f0-547b5469dc-42kll"
Nov 02 12:48:57.983268 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:57.983033    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32309c5a-744b-4225-b1c1-1b92704e4c2a-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-b271f0-547b5469dc-42kll\" (UID: \"32309c5a-744b-4225-b1c1-1b92704e4c2a\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b271f0-547b5469dc-42kll"
Nov 02 12:48:58.033556 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:58.033238    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wplfm\" (UniqueName: \"kubernetes.io/projected/32309c5a-744b-4225-b1c1-1b92704e4c2a-kube-api-access-wplfm\") pod \"payload-pre-processing-e2e-praxis-a-b271f0-547b5469dc-42kll\" (UID: \"32309c5a-744b-4225-b1c1-1b92704e4c2a\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b271f0-547b5469dc-42kll"
Nov 02 12:48:58.048414 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:58.043978    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b271f0-547b5469dc-42kll"
Nov 02 12:48:58.056691 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:58.053911    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-b271f0-6966d4c887-jthwp"]
Nov 02 12:48:58.056691 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:58.054068    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-b271f0-6966d4c887-jthwp"
Nov 02 12:48:58.186269 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:58.186226    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dwlm\" (UniqueName: \"kubernetes.io/projected/599fd828-375f-4169-9e81-cda2e36228a3-kube-api-access-4dwlm\") pod \"payload-processing-e2e-praxis-a-b271f0-6966d4c887-jthwp\" (UID: \"599fd828-375f-4169-9e81-cda2e36228a3\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-b271f0-6966d4c887-jthwp"
Nov 02 12:48:58.186448 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:58.186362    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/599fd828-375f-4169-9e81-cda2e36228a3-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-b271f0-6966d4c887-jthwp\" (UID: \"599fd828-375f-4169-9e81-cda2e36228a3\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-b271f0-6966d4c887-jthwp"
Nov 02 12:48:58.287339 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:58.287244    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dwlm\" (UniqueName: \"kubernetes.io/projected/599fd828-375f-4169-9e81-cda2e36228a3-kube-api-access-4dwlm\") pod \"payload-processing-e2e-praxis-a-b271f0-6966d4c887-jthwp\" (UID: \"599fd828-375f-4169-9e81-cda2e36228a3\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-b271f0-6966d4c887-jthwp"
Nov 02 12:48:58.287339 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:58.287297    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/599fd828-375f-4169-9e81-cda2e36228a3-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-b271f0-6966d4c887-jthwp\" (UID: \"599fd828-375f-4169-9e81-cda2e36228a3\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-b271f0-6966d4c887-jthwp"
Nov 02 12:48:58.288053 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:58.288034    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/599fd828-375f-4169-9e81-cda2e36228a3-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-b271f0-6966d4c887-jthwp\" (UID: \"599fd828-375f-4169-9e81-cda2e36228a3\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-b271f0-6966d4c887-jthwp"
Nov 02 12:48:58.312301 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:58.312274    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dwlm\" (UniqueName: \"kubernetes.io/projected/599fd828-375f-4169-9e81-cda2e36228a3-kube-api-access-4dwlm\") pod \"payload-processing-e2e-praxis-a-b271f0-6966d4c887-jthwp\" (UID: \"599fd828-375f-4169-9e81-cda2e36228a3\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-b271f0-6966d4c887-jthwp"
Nov 02 12:48:58.357729 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:58.357697    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-e93a4b-openshift-default-7f8c56bf9c-4pjxc"]
Nov 02 12:48:58.369278 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:48:58.369246    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod624a0494_692e_4864_810e_42ba5306a5f0.slice/crio-d5b53ecdf084c62a078df6fffe2b26241e0519c7bf1d79f91df2b2f423722f99 WatchSource:0}: Error finding container d5b53ecdf084c62a078df6fffe2b26241e0519c7bf1d79f91df2b2f423722f99: Status 404 returned error can't find the container with id d5b53ecdf084c62a078df6fffe2b26241e0519c7bf1d79f91df2b2f423722f99
Nov 02 12:48:58.371880 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:58.371844    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:48:58.371963 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:58.371915    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:48:58.371963 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:58.371946    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:48:58.378323 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:58.378302    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-b271f0-6966d4c887-jthwp"
Nov 02 12:48:58.399715 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:58.399685    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-b271f0-547b5469dc-42kll"]
Nov 02 12:48:58.706442 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:58.706391    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-b271f0-6966d4c887-jthwp"]
Nov 02 12:48:58.708467 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:48:58.708433    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod599fd828_375f_4169_9e81_cda2e36228a3.slice/crio-251d18f2f248c13737dc2ac258ebdca65bfa7baa997903ed1a4a097f94fc2021 WatchSource:0}: Error finding container 251d18f2f248c13737dc2ac258ebdca65bfa7baa997903ed1a4a097f94fc2021: Status 404 returned error can't find the container with id 251d18f2f248c13737dc2ac258ebdca65bfa7baa997903ed1a4a097f94fc2021
Nov 02 12:48:59.307141 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:59.307097    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b271f0-547b5469dc-42kll" event={"ID":"32309c5a-744b-4225-b1c1-1b92704e4c2a","Type":"ContainerStarted","Data":"b213bfe5c5bc908ee1729837867ddfa08ae9f63fb7ec3f0acde24226cdfc9fc7"}
Nov 02 12:48:59.307319 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:59.307153    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b271f0-547b5469dc-42kll" event={"ID":"32309c5a-744b-4225-b1c1-1b92704e4c2a","Type":"ContainerStarted","Data":"1154da98a1d1a1c9a55498d111c1e3d93d49623034832da51a3bb05929b53c1c"}
Nov 02 12:48:59.307319 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:59.307297    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b271f0-547b5469dc-42kll"
Nov 02 12:48:59.308877 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:59.308848    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-e93a4b-openshift-default-7f8c56bf9c-4pjxc" event={"ID":"624a0494-692e-4864-810e-42ba5306a5f0","Type":"ContainerStarted","Data":"55f6f8b469fdf1e3896bdc3ce5359712e1a572fe5b7d996b9744071c52c96126"}
Nov 02 12:48:59.309015 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:59.308903    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-e93a4b-openshift-default-7f8c56bf9c-4pjxc" event={"ID":"624a0494-692e-4864-810e-42ba5306a5f0","Type":"ContainerStarted","Data":"d5b53ecdf084c62a078df6fffe2b26241e0519c7bf1d79f91df2b2f423722f99"}
Nov 02 12:48:59.310707 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:59.310685    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-b271f0-6966d4c887-jthwp" event={"ID":"599fd828-375f-4169-9e81-cda2e36228a3","Type":"ContainerStarted","Data":"4d640c29cff4a25c8c77dc34d5b4bc5a98b1ed24ab198fe9f07087be864c287a"}
Nov 02 12:48:59.310827 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:59.310725    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-b271f0-6966d4c887-jthwp" event={"ID":"599fd828-375f-4169-9e81-cda2e36228a3","Type":"ContainerStarted","Data":"251d18f2f248c13737dc2ac258ebdca65bfa7baa997903ed1a4a097f94fc2021"}
Nov 02 12:48:59.310827 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:59.310772    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-b271f0-6966d4c887-jthwp"
Nov 02 12:48:59.352911 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:59.352849    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b271f0-547b5469dc-42kll" podStartSLOduration=2.352835504 podStartE2EDuration="2.352835504s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:48:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:48:59.350566611 +0000 UTC m=+1335.605965729" watchObservedRunningTime="2026-11-02 12:48:59.352835504 +0000 UTC m=+1335.608234599"
Nov 02 12:48:59.398846 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:59.398789    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-b271f0-6966d4c887-jthwp" podStartSLOduration=2.398771017 podStartE2EDuration="2.398771017s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:48:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:48:59.395728388 +0000 UTC m=+1335.651127483" watchObservedRunningTime="2026-11-02 12:48:59.398771017 +0000 UTC m=+1335.654170115"
Nov 02 12:48:59.446225 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:59.446180    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-e93a4b-openshift-default-7f8c56bf9c-4pjxc" podStartSLOduration=2.446167813 podStartE2EDuration="2.446167813s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:48:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:48:59.444206989 +0000 UTC m=+1335.699606084" watchObservedRunningTime="2026-11-02 12:48:59.446167813 +0000 UTC m=+1335.701566907"
Nov 02 12:48:59.818102 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:59.818069    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-e93a4b-openshift-default-7f8c56bf9c-4pjxc"
Nov 02 12:48:59.823890 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:48:59.823866    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-e93a4b-openshift-default-7f8c56bf9c-4pjxc"
Nov 02 12:49:00.317174 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:00.317144    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-e93a4b-openshift-default-7f8c56bf9c-4pjxc"
Nov 02 12:49:00.318088 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:00.318063    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-e93a4b-openshift-default-7f8c56bf9c-4pjxc"
Nov 02 12:49:00.912733 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:00.912693    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"]
Nov 02 12:49:01.087745 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:01.087712    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f7f44a-84dc986db4-q8cww"]
Nov 02 12:49:01.087967 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:01.087943    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f7f44a-84dc986db4-q8cww" podUID="61ae8a44-679a-4df1-abdd-3e5decbd2403" containerName="maas-api" containerID="cri-o://5ff6fb71b2273cf2ce0f4f10009dec40aeece72ffeaedf2238900850bd0b3227" gracePeriod=30
Nov 02 12:49:01.336027 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:01.335998    4559 generic.go:356] "Generic (PLEG): container finished" podID="61ae8a44-679a-4df1-abdd-3e5decbd2403" containerID="5ff6fb71b2273cf2ce0f4f10009dec40aeece72ffeaedf2238900850bd0b3227" exitCode=0
Nov 02 12:49:01.336210 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:01.336086    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f7f44a-84dc986db4-q8cww" event={"ID":"61ae8a44-679a-4df1-abdd-3e5decbd2403","Type":"ContainerDied","Data":"5ff6fb71b2273cf2ce0f4f10009dec40aeece72ffeaedf2238900850bd0b3227"}
Nov 02 12:49:01.371112 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:01.371080    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f7f44a-84dc986db4-q8cww"
Nov 02 12:49:01.497367 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:01.497280    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/61ae8a44-679a-4df1-abdd-3e5decbd2403-metrics-certs\" (UniqueName: \"kubernetes.io/secret/61ae8a44-679a-4df1-abdd-3e5decbd2403-metrics-certs\") pod \"61ae8a44-679a-4df1-abdd-3e5decbd2403\" (UID: \"61ae8a44-679a-4df1-abdd-3e5decbd2403\") "
Nov 02 12:49:01.497367 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:01.497353    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61ae8a44-679a-4df1-abdd-3e5decbd2403-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/61ae8a44-679a-4df1-abdd-3e5decbd2403-openshift-service-ca\") pod \"61ae8a44-679a-4df1-abdd-3e5decbd2403\" (UID: \"61ae8a44-679a-4df1-abdd-3e5decbd2403\") "
Nov 02 12:49:01.497619 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:01.497420    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/61ae8a44-679a-4df1-abdd-3e5decbd2403-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/61ae8a44-679a-4df1-abdd-3e5decbd2403-maas-api-tls\") pod \"61ae8a44-679a-4df1-abdd-3e5decbd2403\" (UID: \"61ae8a44-679a-4df1-abdd-3e5decbd2403\") "
Nov 02 12:49:01.497619 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:01.497444    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61ae8a44-679a-4df1-abdd-3e5decbd2403-kube-api-access-kq7zj\" (UniqueName: \"kubernetes.io/projected/61ae8a44-679a-4df1-abdd-3e5decbd2403-kube-api-access-kq7zj\") pod \"61ae8a44-679a-4df1-abdd-3e5decbd2403\" (UID: \"61ae8a44-679a-4df1-abdd-3e5decbd2403\") "
Nov 02 12:49:01.497776 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:01.497724    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61ae8a44-679a-4df1-abdd-3e5decbd2403-openshift-service-ca" pod "61ae8a44-679a-4df1-abdd-3e5decbd2403" (UID: "61ae8a44-679a-4df1-abdd-3e5decbd2403"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:49:01.498369 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:01.498294    4559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/61ae8a44-679a-4df1-abdd-3e5decbd2403-openshift-service-ca\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:49:01.499603 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:01.499578    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61ae8a44-679a-4df1-abdd-3e5decbd2403-metrics-certs" pod "61ae8a44-679a-4df1-abdd-3e5decbd2403" (UID: "61ae8a44-679a-4df1-abdd-3e5decbd2403"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:49:01.500043 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:01.500017    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61ae8a44-679a-4df1-abdd-3e5decbd2403-maas-api-tls" pod "61ae8a44-679a-4df1-abdd-3e5decbd2403" (UID: "61ae8a44-679a-4df1-abdd-3e5decbd2403"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:49:01.500131 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:01.500043    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61ae8a44-679a-4df1-abdd-3e5decbd2403-kube-api-access-kq7zj" pod "61ae8a44-679a-4df1-abdd-3e5decbd2403" (UID: "61ae8a44-679a-4df1-abdd-3e5decbd2403"). InnerVolumeSpecName "kube-api-access-kq7zj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:49:01.617811 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:01.599609    4559 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/61ae8a44-679a-4df1-abdd-3e5decbd2403-metrics-certs\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:49:01.618063 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:01.618038    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/61ae8a44-679a-4df1-abdd-3e5decbd2403-maas-api-tls\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:49:01.618341 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:01.618323    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kq7zj\" (UniqueName: \"kubernetes.io/projected/61ae8a44-679a-4df1-abdd-3e5decbd2403-kube-api-access-kq7zj\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:49:01.923695 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:01.923660    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-299e74-865787d6ff-c8lt4"]
Nov 02 12:49:01.960525 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:01.956925    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-299e74-7c7d6bb879-bkw2f"]
Nov 02 12:49:02.381815 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:02.381739    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f7f44a-84dc986db4-q8cww"
Nov 02 12:49:02.382750 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:02.382628    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f7f44a-84dc986db4-q8cww" event={"ID":"61ae8a44-679a-4df1-abdd-3e5decbd2403","Type":"ContainerDied","Data":"372861f1d00c598b8593cff8094609103dc185c1a0d046621a454ee28003e9d5"}
Nov 02 12:49:02.383094 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:02.382985    4559 scope.go:122] "RemoveContainer" containerID="5ff6fb71b2273cf2ce0f4f10009dec40aeece72ffeaedf2238900850bd0b3227"
Nov 02 12:49:02.477668 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:02.474279    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f7f44a-84dc986db4-q8cww"]
Nov 02 12:49:02.480465 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:02.480294    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f7f44a-84dc986db4-q8cww"]
Nov 02 12:49:03.309958 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:03.309921    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-d6d314-openshift-default-5b99b69565-2kkgt"]
Nov 02 12:49:03.310830 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:03.310808    4559 cpu_manager.go:414] "containerMap: removing container" podUID="61ae8a44-679a-4df1-abdd-3e5decbd2403" containerName="maas-api"
Nov 02 12:49:03.310830 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:03.310830    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61ae8a44-679a-4df1-abdd-3e5decbd2403" containerName="maas-api"
Nov 02 12:49:03.311030 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:03.311014    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="61ae8a44-679a-4df1-abdd-3e5decbd2403" containerName="maas-api"
Nov 02 12:49:03.316089 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:03.316062    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-d6d314-openshift-default-5b99b69565-2kkgt"
Nov 02 12:49:03.367008 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:03.366975    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-d6d314-openshift-default-dockercfg-4n6vn\""
Nov 02 12:49:03.367767 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:03.367741    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-d6d314-openshift-default-5b99b69565-2kkgt"]
Nov 02 12:49:03.456662 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:03.456472    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ac6508c-24db-40f7-a14c-0964dac7768b-istio-token\") pod \"e2e-praxis-a-d6d314-openshift-default-5b99b69565-2kkgt\" (UID: \"4ac6508c-24db-40f7-a14c-0964dac7768b\") " pod="openshift-ingress/e2e-praxis-a-d6d314-openshift-default-5b99b69565-2kkgt"
Nov 02 12:49:03.456662 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:03.456555    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2qfg\" (UniqueName: \"kubernetes.io/projected/4ac6508c-24db-40f7-a14c-0964dac7768b-kube-api-access-t2qfg\") pod \"e2e-praxis-a-d6d314-openshift-default-5b99b69565-2kkgt\" (UID: \"4ac6508c-24db-40f7-a14c-0964dac7768b\") " pod="openshift-ingress/e2e-praxis-a-d6d314-openshift-default-5b99b69565-2kkgt"
Nov 02 12:49:03.456923 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:03.456724    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ac6508c-24db-40f7-a14c-0964dac7768b-istio-envoy\") pod \"e2e-praxis-a-d6d314-openshift-default-5b99b69565-2kkgt\" (UID: \"4ac6508c-24db-40f7-a14c-0964dac7768b\") " pod="openshift-ingress/e2e-praxis-a-d6d314-openshift-default-5b99b69565-2kkgt"
Nov 02 12:49:03.456923 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:03.456792    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ac6508c-24db-40f7-a14c-0964dac7768b-credential-socket\") pod \"e2e-praxis-a-d6d314-openshift-default-5b99b69565-2kkgt\" (UID: \"4ac6508c-24db-40f7-a14c-0964dac7768b\") " pod="openshift-ingress/e2e-praxis-a-d6d314-openshift-default-5b99b69565-2kkgt"
Nov 02 12:49:03.456923 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:03.456847    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ac6508c-24db-40f7-a14c-0964dac7768b-workload-socket\") pod \"e2e-praxis-a-d6d314-openshift-default-5b99b69565-2kkgt\" (UID: \"4ac6508c-24db-40f7-a14c-0964dac7768b\") " pod="openshift-ingress/e2e-praxis-a-d6d314-openshift-default-5b99b69565-2kkgt"
Nov 02 12:49:03.456923 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:03.456877    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ac6508c-24db-40f7-a14c-0964dac7768b-istio-podinfo\") pod \"e2e-praxis-a-d6d314-openshift-default-5b99b69565-2kkgt\" (UID: \"4ac6508c-24db-40f7-a14c-0964dac7768b\") " pod="openshift-ingress/e2e-praxis-a-d6d314-openshift-default-5b99b69565-2kkgt"
Nov 02 12:49:03.457150 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:03.456993    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ac6508c-24db-40f7-a14c-0964dac7768b-istiod-ca-cert\") pod \"e2e-praxis-a-d6d314-openshift-default-5b99b69565-2kkgt\" (UID: \"4ac6508c-24db-40f7-a14c-0964dac7768b\") " pod="openshift-ingress/e2e-praxis-a-d6d314-openshift-default-5b99b69565-2kkgt"
Nov 02 12:49:03.457150 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:03.457035    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ac6508c-24db-40f7-a14c-0964dac7768b-workload-certs\") pod \"e2e-praxis-a-d6d314-openshift-default-5b99b69565-2kkgt\" (UID: \"4ac6508c-24db-40f7-a14c-0964dac7768b\") " pod="openshift-ingress/e2e-praxis-a-d6d314-openshift-default-5b99b69565-2kkgt"
Nov 02 12:49:03.457150 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:03.457114    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ac6508c-24db-40f7-a14c-0964dac7768b-istio-data\") pod \"e2e-praxis-a-d6d314-openshift-default-5b99b69565-2kkgt\" (UID: \"4ac6508c-24db-40f7-a14c-0964dac7768b\") " pod="openshift-ingress/e2e-praxis-a-d6d314-openshift-default-5b99b69565-2kkgt"
Nov 02 12:49:03.561711 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:03.558522    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ac6508c-24db-40f7-a14c-0964dac7768b-istiod-ca-cert\") pod \"e2e-praxis-a-d6d314-openshift-default-5b99b69565-2kkgt\" (UID: \"4ac6508c-24db-40f7-a14c-0964dac7768b\") " pod="openshift-ingress/e2e-praxis-a-d6d314-openshift-default-5b99b69565-2kkgt"
Nov 02 12:49:03.561711 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:03.558579    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ac6508c-24db-40f7-a14c-0964dac7768b-workload-certs\") pod \"e2e-praxis-a-d6d314-openshift-default-5b99b69565-2kkgt\" (UID: \"4ac6508c-24db-40f7-a14c-0964dac7768b\") " pod="openshift-ingress/e2e-praxis-a-d6d314-openshift-default-5b99b69565-2kkgt"
Nov 02 12:49:03.561711 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:03.558670    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ac6508c-24db-40f7-a14c-0964dac7768b-istio-data\") pod \"e2e-praxis-a-d6d314-openshift-default-5b99b69565-2kkgt\" (UID: \"4ac6508c-24db-40f7-a14c-0964dac7768b\") " pod="openshift-ingress/e2e-praxis-a-d6d314-openshift-default-5b99b69565-2kkgt"
Nov 02 12:49:03.561711 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:03.558727    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ac6508c-24db-40f7-a14c-0964dac7768b-istio-token\") pod \"e2e-praxis-a-d6d314-openshift-default-5b99b69565-2kkgt\" (UID: \"4ac6508c-24db-40f7-a14c-0964dac7768b\") " pod="openshift-ingress/e2e-praxis-a-d6d314-openshift-default-5b99b69565-2kkgt"
Nov 02 12:49:03.561711 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:03.558779    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2qfg\" (UniqueName: \"kubernetes.io/projected/4ac6508c-24db-40f7-a14c-0964dac7768b-kube-api-access-t2qfg\") pod \"e2e-praxis-a-d6d314-openshift-default-5b99b69565-2kkgt\" (UID: \"4ac6508c-24db-40f7-a14c-0964dac7768b\") " pod="openshift-ingress/e2e-praxis-a-d6d314-openshift-default-5b99b69565-2kkgt"
Nov 02 12:49:03.561711 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:03.558885    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ac6508c-24db-40f7-a14c-0964dac7768b-istio-envoy\") pod \"e2e-praxis-a-d6d314-openshift-default-5b99b69565-2kkgt\" (UID: \"4ac6508c-24db-40f7-a14c-0964dac7768b\") " pod="openshift-ingress/e2e-praxis-a-d6d314-openshift-default-5b99b69565-2kkgt"
Nov 02 12:49:03.561711 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:03.558944    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ac6508c-24db-40f7-a14c-0964dac7768b-credential-socket\") pod \"e2e-praxis-a-d6d314-openshift-default-5b99b69565-2kkgt\" (UID: \"4ac6508c-24db-40f7-a14c-0964dac7768b\") " pod="openshift-ingress/e2e-praxis-a-d6d314-openshift-default-5b99b69565-2kkgt"
Nov 02 12:49:03.561711 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:03.558988    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ac6508c-24db-40f7-a14c-0964dac7768b-workload-socket\") pod \"e2e-praxis-a-d6d314-openshift-default-5b99b69565-2kkgt\" (UID: \"4ac6508c-24db-40f7-a14c-0964dac7768b\") " pod="openshift-ingress/e2e-praxis-a-d6d314-openshift-default-5b99b69565-2kkgt"
Nov 02 12:49:03.561711 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:03.559015    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ac6508c-24db-40f7-a14c-0964dac7768b-istio-podinfo\") pod \"e2e-praxis-a-d6d314-openshift-default-5b99b69565-2kkgt\" (UID: \"4ac6508c-24db-40f7-a14c-0964dac7768b\") " pod="openshift-ingress/e2e-praxis-a-d6d314-openshift-default-5b99b69565-2kkgt"
Nov 02 12:49:03.561711 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:03.560933    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ac6508c-24db-40f7-a14c-0964dac7768b-istiod-ca-cert\") pod \"e2e-praxis-a-d6d314-openshift-default-5b99b69565-2kkgt\" (UID: \"4ac6508c-24db-40f7-a14c-0964dac7768b\") " pod="openshift-ingress/e2e-praxis-a-d6d314-openshift-default-5b99b69565-2kkgt"
Nov 02 12:49:03.561711 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:03.561218    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ac6508c-24db-40f7-a14c-0964dac7768b-workload-certs\") pod \"e2e-praxis-a-d6d314-openshift-default-5b99b69565-2kkgt\" (UID: \"4ac6508c-24db-40f7-a14c-0964dac7768b\") " pod="openshift-ingress/e2e-praxis-a-d6d314-openshift-default-5b99b69565-2kkgt"
Nov 02 12:49:03.561711 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:03.561431    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ac6508c-24db-40f7-a14c-0964dac7768b-istio-data\") pod \"e2e-praxis-a-d6d314-openshift-default-5b99b69565-2kkgt\" (UID: \"4ac6508c-24db-40f7-a14c-0964dac7768b\") " pod="openshift-ingress/e2e-praxis-a-d6d314-openshift-default-5b99b69565-2kkgt"
Nov 02 12:49:03.580666 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:03.577127    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ac6508c-24db-40f7-a14c-0964dac7768b-credential-socket\") pod \"e2e-praxis-a-d6d314-openshift-default-5b99b69565-2kkgt\" (UID: \"4ac6508c-24db-40f7-a14c-0964dac7768b\") " pod="openshift-ingress/e2e-praxis-a-d6d314-openshift-default-5b99b69565-2kkgt"
Nov 02 12:49:03.580666 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:03.577873    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ac6508c-24db-40f7-a14c-0964dac7768b-workload-socket\") pod \"e2e-praxis-a-d6d314-openshift-default-5b99b69565-2kkgt\" (UID: \"4ac6508c-24db-40f7-a14c-0964dac7768b\") " pod="openshift-ingress/e2e-praxis-a-d6d314-openshift-default-5b99b69565-2kkgt"
Nov 02 12:49:03.593498 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:03.593469    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ac6508c-24db-40f7-a14c-0964dac7768b-istio-podinfo\") pod \"e2e-praxis-a-d6d314-openshift-default-5b99b69565-2kkgt\" (UID: \"4ac6508c-24db-40f7-a14c-0964dac7768b\") " pod="openshift-ingress/e2e-praxis-a-d6d314-openshift-default-5b99b69565-2kkgt"
Nov 02 12:49:03.618320 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:03.618291    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2qfg\" (UniqueName: \"kubernetes.io/projected/4ac6508c-24db-40f7-a14c-0964dac7768b-kube-api-access-t2qfg\") pod \"e2e-praxis-a-d6d314-openshift-default-5b99b69565-2kkgt\" (UID: \"4ac6508c-24db-40f7-a14c-0964dac7768b\") " pod="openshift-ingress/e2e-praxis-a-d6d314-openshift-default-5b99b69565-2kkgt"
Nov 02 12:49:03.622661 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:03.618965    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ac6508c-24db-40f7-a14c-0964dac7768b-istio-envoy\") pod \"e2e-praxis-a-d6d314-openshift-default-5b99b69565-2kkgt\" (UID: \"4ac6508c-24db-40f7-a14c-0964dac7768b\") " pod="openshift-ingress/e2e-praxis-a-d6d314-openshift-default-5b99b69565-2kkgt"
Nov 02 12:49:03.633598 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:03.633570    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ac6508c-24db-40f7-a14c-0964dac7768b-istio-token\") pod \"e2e-praxis-a-d6d314-openshift-default-5b99b69565-2kkgt\" (UID: \"4ac6508c-24db-40f7-a14c-0964dac7768b\") " pod="openshift-ingress/e2e-praxis-a-d6d314-openshift-default-5b99b69565-2kkgt"
Nov 02 12:49:03.636398 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:03.636376    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-d6d314-openshift-default-5b99b69565-2kkgt"
Nov 02 12:49:03.906818 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:03.906740    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61ae8a44-679a-4df1-abdd-3e5decbd2403" path="/var/lib/kubelet/pods/61ae8a44-679a-4df1-abdd-3e5decbd2403/volumes"
Nov 02 12:49:04.058857 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:49:04.058824    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ac6508c_24db_40f7_a14c_0964dac7768b.slice/crio-028afb01e37fa5cd0f88b256ac66eb54fc7c15aef627bda2d9dfff456f109e1e WatchSource:0}: Error finding container 028afb01e37fa5cd0f88b256ac66eb54fc7c15aef627bda2d9dfff456f109e1e: Status 404 returned error can't find the container with id 028afb01e37fa5cd0f88b256ac66eb54fc7c15aef627bda2d9dfff456f109e1e
Nov 02 12:49:04.059176 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:04.059158    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-d6d314-openshift-default-5b99b69565-2kkgt"]
Nov 02 12:49:04.061772 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:04.061735    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:49:04.061871 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:04.061804    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:49:04.061871 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:04.061833    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:49:04.401960 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:04.401926    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-d6d314-openshift-default-5b99b69565-2kkgt" event={"ID":"4ac6508c-24db-40f7-a14c-0964dac7768b","Type":"ContainerStarted","Data":"41768043411c52609ed00008bc4c1517796cd57983acc45534761dacdf6cf1d0"}
Nov 02 12:49:04.402463 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:04.401986    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-d6d314-openshift-default-5b99b69565-2kkgt" event={"ID":"4ac6508c-24db-40f7-a14c-0964dac7768b","Type":"ContainerStarted","Data":"028afb01e37fa5cd0f88b256ac66eb54fc7c15aef627bda2d9dfff456f109e1e"}
Nov 02 12:49:04.456991 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:04.456940    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-d6d314-openshift-default-5b99b69565-2kkgt" podStartSLOduration=1.456924167 podStartE2EDuration="1.456924167s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:49:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:49:04.454900022 +0000 UTC m=+1340.710299118" watchObservedRunningTime="2026-11-02 12:49:04.456924167 +0000 UTC m=+1340.712323261"
Nov 02 12:49:04.618433 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:04.618399    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-e93a4b-bd984bc95-7fwhg"]
Nov 02 12:49:04.622694 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:04.622675    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e93a4b-bd984bc95-7fwhg"
Nov 02 12:49:04.636671 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:04.636649    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-d6d314-openshift-default-5b99b69565-2kkgt"
Nov 02 12:49:04.640383 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:04.640360    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-e93a4b\""
Nov 02 12:49:04.640670 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:04.640620    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-e93a4b-dockercfg-5t6jc\""
Nov 02 12:49:04.643203 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:04.643179    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-e93a4b-bd984bc95-7fwhg"]
Nov 02 12:49:04.726672 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:04.707900    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gg5pg\" (UniqueName: \"kubernetes.io/projected/fbce1ffb-dfe6-47b1-bd72-d86787cc1a4b-kube-api-access-gg5pg\") pod \"payload-pre-processing-e2e-praxis-b-e93a4b-bd984bc95-7fwhg\" (UID: \"fbce1ffb-dfe6-47b1-bd72-d86787cc1a4b\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e93a4b-bd984bc95-7fwhg"
Nov 02 12:49:04.726672 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:04.707991    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbce1ffb-dfe6-47b1-bd72-d86787cc1a4b-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-e93a4b-bd984bc95-7fwhg\" (UID: \"fbce1ffb-dfe6-47b1-bd72-d86787cc1a4b\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e93a4b-bd984bc95-7fwhg"
Nov 02 12:49:04.756534 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:04.754376    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-e93a4b-6947c6444f-8cc8j"]
Nov 02 12:49:04.760575 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:04.760228    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-e93a4b-6947c6444f-8cc8j"
Nov 02 12:49:04.785959 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:04.785902    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-e93a4b-6947c6444f-8cc8j"]
Nov 02 12:49:04.878399 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:04.858564    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbce1ffb-dfe6-47b1-bd72-d86787cc1a4b-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-e93a4b-bd984bc95-7fwhg\" (UID: \"fbce1ffb-dfe6-47b1-bd72-d86787cc1a4b\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e93a4b-bd984bc95-7fwhg"
Nov 02 12:49:04.878907 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:04.878867    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5sx5v\" (UniqueName: \"kubernetes.io/projected/a7b7d167-6e3d-4632-822e-597f0578a3b0-kube-api-access-5sx5v\") pod \"payload-processing-e2e-praxis-b-e93a4b-6947c6444f-8cc8j\" (UID: \"a7b7d167-6e3d-4632-822e-597f0578a3b0\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-e93a4b-6947c6444f-8cc8j"
Nov 02 12:49:04.879373 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:04.879222    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7b7d167-6e3d-4632-822e-597f0578a3b0-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-e93a4b-6947c6444f-8cc8j\" (UID: \"a7b7d167-6e3d-4632-822e-597f0578a3b0\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-e93a4b-6947c6444f-8cc8j"
Nov 02 12:49:04.879592 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:04.879573    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gg5pg\" (UniqueName: \"kubernetes.io/projected/fbce1ffb-dfe6-47b1-bd72-d86787cc1a4b-kube-api-access-gg5pg\") pod \"payload-pre-processing-e2e-praxis-b-e93a4b-bd984bc95-7fwhg\" (UID: \"fbce1ffb-dfe6-47b1-bd72-d86787cc1a4b\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e93a4b-bd984bc95-7fwhg"
Nov 02 12:49:04.882452 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:04.882427    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbce1ffb-dfe6-47b1-bd72-d86787cc1a4b-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-e93a4b-bd984bc95-7fwhg\" (UID: \"fbce1ffb-dfe6-47b1-bd72-d86787cc1a4b\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e93a4b-bd984bc95-7fwhg"
Nov 02 12:49:04.929769 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:04.927537    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gg5pg\" (UniqueName: \"kubernetes.io/projected/fbce1ffb-dfe6-47b1-bd72-d86787cc1a4b-kube-api-access-gg5pg\") pod \"payload-pre-processing-e2e-praxis-b-e93a4b-bd984bc95-7fwhg\" (UID: \"fbce1ffb-dfe6-47b1-bd72-d86787cc1a4b\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e93a4b-bd984bc95-7fwhg"
Nov 02 12:49:04.953587 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:04.953552    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e93a4b-bd984bc95-7fwhg"
Nov 02 12:49:04.982518 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:04.980971    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7b7d167-6e3d-4632-822e-597f0578a3b0-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-e93a4b-6947c6444f-8cc8j\" (UID: \"a7b7d167-6e3d-4632-822e-597f0578a3b0\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-e93a4b-6947c6444f-8cc8j"
Nov 02 12:49:04.982518 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:04.981229    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5sx5v\" (UniqueName: \"kubernetes.io/projected/a7b7d167-6e3d-4632-822e-597f0578a3b0-kube-api-access-5sx5v\") pod \"payload-processing-e2e-praxis-b-e93a4b-6947c6444f-8cc8j\" (UID: \"a7b7d167-6e3d-4632-822e-597f0578a3b0\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-e93a4b-6947c6444f-8cc8j"
Nov 02 12:49:04.982518 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:04.982432    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7b7d167-6e3d-4632-822e-597f0578a3b0-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-e93a4b-6947c6444f-8cc8j\" (UID: \"a7b7d167-6e3d-4632-822e-597f0578a3b0\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-e93a4b-6947c6444f-8cc8j"
Nov 02 12:49:05.059681 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:05.056586    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5sx5v\" (UniqueName: \"kubernetes.io/projected/a7b7d167-6e3d-4632-822e-597f0578a3b0-kube-api-access-5sx5v\") pod \"payload-processing-e2e-praxis-b-e93a4b-6947c6444f-8cc8j\" (UID: \"a7b7d167-6e3d-4632-822e-597f0578a3b0\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-e93a4b-6947c6444f-8cc8j"
Nov 02 12:49:05.099174 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:05.098886    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-e93a4b-6947c6444f-8cc8j"
Nov 02 12:49:05.386047 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:49:05.386013    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbce1ffb_dfe6_47b1_bd72_d86787cc1a4b.slice/crio-022bbaed823e9b15b3f95eda0f43392fcf0b4a71ccbf5a136c6dc048a49ad71b WatchSource:0}: Error finding container 022bbaed823e9b15b3f95eda0f43392fcf0b4a71ccbf5a136c6dc048a49ad71b: Status 404 returned error can't find the container with id 022bbaed823e9b15b3f95eda0f43392fcf0b4a71ccbf5a136c6dc048a49ad71b
Nov 02 12:49:05.386387 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:05.386366    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-e93a4b-bd984bc95-7fwhg"]
Nov 02 12:49:05.409802 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:05.409774    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e93a4b-bd984bc95-7fwhg" event={"ID":"fbce1ffb-dfe6-47b1-bd72-d86787cc1a4b","Type":"ContainerStarted","Data":"022bbaed823e9b15b3f95eda0f43392fcf0b4a71ccbf5a136c6dc048a49ad71b"}
Nov 02 12:49:05.432809 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:49:05.432783    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7b7d167_6e3d_4632_822e_597f0578a3b0.slice/crio-8a4ab8fe54ea0ad650395630f2d8fe3e0faea0c5f7b8eed383d61d4625a8033e WatchSource:0}: Error finding container 8a4ab8fe54ea0ad650395630f2d8fe3e0faea0c5f7b8eed383d61d4625a8033e: Status 404 returned error can't find the container with id 8a4ab8fe54ea0ad650395630f2d8fe3e0faea0c5f7b8eed383d61d4625a8033e
Nov 02 12:49:05.434083 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:05.434062    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-e93a4b-6947c6444f-8cc8j"]
Nov 02 12:49:05.643366 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:05.643282    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-d6d314-openshift-default-5b99b69565-2kkgt"
Nov 02 12:49:06.417155 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:06.417112    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-e93a4b-6947c6444f-8cc8j" event={"ID":"a7b7d167-6e3d-4632-822e-597f0578a3b0","Type":"ContainerStarted","Data":"781a4b8347a3ba231d1cd5d969497d10c81ce681b569f179f1714c9225989539"}
Nov 02 12:49:06.417633 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:06.417183    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-e93a4b-6947c6444f-8cc8j"
Nov 02 12:49:06.417633 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:06.417206    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-e93a4b-6947c6444f-8cc8j" event={"ID":"a7b7d167-6e3d-4632-822e-597f0578a3b0","Type":"ContainerStarted","Data":"8a4ab8fe54ea0ad650395630f2d8fe3e0faea0c5f7b8eed383d61d4625a8033e"}
Nov 02 12:49:06.418871 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:06.418844    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e93a4b-bd984bc95-7fwhg" event={"ID":"fbce1ffb-dfe6-47b1-bd72-d86787cc1a4b","Type":"ContainerStarted","Data":"a9bdff8f7e4e340e7050c063e9eed857152b2114f6d51bdfaef0643fc624f3ca"}
Nov 02 12:49:06.419088 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:06.419073    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-d6d314-openshift-default-5b99b69565-2kkgt"
Nov 02 12:49:06.419169 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:06.419121    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e93a4b-bd984bc95-7fwhg"
Nov 02 12:49:06.419990 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:06.419971    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-d6d314-openshift-default-5b99b69565-2kkgt"
Nov 02 12:49:06.460937 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:06.460894    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-e93a4b-6947c6444f-8cc8j" podStartSLOduration=2.460880365 podStartE2EDuration="2.460880365s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:49:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:49:06.459684373 +0000 UTC m=+1342.715083462" watchObservedRunningTime="2026-11-02 12:49:06.460880365 +0000 UTC m=+1342.716279460"
Nov 02 12:49:06.504577 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:06.504531    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e93a4b-bd984bc95-7fwhg" podStartSLOduration=2.50451774 podStartE2EDuration="2.50451774s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:49:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:49:06.501975837 +0000 UTC m=+1342.757374933" watchObservedRunningTime="2026-11-02 12:49:06.50451774 +0000 UTC m=+1342.759916834"
Nov 02 12:49:06.958370 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:06.948811    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-299e74-865787d6ff-c8lt4" podUID="17f700b6-f498-497e-a292-3ae243f65858" containerName="payload-processing" containerID="cri-o://2b171615ed74f01ddd8f91a6543e7981cae93a7e5de9d5368e648c7701cfcfeb" gracePeriod=25
Nov 02 12:49:06.958370 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:06.957797    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-299e74-7c7d6bb879-bkw2f" podUID="545d586e-9e3a-452a-8b0b-2775d5e49317" containerName="payload-pre-processing" containerID="cri-o://9c33c19e71901fcb230c538093285f9890a97fa4e8117e2f7f96da7ddb163ba5" gracePeriod=25
Nov 02 12:49:07.240400 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:07.240371    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-299e74-865787d6ff-c8lt4"
Nov 02 12:49:07.270551 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:07.270521    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-299e74-7c7d6bb879-bkw2f"
Nov 02 12:49:07.324434 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:07.324390    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17f700b6-f498-497e-a292-3ae243f65858-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17f700b6-f498-497e-a292-3ae243f65858-plugins-config-volume\") pod \"17f700b6-f498-497e-a292-3ae243f65858\" (UID: \"17f700b6-f498-497e-a292-3ae243f65858\") "
Nov 02 12:49:07.324603 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:07.324553    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/545d586e-9e3a-452a-8b0b-2775d5e49317-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/545d586e-9e3a-452a-8b0b-2775d5e49317-plugins-config-volume\") pod \"545d586e-9e3a-452a-8b0b-2775d5e49317\" (UID: \"545d586e-9e3a-452a-8b0b-2775d5e49317\") "
Nov 02 12:49:07.324715 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:07.324675    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/545d586e-9e3a-452a-8b0b-2775d5e49317-kube-api-access-hpbtc\" (UniqueName: \"kubernetes.io/projected/545d586e-9e3a-452a-8b0b-2775d5e49317-kube-api-access-hpbtc\") pod \"545d586e-9e3a-452a-8b0b-2775d5e49317\" (UID: \"545d586e-9e3a-452a-8b0b-2775d5e49317\") "
Nov 02 12:49:07.324876 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:07.324848    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17f700b6-f498-497e-a292-3ae243f65858-plugins-config-volume" pod "17f700b6-f498-497e-a292-3ae243f65858" (UID: "17f700b6-f498-497e-a292-3ae243f65858"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:49:07.324946 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:07.324887    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17f700b6-f498-497e-a292-3ae243f65858-kube-api-access-9x8sp\" (UniqueName: \"kubernetes.io/projected/17f700b6-f498-497e-a292-3ae243f65858-kube-api-access-9x8sp\") pod \"17f700b6-f498-497e-a292-3ae243f65858\" (UID: \"17f700b6-f498-497e-a292-3ae243f65858\") "
Nov 02 12:49:07.324946 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:07.324925    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/545d586e-9e3a-452a-8b0b-2775d5e49317-plugins-config-volume" pod "545d586e-9e3a-452a-8b0b-2775d5e49317" (UID: "545d586e-9e3a-452a-8b0b-2775d5e49317"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:49:07.325456 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:07.325412    4559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/545d586e-9e3a-452a-8b0b-2775d5e49317-plugins-config-volume\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:49:07.325456 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:07.325436    4559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17f700b6-f498-497e-a292-3ae243f65858-plugins-config-volume\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:49:07.327008 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:07.326975    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/545d586e-9e3a-452a-8b0b-2775d5e49317-kube-api-access-hpbtc" pod "545d586e-9e3a-452a-8b0b-2775d5e49317" (UID: "545d586e-9e3a-452a-8b0b-2775d5e49317"). InnerVolumeSpecName "kube-api-access-hpbtc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:49:07.327687 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:07.327665    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17f700b6-f498-497e-a292-3ae243f65858-kube-api-access-9x8sp" pod "17f700b6-f498-497e-a292-3ae243f65858" (UID: "17f700b6-f498-497e-a292-3ae243f65858"). InnerVolumeSpecName "kube-api-access-9x8sp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:49:07.426344 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:07.426312    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9x8sp\" (UniqueName: \"kubernetes.io/projected/17f700b6-f498-497e-a292-3ae243f65858-kube-api-access-9x8sp\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:49:07.426344 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:07.426342    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hpbtc\" (UniqueName: \"kubernetes.io/projected/545d586e-9e3a-452a-8b0b-2775d5e49317-kube-api-access-hpbtc\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:49:07.429005 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:07.428975    4559 generic.go:356] "Generic (PLEG): container finished" podID="545d586e-9e3a-452a-8b0b-2775d5e49317" containerID="9c33c19e71901fcb230c538093285f9890a97fa4e8117e2f7f96da7ddb163ba5" exitCode=0
Nov 02 12:49:07.429136 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:07.429028    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-299e74-7c7d6bb879-bkw2f"
Nov 02 12:49:07.429136 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:07.429064    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-299e74-7c7d6bb879-bkw2f" event={"ID":"545d586e-9e3a-452a-8b0b-2775d5e49317","Type":"ContainerDied","Data":"9c33c19e71901fcb230c538093285f9890a97fa4e8117e2f7f96da7ddb163ba5"}
Nov 02 12:49:07.429136 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:07.429130    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-299e74-7c7d6bb879-bkw2f" event={"ID":"545d586e-9e3a-452a-8b0b-2775d5e49317","Type":"ContainerDied","Data":"965387834e53617ca8764304592c70bd6f6d27c7f7547e53414c43056dee9c5b"}
Nov 02 12:49:07.429314 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:07.429161    4559 scope.go:122] "RemoveContainer" containerID="9c33c19e71901fcb230c538093285f9890a97fa4e8117e2f7f96da7ddb163ba5"
Nov 02 12:49:07.430427 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:07.430405    4559 generic.go:356] "Generic (PLEG): container finished" podID="17f700b6-f498-497e-a292-3ae243f65858" containerID="2b171615ed74f01ddd8f91a6543e7981cae93a7e5de9d5368e648c7701cfcfeb" exitCode=0
Nov 02 12:49:07.430513 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:07.430448    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-299e74-865787d6ff-c8lt4"
Nov 02 12:49:07.430573 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:07.430499    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-299e74-865787d6ff-c8lt4" event={"ID":"17f700b6-f498-497e-a292-3ae243f65858","Type":"ContainerDied","Data":"2b171615ed74f01ddd8f91a6543e7981cae93a7e5de9d5368e648c7701cfcfeb"}
Nov 02 12:49:07.430695 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:07.430675    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-299e74-865787d6ff-c8lt4" event={"ID":"17f700b6-f498-497e-a292-3ae243f65858","Type":"ContainerDied","Data":"3f1fae93371e027e01f710c957d8b31843ec4cd6f3827ce658eb6178c954c1c8"}
Nov 02 12:49:07.440285 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:07.440267    4559 scope.go:122] "RemoveContainer" containerID="9c33c19e71901fcb230c538093285f9890a97fa4e8117e2f7f96da7ddb163ba5"
Nov 02 12:49:07.440614 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:49:07.440578    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c33c19e71901fcb230c538093285f9890a97fa4e8117e2f7f96da7ddb163ba5\": container with ID starting with 9c33c19e71901fcb230c538093285f9890a97fa4e8117e2f7f96da7ddb163ba5 not found: ID does not exist" containerID="9c33c19e71901fcb230c538093285f9890a97fa4e8117e2f7f96da7ddb163ba5"
Nov 02 12:49:07.440740 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:07.440617    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c33c19e71901fcb230c538093285f9890a97fa4e8117e2f7f96da7ddb163ba5"} err="failed to get container status \"9c33c19e71901fcb230c538093285f9890a97fa4e8117e2f7f96da7ddb163ba5\": rpc error: code = NotFound desc = could not find container \"9c33c19e71901fcb230c538093285f9890a97fa4e8117e2f7f96da7ddb163ba5\": container with ID starting with 9c33c19e71901fcb230c538093285f9890a97fa4e8117e2f7f96da7ddb163ba5 not found: ID does not exist"
Nov 02 12:49:07.440806 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:07.440745    4559 scope.go:122] "RemoveContainer" containerID="2b171615ed74f01ddd8f91a6543e7981cae93a7e5de9d5368e648c7701cfcfeb"
Nov 02 12:49:07.449526 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:07.449504    4559 scope.go:122] "RemoveContainer" containerID="2b171615ed74f01ddd8f91a6543e7981cae93a7e5de9d5368e648c7701cfcfeb"
Nov 02 12:49:07.449816 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:49:07.449798    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b171615ed74f01ddd8f91a6543e7981cae93a7e5de9d5368e648c7701cfcfeb\": container with ID starting with 2b171615ed74f01ddd8f91a6543e7981cae93a7e5de9d5368e648c7701cfcfeb not found: ID does not exist" containerID="2b171615ed74f01ddd8f91a6543e7981cae93a7e5de9d5368e648c7701cfcfeb"
Nov 02 12:49:07.449889 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:07.449825    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b171615ed74f01ddd8f91a6543e7981cae93a7e5de9d5368e648c7701cfcfeb"} err="failed to get container status \"2b171615ed74f01ddd8f91a6543e7981cae93a7e5de9d5368e648c7701cfcfeb\": rpc error: code = NotFound desc = could not find container \"2b171615ed74f01ddd8f91a6543e7981cae93a7e5de9d5368e648c7701cfcfeb\": container with ID starting with 2b171615ed74f01ddd8f91a6543e7981cae93a7e5de9d5368e648c7701cfcfeb not found: ID does not exist"
Nov 02 12:49:07.489272 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:07.489208    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-299e74-865787d6ff-c8lt4"]
Nov 02 12:49:07.493763 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:07.493743    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-299e74-865787d6ff-c8lt4"]
Nov 02 12:49:07.531795 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:07.531768    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-299e74-7c7d6bb879-bkw2f"]
Nov 02 12:49:07.538181 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:07.538154    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-299e74-7c7d6bb879-bkw2f"]
Nov 02 12:49:07.893414 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:07.893329    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17f700b6-f498-497e-a292-3ae243f65858" path="/var/lib/kubelet/pods/17f700b6-f498-497e-a292-3ae243f65858/volumes"
Nov 02 12:49:07.893888 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:07.893866    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="545d586e-9e3a-452a-8b0b-2775d5e49317" path="/var/lib/kubelet/pods/545d586e-9e3a-452a-8b0b-2775d5e49317/volumes"
Nov 02 12:49:08.837576 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:08.837545    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-d6d314-77744649f9-rf95p"]
Nov 02 12:49:08.838174 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:08.838159    4559 cpu_manager.go:414] "containerMap: removing container" podUID="17f700b6-f498-497e-a292-3ae243f65858" containerName="payload-processing"
Nov 02 12:49:08.838222 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:08.838176    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17f700b6-f498-497e-a292-3ae243f65858" containerName="payload-processing"
Nov 02 12:49:08.838222 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:08.838210    4559 cpu_manager.go:414] "containerMap: removing container" podUID="545d586e-9e3a-452a-8b0b-2775d5e49317" containerName="payload-pre-processing"
Nov 02 12:49:08.838222 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:08.838215    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="545d586e-9e3a-452a-8b0b-2775d5e49317" containerName="payload-pre-processing"
Nov 02 12:49:08.838315 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:08.838305    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="545d586e-9e3a-452a-8b0b-2775d5e49317" containerName="payload-pre-processing"
Nov 02 12:49:08.838346 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:08.838316    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="17f700b6-f498-497e-a292-3ae243f65858" containerName="payload-processing"
Nov 02 12:49:08.843070 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:08.843052    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d6d314-77744649f9-rf95p"
Nov 02 12:49:08.860903 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:08.860879    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-d6d314-dockercfg-zhxlw\""
Nov 02 12:49:08.862412 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:08.862385    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-d6d314-77744649f9-rf95p"]
Nov 02 12:49:08.871057 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:08.871032    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-d6d314\""
Nov 02 12:49:09.011662 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:09.011301    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d504a5ab-c0f3-4f82-8385-9c11624edc60-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-d6d314-77744649f9-rf95p\" (UID: \"d504a5ab-c0f3-4f82-8385-9c11624edc60\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d6d314-77744649f9-rf95p"
Nov 02 12:49:09.011662 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:09.011506    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dpf9\" (UniqueName: \"kubernetes.io/projected/d504a5ab-c0f3-4f82-8385-9c11624edc60-kube-api-access-4dpf9\") pod \"payload-pre-processing-e2e-praxis-a-d6d314-77744649f9-rf95p\" (UID: \"d504a5ab-c0f3-4f82-8385-9c11624edc60\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d6d314-77744649f9-rf95p"
Nov 02 12:49:09.115712 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:09.115521    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-d6d314-767dbf49bc-55rvd"]
Nov 02 12:49:09.138267 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:09.135946    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-d6d314-767dbf49bc-55rvd"]
Nov 02 12:49:09.138267 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:09.136073    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-d6d314-767dbf49bc-55rvd"
Nov 02 12:49:09.148552 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:09.148520    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dpf9\" (UniqueName: \"kubernetes.io/projected/d504a5ab-c0f3-4f82-8385-9c11624edc60-kube-api-access-4dpf9\") pod \"payload-pre-processing-e2e-praxis-a-d6d314-77744649f9-rf95p\" (UID: \"d504a5ab-c0f3-4f82-8385-9c11624edc60\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d6d314-77744649f9-rf95p"
Nov 02 12:49:09.148772 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:09.148737    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d504a5ab-c0f3-4f82-8385-9c11624edc60-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-d6d314-77744649f9-rf95p\" (UID: \"d504a5ab-c0f3-4f82-8385-9c11624edc60\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d6d314-77744649f9-rf95p"
Nov 02 12:49:09.149460 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:09.149434    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d504a5ab-c0f3-4f82-8385-9c11624edc60-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-d6d314-77744649f9-rf95p\" (UID: \"d504a5ab-c0f3-4f82-8385-9c11624edc60\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d6d314-77744649f9-rf95p"
Nov 02 12:49:09.173166 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:09.173114    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dpf9\" (UniqueName: \"kubernetes.io/projected/d504a5ab-c0f3-4f82-8385-9c11624edc60-kube-api-access-4dpf9\") pod \"payload-pre-processing-e2e-praxis-a-d6d314-77744649f9-rf95p\" (UID: \"d504a5ab-c0f3-4f82-8385-9c11624edc60\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d6d314-77744649f9-rf95p"
Nov 02 12:49:09.250240 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:09.250205    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c678c\" (UniqueName: \"kubernetes.io/projected/ced18836-7a8d-4913-bcd3-24d3d75592d6-kube-api-access-c678c\") pod \"payload-processing-e2e-praxis-a-d6d314-767dbf49bc-55rvd\" (UID: \"ced18836-7a8d-4913-bcd3-24d3d75592d6\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-d6d314-767dbf49bc-55rvd"
Nov 02 12:49:09.250455 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:09.250434    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ced18836-7a8d-4913-bcd3-24d3d75592d6-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-d6d314-767dbf49bc-55rvd\" (UID: \"ced18836-7a8d-4913-bcd3-24d3d75592d6\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-d6d314-767dbf49bc-55rvd"
Nov 02 12:49:09.365659 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:09.365233    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ced18836-7a8d-4913-bcd3-24d3d75592d6-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-d6d314-767dbf49bc-55rvd\" (UID: \"ced18836-7a8d-4913-bcd3-24d3d75592d6\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-d6d314-767dbf49bc-55rvd"
Nov 02 12:49:09.365659 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:09.365342    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c678c\" (UniqueName: \"kubernetes.io/projected/ced18836-7a8d-4913-bcd3-24d3d75592d6-kube-api-access-c678c\") pod \"payload-processing-e2e-praxis-a-d6d314-767dbf49bc-55rvd\" (UID: \"ced18836-7a8d-4913-bcd3-24d3d75592d6\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-d6d314-767dbf49bc-55rvd"
Nov 02 12:49:09.367594 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:09.366803    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ced18836-7a8d-4913-bcd3-24d3d75592d6-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-d6d314-767dbf49bc-55rvd\" (UID: \"ced18836-7a8d-4913-bcd3-24d3d75592d6\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-d6d314-767dbf49bc-55rvd"
Nov 02 12:49:09.405237 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:09.405203    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c678c\" (UniqueName: \"kubernetes.io/projected/ced18836-7a8d-4913-bcd3-24d3d75592d6-kube-api-access-c678c\") pod \"payload-processing-e2e-praxis-a-d6d314-767dbf49bc-55rvd\" (UID: \"ced18836-7a8d-4913-bcd3-24d3d75592d6\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-d6d314-767dbf49bc-55rvd"
Nov 02 12:49:09.456751 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:09.456720    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d6d314-77744649f9-rf95p"
Nov 02 12:49:09.457783 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:09.457762    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-d6d314-767dbf49bc-55rvd"
Nov 02 12:49:09.762240 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:49:09.762212    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd504a5ab_c0f3_4f82_8385_9c11624edc60.slice/crio-c3ac599317f9b9ddc3e038a2fe6da52796c8e239ef047d3f4e794ace0af7022e WatchSource:0}: Error finding container c3ac599317f9b9ddc3e038a2fe6da52796c8e239ef047d3f4e794ace0af7022e: Status 404 returned error can't find the container with id c3ac599317f9b9ddc3e038a2fe6da52796c8e239ef047d3f4e794ace0af7022e
Nov 02 12:49:09.762364 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:09.762302    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-d6d314-77744649f9-rf95p"]
Nov 02 12:49:09.770757 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:49:09.770730    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podced18836_7a8d_4913_bcd3_24d3d75592d6.slice/crio-43738b648eb7131273d931068f0e496d960c982d5cb32f1c7827aabe19a922a2 WatchSource:0}: Error finding container 43738b648eb7131273d931068f0e496d960c982d5cb32f1c7827aabe19a922a2: Status 404 returned error can't find the container with id 43738b648eb7131273d931068f0e496d960c982d5cb32f1c7827aabe19a922a2
Nov 02 12:49:09.774608 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:09.774586    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-d6d314-767dbf49bc-55rvd"]
Nov 02 12:49:10.320287 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:10.320261    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b271f0-547b5469dc-42kll"
Nov 02 12:49:10.320692 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:10.320339    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-b271f0-6966d4c887-jthwp"
Nov 02 12:49:10.434590 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:10.434557    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-f7f44a-openshift-default-765dbcd8cf-t8fz5"]
Nov 02 12:49:10.434830 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:10.434793    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-f7f44a-openshift-default-765dbcd8cf-t8fz5" podUID="4906067b-634f-4251-9953-e4c70616400b" containerName="istio-proxy" containerID="cri-o://b3c98a866162e106ba9faa3937f33559acd6ad622d7778dbb5ba2265d5c4b042" gracePeriod=30
Nov 02 12:49:10.456949 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:10.456917    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d6d314-77744649f9-rf95p" event={"ID":"d504a5ab-c0f3-4f82-8385-9c11624edc60","Type":"ContainerStarted","Data":"0c96e44c6b992653b29a3e361bf2ce1f071b8cc824e80a7d4239a5936491e2c3"}
Nov 02 12:49:10.457109 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:10.456975    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d6d314-77744649f9-rf95p" event={"ID":"d504a5ab-c0f3-4f82-8385-9c11624edc60","Type":"ContainerStarted","Data":"c3ac599317f9b9ddc3e038a2fe6da52796c8e239ef047d3f4e794ace0af7022e"}
Nov 02 12:49:10.457109 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:10.457105    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d6d314-77744649f9-rf95p"
Nov 02 12:49:10.459347 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:10.459320    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-d6d314-767dbf49bc-55rvd" event={"ID":"ced18836-7a8d-4913-bcd3-24d3d75592d6","Type":"ContainerStarted","Data":"bfc46b31db75ba10f0296415d523502e2cf188f0e88ecb8fceb85858bc6d31bb"}
Nov 02 12:49:10.459454 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:10.459378    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-d6d314-767dbf49bc-55rvd"
Nov 02 12:49:10.459454 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:10.459406    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-d6d314-767dbf49bc-55rvd" event={"ID":"ced18836-7a8d-4913-bcd3-24d3d75592d6","Type":"ContainerStarted","Data":"43738b648eb7131273d931068f0e496d960c982d5cb32f1c7827aabe19a922a2"}
Nov 02 12:49:10.573712 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:10.573554    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d6d314-77744649f9-rf95p" podStartSLOduration=2.573529224 podStartE2EDuration="2.573529224s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:49:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:49:10.544931181 +0000 UTC m=+1346.800330277" watchObservedRunningTime="2026-11-02 12:49:10.573529224 +0000 UTC m=+1346.828928320"
Nov 02 12:49:10.695762 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:10.695698    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-d6d314-767dbf49bc-55rvd" podStartSLOduration=2.695677582 podStartE2EDuration="2.695677582s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:49:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:49:10.667043259 +0000 UTC m=+1346.922442410" watchObservedRunningTime="2026-11-02 12:49:10.695677582 +0000 UTC m=+1346.951076685"
Nov 02 12:49:11.809111 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:11.809077    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-eb24df-openshift-default-67f48b7f65-fg527"]
Nov 02 12:49:11.814443 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:11.814417    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-eb24df-openshift-default-67f48b7f65-fg527"
Nov 02 12:49:11.833716 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:11.833695    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-eb24df-openshift-default-dockercfg-6tpwc\""
Nov 02 12:49:11.838592 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:11.838559    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-eb24df-openshift-default-67f48b7f65-fg527"]
Nov 02 12:49:11.910625 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:11.910590    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ct826\" (UniqueName: \"kubernetes.io/projected/f19dde18-978d-4894-83d3-5c39478c866c-kube-api-access-ct826\") pod \"e2e-ipp-a-eb24df-openshift-default-67f48b7f65-fg527\" (UID: \"f19dde18-978d-4894-83d3-5c39478c866c\") " pod="openshift-ingress/e2e-ipp-a-eb24df-openshift-default-67f48b7f65-fg527"
Nov 02 12:49:11.910806 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:11.910774    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f19dde18-978d-4894-83d3-5c39478c866c-credential-socket\") pod \"e2e-ipp-a-eb24df-openshift-default-67f48b7f65-fg527\" (UID: \"f19dde18-978d-4894-83d3-5c39478c866c\") " pod="openshift-ingress/e2e-ipp-a-eb24df-openshift-default-67f48b7f65-fg527"
Nov 02 12:49:11.910859 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:11.910840    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f19dde18-978d-4894-83d3-5c39478c866c-workload-certs\") pod \"e2e-ipp-a-eb24df-openshift-default-67f48b7f65-fg527\" (UID: \"f19dde18-978d-4894-83d3-5c39478c866c\") " pod="openshift-ingress/e2e-ipp-a-eb24df-openshift-default-67f48b7f65-fg527"
Nov 02 12:49:11.910919 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:11.910871    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f19dde18-978d-4894-83d3-5c39478c866c-istio-podinfo\") pod \"e2e-ipp-a-eb24df-openshift-default-67f48b7f65-fg527\" (UID: \"f19dde18-978d-4894-83d3-5c39478c866c\") " pod="openshift-ingress/e2e-ipp-a-eb24df-openshift-default-67f48b7f65-fg527"
Nov 02 12:49:11.910919 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:11.910908    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f19dde18-978d-4894-83d3-5c39478c866c-istio-token\") pod \"e2e-ipp-a-eb24df-openshift-default-67f48b7f65-fg527\" (UID: \"f19dde18-978d-4894-83d3-5c39478c866c\") " pod="openshift-ingress/e2e-ipp-a-eb24df-openshift-default-67f48b7f65-fg527"
Nov 02 12:49:11.911031 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:11.911012    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f19dde18-978d-4894-83d3-5c39478c866c-istio-envoy\") pod \"e2e-ipp-a-eb24df-openshift-default-67f48b7f65-fg527\" (UID: \"f19dde18-978d-4894-83d3-5c39478c866c\") " pod="openshift-ingress/e2e-ipp-a-eb24df-openshift-default-67f48b7f65-fg527"
Nov 02 12:49:11.911095 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:11.911053    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f19dde18-978d-4894-83d3-5c39478c866c-workload-socket\") pod \"e2e-ipp-a-eb24df-openshift-default-67f48b7f65-fg527\" (UID: \"f19dde18-978d-4894-83d3-5c39478c866c\") " pod="openshift-ingress/e2e-ipp-a-eb24df-openshift-default-67f48b7f65-fg527"
Nov 02 12:49:11.911095 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:11.911079    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f19dde18-978d-4894-83d3-5c39478c866c-istio-data\") pod \"e2e-ipp-a-eb24df-openshift-default-67f48b7f65-fg527\" (UID: \"f19dde18-978d-4894-83d3-5c39478c866c\") " pod="openshift-ingress/e2e-ipp-a-eb24df-openshift-default-67f48b7f65-fg527"
Nov 02 12:49:11.911265 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:11.911103    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f19dde18-978d-4894-83d3-5c39478c866c-istiod-ca-cert\") pod \"e2e-ipp-a-eb24df-openshift-default-67f48b7f65-fg527\" (UID: \"f19dde18-978d-4894-83d3-5c39478c866c\") " pod="openshift-ingress/e2e-ipp-a-eb24df-openshift-default-67f48b7f65-fg527"
Nov 02 12:49:12.017700 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:12.013360    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f19dde18-978d-4894-83d3-5c39478c866c-istio-podinfo\") pod \"e2e-ipp-a-eb24df-openshift-default-67f48b7f65-fg527\" (UID: \"f19dde18-978d-4894-83d3-5c39478c866c\") " pod="openshift-ingress/e2e-ipp-a-eb24df-openshift-default-67f48b7f65-fg527"
Nov 02 12:49:12.017700 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:12.013428    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f19dde18-978d-4894-83d3-5c39478c866c-istio-token\") pod \"e2e-ipp-a-eb24df-openshift-default-67f48b7f65-fg527\" (UID: \"f19dde18-978d-4894-83d3-5c39478c866c\") " pod="openshift-ingress/e2e-ipp-a-eb24df-openshift-default-67f48b7f65-fg527"
Nov 02 12:49:12.017700 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:12.013499    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f19dde18-978d-4894-83d3-5c39478c866c-istio-envoy\") pod \"e2e-ipp-a-eb24df-openshift-default-67f48b7f65-fg527\" (UID: \"f19dde18-978d-4894-83d3-5c39478c866c\") " pod="openshift-ingress/e2e-ipp-a-eb24df-openshift-default-67f48b7f65-fg527"
Nov 02 12:49:12.017700 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:12.013537    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f19dde18-978d-4894-83d3-5c39478c866c-workload-socket\") pod \"e2e-ipp-a-eb24df-openshift-default-67f48b7f65-fg527\" (UID: \"f19dde18-978d-4894-83d3-5c39478c866c\") " pod="openshift-ingress/e2e-ipp-a-eb24df-openshift-default-67f48b7f65-fg527"
Nov 02 12:49:12.017700 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:12.013578    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f19dde18-978d-4894-83d3-5c39478c866c-istio-data\") pod \"e2e-ipp-a-eb24df-openshift-default-67f48b7f65-fg527\" (UID: \"f19dde18-978d-4894-83d3-5c39478c866c\") " pod="openshift-ingress/e2e-ipp-a-eb24df-openshift-default-67f48b7f65-fg527"
Nov 02 12:49:12.017700 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:12.013610    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f19dde18-978d-4894-83d3-5c39478c866c-istiod-ca-cert\") pod \"e2e-ipp-a-eb24df-openshift-default-67f48b7f65-fg527\" (UID: \"f19dde18-978d-4894-83d3-5c39478c866c\") " pod="openshift-ingress/e2e-ipp-a-eb24df-openshift-default-67f48b7f65-fg527"
Nov 02 12:49:12.017700 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:12.013772    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ct826\" (UniqueName: \"kubernetes.io/projected/f19dde18-978d-4894-83d3-5c39478c866c-kube-api-access-ct826\") pod \"e2e-ipp-a-eb24df-openshift-default-67f48b7f65-fg527\" (UID: \"f19dde18-978d-4894-83d3-5c39478c866c\") " pod="openshift-ingress/e2e-ipp-a-eb24df-openshift-default-67f48b7f65-fg527"
Nov 02 12:49:12.017700 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:12.013898    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f19dde18-978d-4894-83d3-5c39478c866c-credential-socket\") pod \"e2e-ipp-a-eb24df-openshift-default-67f48b7f65-fg527\" (UID: \"f19dde18-978d-4894-83d3-5c39478c866c\") " pod="openshift-ingress/e2e-ipp-a-eb24df-openshift-default-67f48b7f65-fg527"
Nov 02 12:49:12.017700 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:12.013928    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f19dde18-978d-4894-83d3-5c39478c866c-workload-certs\") pod \"e2e-ipp-a-eb24df-openshift-default-67f48b7f65-fg527\" (UID: \"f19dde18-978d-4894-83d3-5c39478c866c\") " pod="openshift-ingress/e2e-ipp-a-eb24df-openshift-default-67f48b7f65-fg527"
Nov 02 12:49:12.017700 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:12.014370    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f19dde18-978d-4894-83d3-5c39478c866c-workload-certs\") pod \"e2e-ipp-a-eb24df-openshift-default-67f48b7f65-fg527\" (UID: \"f19dde18-978d-4894-83d3-5c39478c866c\") " pod="openshift-ingress/e2e-ipp-a-eb24df-openshift-default-67f48b7f65-fg527"
Nov 02 12:49:12.027658 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:12.026327    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f19dde18-978d-4894-83d3-5c39478c866c-istio-data\") pod \"e2e-ipp-a-eb24df-openshift-default-67f48b7f65-fg527\" (UID: \"f19dde18-978d-4894-83d3-5c39478c866c\") " pod="openshift-ingress/e2e-ipp-a-eb24df-openshift-default-67f48b7f65-fg527"
Nov 02 12:49:12.027658 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:12.027606    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f19dde18-978d-4894-83d3-5c39478c866c-workload-socket\") pod \"e2e-ipp-a-eb24df-openshift-default-67f48b7f65-fg527\" (UID: \"f19dde18-978d-4894-83d3-5c39478c866c\") " pod="openshift-ingress/e2e-ipp-a-eb24df-openshift-default-67f48b7f65-fg527"
Nov 02 12:49:12.054671 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:12.028516    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f19dde18-978d-4894-83d3-5c39478c866c-istiod-ca-cert\") pod \"e2e-ipp-a-eb24df-openshift-default-67f48b7f65-fg527\" (UID: \"f19dde18-978d-4894-83d3-5c39478c866c\") " pod="openshift-ingress/e2e-ipp-a-eb24df-openshift-default-67f48b7f65-fg527"
Nov 02 12:49:12.078865 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:12.039963    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f19dde18-978d-4894-83d3-5c39478c866c-istio-envoy\") pod \"e2e-ipp-a-eb24df-openshift-default-67f48b7f65-fg527\" (UID: \"f19dde18-978d-4894-83d3-5c39478c866c\") " pod="openshift-ingress/e2e-ipp-a-eb24df-openshift-default-67f48b7f65-fg527"
Nov 02 12:49:12.079069 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:12.040209    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f19dde18-978d-4894-83d3-5c39478c866c-credential-socket\") pod \"e2e-ipp-a-eb24df-openshift-default-67f48b7f65-fg527\" (UID: \"f19dde18-978d-4894-83d3-5c39478c866c\") " pod="openshift-ingress/e2e-ipp-a-eb24df-openshift-default-67f48b7f65-fg527"
Nov 02 12:49:12.079178 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:12.062404    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ct826\" (UniqueName: \"kubernetes.io/projected/f19dde18-978d-4894-83d3-5c39478c866c-kube-api-access-ct826\") pod \"e2e-ipp-a-eb24df-openshift-default-67f48b7f65-fg527\" (UID: \"f19dde18-978d-4894-83d3-5c39478c866c\") " pod="openshift-ingress/e2e-ipp-a-eb24df-openshift-default-67f48b7f65-fg527"
Nov 02 12:49:12.079265 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:12.070246    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f19dde18-978d-4894-83d3-5c39478c866c-istio-token\") pod \"e2e-ipp-a-eb24df-openshift-default-67f48b7f65-fg527\" (UID: \"f19dde18-978d-4894-83d3-5c39478c866c\") " pod="openshift-ingress/e2e-ipp-a-eb24df-openshift-default-67f48b7f65-fg527"
Nov 02 12:49:12.079349 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:12.078736    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f19dde18-978d-4894-83d3-5c39478c866c-istio-podinfo\") pod \"e2e-ipp-a-eb24df-openshift-default-67f48b7f65-fg527\" (UID: \"f19dde18-978d-4894-83d3-5c39478c866c\") " pod="openshift-ingress/e2e-ipp-a-eb24df-openshift-default-67f48b7f65-fg527"
Nov 02 12:49:12.139666 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:12.133590    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-eb24df-openshift-default-67f48b7f65-fg527"
Nov 02 12:49:12.543708 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:49:12.543619    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf19dde18_978d_4894_83d3_5c39478c866c.slice/crio-b309cc72a8fe1c0e05c37031a9c221b26153803e374eb337ffd236842c32a3aa WatchSource:0}: Error finding container b309cc72a8fe1c0e05c37031a9c221b26153803e374eb337ffd236842c32a3aa: Status 404 returned error can't find the container with id b309cc72a8fe1c0e05c37031a9c221b26153803e374eb337ffd236842c32a3aa
Nov 02 12:49:12.546471 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:12.545113    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-eb24df-openshift-default-67f48b7f65-fg527"]
Nov 02 12:49:12.550771 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:12.550736    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:49:12.550846 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:12.550811    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:49:12.550902 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:12.550842    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:49:13.479728 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:13.479681    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-eb24df-openshift-default-67f48b7f65-fg527" event={"ID":"f19dde18-978d-4894-83d3-5c39478c866c","Type":"ContainerStarted","Data":"6ec22c25fd91edf8dac1aa8536d95431284d95d3c6d0009924cbe8851f1f9949"}
Nov 02 12:49:13.480123 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:13.479748    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-eb24df-openshift-default-67f48b7f65-fg527" event={"ID":"f19dde18-978d-4894-83d3-5c39478c866c","Type":"ContainerStarted","Data":"b309cc72a8fe1c0e05c37031a9c221b26153803e374eb337ffd236842c32a3aa"}
Nov 02 12:49:13.551249 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:13.551191    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-eb24df-openshift-default-67f48b7f65-fg527" podStartSLOduration=2.551178254 podStartE2EDuration="2.551178254s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:49:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:49:13.539907806 +0000 UTC m=+1349.795306902" watchObservedRunningTime="2026-11-02 12:49:13.551178254 +0000 UTC m=+1349.806577413"
Nov 02 12:49:14.135097 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:14.135059    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-eb24df-openshift-default-67f48b7f65-fg527"
Nov 02 12:49:14.141015 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:14.140987    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-eb24df-openshift-default-67f48b7f65-fg527"
Nov 02 12:49:14.297330 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:14.297296    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-b271f0-578d584488-vwllx"]
Nov 02 12:49:14.302988 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:14.302950    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b271f0-578d584488-vwllx"
Nov 02 12:49:14.322591 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:14.321532    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-b271f0-serving-cert\""
Nov 02 12:49:14.325523 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:14.325493    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-b271f0-578d584488-vwllx"]
Nov 02 12:49:14.465198 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:14.465163    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a0ab432-145b-45a0-8629-fa4dcb129694-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-b271f0-578d584488-vwllx\" (UID: \"0a0ab432-145b-45a0-8629-fa4dcb129694\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b271f0-578d584488-vwllx"
Nov 02 12:49:14.465381 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:14.465326    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wp667\" (UniqueName: \"kubernetes.io/projected/0a0ab432-145b-45a0-8629-fa4dcb129694-kube-api-access-wp667\") pod \"maas-api-e2e-praxis-a-b271f0-578d584488-vwllx\" (UID: \"0a0ab432-145b-45a0-8629-fa4dcb129694\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b271f0-578d584488-vwllx"
Nov 02 12:49:14.465431 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:14.465418    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a0ab432-145b-45a0-8629-fa4dcb129694-metrics-certs\") pod \"maas-api-e2e-praxis-a-b271f0-578d584488-vwllx\" (UID: \"0a0ab432-145b-45a0-8629-fa4dcb129694\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b271f0-578d584488-vwllx"
Nov 02 12:49:14.465551 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:14.465528    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a0ab432-145b-45a0-8629-fa4dcb129694-maas-api-tls\") pod \"maas-api-e2e-praxis-a-b271f0-578d584488-vwllx\" (UID: \"0a0ab432-145b-45a0-8629-fa4dcb129694\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b271f0-578d584488-vwllx"
Nov 02 12:49:14.558674 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:14.554533    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-eb24df-openshift-default-67f48b7f65-fg527"
Nov 02 12:49:14.565004 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:14.564973    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-eb24df-openshift-default-67f48b7f65-fg527"
Nov 02 12:49:14.572691 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:14.567173    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wp667\" (UniqueName: \"kubernetes.io/projected/0a0ab432-145b-45a0-8629-fa4dcb129694-kube-api-access-wp667\") pod \"maas-api-e2e-praxis-a-b271f0-578d584488-vwllx\" (UID: \"0a0ab432-145b-45a0-8629-fa4dcb129694\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b271f0-578d584488-vwllx"
Nov 02 12:49:14.572691 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:14.567300    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a0ab432-145b-45a0-8629-fa4dcb129694-metrics-certs\") pod \"maas-api-e2e-praxis-a-b271f0-578d584488-vwllx\" (UID: \"0a0ab432-145b-45a0-8629-fa4dcb129694\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b271f0-578d584488-vwllx"
Nov 02 12:49:14.572691 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:14.567396    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a0ab432-145b-45a0-8629-fa4dcb129694-maas-api-tls\") pod \"maas-api-e2e-praxis-a-b271f0-578d584488-vwllx\" (UID: \"0a0ab432-145b-45a0-8629-fa4dcb129694\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b271f0-578d584488-vwllx"
Nov 02 12:49:14.572691 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:14.567444    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a0ab432-145b-45a0-8629-fa4dcb129694-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-b271f0-578d584488-vwllx\" (UID: \"0a0ab432-145b-45a0-8629-fa4dcb129694\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b271f0-578d584488-vwllx"
Nov 02 12:49:14.572691 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:14.568380    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a0ab432-145b-45a0-8629-fa4dcb129694-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-b271f0-578d584488-vwllx\" (UID: \"0a0ab432-145b-45a0-8629-fa4dcb129694\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b271f0-578d584488-vwllx"
Nov 02 12:49:14.610720 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:14.610689    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wp667\" (UniqueName: \"kubernetes.io/projected/0a0ab432-145b-45a0-8629-fa4dcb129694-kube-api-access-wp667\") pod \"maas-api-e2e-praxis-a-b271f0-578d584488-vwllx\" (UID: \"0a0ab432-145b-45a0-8629-fa4dcb129694\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b271f0-578d584488-vwllx"
Nov 02 12:49:14.611222 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:14.611186    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a0ab432-145b-45a0-8629-fa4dcb129694-metrics-certs\") pod \"maas-api-e2e-praxis-a-b271f0-578d584488-vwllx\" (UID: \"0a0ab432-145b-45a0-8629-fa4dcb129694\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b271f0-578d584488-vwllx"
Nov 02 12:49:14.611446 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:14.611424    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a0ab432-145b-45a0-8629-fa4dcb129694-maas-api-tls\") pod \"maas-api-e2e-praxis-a-b271f0-578d584488-vwllx\" (UID: \"0a0ab432-145b-45a0-8629-fa4dcb129694\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b271f0-578d584488-vwllx"
Nov 02 12:49:14.627713 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:14.627684    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b271f0-578d584488-vwllx"
Nov 02 12:49:14.945977 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:49:14.945950    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a0ab432_145b_45a0_8629_fa4dcb129694.slice/crio-404c47666f9aa740bfcf0fa8aa2a402b9dd9c6881e8da316acce45b396f0a8e6 WatchSource:0}: Error finding container 404c47666f9aa740bfcf0fa8aa2a402b9dd9c6881e8da316acce45b396f0a8e6: Status 404 returned error can't find the container with id 404c47666f9aa740bfcf0fa8aa2a402b9dd9c6881e8da316acce45b396f0a8e6
Nov 02 12:49:14.946670 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:14.946631    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-b271f0-578d584488-vwllx"]
Nov 02 12:49:14.947791 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:14.947774    4559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 12:49:15.086189 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:49:15.086154    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqcmj" containerName="extractor"
Nov 02 12:49:15.649677 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:15.638477    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b271f0-578d584488-vwllx" event={"ID":"0a0ab432-145b-45a0-8629-fa4dcb129694","Type":"ContainerStarted","Data":"f02fa12fe8defe52eda8006fefea34ffdb32973dc63f7f6bd287cc967efa68de"}
Nov 02 12:49:15.649677 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:15.647658    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b271f0-578d584488-vwllx" event={"ID":"0a0ab432-145b-45a0-8629-fa4dcb129694","Type":"ContainerStarted","Data":"404c47666f9aa740bfcf0fa8aa2a402b9dd9c6881e8da316acce45b396f0a8e6"}
Nov 02 12:49:15.649677 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:15.647745    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b271f0-578d584488-vwllx"
Nov 02 12:49:15.680676 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:15.680493    4559 patch_prober.go:29] interesting pod/e2e-api-b-f7f44a-openshift-default-765dbcd8cf-t8fz5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body=
Nov 02 12:49:15.680676 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:15.680553    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-f7f44a-openshift-default-765dbcd8cf-t8fz5" podUID="4906067b-634f-4251-9953-e4c70616400b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused"
Nov 02 12:49:15.689651 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:15.689575    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b271f0-578d584488-vwllx" podStartSLOduration=1.3274535379999999 podStartE2EDuration="1.689558758s" totalImagesPullingTime="362.10522ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:49:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:49:15.686338855 +0000 UTC m=+1351.941737951" watchObservedRunningTime="2026-11-02 12:49:15.689558758 +0000 UTC m=+1351.944957853"
Nov 02 12:49:15.874500 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:15.870966    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-f7f44a-openshift-default-765dbcd8cf-t8fz5"
Nov 02 12:49:16.009391 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:16.009358    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4906067b-634f-4251-9953-e4c70616400b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4906067b-634f-4251-9953-e4c70616400b-istio-envoy\") pod \"4906067b-634f-4251-9953-e4c70616400b\" (UID: \"4906067b-634f-4251-9953-e4c70616400b\") "
Nov 02 12:49:16.009596 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:16.009448    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4906067b-634f-4251-9953-e4c70616400b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4906067b-634f-4251-9953-e4c70616400b-workload-socket\") pod \"4906067b-634f-4251-9953-e4c70616400b\" (UID: \"4906067b-634f-4251-9953-e4c70616400b\") "
Nov 02 12:49:16.009596 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:16.009468    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4906067b-634f-4251-9953-e4c70616400b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4906067b-634f-4251-9953-e4c70616400b-istiod-ca-cert\") pod \"4906067b-634f-4251-9953-e4c70616400b\" (UID: \"4906067b-634f-4251-9953-e4c70616400b\") "
Nov 02 12:49:16.009596 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:16.009508    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4906067b-634f-4251-9953-e4c70616400b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4906067b-634f-4251-9953-e4c70616400b-workload-certs\") pod \"4906067b-634f-4251-9953-e4c70616400b\" (UID: \"4906067b-634f-4251-9953-e4c70616400b\") "
Nov 02 12:49:16.009801 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:16.009602    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4906067b-634f-4251-9953-e4c70616400b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4906067b-634f-4251-9953-e4c70616400b-credential-socket\") pod \"4906067b-634f-4251-9953-e4c70616400b\" (UID: \"4906067b-634f-4251-9953-e4c70616400b\") "
Nov 02 12:49:16.009801 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:16.009714    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4906067b-634f-4251-9953-e4c70616400b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4906067b-634f-4251-9953-e4c70616400b-istio-data\") pod \"4906067b-634f-4251-9953-e4c70616400b\" (UID: \"4906067b-634f-4251-9953-e4c70616400b\") "
Nov 02 12:49:16.009801 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:16.009733    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4906067b-634f-4251-9953-e4c70616400b-workload-socket" pod "4906067b-634f-4251-9953-e4c70616400b" (UID: "4906067b-634f-4251-9953-e4c70616400b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:49:16.009944 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:16.009844    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4906067b-634f-4251-9953-e4c70616400b-istio-token\" (UniqueName: \"kubernetes.io/projected/4906067b-634f-4251-9953-e4c70616400b-istio-token\") pod \"4906067b-634f-4251-9953-e4c70616400b\" (UID: \"4906067b-634f-4251-9953-e4c70616400b\") "
Nov 02 12:49:16.009944 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:16.009911    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4906067b-634f-4251-9953-e4c70616400b-workload-certs" pod "4906067b-634f-4251-9953-e4c70616400b" (UID: "4906067b-634f-4251-9953-e4c70616400b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:49:16.010049 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:16.009959    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4906067b-634f-4251-9953-e4c70616400b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4906067b-634f-4251-9953-e4c70616400b-istio-podinfo\") pod \"4906067b-634f-4251-9953-e4c70616400b\" (UID: \"4906067b-634f-4251-9953-e4c70616400b\") "
Nov 02 12:49:16.010049 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:16.010010    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4906067b-634f-4251-9953-e4c70616400b-kube-api-access-q7hft\" (UniqueName: \"kubernetes.io/projected/4906067b-634f-4251-9953-e4c70616400b-kube-api-access-q7hft\") pod \"4906067b-634f-4251-9953-e4c70616400b\" (UID: \"4906067b-634f-4251-9953-e4c70616400b\") "
Nov 02 12:49:16.010049 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:16.010018    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4906067b-634f-4251-9953-e4c70616400b-istio-data" pod "4906067b-634f-4251-9953-e4c70616400b" (UID: "4906067b-634f-4251-9953-e4c70616400b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:49:16.010189 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:16.010063    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4906067b-634f-4251-9953-e4c70616400b-istiod-ca-cert" pod "4906067b-634f-4251-9953-e4c70616400b" (UID: "4906067b-634f-4251-9953-e4c70616400b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:49:16.010321 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:16.010299    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4906067b-634f-4251-9953-e4c70616400b-credential-socket" pod "4906067b-634f-4251-9953-e4c70616400b" (UID: "4906067b-634f-4251-9953-e4c70616400b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:49:16.010932 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:16.010907    4559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4906067b-634f-4251-9953-e4c70616400b-workload-socket\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:49:16.011056 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:16.010937    4559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4906067b-634f-4251-9953-e4c70616400b-istiod-ca-cert\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:49:16.011056 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:16.010954    4559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4906067b-634f-4251-9953-e4c70616400b-workload-certs\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:49:16.011056 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:16.010968    4559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4906067b-634f-4251-9953-e4c70616400b-credential-socket\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:49:16.011056 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:16.010982    4559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4906067b-634f-4251-9953-e4c70616400b-istio-data\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:49:16.012075 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:16.012054    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4906067b-634f-4251-9953-e4c70616400b-istio-envoy" pod "4906067b-634f-4251-9953-e4c70616400b" (UID: "4906067b-634f-4251-9953-e4c70616400b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:49:16.012182 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:16.012148    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4906067b-634f-4251-9953-e4c70616400b-istio-token" pod "4906067b-634f-4251-9953-e4c70616400b" (UID: "4906067b-634f-4251-9953-e4c70616400b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:49:16.012246 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:16.012199    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4906067b-634f-4251-9953-e4c70616400b-kube-api-access-q7hft" pod "4906067b-634f-4251-9953-e4c70616400b" (UID: "4906067b-634f-4251-9953-e4c70616400b"). InnerVolumeSpecName "kube-api-access-q7hft". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:49:16.012560 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:16.012534    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4906067b-634f-4251-9953-e4c70616400b-istio-podinfo" pod "4906067b-634f-4251-9953-e4c70616400b" (UID: "4906067b-634f-4251-9953-e4c70616400b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 12:49:16.126014 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:16.125849    4559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4906067b-634f-4251-9953-e4c70616400b-istio-token\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:49:16.126014 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:16.125890    4559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4906067b-634f-4251-9953-e4c70616400b-istio-podinfo\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:49:16.126014 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:16.125908    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q7hft\" (UniqueName: \"kubernetes.io/projected/4906067b-634f-4251-9953-e4c70616400b-kube-api-access-q7hft\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:49:16.126014 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:16.125926    4559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4906067b-634f-4251-9953-e4c70616400b-istio-envoy\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:49:16.646026 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:16.645990    4559 generic.go:356] "Generic (PLEG): container finished" podID="4906067b-634f-4251-9953-e4c70616400b" containerID="b3c98a866162e106ba9faa3937f33559acd6ad622d7778dbb5ba2265d5c4b042" exitCode=0
Nov 02 12:49:16.646179 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:16.646041    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-f7f44a-openshift-default-765dbcd8cf-t8fz5"
Nov 02 12:49:16.646179 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:16.646065    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f7f44a-openshift-default-765dbcd8cf-t8fz5" event={"ID":"4906067b-634f-4251-9953-e4c70616400b","Type":"ContainerDied","Data":"b3c98a866162e106ba9faa3937f33559acd6ad622d7778dbb5ba2265d5c4b042"}
Nov 02 12:49:16.646179 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:16.646123    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f7f44a-openshift-default-765dbcd8cf-t8fz5" event={"ID":"4906067b-634f-4251-9953-e4c70616400b","Type":"ContainerDied","Data":"e3bc190555f788f68a1c8a1e5f98e6e7fd27efe726b347942448ba8fe6b0ccca"}
Nov 02 12:49:16.646179 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:16.646147    4559 scope.go:122] "RemoveContainer" containerID="b3c98a866162e106ba9faa3937f33559acd6ad622d7778dbb5ba2265d5c4b042"
Nov 02 12:49:16.658004 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:16.657820    4559 scope.go:122] "RemoveContainer" containerID="dc9e4f833b5068cfdc384b48c192cdc27e491ab61fbc76d4335a6ddfc342be46"
Nov 02 12:49:16.667091 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:16.667070    4559 scope.go:122] "RemoveContainer" containerID="b3c98a866162e106ba9faa3937f33559acd6ad622d7778dbb5ba2265d5c4b042"
Nov 02 12:49:16.667381 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:49:16.667363    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3c98a866162e106ba9faa3937f33559acd6ad622d7778dbb5ba2265d5c4b042\": container with ID starting with b3c98a866162e106ba9faa3937f33559acd6ad622d7778dbb5ba2265d5c4b042 not found: ID does not exist" containerID="b3c98a866162e106ba9faa3937f33559acd6ad622d7778dbb5ba2265d5c4b042"
Nov 02 12:49:16.667437 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:16.667389    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3c98a866162e106ba9faa3937f33559acd6ad622d7778dbb5ba2265d5c4b042"} err="failed to get container status \"b3c98a866162e106ba9faa3937f33559acd6ad622d7778dbb5ba2265d5c4b042\": rpc error: code = NotFound desc = could not find container \"b3c98a866162e106ba9faa3937f33559acd6ad622d7778dbb5ba2265d5c4b042\": container with ID starting with b3c98a866162e106ba9faa3937f33559acd6ad622d7778dbb5ba2265d5c4b042 not found: ID does not exist"
Nov 02 12:49:16.667437 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:16.667405    4559 scope.go:122] "RemoveContainer" containerID="dc9e4f833b5068cfdc384b48c192cdc27e491ab61fbc76d4335a6ddfc342be46"
Nov 02 12:49:16.667705 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:49:16.667627    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc9e4f833b5068cfdc384b48c192cdc27e491ab61fbc76d4335a6ddfc342be46\": container with ID starting with dc9e4f833b5068cfdc384b48c192cdc27e491ab61fbc76d4335a6ddfc342be46 not found: ID does not exist" containerID="dc9e4f833b5068cfdc384b48c192cdc27e491ab61fbc76d4335a6ddfc342be46"
Nov 02 12:49:16.667768 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:16.667715    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc9e4f833b5068cfdc384b48c192cdc27e491ab61fbc76d4335a6ddfc342be46"} err="failed to get container status \"dc9e4f833b5068cfdc384b48c192cdc27e491ab61fbc76d4335a6ddfc342be46\": rpc error: code = NotFound desc = could not find container \"dc9e4f833b5068cfdc384b48c192cdc27e491ab61fbc76d4335a6ddfc342be46\": container with ID starting with dc9e4f833b5068cfdc384b48c192cdc27e491ab61fbc76d4335a6ddfc342be46 not found: ID does not exist"
Nov 02 12:49:16.703546 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:16.703524    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-f7f44a-openshift-default-765dbcd8cf-t8fz5"]
Nov 02 12:49:16.735657 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:16.726738    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-f7f44a-openshift-default-765dbcd8cf-t8fz5"]
Nov 02 12:49:17.433414 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:17.433385    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e93a4b-bd984bc95-7fwhg"
Nov 02 12:49:17.433585 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:17.433451    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-e93a4b-6947c6444f-8cc8j"
Nov 02 12:49:17.894560 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:17.894488    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4906067b-634f-4251-9953-e4c70616400b" path="/var/lib/kubelet/pods/4906067b-634f-4251-9953-e4c70616400b/volumes"
Nov 02 12:49:17.901012 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:17.900987    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-eb24df-748477f8f5-67vfv"]
Nov 02 12:49:17.901587 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:17.901572    4559 cpu_manager.go:414] "containerMap: removing container" podUID="4906067b-634f-4251-9953-e4c70616400b" containerName="istio-proxy"
Nov 02 12:49:17.901635 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:17.901589    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4906067b-634f-4251-9953-e4c70616400b" containerName="istio-proxy"
Nov 02 12:49:17.901742 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:17.901732    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="4906067b-634f-4251-9953-e4c70616400b" containerName="istio-proxy"
Nov 02 12:49:17.906846 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:17.906825    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-eb24df-748477f8f5-67vfv"
Nov 02 12:49:17.924535 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:17.924512    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-eb24df-dockercfg-rrrgq\""
Nov 02 12:49:17.928902 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:17.928884    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-eb24df\""
Nov 02 12:49:17.934352 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:17.934334    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-eb24df-748477f8f5-67vfv"]
Nov 02 12:49:18.086666 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:18.079728    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f72904b-21b5-40f8-804d-cd747e649e02-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-eb24df-748477f8f5-67vfv\" (UID: \"9f72904b-21b5-40f8-804d-cd747e649e02\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-eb24df-748477f8f5-67vfv"
Nov 02 12:49:18.086666 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:18.079962    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smzn6\" (UniqueName: \"kubernetes.io/projected/9f72904b-21b5-40f8-804d-cd747e649e02-kube-api-access-smzn6\") pod \"payload-pre-processing-e2e-ipp-a-eb24df-748477f8f5-67vfv\" (UID: \"9f72904b-21b5-40f8-804d-cd747e649e02\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-eb24df-748477f8f5-67vfv"
Nov 02 12:49:18.120765 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:18.120732    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-eb24df-77b8d4b6bc-b7fx5"]
Nov 02 12:49:18.193915 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:18.186147    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smzn6\" (UniqueName: \"kubernetes.io/projected/9f72904b-21b5-40f8-804d-cd747e649e02-kube-api-access-smzn6\") pod \"payload-pre-processing-e2e-ipp-a-eb24df-748477f8f5-67vfv\" (UID: \"9f72904b-21b5-40f8-804d-cd747e649e02\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-eb24df-748477f8f5-67vfv"
Nov 02 12:49:18.193915 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:18.186320    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f72904b-21b5-40f8-804d-cd747e649e02-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-eb24df-748477f8f5-67vfv\" (UID: \"9f72904b-21b5-40f8-804d-cd747e649e02\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-eb24df-748477f8f5-67vfv"
Nov 02 12:49:18.193915 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:18.187614    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f72904b-21b5-40f8-804d-cd747e649e02-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-eb24df-748477f8f5-67vfv\" (UID: \"9f72904b-21b5-40f8-804d-cd747e649e02\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-eb24df-748477f8f5-67vfv"
Nov 02 12:49:18.214210 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:18.205384    4559 cpu_manager.go:414] "containerMap: removing container" podUID="4906067b-634f-4251-9953-e4c70616400b" containerName="istio-proxy"
Nov 02 12:49:18.214210 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:18.205413    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4906067b-634f-4251-9953-e4c70616400b" containerName="istio-proxy"
Nov 02 12:49:18.214210 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:18.205623    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="4906067b-634f-4251-9953-e4c70616400b" containerName="istio-proxy"
Nov 02 12:49:18.214210 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:18.211624    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-eb24df-77b8d4b6bc-b7fx5"]
Nov 02 12:49:18.214210 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:18.211767    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-eb24df-77b8d4b6bc-b7fx5"
Nov 02 12:49:18.215743 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:18.215682    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-smzn6\" (UniqueName: \"kubernetes.io/projected/9f72904b-21b5-40f8-804d-cd747e649e02-kube-api-access-smzn6\") pod \"payload-pre-processing-e2e-ipp-a-eb24df-748477f8f5-67vfv\" (UID: \"9f72904b-21b5-40f8-804d-cd747e649e02\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-eb24df-748477f8f5-67vfv"
Nov 02 12:49:18.225427 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:18.224863    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-eb24df-748477f8f5-67vfv"
Nov 02 12:49:18.388922 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:18.388890    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b7aacab-55fe-482d-8e15-a8f847678e48-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-eb24df-77b8d4b6bc-b7fx5\" (UID: \"6b7aacab-55fe-482d-8e15-a8f847678e48\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-eb24df-77b8d4b6bc-b7fx5"
Nov 02 12:49:18.389132 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:18.389113    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79cpp\" (UniqueName: \"kubernetes.io/projected/6b7aacab-55fe-482d-8e15-a8f847678e48-kube-api-access-79cpp\") pod \"payload-processing-e2e-ipp-a-eb24df-77b8d4b6bc-b7fx5\" (UID: \"6b7aacab-55fe-482d-8e15-a8f847678e48\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-eb24df-77b8d4b6bc-b7fx5"
Nov 02 12:49:18.490984 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:18.490905    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b7aacab-55fe-482d-8e15-a8f847678e48-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-eb24df-77b8d4b6bc-b7fx5\" (UID: \"6b7aacab-55fe-482d-8e15-a8f847678e48\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-eb24df-77b8d4b6bc-b7fx5"
Nov 02 12:49:18.491130 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:18.491091    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79cpp\" (UniqueName: \"kubernetes.io/projected/6b7aacab-55fe-482d-8e15-a8f847678e48-kube-api-access-79cpp\") pod \"payload-processing-e2e-ipp-a-eb24df-77b8d4b6bc-b7fx5\" (UID: \"6b7aacab-55fe-482d-8e15-a8f847678e48\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-eb24df-77b8d4b6bc-b7fx5"
Nov 02 12:49:18.491668 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:18.491625    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b7aacab-55fe-482d-8e15-a8f847678e48-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-eb24df-77b8d4b6bc-b7fx5\" (UID: \"6b7aacab-55fe-482d-8e15-a8f847678e48\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-eb24df-77b8d4b6bc-b7fx5"
Nov 02 12:49:18.516035 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:18.516009    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79cpp\" (UniqueName: \"kubernetes.io/projected/6b7aacab-55fe-482d-8e15-a8f847678e48-kube-api-access-79cpp\") pod \"payload-processing-e2e-ipp-a-eb24df-77b8d4b6bc-b7fx5\" (UID: \"6b7aacab-55fe-482d-8e15-a8f847678e48\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-eb24df-77b8d4b6bc-b7fx5"
Nov 02 12:49:18.553223 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:18.553195    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-eb24df-748477f8f5-67vfv"]
Nov 02 12:49:18.554062 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:49:18.554040    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f72904b_21b5_40f8_804d_cd747e649e02.slice/crio-dd8c60e5de2807b9df4eb4b5acc6450f37b7fba20c78715fbe2288a56360dab8 WatchSource:0}: Error finding container dd8c60e5de2807b9df4eb4b5acc6450f37b7fba20c78715fbe2288a56360dab8: Status 404 returned error can't find the container with id dd8c60e5de2807b9df4eb4b5acc6450f37b7fba20c78715fbe2288a56360dab8
Nov 02 12:49:18.581665 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:18.574902    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-eb24df-77b8d4b6bc-b7fx5"
Nov 02 12:49:18.675083 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:18.675053    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-eb24df-748477f8f5-67vfv" event={"ID":"9f72904b-21b5-40f8-804d-cd747e649e02","Type":"ContainerStarted","Data":"dd8c60e5de2807b9df4eb4b5acc6450f37b7fba20c78715fbe2288a56360dab8"}
Nov 02 12:49:18.961543 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:49:18.961515    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b7aacab_55fe_482d_8e15_a8f847678e48.slice/crio-88272a1ac8069002b4d33897a05d369c2636f001b9aaa5d51377843e5d6b9ecb WatchSource:0}: Error finding container 88272a1ac8069002b4d33897a05d369c2636f001b9aaa5d51377843e5d6b9ecb: Status 404 returned error can't find the container with id 88272a1ac8069002b4d33897a05d369c2636f001b9aaa5d51377843e5d6b9ecb
Nov 02 12:49:18.961946 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:18.961926    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-eb24df-77b8d4b6bc-b7fx5"]
Nov 02 12:49:19.683701 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:19.683663    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-eb24df-748477f8f5-67vfv" event={"ID":"9f72904b-21b5-40f8-804d-cd747e649e02","Type":"ContainerStarted","Data":"a2766108e68f2f54be02ba0f63c6e1c596ebb23a22966cf318d61082a32c1b33"}
Nov 02 12:49:19.683905 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:19.683833    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-eb24df-748477f8f5-67vfv"
Nov 02 12:49:19.685367 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:19.685340    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-eb24df-77b8d4b6bc-b7fx5" event={"ID":"6b7aacab-55fe-482d-8e15-a8f847678e48","Type":"ContainerStarted","Data":"6269a8abd8b22acd4ad898ef7b00f83a21595260625526b4ed2ce4843f31cf23"}
Nov 02 12:49:19.685507 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:19.685390    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-eb24df-77b8d4b6bc-b7fx5" event={"ID":"6b7aacab-55fe-482d-8e15-a8f847678e48","Type":"ContainerStarted","Data":"88272a1ac8069002b4d33897a05d369c2636f001b9aaa5d51377843e5d6b9ecb"}
Nov 02 12:49:19.685507 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:19.685436    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-eb24df-77b8d4b6bc-b7fx5"
Nov 02 12:49:19.754170 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:19.754115    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-eb24df-748477f8f5-67vfv" podStartSLOduration=2.754098681 podStartE2EDuration="2.754098681s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:49:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:49:19.753358451 +0000 UTC m=+1356.008757547" watchObservedRunningTime="2026-11-02 12:49:19.754098681 +0000 UTC m=+1356.009497779"
Nov 02 12:49:19.800831 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:19.800780    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-eb24df-77b8d4b6bc-b7fx5" podStartSLOduration=2.800766037 podStartE2EDuration="2.800766037s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:49:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:49:19.798210709 +0000 UTC m=+1356.053609803" watchObservedRunningTime="2026-11-02 12:49:19.800766037 +0000 UTC m=+1356.056165131"
Nov 02 12:49:20.329726 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:20.321482    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-299e74-f6d59fd87-5s5pn"]
Nov 02 12:49:20.361985 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:20.361953    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-299e74-f6d59fd87-5s5pn"
Nov 02 12:49:20.413793 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:20.413755    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-299e74-f6d59fd87-5s5pn"]
Nov 02 12:49:20.462156 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:20.454310    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-299e74\""
Nov 02 12:49:20.462156 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:20.454502    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-299e74-dockercfg-rfrlq\""
Nov 02 12:49:20.462156 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:20.458880    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-299e74-78c848cbc6-xk4wx"]
Nov 02 12:49:20.464442 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:20.463820    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-299e74-78c848cbc6-xk4wx"
Nov 02 12:49:20.473959 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:20.473925    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cq8jq\" (UniqueName: \"kubernetes.io/projected/6f136bff-02b6-41b6-9bee-cf5626c366dc-kube-api-access-cq8jq\") pod \"payload-pre-processing-e2e-ipp-b-299e74-f6d59fd87-5s5pn\" (UID: \"6f136bff-02b6-41b6-9bee-cf5626c366dc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-299e74-f6d59fd87-5s5pn"
Nov 02 12:49:20.474124 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:20.474062    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f136bff-02b6-41b6-9bee-cf5626c366dc-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-299e74-f6d59fd87-5s5pn\" (UID: \"6f136bff-02b6-41b6-9bee-cf5626c366dc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-299e74-f6d59fd87-5s5pn"
Nov 02 12:49:20.480891 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:20.480861    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-299e74-78c848cbc6-xk4wx"]
Nov 02 12:49:20.575838 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:20.575799    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qggst\" (UniqueName: \"kubernetes.io/projected/9d93eaf1-b398-4b86-9156-845a73b24aa9-kube-api-access-qggst\") pod \"payload-processing-e2e-ipp-b-299e74-78c848cbc6-xk4wx\" (UID: \"9d93eaf1-b398-4b86-9156-845a73b24aa9\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-299e74-78c848cbc6-xk4wx"
Nov 02 12:49:20.576056 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:20.575868    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cq8jq\" (UniqueName: \"kubernetes.io/projected/6f136bff-02b6-41b6-9bee-cf5626c366dc-kube-api-access-cq8jq\") pod \"payload-pre-processing-e2e-ipp-b-299e74-f6d59fd87-5s5pn\" (UID: \"6f136bff-02b6-41b6-9bee-cf5626c366dc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-299e74-f6d59fd87-5s5pn"
Nov 02 12:49:20.576056 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:20.576005    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f136bff-02b6-41b6-9bee-cf5626c366dc-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-299e74-f6d59fd87-5s5pn\" (UID: \"6f136bff-02b6-41b6-9bee-cf5626c366dc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-299e74-f6d59fd87-5s5pn"
Nov 02 12:49:20.576153 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:20.576110    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d93eaf1-b398-4b86-9156-845a73b24aa9-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-299e74-78c848cbc6-xk4wx\" (UID: \"9d93eaf1-b398-4b86-9156-845a73b24aa9\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-299e74-78c848cbc6-xk4wx"
Nov 02 12:49:20.576734 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:20.576715    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f136bff-02b6-41b6-9bee-cf5626c366dc-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-299e74-f6d59fd87-5s5pn\" (UID: \"6f136bff-02b6-41b6-9bee-cf5626c366dc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-299e74-f6d59fd87-5s5pn"
Nov 02 12:49:20.600877 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:20.600818    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cq8jq\" (UniqueName: \"kubernetes.io/projected/6f136bff-02b6-41b6-9bee-cf5626c366dc-kube-api-access-cq8jq\") pod \"payload-pre-processing-e2e-ipp-b-299e74-f6d59fd87-5s5pn\" (UID: \"6f136bff-02b6-41b6-9bee-cf5626c366dc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-299e74-f6d59fd87-5s5pn"
Nov 02 12:49:20.677516 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:20.677486    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d93eaf1-b398-4b86-9156-845a73b24aa9-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-299e74-78c848cbc6-xk4wx\" (UID: \"9d93eaf1-b398-4b86-9156-845a73b24aa9\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-299e74-78c848cbc6-xk4wx"
Nov 02 12:49:20.677703 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:20.677575    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qggst\" (UniqueName: \"kubernetes.io/projected/9d93eaf1-b398-4b86-9156-845a73b24aa9-kube-api-access-qggst\") pod \"payload-processing-e2e-ipp-b-299e74-78c848cbc6-xk4wx\" (UID: \"9d93eaf1-b398-4b86-9156-845a73b24aa9\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-299e74-78c848cbc6-xk4wx"
Nov 02 12:49:20.678240 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:20.678217    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d93eaf1-b398-4b86-9156-845a73b24aa9-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-299e74-78c848cbc6-xk4wx\" (UID: \"9d93eaf1-b398-4b86-9156-845a73b24aa9\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-299e74-78c848cbc6-xk4wx"
Nov 02 12:49:20.703007 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:20.702982    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qggst\" (UniqueName: \"kubernetes.io/projected/9d93eaf1-b398-4b86-9156-845a73b24aa9-kube-api-access-qggst\") pod \"payload-processing-e2e-ipp-b-299e74-78c848cbc6-xk4wx\" (UID: \"9d93eaf1-b398-4b86-9156-845a73b24aa9\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-299e74-78c848cbc6-xk4wx"
Nov 02 12:49:20.744316 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:20.744287    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-299e74-f6d59fd87-5s5pn"
Nov 02 12:49:20.789914 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:20.789883    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-299e74-78c848cbc6-xk4wx"
Nov 02 12:49:21.047344 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:49:21.047318    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f136bff_02b6_41b6_9bee_cf5626c366dc.slice/crio-8109b794553a14c836500c097089f3c44064ff8f52ea4203d1f59e67820a6179 WatchSource:0}: Error finding container 8109b794553a14c836500c097089f3c44064ff8f52ea4203d1f59e67820a6179: Status 404 returned error can't find the container with id 8109b794553a14c836500c097089f3c44064ff8f52ea4203d1f59e67820a6179
Nov 02 12:49:21.049207 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:21.049180    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-299e74-f6d59fd87-5s5pn"]
Nov 02 12:49:21.186298 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:49:21.186267    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d93eaf1_b398_4b86_9156_845a73b24aa9.slice/crio-03d1db20dd219134dc90c6f8dfab1e7def962c3742fd7da32709a27ea20c5fdc WatchSource:0}: Error finding container 03d1db20dd219134dc90c6f8dfab1e7def962c3742fd7da32709a27ea20c5fdc: Status 404 returned error can't find the container with id 03d1db20dd219134dc90c6f8dfab1e7def962c3742fd7da32709a27ea20c5fdc
Nov 02 12:49:21.187654 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:21.187617    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-299e74-78c848cbc6-xk4wx"]
Nov 02 12:49:21.469297 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:21.469267    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d6d314-77744649f9-rf95p"
Nov 02 12:49:21.469753 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:21.469350    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-d6d314-767dbf49bc-55rvd"
Nov 02 12:49:21.702236 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:21.702194    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-299e74-f6d59fd87-5s5pn" event={"ID":"6f136bff-02b6-41b6-9bee-cf5626c366dc","Type":"ContainerStarted","Data":"fa3e160d1374c182c1eef451397023b8a8a89f3dc776de7c5cd24574d1130d86"}
Nov 02 12:49:21.702435 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:21.702264    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-299e74-f6d59fd87-5s5pn" event={"ID":"6f136bff-02b6-41b6-9bee-cf5626c366dc","Type":"ContainerStarted","Data":"8109b794553a14c836500c097089f3c44064ff8f52ea4203d1f59e67820a6179"}
Nov 02 12:49:21.702435 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:21.702304    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-299e74-f6d59fd87-5s5pn"
Nov 02 12:49:21.704514 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:21.704479    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-299e74-78c848cbc6-xk4wx" event={"ID":"9d93eaf1-b398-4b86-9156-845a73b24aa9","Type":"ContainerStarted","Data":"712feb6225923324880ddeea02e69d552b0dd7f3428ae50d233012ccf9f4f478"}
Nov 02 12:49:21.704694 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:21.704538    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-299e74-78c848cbc6-xk4wx"
Nov 02 12:49:21.704694 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:21.704561    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-299e74-78c848cbc6-xk4wx" event={"ID":"9d93eaf1-b398-4b86-9156-845a73b24aa9","Type":"ContainerStarted","Data":"03d1db20dd219134dc90c6f8dfab1e7def962c3742fd7da32709a27ea20c5fdc"}
Nov 02 12:49:21.746543 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:21.746440    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-299e74-f6d59fd87-5s5pn" podStartSLOduration=1.746423333 podStartE2EDuration="1.746423333s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:49:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:49:21.745425759 +0000 UTC m=+1358.000824853" watchObservedRunningTime="2026-11-02 12:49:21.746423333 +0000 UTC m=+1358.001822433"
Nov 02 12:49:21.790869 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:21.790814    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-299e74-78c848cbc6-xk4wx" podStartSLOduration=1.790796423 podStartE2EDuration="1.790796423s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:49:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:49:21.788386758 +0000 UTC m=+1358.043785857" watchObservedRunningTime="2026-11-02 12:49:21.790796423 +0000 UTC m=+1358.046195521"
Nov 02 12:49:22.607070 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:22.607035    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-e93a4b-7697fd684f-gb8s9"]
Nov 02 12:49:22.611442 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:22.611422    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e93a4b-7697fd684f-gb8s9"
Nov 02 12:49:22.633035 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:22.633003    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-e93a4b-7697fd684f-gb8s9"]
Nov 02 12:49:22.643670 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:22.641880    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-e93a4b-serving-cert\""
Nov 02 12:49:22.663813 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:22.663789    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b271f0-578d584488-vwllx"
Nov 02 12:49:22.734583 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:22.734508    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a673851-931d-4634-8b10-d26a2518552d-metrics-certs\") pod \"maas-api-e2e-praxis-b-e93a4b-7697fd684f-gb8s9\" (UID: \"7a673851-931d-4634-8b10-d26a2518552d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e93a4b-7697fd684f-gb8s9"
Nov 02 12:49:22.734798 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:22.734738    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a673851-931d-4634-8b10-d26a2518552d-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-e93a4b-7697fd684f-gb8s9\" (UID: \"7a673851-931d-4634-8b10-d26a2518552d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e93a4b-7697fd684f-gb8s9"
Nov 02 12:49:22.734876 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:22.734797    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a673851-931d-4634-8b10-d26a2518552d-maas-api-tls\") pod \"maas-api-e2e-praxis-b-e93a4b-7697fd684f-gb8s9\" (UID: \"7a673851-931d-4634-8b10-d26a2518552d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e93a4b-7697fd684f-gb8s9"
Nov 02 12:49:22.734931 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:22.734889    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9pqf\" (UniqueName: \"kubernetes.io/projected/7a673851-931d-4634-8b10-d26a2518552d-kube-api-access-k9pqf\") pod \"maas-api-e2e-praxis-b-e93a4b-7697fd684f-gb8s9\" (UID: \"7a673851-931d-4634-8b10-d26a2518552d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e93a4b-7697fd684f-gb8s9"
Nov 02 12:49:22.841126 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:22.841079    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9pqf\" (UniqueName: \"kubernetes.io/projected/7a673851-931d-4634-8b10-d26a2518552d-kube-api-access-k9pqf\") pod \"maas-api-e2e-praxis-b-e93a4b-7697fd684f-gb8s9\" (UID: \"7a673851-931d-4634-8b10-d26a2518552d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e93a4b-7697fd684f-gb8s9"
Nov 02 12:49:22.842659 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:22.842619    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a673851-931d-4634-8b10-d26a2518552d-metrics-certs\") pod \"maas-api-e2e-praxis-b-e93a4b-7697fd684f-gb8s9\" (UID: \"7a673851-931d-4634-8b10-d26a2518552d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e93a4b-7697fd684f-gb8s9"
Nov 02 12:49:22.844211 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:22.844144    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a673851-931d-4634-8b10-d26a2518552d-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-e93a4b-7697fd684f-gb8s9\" (UID: \"7a673851-931d-4634-8b10-d26a2518552d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e93a4b-7697fd684f-gb8s9"
Nov 02 12:49:22.846043 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:22.846021    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a673851-931d-4634-8b10-d26a2518552d-maas-api-tls\") pod \"maas-api-e2e-praxis-b-e93a4b-7697fd684f-gb8s9\" (UID: \"7a673851-931d-4634-8b10-d26a2518552d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e93a4b-7697fd684f-gb8s9"
Nov 02 12:49:22.847252 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:22.845932    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a673851-931d-4634-8b10-d26a2518552d-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-e93a4b-7697fd684f-gb8s9\" (UID: \"7a673851-931d-4634-8b10-d26a2518552d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e93a4b-7697fd684f-gb8s9"
Nov 02 12:49:22.892450 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:22.892376    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a673851-931d-4634-8b10-d26a2518552d-maas-api-tls\") pod \"maas-api-e2e-praxis-b-e93a4b-7697fd684f-gb8s9\" (UID: \"7a673851-931d-4634-8b10-d26a2518552d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e93a4b-7697fd684f-gb8s9"
Nov 02 12:49:22.900594 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:22.895215    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a673851-931d-4634-8b10-d26a2518552d-metrics-certs\") pod \"maas-api-e2e-praxis-b-e93a4b-7697fd684f-gb8s9\" (UID: \"7a673851-931d-4634-8b10-d26a2518552d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e93a4b-7697fd684f-gb8s9"
Nov 02 12:49:22.906505 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:22.906378    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9pqf\" (UniqueName: \"kubernetes.io/projected/7a673851-931d-4634-8b10-d26a2518552d-kube-api-access-k9pqf\") pod \"maas-api-e2e-praxis-b-e93a4b-7697fd684f-gb8s9\" (UID: \"7a673851-931d-4634-8b10-d26a2518552d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e93a4b-7697fd684f-gb8s9"
Nov 02 12:49:22.927190 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:22.926907    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e93a4b-7697fd684f-gb8s9"
Nov 02 12:49:23.262945 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:49:23.262918    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a673851_931d_4634_8b10_d26a2518552d.slice/crio-6f5e19279eba69dd9d9e1b232a655496fdf5bfe65447469b3abf53262fd8d784 WatchSource:0}: Error finding container 6f5e19279eba69dd9d9e1b232a655496fdf5bfe65447469b3abf53262fd8d784: Status 404 returned error can't find the container with id 6f5e19279eba69dd9d9e1b232a655496fdf5bfe65447469b3abf53262fd8d784
Nov 02 12:49:23.264290 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:23.264268    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-e93a4b-7697fd684f-gb8s9"]
Nov 02 12:49:23.718516 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:23.718482    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e93a4b-7697fd684f-gb8s9" event={"ID":"7a673851-931d-4634-8b10-d26a2518552d","Type":"ContainerStarted","Data":"30fe7da74ba6e9084990cbc7fc3bc0d546f5df2580fdec2fd30b4691bcc05cb0"}
Nov 02 12:49:23.718937 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:23.718538    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e93a4b-7697fd684f-gb8s9" event={"ID":"7a673851-931d-4634-8b10-d26a2518552d","Type":"ContainerStarted","Data":"6f5e19279eba69dd9d9e1b232a655496fdf5bfe65447469b3abf53262fd8d784"}
Nov 02 12:49:23.718937 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:23.718567    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e93a4b-7697fd684f-gb8s9"
Nov 02 12:49:23.773414 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:23.773231    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e93a4b-7697fd684f-gb8s9" podStartSLOduration=1.425404584 podStartE2EDuration="1.773211375s" totalImagesPullingTime="347.806791ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:49:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:49:23.770979002 +0000 UTC m=+1360.026378098" watchObservedRunningTime="2026-11-02 12:49:23.773211375 +0000 UTC m=+1360.028610470"
Nov 02 12:49:26.430662 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:26.422010    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"]
Nov 02 12:49:28.335435 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:28.335392    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-d6d314-67c7c65cdd-hb2gt"]
Nov 02 12:49:28.343528 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:28.343503    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d6d314-67c7c65cdd-hb2gt"
Nov 02 12:49:28.360080 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:28.360045    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-d6d314-67c7c65cdd-hb2gt"]
Nov 02 12:49:28.369600 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:28.369570    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-d6d314-serving-cert\""
Nov 02 12:49:28.452323 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:28.452289    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/af8db105-0f50-418c-b155-5bbef3014b00-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-d6d314-67c7c65cdd-hb2gt\" (UID: \"af8db105-0f50-418c-b155-5bbef3014b00\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d6d314-67c7c65cdd-hb2gt"
Nov 02 12:49:28.452499 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:28.452331    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af8db105-0f50-418c-b155-5bbef3014b00-metrics-certs\") pod \"maas-api-e2e-praxis-a-d6d314-67c7c65cdd-hb2gt\" (UID: \"af8db105-0f50-418c-b155-5bbef3014b00\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d6d314-67c7c65cdd-hb2gt"
Nov 02 12:49:28.452499 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:28.452473    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af8db105-0f50-418c-b155-5bbef3014b00-maas-api-tls\") pod \"maas-api-e2e-praxis-a-d6d314-67c7c65cdd-hb2gt\" (UID: \"af8db105-0f50-418c-b155-5bbef3014b00\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d6d314-67c7c65cdd-hb2gt"
Nov 02 12:49:28.452583 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:28.452506    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9rx4\" (UniqueName: \"kubernetes.io/projected/af8db105-0f50-418c-b155-5bbef3014b00-kube-api-access-f9rx4\") pod \"maas-api-e2e-praxis-a-d6d314-67c7c65cdd-hb2gt\" (UID: \"af8db105-0f50-418c-b155-5bbef3014b00\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d6d314-67c7c65cdd-hb2gt"
Nov 02 12:49:28.553812 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:28.553778    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/af8db105-0f50-418c-b155-5bbef3014b00-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-d6d314-67c7c65cdd-hb2gt\" (UID: \"af8db105-0f50-418c-b155-5bbef3014b00\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d6d314-67c7c65cdd-hb2gt"
Nov 02 12:49:28.553812 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:28.553817    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af8db105-0f50-418c-b155-5bbef3014b00-metrics-certs\") pod \"maas-api-e2e-praxis-a-d6d314-67c7c65cdd-hb2gt\" (UID: \"af8db105-0f50-418c-b155-5bbef3014b00\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d6d314-67c7c65cdd-hb2gt"
Nov 02 12:49:28.554028 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:28.553912    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af8db105-0f50-418c-b155-5bbef3014b00-maas-api-tls\") pod \"maas-api-e2e-praxis-a-d6d314-67c7c65cdd-hb2gt\" (UID: \"af8db105-0f50-418c-b155-5bbef3014b00\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d6d314-67c7c65cdd-hb2gt"
Nov 02 12:49:28.554028 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:28.553961    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9rx4\" (UniqueName: \"kubernetes.io/projected/af8db105-0f50-418c-b155-5bbef3014b00-kube-api-access-f9rx4\") pod \"maas-api-e2e-praxis-a-d6d314-67c7c65cdd-hb2gt\" (UID: \"af8db105-0f50-418c-b155-5bbef3014b00\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d6d314-67c7c65cdd-hb2gt"
Nov 02 12:49:28.554524 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:28.554503    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/af8db105-0f50-418c-b155-5bbef3014b00-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-d6d314-67c7c65cdd-hb2gt\" (UID: \"af8db105-0f50-418c-b155-5bbef3014b00\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d6d314-67c7c65cdd-hb2gt"
Nov 02 12:49:28.556434 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:28.556410    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af8db105-0f50-418c-b155-5bbef3014b00-metrics-certs\") pod \"maas-api-e2e-praxis-a-d6d314-67c7c65cdd-hb2gt\" (UID: \"af8db105-0f50-418c-b155-5bbef3014b00\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d6d314-67c7c65cdd-hb2gt"
Nov 02 12:49:28.556521 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:28.556466    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af8db105-0f50-418c-b155-5bbef3014b00-maas-api-tls\") pod \"maas-api-e2e-praxis-a-d6d314-67c7c65cdd-hb2gt\" (UID: \"af8db105-0f50-418c-b155-5bbef3014b00\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d6d314-67c7c65cdd-hb2gt"
Nov 02 12:49:28.578517 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:28.578488    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9rx4\" (UniqueName: \"kubernetes.io/projected/af8db105-0f50-418c-b155-5bbef3014b00-kube-api-access-f9rx4\") pod \"maas-api-e2e-praxis-a-d6d314-67c7c65cdd-hb2gt\" (UID: \"af8db105-0f50-418c-b155-5bbef3014b00\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d6d314-67c7c65cdd-hb2gt"
Nov 02 12:49:28.660476 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:28.660323    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d6d314-67c7c65cdd-hb2gt"
Nov 02 12:49:29.078181 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:49:29.078152    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf8db105_0f50_418c_b155_5bbef3014b00.slice/crio-75dc2f512c0879650c76518d560668b5234feb85645f3f290f73a782988b4128 WatchSource:0}: Error finding container 75dc2f512c0879650c76518d560668b5234feb85645f3f290f73a782988b4128: Status 404 returned error can't find the container with id 75dc2f512c0879650c76518d560668b5234feb85645f3f290f73a782988b4128
Nov 02 12:49:29.078434 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:29.078410    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-d6d314-67c7c65cdd-hb2gt"]
Nov 02 12:49:29.736307 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:29.736223    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e93a4b-7697fd684f-gb8s9"
Nov 02 12:49:29.805670 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:29.781821    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d6d314-67c7c65cdd-hb2gt" event={"ID":"af8db105-0f50-418c-b155-5bbef3014b00","Type":"ContainerStarted","Data":"babc7e55f3903f5c607674835f3a6b29d7e27f080634cbcb8505380b4249784e"}
Nov 02 12:49:29.805670 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:29.781893    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d6d314-67c7c65cdd-hb2gt" event={"ID":"af8db105-0f50-418c-b155-5bbef3014b00","Type":"ContainerStarted","Data":"75dc2f512c0879650c76518d560668b5234feb85645f3f290f73a782988b4128"}
Nov 02 12:49:29.805670 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:29.782018    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d6d314-67c7c65cdd-hb2gt"
Nov 02 12:49:29.832305 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:29.832251    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d6d314-67c7c65cdd-hb2gt" podStartSLOduration=1.48241833 podStartE2EDuration="1.832237716s" totalImagesPullingTime="349.819386ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:49:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:49:29.829791007 +0000 UTC m=+1366.085190104" watchObservedRunningTime="2026-11-02 12:49:29.832237716 +0000 UTC m=+1366.087636810"
Nov 02 12:49:30.708690 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:30.708660    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-eb24df-748477f8f5-67vfv"
Nov 02 12:49:30.709023 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:30.708992    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-eb24df-77b8d4b6bc-b7fx5"
Nov 02 12:49:32.713022 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:32.712989    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-299e74-f6d59fd87-5s5pn"
Nov 02 12:49:32.713513 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:32.713077    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-299e74-78c848cbc6-xk4wx"
Nov 02 12:49:35.825753 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:35.825723    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d6d314-67c7c65cdd-hb2gt"
Nov 02 12:49:36.866625 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:36.866547    4559 generic.go:356] "Generic (PLEG): container finished" podID="e5435e4b-b8a8-45b9-a90a-bca4b5a65506" containerID="b79c65999f4b4d8153087bcfb6f1e7caf141c10c574e6be52b7157b40c272036" exitCode=0
Nov 02 12:49:36.866625 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:36.866593    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-b271f0-openshift-default-5544549cdf-q9c6j" event={"ID":"e5435e4b-b8a8-45b9-a90a-bca4b5a65506","Type":"ContainerDied","Data":"b79c65999f4b4d8153087bcfb6f1e7caf141c10c574e6be52b7157b40c272036"}
Nov 02 12:49:36.867135 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:36.867117    4559 scope.go:122] "RemoveContainer" containerID="b79c65999f4b4d8153087bcfb6f1e7caf141c10c574e6be52b7157b40c272036"
Nov 02 12:49:36.868814 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:36.868784    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:49:36.868901 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:36.868863    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:49:36.868901 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:36.868897    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:49:37.876173 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:37.876138    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-b271f0-openshift-default-5544549cdf-q9c6j" event={"ID":"e5435e4b-b8a8-45b9-a90a-bca4b5a65506","Type":"ContainerStarted","Data":"860a818cf8db5e8a6f7f52c2740074327c73bf66f4c85bb806b694f683003b7a"}
Nov 02 12:49:38.705662 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:38.705365    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-b271f0-openshift-default-5544549cdf-q9c6j"
Nov 02 12:49:38.705662 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:38.705500    4559 patch_prober.go:29] interesting pod/e2e-praxis-a-b271f0-openshift-default-5544549cdf-q9c6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body=
Nov 02 12:49:38.705662 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:38.705542    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-b271f0-openshift-default-5544549cdf-q9c6j" podUID="e5435e4b-b8a8-45b9-a90a-bca4b5a65506" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused"
Nov 02 12:49:39.696672 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:39.694803    4559 patch_prober.go:29] interesting pod/e2e-praxis-a-b271f0-openshift-default-5544549cdf-q9c6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body=
Nov 02 12:49:39.696672 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:39.694897    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-b271f0-openshift-default-5544549cdf-q9c6j" podUID="e5435e4b-b8a8-45b9-a90a-bca4b5a65506" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused"
Nov 02 12:49:40.274187 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:40.274150    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-b271f0-openshift-default-5544549cdf-q9c6j"
Nov 02 12:49:40.698665 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:40.695605    4559 patch_prober.go:29] interesting pod/e2e-praxis-a-b271f0-openshift-default-5544549cdf-q9c6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body=
Nov 02 12:49:40.698665 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:40.695692    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-b271f0-openshift-default-5544549cdf-q9c6j" podUID="e5435e4b-b8a8-45b9-a90a-bca4b5a65506" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused"
Nov 02 12:49:41.700041 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:41.700000    4559 patch_prober.go:29] interesting pod/e2e-praxis-a-b271f0-openshift-default-5544549cdf-q9c6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body=
Nov 02 12:49:41.702405 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:41.702372    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-b271f0-openshift-default-5544549cdf-q9c6j" podUID="e5435e4b-b8a8-45b9-a90a-bca4b5a65506" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused"
Nov 02 12:49:42.706664 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:42.700004    4559 patch_prober.go:29] interesting pod/e2e-praxis-a-b271f0-openshift-default-5544549cdf-q9c6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body=
Nov 02 12:49:42.706664 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:42.700068    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-b271f0-openshift-default-5544549cdf-q9c6j" podUID="e5435e4b-b8a8-45b9-a90a-bca4b5a65506" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused"
Nov 02 12:49:43.065014 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:43.064909    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-s8hnh"]
Nov 02 12:49:43.093691 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:43.088618    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-s8hnh"
Nov 02 12:49:43.104585 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:43.104555    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-s8hnh"]
Nov 02 12:49:43.105415 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:43.105394    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-9750c7-openshift-default-dockercfg-6f96j\""
Nov 02 12:49:43.234822 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:43.234787    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7666985a-ed93-432a-9bd8-c668a7bfd8f5-istio-token\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-s8hnh\" (UID: \"7666985a-ed93-432a-9bd8-c668a7bfd8f5\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-s8hnh"
Nov 02 12:49:43.235008 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:43.234859    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7666985a-ed93-432a-9bd8-c668a7bfd8f5-credential-socket\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-s8hnh\" (UID: \"7666985a-ed93-432a-9bd8-c668a7bfd8f5\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-s8hnh"
Nov 02 12:49:43.235072 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:43.234995    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7666985a-ed93-432a-9bd8-c668a7bfd8f5-istio-envoy\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-s8hnh\" (UID: \"7666985a-ed93-432a-9bd8-c668a7bfd8f5\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-s8hnh"
Nov 02 12:49:43.235072 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:43.235058    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7666985a-ed93-432a-9bd8-c668a7bfd8f5-istiod-ca-cert\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-s8hnh\" (UID: \"7666985a-ed93-432a-9bd8-c668a7bfd8f5\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-s8hnh"
Nov 02 12:49:43.235205 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:43.235179    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7666985a-ed93-432a-9bd8-c668a7bfd8f5-workload-socket\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-s8hnh\" (UID: \"7666985a-ed93-432a-9bd8-c668a7bfd8f5\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-s8hnh"
Nov 02 12:49:43.235274 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:43.235261    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7666985a-ed93-432a-9bd8-c668a7bfd8f5-workload-certs\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-s8hnh\" (UID: \"7666985a-ed93-432a-9bd8-c668a7bfd8f5\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-s8hnh"
Nov 02 12:49:43.235334 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:43.235313    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4m4q\" (UniqueName: \"kubernetes.io/projected/7666985a-ed93-432a-9bd8-c668a7bfd8f5-kube-api-access-j4m4q\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-s8hnh\" (UID: \"7666985a-ed93-432a-9bd8-c668a7bfd8f5\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-s8hnh"
Nov 02 12:49:43.235416 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:43.235402    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7666985a-ed93-432a-9bd8-c668a7bfd8f5-istio-podinfo\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-s8hnh\" (UID: \"7666985a-ed93-432a-9bd8-c668a7bfd8f5\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-s8hnh"
Nov 02 12:49:43.235511 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:43.235490    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7666985a-ed93-432a-9bd8-c668a7bfd8f5-istio-data\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-s8hnh\" (UID: \"7666985a-ed93-432a-9bd8-c668a7bfd8f5\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-s8hnh"
Nov 02 12:49:43.378717 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:43.376624    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7666985a-ed93-432a-9bd8-c668a7bfd8f5-credential-socket\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-s8hnh\" (UID: \"7666985a-ed93-432a-9bd8-c668a7bfd8f5\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-s8hnh"
Nov 02 12:49:43.378717 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:43.376766    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7666985a-ed93-432a-9bd8-c668a7bfd8f5-istio-envoy\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-s8hnh\" (UID: \"7666985a-ed93-432a-9bd8-c668a7bfd8f5\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-s8hnh"
Nov 02 12:49:43.378717 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:43.376837    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7666985a-ed93-432a-9bd8-c668a7bfd8f5-istiod-ca-cert\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-s8hnh\" (UID: \"7666985a-ed93-432a-9bd8-c668a7bfd8f5\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-s8hnh"
Nov 02 12:49:43.378717 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:43.377024    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7666985a-ed93-432a-9bd8-c668a7bfd8f5-workload-socket\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-s8hnh\" (UID: \"7666985a-ed93-432a-9bd8-c668a7bfd8f5\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-s8hnh"
Nov 02 12:49:43.378717 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:43.377109    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7666985a-ed93-432a-9bd8-c668a7bfd8f5-workload-certs\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-s8hnh\" (UID: \"7666985a-ed93-432a-9bd8-c668a7bfd8f5\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-s8hnh"
Nov 02 12:49:43.378717 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:43.377169    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4m4q\" (UniqueName: \"kubernetes.io/projected/7666985a-ed93-432a-9bd8-c668a7bfd8f5-kube-api-access-j4m4q\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-s8hnh\" (UID: \"7666985a-ed93-432a-9bd8-c668a7bfd8f5\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-s8hnh"
Nov 02 12:49:43.378717 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:43.377245    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7666985a-ed93-432a-9bd8-c668a7bfd8f5-istio-podinfo\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-s8hnh\" (UID: \"7666985a-ed93-432a-9bd8-c668a7bfd8f5\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-s8hnh"
Nov 02 12:49:43.378717 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:43.377358    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7666985a-ed93-432a-9bd8-c668a7bfd8f5-istio-data\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-s8hnh\" (UID: \"7666985a-ed93-432a-9bd8-c668a7bfd8f5\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-s8hnh"
Nov 02 12:49:43.378717 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:43.377480    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7666985a-ed93-432a-9bd8-c668a7bfd8f5-istio-token\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-s8hnh\" (UID: \"7666985a-ed93-432a-9bd8-c668a7bfd8f5\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-s8hnh"
Nov 02 12:49:43.392325 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:43.392290    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7666985a-ed93-432a-9bd8-c668a7bfd8f5-workload-socket\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-s8hnh\" (UID: \"7666985a-ed93-432a-9bd8-c668a7bfd8f5\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-s8hnh"
Nov 02 12:49:43.392908 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:43.392884    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7666985a-ed93-432a-9bd8-c668a7bfd8f5-credential-socket\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-s8hnh\" (UID: \"7666985a-ed93-432a-9bd8-c668a7bfd8f5\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-s8hnh"
Nov 02 12:49:43.416662 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:43.398113    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7666985a-ed93-432a-9bd8-c668a7bfd8f5-workload-certs\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-s8hnh\" (UID: \"7666985a-ed93-432a-9bd8-c668a7bfd8f5\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-s8hnh"
Nov 02 12:49:43.416662 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:43.400152    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7666985a-ed93-432a-9bd8-c668a7bfd8f5-istio-data\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-s8hnh\" (UID: \"7666985a-ed93-432a-9bd8-c668a7bfd8f5\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-s8hnh"
Nov 02 12:49:43.416662 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:43.400566    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7666985a-ed93-432a-9bd8-c668a7bfd8f5-istiod-ca-cert\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-s8hnh\" (UID: \"7666985a-ed93-432a-9bd8-c668a7bfd8f5\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-s8hnh"
Nov 02 12:49:43.439661 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:43.420305    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7666985a-ed93-432a-9bd8-c668a7bfd8f5-istio-podinfo\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-s8hnh\" (UID: \"7666985a-ed93-432a-9bd8-c668a7bfd8f5\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-s8hnh"
Nov 02 12:49:43.439916 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:43.425309    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7666985a-ed93-432a-9bd8-c668a7bfd8f5-istio-envoy\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-s8hnh\" (UID: \"7666985a-ed93-432a-9bd8-c668a7bfd8f5\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-s8hnh"
Nov 02 12:49:43.468220 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:43.468097    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7666985a-ed93-432a-9bd8-c668a7bfd8f5-istio-token\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-s8hnh\" (UID: \"7666985a-ed93-432a-9bd8-c668a7bfd8f5\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-s8hnh"
Nov 02 12:49:43.468654 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:43.468612    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4m4q\" (UniqueName: \"kubernetes.io/projected/7666985a-ed93-432a-9bd8-c668a7bfd8f5-kube-api-access-j4m4q\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-s8hnh\" (UID: \"7666985a-ed93-432a-9bd8-c668a7bfd8f5\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-s8hnh"
Nov 02 12:49:43.699074 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:43.699044    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-b271f0-openshift-default-5544549cdf-q9c6j"
Nov 02 12:49:43.700283 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:43.700257    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-b271f0-openshift-default-5544549cdf-q9c6j"
Nov 02 12:49:43.724249 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:43.724226    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-s8hnh"
Nov 02 12:49:44.044026 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:49:44.043984    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7666985a_ed93_432a_9bd8_c668a7bfd8f5.slice/crio-59d5eca8ec2c4160076dd57f8d29c2f552b11a7873c3699425034e618f19394a WatchSource:0}: Error finding container 59d5eca8ec2c4160076dd57f8d29c2f552b11a7873c3699425034e618f19394a: Status 404 returned error can't find the container with id 59d5eca8ec2c4160076dd57f8d29c2f552b11a7873c3699425034e618f19394a
Nov 02 12:49:44.044301 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:44.044281    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-s8hnh"]
Nov 02 12:49:44.046713 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:44.046678    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:49:44.046801 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:44.046745    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:49:44.046801 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:44.046773    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:49:44.995337 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:44.995300    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-s8hnh" event={"ID":"7666985a-ed93-432a-9bd8-c668a7bfd8f5","Type":"ContainerStarted","Data":"e1d6e40836826a764fb8d275885e3e19d0e23253fe2869d187becd49b38acd28"}
Nov 02 12:49:44.995760 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:44.995363    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-s8hnh" event={"ID":"7666985a-ed93-432a-9bd8-c668a7bfd8f5","Type":"ContainerStarted","Data":"59d5eca8ec2c4160076dd57f8d29c2f552b11a7873c3699425034e618f19394a"}
Nov 02 12:49:45.058650 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:45.058588    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-s8hnh" podStartSLOduration=2.058575738 podStartE2EDuration="2.058575738s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:49:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:49:45.056697869 +0000 UTC m=+1381.312096965" watchObservedRunningTime="2026-11-02 12:49:45.058575738 +0000 UTC m=+1381.313974833"
Nov 02 12:49:45.086605 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:49:45.086569    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqcmj" containerName="extractor"
Nov 02 12:49:45.119380 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:45.119352    4559 scope.go:122] "RemoveContainer" containerID="6e3956e5bcf36217b7354b75a1e0890ee6a3272d127e26d9d2cb4cc6b50c10a5"
Nov 02 12:49:45.724437 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:45.724404    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-s8hnh"
Nov 02 12:49:45.732349 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:45.732322    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-s8hnh"
Nov 02 12:49:46.002312 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:46.002230    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-s8hnh"
Nov 02 12:49:46.003287 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:46.003267    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-s8hnh"
Nov 02 12:49:49.419349 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:49.419302    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"]
Nov 02 12:49:49.457664 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:49.457610    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-9750c7-5df98f984c-mplnn"]
Nov 02 12:49:49.463984 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:49.463954    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-9750c7-5df98f984c-mplnn"
Nov 02 12:49:49.482375 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:49.482339    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-9750c7-5df98f984c-mplnn"]
Nov 02 12:49:49.505461 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:49.505412    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-9750c7\""
Nov 02 12:49:49.505662 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:49.505585    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-9750c7-dockercfg-gclk9\""
Nov 02 12:49:49.639663 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:49.635162    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlnng\" (UniqueName: \"kubernetes.io/projected/19089e8c-1e56-4d5a-9697-3c3477bcd803-kube-api-access-tlnng\") pod \"payload-processing-e2e-praxis-b-9750c7-5df98f984c-mplnn\" (UID: \"19089e8c-1e56-4d5a-9697-3c3477bcd803\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-9750c7-5df98f984c-mplnn"
Nov 02 12:49:49.639663 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:49.635377    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19089e8c-1e56-4d5a-9697-3c3477bcd803-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-9750c7-5df98f984c-mplnn\" (UID: \"19089e8c-1e56-4d5a-9697-3c3477bcd803\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-9750c7-5df98f984c-mplnn"
Nov 02 12:49:49.738902 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:49.738164    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlnng\" (UniqueName: \"kubernetes.io/projected/19089e8c-1e56-4d5a-9697-3c3477bcd803-kube-api-access-tlnng\") pod \"payload-processing-e2e-praxis-b-9750c7-5df98f984c-mplnn\" (UID: \"19089e8c-1e56-4d5a-9697-3c3477bcd803\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-9750c7-5df98f984c-mplnn"
Nov 02 12:49:49.738902 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:49.738289    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19089e8c-1e56-4d5a-9697-3c3477bcd803-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-9750c7-5df98f984c-mplnn\" (UID: \"19089e8c-1e56-4d5a-9697-3c3477bcd803\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-9750c7-5df98f984c-mplnn"
Nov 02 12:49:49.739693 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:49.739256    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19089e8c-1e56-4d5a-9697-3c3477bcd803-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-9750c7-5df98f984c-mplnn\" (UID: \"19089e8c-1e56-4d5a-9697-3c3477bcd803\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-9750c7-5df98f984c-mplnn"
Nov 02 12:49:49.809519 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:49.809487    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlnng\" (UniqueName: \"kubernetes.io/projected/19089e8c-1e56-4d5a-9697-3c3477bcd803-kube-api-access-tlnng\") pod \"payload-processing-e2e-praxis-b-9750c7-5df98f984c-mplnn\" (UID: \"19089e8c-1e56-4d5a-9697-3c3477bcd803\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-9750c7-5df98f984c-mplnn"
Nov 02 12:49:50.093716 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:50.090217    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-9750c7-5df98f984c-mplnn"
Nov 02 12:49:50.533000 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:50.532964    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-9750c7-5df98f984c-mplnn"]
Nov 02 12:49:50.535001 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:49:50.534966    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19089e8c_1e56_4d5a_9697_3c3477bcd803.slice/crio-3876c6a7b8caaa2c21f2fa94207721f6459d4318f822df26329d7d0180b96da0 WatchSource:0}: Error finding container 3876c6a7b8caaa2c21f2fa94207721f6459d4318f822df26329d7d0180b96da0: Status 404 returned error can't find the container with id 3876c6a7b8caaa2c21f2fa94207721f6459d4318f822df26329d7d0180b96da0
Nov 02 12:49:51.079604 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:51.079571    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-9750c7-5df98f984c-mplnn" event={"ID":"19089e8c-1e56-4d5a-9697-3c3477bcd803","Type":"ContainerStarted","Data":"47dbc90563b222dc0e8db6fc0241a7cdf0917bcf2ebfd47b358cc1a739fc0efd"}
Nov 02 12:49:51.079817 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:51.079622    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-9750c7-5df98f984c-mplnn" event={"ID":"19089e8c-1e56-4d5a-9697-3c3477bcd803","Type":"ContainerStarted","Data":"3876c6a7b8caaa2c21f2fa94207721f6459d4318f822df26329d7d0180b96da0"}
Nov 02 12:49:51.079817 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:51.079692    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-9750c7-5df98f984c-mplnn"
Nov 02 12:49:51.139594 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:49:51.139531    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-9750c7-5df98f984c-mplnn" podStartSLOduration=2.139510584 podStartE2EDuration="2.139510584s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:49:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:49:51.139067183 +0000 UTC m=+1387.394466279" watchObservedRunningTime="2026-11-02 12:49:51.139510584 +0000 UTC m=+1387.394909680"
Nov 02 12:50:00.609798 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:00.609747    4559 patch_prober.go:29] interesting pod/e2e-ipp-a-eb24df-openshift-default-67f48b7f65-fg527 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 12:50:00.610454 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:00.609831    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-eb24df-openshift-default-67f48b7f65-fg527" podUID="f19dde18-978d-4894-83d3-5c39478c866c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:50:01.373851 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:01.373804    4559 patch_prober.go:29] interesting pod/e2e-praxis-b-e93a4b-openshift-default-7f8c56bf9c-4pjxc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 12:50:01.374067 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:01.373871    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-e93a4b-openshift-default-7f8c56bf9c-4pjxc" podUID="624a0494-692e-4864-810e-42ba5306a5f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:50:02.132625 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:02.132554    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-9750c7-5df98f984c-mplnn"
Nov 02 12:50:03.088796 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:03.088766    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-ee509d-6d46494d8c-q2qwp"]
Nov 02 12:50:03.125072 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:03.125045    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-ee509d-5568559575-g8848"]
Nov 02 12:50:03.253575 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:03.253545    4559 generic.go:356] "Generic (PLEG): container finished" podID="f19dde18-978d-4894-83d3-5c39478c866c" containerID="6ec22c25fd91edf8dac1aa8536d95431284d95d3c6d0009924cbe8851f1f9949" exitCode=0
Nov 02 12:50:03.253923 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:03.253614    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-eb24df-openshift-default-67f48b7f65-fg527" event={"ID":"f19dde18-978d-4894-83d3-5c39478c866c","Type":"ContainerDied","Data":"6ec22c25fd91edf8dac1aa8536d95431284d95d3c6d0009924cbe8851f1f9949"}
Nov 02 12:50:03.254177 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:03.254159    4559 scope.go:122] "RemoveContainer" containerID="6ec22c25fd91edf8dac1aa8536d95431284d95d3c6d0009924cbe8851f1f9949"
Nov 02 12:50:03.255990 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:03.255961    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:50:03.256097 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:03.256033    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:50:03.256097 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:03.256068    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:50:04.261351 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:04.261320    4559 generic.go:356] "Generic (PLEG): container finished" podID="4ac6508c-24db-40f7-a14c-0964dac7768b" containerID="41768043411c52609ed00008bc4c1517796cd57983acc45534761dacdf6cf1d0" exitCode=0
Nov 02 12:50:04.261798 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:04.261389    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-d6d314-openshift-default-5b99b69565-2kkgt" event={"ID":"4ac6508c-24db-40f7-a14c-0964dac7768b","Type":"ContainerDied","Data":"41768043411c52609ed00008bc4c1517796cd57983acc45534761dacdf6cf1d0"}
Nov 02 12:50:04.261903 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:04.261888    4559 scope.go:122] "RemoveContainer" containerID="41768043411c52609ed00008bc4c1517796cd57983acc45534761dacdf6cf1d0"
Nov 02 12:50:04.263613 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:04.263594    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-eb24df-openshift-default-67f48b7f65-fg527" event={"ID":"f19dde18-978d-4894-83d3-5c39478c866c","Type":"ContainerStarted","Data":"cbb8fc2b4dd1bab56485740516f2c420f0785c1f1bd528be5a84f6ac0fd9ccb6"}
Nov 02 12:50:04.263734 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:04.263621    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:50:04.263791 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:04.263743    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:50:04.263829 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:04.263792    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:50:04.265648 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:04.265610    4559 generic.go:356] "Generic (PLEG): container finished" podID="624a0494-692e-4864-810e-42ba5306a5f0" containerID="55f6f8b469fdf1e3896bdc3ce5359712e1a572fe5b7d996b9744071c52c96126" exitCode=0
Nov 02 12:50:04.265743 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:04.265722    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-e93a4b-openshift-default-7f8c56bf9c-4pjxc" event={"ID":"624a0494-692e-4864-810e-42ba5306a5f0","Type":"ContainerDied","Data":"55f6f8b469fdf1e3896bdc3ce5359712e1a572fe5b7d996b9744071c52c96126"}
Nov 02 12:50:04.266111 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:04.266093    4559 scope.go:122] "RemoveContainer" containerID="55f6f8b469fdf1e3896bdc3ce5359712e1a572fe5b7d996b9744071c52c96126"
Nov 02 12:50:04.267518 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:04.267493    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:50:04.267569 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:04.267557    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:50:04.267605 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:04.267590    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:50:05.143743 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:05.142978    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-eb24df-openshift-default-67f48b7f65-fg527"
Nov 02 12:50:05.143743 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:05.143102    4559 patch_prober.go:29] interesting pod/e2e-ipp-a-eb24df-openshift-default-67f48b7f65-fg527 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body=
Nov 02 12:50:05.143743 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:05.143145    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-eb24df-openshift-default-67f48b7f65-fg527" podUID="f19dde18-978d-4894-83d3-5c39478c866c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused"
Nov 02 12:50:05.275867 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:05.275830    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-e93a4b-openshift-default-7f8c56bf9c-4pjxc" event={"ID":"624a0494-692e-4864-810e-42ba5306a5f0","Type":"ContainerStarted","Data":"3d0e1a2a8726fba0000726f99720499ac2487a10d6b8d0553405a1f0ebe072ef"}
Nov 02 12:50:05.277975 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:05.277951    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-d6d314-openshift-default-5b99b69565-2kkgt" event={"ID":"4ac6508c-24db-40f7-a14c-0964dac7768b","Type":"ContainerStarted","Data":"7cfb9e4cf80abe147a36dfb196732604cd012e5a6b624875c3b2451d3f18a047"}
Nov 02 12:50:05.629654 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:05.629593    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-d3a281-844b5468d7-hvsnh"]
Nov 02 12:50:05.682354 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:05.678084    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-d6d314-openshift-default-5b99b69565-2kkgt"
Nov 02 12:50:05.700684 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:05.699931    4559 patch_prober.go:29] interesting pod/e2e-praxis-a-d6d314-openshift-default-5b99b69565-2kkgt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body=
Nov 02 12:50:05.700684 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:05.699993    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-d6d314-openshift-default-5b99b69565-2kkgt" podUID="4ac6508c-24db-40f7-a14c-0964dac7768b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused"
Nov 02 12:50:05.748152 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:05.741509    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-d3a281-844b5468d7-hvsnh"
Nov 02 12:50:05.781116 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:05.780797    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-d3a281-dockercfg-5glrg\""
Nov 02 12:50:05.781116 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:05.780892    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-d3a281\""
Nov 02 12:50:05.810210 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:05.810160    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19ffc8e8-e659-4879-b936-29bd33dfc5a9-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-d3a281-844b5468d7-hvsnh\" (UID: \"19ffc8e8-e659-4879-b936-29bd33dfc5a9\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-d3a281-844b5468d7-hvsnh"
Nov 02 12:50:05.814847 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:05.814805    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njp6m\" (UniqueName: \"kubernetes.io/projected/19ffc8e8-e659-4879-b936-29bd33dfc5a9-kube-api-access-njp6m\") pod \"payload-processing-e2e-praxis-b-d3a281-844b5468d7-hvsnh\" (UID: \"19ffc8e8-e659-4879-b936-29bd33dfc5a9\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-d3a281-844b5468d7-hvsnh"
Nov 02 12:50:05.840818 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:05.840765    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-e93a4b-openshift-default-7f8c56bf9c-4pjxc"
Nov 02 12:50:05.878724 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:05.872694    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-d3a281-844b5468d7-hvsnh"]
Nov 02 12:50:05.878724 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:05.873244    4559 patch_prober.go:29] interesting pod/e2e-praxis-b-e93a4b-openshift-default-7f8c56bf9c-4pjxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body=
Nov 02 12:50:05.878724 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:05.873295    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-e93a4b-openshift-default-7f8c56bf9c-4pjxc" podUID="624a0494-692e-4864-810e-42ba5306a5f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused"
Nov 02 12:50:05.981715 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:05.979779    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19ffc8e8-e659-4879-b936-29bd33dfc5a9-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-d3a281-844b5468d7-hvsnh\" (UID: \"19ffc8e8-e659-4879-b936-29bd33dfc5a9\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-d3a281-844b5468d7-hvsnh"
Nov 02 12:50:05.981715 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:05.979901    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njp6m\" (UniqueName: \"kubernetes.io/projected/19ffc8e8-e659-4879-b936-29bd33dfc5a9-kube-api-access-njp6m\") pod \"payload-processing-e2e-praxis-b-d3a281-844b5468d7-hvsnh\" (UID: \"19ffc8e8-e659-4879-b936-29bd33dfc5a9\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-d3a281-844b5468d7-hvsnh"
Nov 02 12:50:05.981715 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:05.980742    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19ffc8e8-e659-4879-b936-29bd33dfc5a9-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-d3a281-844b5468d7-hvsnh\" (UID: \"19ffc8e8-e659-4879-b936-29bd33dfc5a9\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-d3a281-844b5468d7-hvsnh"
Nov 02 12:50:06.053582 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:06.053492    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-njp6m\" (UniqueName: \"kubernetes.io/projected/19ffc8e8-e659-4879-b936-29bd33dfc5a9-kube-api-access-njp6m\") pod \"payload-processing-e2e-praxis-b-d3a281-844b5468d7-hvsnh\" (UID: \"19ffc8e8-e659-4879-b936-29bd33dfc5a9\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-d3a281-844b5468d7-hvsnh"
Nov 02 12:50:06.137321 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:06.136856    4559 patch_prober.go:29] interesting pod/e2e-ipp-a-eb24df-openshift-default-67f48b7f65-fg527 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body=
Nov 02 12:50:06.137321 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:06.136935    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-eb24df-openshift-default-67f48b7f65-fg527" podUID="f19dde18-978d-4894-83d3-5c39478c866c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused"
Nov 02 12:50:06.281629 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:06.280861    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-d3a281-844b5468d7-hvsnh"
Nov 02 12:50:06.433677 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:06.420756    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-d6d314-openshift-default-5b99b69565-2kkgt"
Nov 02 12:50:06.664663 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:06.661216    4559 patch_prober.go:29] interesting pod/e2e-praxis-a-d6d314-openshift-default-5b99b69565-2kkgt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body=
Nov 02 12:50:06.664663 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:06.661948    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-d6d314-openshift-default-5b99b69565-2kkgt" podUID="4ac6508c-24db-40f7-a14c-0964dac7768b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused"
Nov 02 12:50:06.957659 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:06.946018    4559 patch_prober.go:29] interesting pod/e2e-praxis-b-e93a4b-openshift-default-7f8c56bf9c-4pjxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body=
Nov 02 12:50:06.957659 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:06.946079    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-e93a4b-openshift-default-7f8c56bf9c-4pjxc" podUID="624a0494-692e-4864-810e-42ba5306a5f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused"
Nov 02 12:50:07.068185 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:07.068147    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-d3a281-844b5468d7-hvsnh"]
Nov 02 12:50:07.146459 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:07.146427    4559 patch_prober.go:29] interesting pod/e2e-ipp-a-eb24df-openshift-default-67f48b7f65-fg527 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body=
Nov 02 12:50:07.146827 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:07.146780    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-eb24df-openshift-default-67f48b7f65-fg527" podUID="f19dde18-978d-4894-83d3-5c39478c866c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused"
Nov 02 12:50:07.369701 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:07.367770    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-d3a281-844b5468d7-hvsnh" event={"ID":"19ffc8e8-e659-4879-b936-29bd33dfc5a9","Type":"ContainerStarted","Data":"a1564eb0e74a1fd8fb7cfb744aad5a34a8961cdb030c775f26ffbcce4756c3ed"}
Nov 02 12:50:07.640663 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:07.639975    4559 patch_prober.go:29] interesting pod/e2e-praxis-a-d6d314-openshift-default-5b99b69565-2kkgt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body=
Nov 02 12:50:07.640663 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:07.640040    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-d6d314-openshift-default-5b99b69565-2kkgt" podUID="4ac6508c-24db-40f7-a14c-0964dac7768b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused"
Nov 02 12:50:07.828721 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:07.827914    4559 patch_prober.go:29] interesting pod/e2e-praxis-b-e93a4b-openshift-default-7f8c56bf9c-4pjxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body=
Nov 02 12:50:07.828721 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:07.827976    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-e93a4b-openshift-default-7f8c56bf9c-4pjxc" podUID="624a0494-692e-4864-810e-42ba5306a5f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused"
Nov 02 12:50:08.109750 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:08.104719    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-ee509d-6d46494d8c-q2qwp" podUID="12ce9360-ddff-4f72-acbb-bf9659a5c0a3" containerName="payload-processing" containerID="cri-o://2d279267b8f4d636a8680c93e7a7eabbe24e3c562437c16fd56e64dbcbe85aad" gracePeriod=25
Nov 02 12:50:08.147245 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:08.128717    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ee509d-5568559575-g8848" podUID="14d49a01-92ac-4a12-9c40-82dcb6dc0bfd" containerName="payload-pre-processing" containerID="cri-o://9bd2879074940650dcbe3ab5ac388e44726ff6a87845877eb219ae810ff57854" gracePeriod=25
Nov 02 12:50:08.147245 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:08.144801    4559 patch_prober.go:29] interesting pod/e2e-ipp-a-eb24df-openshift-default-67f48b7f65-fg527 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body=
Nov 02 12:50:08.147245 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:08.144962    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-eb24df-openshift-default-67f48b7f65-fg527" podUID="f19dde18-978d-4894-83d3-5c39478c866c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused"
Nov 02 12:50:08.377858 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:08.377399    4559 generic.go:356] "Generic (PLEG): container finished" podID="12ce9360-ddff-4f72-acbb-bf9659a5c0a3" containerID="2d279267b8f4d636a8680c93e7a7eabbe24e3c562437c16fd56e64dbcbe85aad" exitCode=0
Nov 02 12:50:08.377858 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:08.377720    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-ee509d-6d46494d8c-q2qwp" event={"ID":"12ce9360-ddff-4f72-acbb-bf9659a5c0a3","Type":"ContainerDied","Data":"2d279267b8f4d636a8680c93e7a7eabbe24e3c562437c16fd56e64dbcbe85aad"}
Nov 02 12:50:08.380483 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:08.380456    4559 generic.go:356] "Generic (PLEG): container finished" podID="14d49a01-92ac-4a12-9c40-82dcb6dc0bfd" containerID="9bd2879074940650dcbe3ab5ac388e44726ff6a87845877eb219ae810ff57854" exitCode=0
Nov 02 12:50:08.380659 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:08.380553    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ee509d-5568559575-g8848" event={"ID":"14d49a01-92ac-4a12-9c40-82dcb6dc0bfd","Type":"ContainerDied","Data":"9bd2879074940650dcbe3ab5ac388e44726ff6a87845877eb219ae810ff57854"}
Nov 02 12:50:08.382176 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:08.382132    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-d3a281-844b5468d7-hvsnh" event={"ID":"19ffc8e8-e659-4879-b936-29bd33dfc5a9","Type":"ContainerStarted","Data":"33673c7c6198e753102c1109a80d842d17201f57b4508af3fc7b708c54654279"}
Nov 02 12:50:08.382305 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:08.382289    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-d3a281-844b5468d7-hvsnh"
Nov 02 12:50:08.468688 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:08.467277    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-d3a281-844b5468d7-hvsnh" podStartSLOduration=3.467260052 podStartE2EDuration="3.467260052s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:50:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:50:08.43100389 +0000 UTC m=+1404.686402987" watchObservedRunningTime="2026-11-02 12:50:08.467260052 +0000 UTC m=+1404.722659147"
Nov 02 12:50:08.743913 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:08.729349    4559 patch_prober.go:29] interesting pod/e2e-praxis-a-d6d314-openshift-default-5b99b69565-2kkgt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body=
Nov 02 12:50:08.786728 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:08.744263    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-d6d314-openshift-default-5b99b69565-2kkgt" podUID="4ac6508c-24db-40f7-a14c-0964dac7768b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused"
Nov 02 12:50:08.786728 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:08.743907    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-66rd4"]
Nov 02 12:50:08.786728 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:08.769985    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-66rd4" podUID="e09c5fe2-c1dd-47ab-8464-facefd253549" containerName="istio-proxy" containerID="cri-o://01cb3f6d6d9046ecf876e009e0c30e94347a355c0df84d6ddcc2a15171f82c94" gracePeriod=30
Nov 02 12:50:08.925555 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:08.902098    4559 patch_prober.go:29] interesting pod/e2e-praxis-b-e93a4b-openshift-default-7f8c56bf9c-4pjxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body=
Nov 02 12:50:08.925555 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:08.902158    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-e93a4b-openshift-default-7f8c56bf9c-4pjxc" podUID="624a0494-692e-4864-810e-42ba5306a5f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused"
Nov 02 12:50:08.925555 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:08.902839    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tz7qn"]
Nov 02 12:50:08.925555 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:08.903084    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tz7qn" podUID="597fdf69-8d19-46b2-b61f-bbe2ec731e61" containerName="istio-proxy" containerID="cri-o://d65388fb60ebe4470e7fdd8574df2e564aca716b71e203247453ff63961a0559" gracePeriod=30
Nov 02 12:50:09.002005 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:08.998912    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x9hwg"]
Nov 02 12:50:09.077128 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:09.071739    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x9hwg" podUID="eec5b49d-6cf5-4203-b386-9f9a5fa6c7fa" containerName="istio-proxy" containerID="cri-o://a35152e4368ed7dc8d754d70c001c11136a446cf09d66af582a531290627369a" gracePeriod=30
Nov 02 12:50:09.175883 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:09.175344    4559 patch_prober.go:29] interesting pod/e2e-ipp-a-eb24df-openshift-default-67f48b7f65-fg527 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body=
Nov 02 12:50:09.175883 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:09.175401    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-eb24df-openshift-default-67f48b7f65-fg527" podUID="f19dde18-978d-4894-83d3-5c39478c866c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused"
Nov 02 12:50:09.182261 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:09.182003    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ee509d-5568559575-g8848"
Nov 02 12:50:09.373961 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:09.373909    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14d49a01-92ac-4a12-9c40-82dcb6dc0bfd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14d49a01-92ac-4a12-9c40-82dcb6dc0bfd-plugins-config-volume\") pod \"14d49a01-92ac-4a12-9c40-82dcb6dc0bfd\" (UID: \"14d49a01-92ac-4a12-9c40-82dcb6dc0bfd\") "
Nov 02 12:50:09.374234 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:09.374210    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14d49a01-92ac-4a12-9c40-82dcb6dc0bfd-kube-api-access-6bn9f\" (UniqueName: \"kubernetes.io/projected/14d49a01-92ac-4a12-9c40-82dcb6dc0bfd-kube-api-access-6bn9f\") pod \"14d49a01-92ac-4a12-9c40-82dcb6dc0bfd\" (UID: \"14d49a01-92ac-4a12-9c40-82dcb6dc0bfd\") "
Nov 02 12:50:09.375102 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:09.375073    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14d49a01-92ac-4a12-9c40-82dcb6dc0bfd-plugins-config-volume" pod "14d49a01-92ac-4a12-9c40-82dcb6dc0bfd" (UID: "14d49a01-92ac-4a12-9c40-82dcb6dc0bfd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:50:09.378007 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:09.377980    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14d49a01-92ac-4a12-9c40-82dcb6dc0bfd-kube-api-access-6bn9f" pod "14d49a01-92ac-4a12-9c40-82dcb6dc0bfd" (UID: "14d49a01-92ac-4a12-9c40-82dcb6dc0bfd"). InnerVolumeSpecName "kube-api-access-6bn9f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:50:09.379989 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:09.379963    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-ee509d-6d46494d8c-q2qwp"
Nov 02 12:50:09.475743 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:09.475708    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12ce9360-ddff-4f72-acbb-bf9659a5c0a3-kube-api-access-zrxb8\" (UniqueName: \"kubernetes.io/projected/12ce9360-ddff-4f72-acbb-bf9659a5c0a3-kube-api-access-zrxb8\") pod \"12ce9360-ddff-4f72-acbb-bf9659a5c0a3\" (UID: \"12ce9360-ddff-4f72-acbb-bf9659a5c0a3\") "
Nov 02 12:50:09.476552 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:09.476528    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12ce9360-ddff-4f72-acbb-bf9659a5c0a3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12ce9360-ddff-4f72-acbb-bf9659a5c0a3-plugins-config-volume\") pod \"12ce9360-ddff-4f72-acbb-bf9659a5c0a3\" (UID: \"12ce9360-ddff-4f72-acbb-bf9659a5c0a3\") "
Nov 02 12:50:09.477020 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:09.476954    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12ce9360-ddff-4f72-acbb-bf9659a5c0a3-plugins-config-volume" pod "12ce9360-ddff-4f72-acbb-bf9659a5c0a3" (UID: "12ce9360-ddff-4f72-acbb-bf9659a5c0a3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:50:09.477242 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:09.477219    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6bn9f\" (UniqueName: \"kubernetes.io/projected/14d49a01-92ac-4a12-9c40-82dcb6dc0bfd-kube-api-access-6bn9f\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:50:09.477313 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:09.477247    4559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12ce9360-ddff-4f72-acbb-bf9659a5c0a3-plugins-config-volume\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:50:09.477313 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:09.477262    4559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14d49a01-92ac-4a12-9c40-82dcb6dc0bfd-plugins-config-volume\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:50:09.479250 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:09.479220    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12ce9360-ddff-4f72-acbb-bf9659a5c0a3-kube-api-access-zrxb8" pod "12ce9360-ddff-4f72-acbb-bf9659a5c0a3" (UID: "12ce9360-ddff-4f72-acbb-bf9659a5c0a3"). InnerVolumeSpecName "kube-api-access-zrxb8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:50:09.480380 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:09.480359    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ee509d-5568559575-g8848"
Nov 02 12:50:09.480876 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:09.480845    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ee509d-5568559575-g8848" event={"ID":"14d49a01-92ac-4a12-9c40-82dcb6dc0bfd","Type":"ContainerDied","Data":"22bcaef227f301233a964199f9a25a78a335c531e541c2f78a8e4eae8bfe0cf8"}
Nov 02 12:50:09.481185 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:09.481164    4559 scope.go:122] "RemoveContainer" containerID="9bd2879074940650dcbe3ab5ac388e44726ff6a87845877eb219ae810ff57854"
Nov 02 12:50:09.482139 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:09.482109    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-ee509d-6d46494d8c-q2qwp" event={"ID":"12ce9360-ddff-4f72-acbb-bf9659a5c0a3","Type":"ContainerDied","Data":"129dbacc09e08f55314750f2b0df21c748c591ab3860fbfab611f398e0b20f81"}
Nov 02 12:50:09.482245 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:09.482139    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-ee509d-6d46494d8c-q2qwp"
Nov 02 12:50:09.503921 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:09.503824    4559 scope.go:122] "RemoveContainer" containerID="2d279267b8f4d636a8680c93e7a7eabbe24e3c562437c16fd56e64dbcbe85aad"
Nov 02 12:50:09.578848 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:09.578817    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zrxb8\" (UniqueName: \"kubernetes.io/projected/12ce9360-ddff-4f72-acbb-bf9659a5c0a3-kube-api-access-zrxb8\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:50:09.660759 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:09.646299    4559 patch_prober.go:29] interesting pod/e2e-praxis-a-d6d314-openshift-default-5b99b69565-2kkgt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body=
Nov 02 12:50:09.660759 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:09.646357    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-d6d314-openshift-default-5b99b69565-2kkgt" podUID="4ac6508c-24db-40f7-a14c-0964dac7768b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused"
Nov 02 12:50:09.669304 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:09.667390    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-ee509d-5568559575-g8848"]
Nov 02 12:50:09.673270 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:09.673226    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-ee509d-5568559575-g8848"]
Nov 02 12:50:09.771684 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:09.765729    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-ee509d-6d46494d8c-q2qwp"]
Nov 02 12:50:09.774859 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:09.774829    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-ee509d-6d46494d8c-q2qwp"]
Nov 02 12:50:09.855723 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:09.853809    4559 patch_prober.go:29] interesting pod/e2e-praxis-b-e93a4b-openshift-default-7f8c56bf9c-4pjxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body=
Nov 02 12:50:09.855723 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:09.853877    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-e93a4b-openshift-default-7f8c56bf9c-4pjxc" podUID="624a0494-692e-4864-810e-42ba5306a5f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused"
Nov 02 12:50:09.956081 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:09.956040    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12ce9360-ddff-4f72-acbb-bf9659a5c0a3" path="/var/lib/kubelet/pods/12ce9360-ddff-4f72-acbb-bf9659a5c0a3/volumes"
Nov 02 12:50:09.957117 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:09.957092    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="14d49a01-92ac-4a12-9c40-82dcb6dc0bfd" path="/var/lib/kubelet/pods/14d49a01-92ac-4a12-9c40-82dcb6dc0bfd/volumes"
Nov 02 12:50:10.136353 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:10.136254    4559 patch_prober.go:29] interesting pod/e2e-ipp-a-eb24df-openshift-default-67f48b7f65-fg527 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body=
Nov 02 12:50:10.136623 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:10.136580    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-eb24df-openshift-default-67f48b7f65-fg527" podUID="f19dde18-978d-4894-83d3-5c39478c866c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused"
Nov 02 12:50:10.657115 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:10.655729    4559 patch_prober.go:29] interesting pod/e2e-praxis-a-d6d314-openshift-default-5b99b69565-2kkgt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body=
Nov 02 12:50:10.657115 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:10.655797    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-d6d314-openshift-default-5b99b69565-2kkgt" podUID="4ac6508c-24db-40f7-a14c-0964dac7768b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused"
Nov 02 12:50:10.819662 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:10.819089    4559 patch_prober.go:29] interesting pod/e2e-praxis-b-e93a4b-openshift-default-7f8c56bf9c-4pjxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body=
Nov 02 12:50:10.819662 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:10.819152    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-e93a4b-openshift-default-7f8c56bf9c-4pjxc" podUID="624a0494-692e-4864-810e-42ba5306a5f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused"
Nov 02 12:50:11.165864 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:11.165013    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-eb24df-openshift-default-67f48b7f65-fg527"
Nov 02 12:50:11.166433 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:11.166411    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-eb24df-openshift-default-67f48b7f65-fg527"
Nov 02 12:50:11.168130 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:11.168112    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-eb24df-openshift-default-67f48b7f65-fg527"
Nov 02 12:50:11.637492 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:11.637434    4559 patch_prober.go:29] interesting pod/e2e-praxis-a-d6d314-openshift-default-5b99b69565-2kkgt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body=
Nov 02 12:50:11.637713 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:11.637511    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-d6d314-openshift-default-5b99b69565-2kkgt" podUID="4ac6508c-24db-40f7-a14c-0964dac7768b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused"
Nov 02 12:50:11.819089 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:11.819039    4559 patch_prober.go:29] interesting pod/e2e-praxis-b-e93a4b-openshift-default-7f8c56bf9c-4pjxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body=
Nov 02 12:50:11.819620 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:11.819110    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-e93a4b-openshift-default-7f8c56bf9c-4pjxc" podUID="624a0494-692e-4864-810e-42ba5306a5f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused"
Nov 02 12:50:11.910897 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:11.910811    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-eb24df-77b8d4b6bc-b7fx5"]
Nov 02 12:50:11.926970 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:11.926932    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-eb24df-748477f8f5-67vfv"]
Nov 02 12:50:12.614419 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:12.614388    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-9750c7-84d6d97dc4-cbh2m"]
Nov 02 12:50:12.615156 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:12.615140    4559 cpu_manager.go:414] "containerMap: removing container" podUID="14d49a01-92ac-4a12-9c40-82dcb6dc0bfd" containerName="payload-pre-processing"
Nov 02 12:50:12.615213 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:12.615159    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14d49a01-92ac-4a12-9c40-82dcb6dc0bfd" containerName="payload-pre-processing"
Nov 02 12:50:12.615213 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:12.615172    4559 cpu_manager.go:414] "containerMap: removing container" podUID="12ce9360-ddff-4f72-acbb-bf9659a5c0a3" containerName="payload-processing"
Nov 02 12:50:12.615213 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:12.615177    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12ce9360-ddff-4f72-acbb-bf9659a5c0a3" containerName="payload-processing"
Nov 02 12:50:12.615313 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:12.615277    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="14d49a01-92ac-4a12-9c40-82dcb6dc0bfd" containerName="payload-pre-processing"
Nov 02 12:50:12.615313 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:12.615285    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="12ce9360-ddff-4f72-acbb-bf9659a5c0a3" containerName="payload-processing"
Nov 02 12:50:12.619810 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:12.619794    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9750c7-84d6d97dc4-cbh2m"
Nov 02 12:50:12.634947 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:12.630779    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bcl9c\" (UniqueName: \"kubernetes.io/projected/465410fd-fc85-469e-90fb-842446ce62e4-kube-api-access-bcl9c\") pod \"maas-api-e2e-praxis-b-9750c7-84d6d97dc4-cbh2m\" (UID: \"465410fd-fc85-469e-90fb-842446ce62e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9750c7-84d6d97dc4-cbh2m"
Nov 02 12:50:12.634947 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:12.630842    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/465410fd-fc85-469e-90fb-842446ce62e4-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-9750c7-84d6d97dc4-cbh2m\" (UID: \"465410fd-fc85-469e-90fb-842446ce62e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9750c7-84d6d97dc4-cbh2m"
Nov 02 12:50:12.634947 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:12.630947    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/465410fd-fc85-469e-90fb-842446ce62e4-metrics-certs\") pod \"maas-api-e2e-praxis-b-9750c7-84d6d97dc4-cbh2m\" (UID: \"465410fd-fc85-469e-90fb-842446ce62e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9750c7-84d6d97dc4-cbh2m"
Nov 02 12:50:12.634947 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:12.631094    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/465410fd-fc85-469e-90fb-842446ce62e4-maas-api-tls\") pod \"maas-api-e2e-praxis-b-9750c7-84d6d97dc4-cbh2m\" (UID: \"465410fd-fc85-469e-90fb-842446ce62e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9750c7-84d6d97dc4-cbh2m"
Nov 02 12:50:12.646002 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:12.645966    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-9750c7-84d6d97dc4-cbh2m"]
Nov 02 12:50:12.646859 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:12.646835    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-9750c7-serving-cert\""
Nov 02 12:50:12.650062 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:12.650027    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-d6d314-openshift-default-5b99b69565-2kkgt"
Nov 02 12:50:12.653857 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:12.653840    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-d6d314-openshift-default-5b99b69565-2kkgt"
Nov 02 12:50:12.732158 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:12.732105    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/465410fd-fc85-469e-90fb-842446ce62e4-metrics-certs\") pod \"maas-api-e2e-praxis-b-9750c7-84d6d97dc4-cbh2m\" (UID: \"465410fd-fc85-469e-90fb-842446ce62e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9750c7-84d6d97dc4-cbh2m"
Nov 02 12:50:12.732355 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:12.732290    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/465410fd-fc85-469e-90fb-842446ce62e4-maas-api-tls\") pod \"maas-api-e2e-praxis-b-9750c7-84d6d97dc4-cbh2m\" (UID: \"465410fd-fc85-469e-90fb-842446ce62e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9750c7-84d6d97dc4-cbh2m"
Nov 02 12:50:12.732431 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:12.732414    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcl9c\" (UniqueName: \"kubernetes.io/projected/465410fd-fc85-469e-90fb-842446ce62e4-kube-api-access-bcl9c\") pod \"maas-api-e2e-praxis-b-9750c7-84d6d97dc4-cbh2m\" (UID: \"465410fd-fc85-469e-90fb-842446ce62e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9750c7-84d6d97dc4-cbh2m"
Nov 02 12:50:12.732511 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:12.732494    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/465410fd-fc85-469e-90fb-842446ce62e4-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-9750c7-84d6d97dc4-cbh2m\" (UID: \"465410fd-fc85-469e-90fb-842446ce62e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9750c7-84d6d97dc4-cbh2m"
Nov 02 12:50:12.733365 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:12.733331    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/465410fd-fc85-469e-90fb-842446ce62e4-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-9750c7-84d6d97dc4-cbh2m\" (UID: \"465410fd-fc85-469e-90fb-842446ce62e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9750c7-84d6d97dc4-cbh2m"
Nov 02 12:50:12.734590 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:50:12.734564    4559 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-b-9750c7-serving-cert: secret "maas-api-e2e-praxis-b-9750c7-serving-cert" not found
Nov 02 12:50:12.734917 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:50:12.734631    4559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/465410fd-fc85-469e-90fb-842446ce62e4-maas-api-tls podName:465410fd-fc85-469e-90fb-842446ce62e4 nodeName:}" failed. No retries permitted until 2026-11-02 12:50:13.234613934 +0000 UTC m=+1409.490013021 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/465410fd-fc85-469e-90fb-842446ce62e4-maas-api-tls") pod "maas-api-e2e-praxis-b-9750c7-84d6d97dc4-cbh2m" (UID: "465410fd-fc85-469e-90fb-842446ce62e4") : secret "maas-api-e2e-praxis-b-9750c7-serving-cert" not found
Nov 02 12:50:12.754705 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:12.754191    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/465410fd-fc85-469e-90fb-842446ce62e4-metrics-certs\") pod \"maas-api-e2e-praxis-b-9750c7-84d6d97dc4-cbh2m\" (UID: \"465410fd-fc85-469e-90fb-842446ce62e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9750c7-84d6d97dc4-cbh2m"
Nov 02 12:50:12.778370 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:12.764757    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcl9c\" (UniqueName: \"kubernetes.io/projected/465410fd-fc85-469e-90fb-842446ce62e4-kube-api-access-bcl9c\") pod \"maas-api-e2e-praxis-b-9750c7-84d6d97dc4-cbh2m\" (UID: \"465410fd-fc85-469e-90fb-842446ce62e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9750c7-84d6d97dc4-cbh2m"
Nov 02 12:50:12.874731 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:12.873875    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-e93a4b-openshift-default-7f8c56bf9c-4pjxc"
Nov 02 12:50:12.881746 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:12.881709    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-e93a4b-openshift-default-7f8c56bf9c-4pjxc"
Nov 02 12:50:12.926673 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:12.888875    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-e93a4b-openshift-default-7f8c56bf9c-4pjxc"
Nov 02 12:50:13.277402 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:13.277362    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/465410fd-fc85-469e-90fb-842446ce62e4-maas-api-tls\") pod \"maas-api-e2e-praxis-b-9750c7-84d6d97dc4-cbh2m\" (UID: \"465410fd-fc85-469e-90fb-842446ce62e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9750c7-84d6d97dc4-cbh2m"
Nov 02 12:50:13.283466 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:13.283435    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/465410fd-fc85-469e-90fb-842446ce62e4-maas-api-tls\") pod \"maas-api-e2e-praxis-b-9750c7-84d6d97dc4-cbh2m\" (UID: \"465410fd-fc85-469e-90fb-842446ce62e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9750c7-84d6d97dc4-cbh2m"
Nov 02 12:50:13.552953 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:13.552868    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9750c7-84d6d97dc4-cbh2m"
Nov 02 12:50:13.898276 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:50:13.898227    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod465410fd_fc85_469e_90fb_842446ce62e4.slice/crio-504d95d38bee30e6696443e5ced646f2c7a811aea41de9dd6def8365d0bdf18e WatchSource:0}: Error finding container 504d95d38bee30e6696443e5ced646f2c7a811aea41de9dd6def8365d0bdf18e: Status 404 returned error can't find the container with id 504d95d38bee30e6696443e5ced646f2c7a811aea41de9dd6def8365d0bdf18e
Nov 02 12:50:13.900407 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:13.900383    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-9750c7-84d6d97dc4-cbh2m"]
Nov 02 12:50:14.267566 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:14.267544    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-66rd4"
Nov 02 12:50:14.289846 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:14.289823    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tz7qn"
Nov 02 12:50:14.327257 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:14.327221    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/597fdf69-8d19-46b2-b61f-bbe2ec731e61-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/597fdf69-8d19-46b2-b61f-bbe2ec731e61-workload-certs\") pod \"597fdf69-8d19-46b2-b61f-bbe2ec731e61\" (UID: \"597fdf69-8d19-46b2-b61f-bbe2ec731e61\") "
Nov 02 12:50:14.327443 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:14.327311    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e09c5fe2-c1dd-47ab-8464-facefd253549-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e09c5fe2-c1dd-47ab-8464-facefd253549-workload-socket\") pod \"e09c5fe2-c1dd-47ab-8464-facefd253549\" (UID: \"e09c5fe2-c1dd-47ab-8464-facefd253549\") "
Nov 02 12:50:14.327443 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:14.327347    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e09c5fe2-c1dd-47ab-8464-facefd253549-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e09c5fe2-c1dd-47ab-8464-facefd253549-istiod-ca-cert\") pod \"e09c5fe2-c1dd-47ab-8464-facefd253549\" (UID: \"e09c5fe2-c1dd-47ab-8464-facefd253549\") "
Nov 02 12:50:14.327443 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:14.327428    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e09c5fe2-c1dd-47ab-8464-facefd253549-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e09c5fe2-c1dd-47ab-8464-facefd253549-workload-certs\") pod \"e09c5fe2-c1dd-47ab-8464-facefd253549\" (UID: \"e09c5fe2-c1dd-47ab-8464-facefd253549\") "
Nov 02 12:50:14.327628 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:14.327467    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e09c5fe2-c1dd-47ab-8464-facefd253549-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e09c5fe2-c1dd-47ab-8464-facefd253549-istio-data\") pod \"e09c5fe2-c1dd-47ab-8464-facefd253549\" (UID: \"e09c5fe2-c1dd-47ab-8464-facefd253549\") "
Nov 02 12:50:14.327628 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:14.327511    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e09c5fe2-c1dd-47ab-8464-facefd253549-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e09c5fe2-c1dd-47ab-8464-facefd253549-istio-envoy\") pod \"e09c5fe2-c1dd-47ab-8464-facefd253549\" (UID: \"e09c5fe2-c1dd-47ab-8464-facefd253549\") "
Nov 02 12:50:14.327628 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:14.327538    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/597fdf69-8d19-46b2-b61f-bbe2ec731e61-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/597fdf69-8d19-46b2-b61f-bbe2ec731e61-istio-data\") pod \"597fdf69-8d19-46b2-b61f-bbe2ec731e61\" (UID: \"597fdf69-8d19-46b2-b61f-bbe2ec731e61\") "
Nov 02 12:50:14.327628 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:14.327607    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/597fdf69-8d19-46b2-b61f-bbe2ec731e61-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/597fdf69-8d19-46b2-b61f-bbe2ec731e61-workload-socket\") pod \"597fdf69-8d19-46b2-b61f-bbe2ec731e61\" (UID: \"597fdf69-8d19-46b2-b61f-bbe2ec731e61\") "
Nov 02 12:50:14.327819 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:14.327710    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e09c5fe2-c1dd-47ab-8464-facefd253549-istio-token\" (UniqueName: \"kubernetes.io/projected/e09c5fe2-c1dd-47ab-8464-facefd253549-istio-token\") pod \"e09c5fe2-c1dd-47ab-8464-facefd253549\" (UID: \"e09c5fe2-c1dd-47ab-8464-facefd253549\") "
Nov 02 12:50:14.327819 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:14.327793    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/597fdf69-8d19-46b2-b61f-bbe2ec731e61-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/597fdf69-8d19-46b2-b61f-bbe2ec731e61-istio-podinfo\") pod \"597fdf69-8d19-46b2-b61f-bbe2ec731e61\" (UID: \"597fdf69-8d19-46b2-b61f-bbe2ec731e61\") "
Nov 02 12:50:14.327888 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:14.327833    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/597fdf69-8d19-46b2-b61f-bbe2ec731e61-istio-token\" (UniqueName: \"kubernetes.io/projected/597fdf69-8d19-46b2-b61f-bbe2ec731e61-istio-token\") pod \"597fdf69-8d19-46b2-b61f-bbe2ec731e61\" (UID: \"597fdf69-8d19-46b2-b61f-bbe2ec731e61\") "
Nov 02 12:50:14.327925 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:14.327890    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/597fdf69-8d19-46b2-b61f-bbe2ec731e61-kube-api-access-b98s2\" (UniqueName: \"kubernetes.io/projected/597fdf69-8d19-46b2-b61f-bbe2ec731e61-kube-api-access-b98s2\") pod \"597fdf69-8d19-46b2-b61f-bbe2ec731e61\" (UID: \"597fdf69-8d19-46b2-b61f-bbe2ec731e61\") "
Nov 02 12:50:14.327925 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:14.327919    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/597fdf69-8d19-46b2-b61f-bbe2ec731e61-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/597fdf69-8d19-46b2-b61f-bbe2ec731e61-credential-socket\") pod \"597fdf69-8d19-46b2-b61f-bbe2ec731e61\" (UID: \"597fdf69-8d19-46b2-b61f-bbe2ec731e61\") "
Nov 02 12:50:14.327989 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:14.327952    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e09c5fe2-c1dd-47ab-8464-facefd253549-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e09c5fe2-c1dd-47ab-8464-facefd253549-credential-socket\") pod \"e09c5fe2-c1dd-47ab-8464-facefd253549\" (UID: \"e09c5fe2-c1dd-47ab-8464-facefd253549\") "
Nov 02 12:50:14.328022 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:14.327992    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e09c5fe2-c1dd-47ab-8464-facefd253549-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e09c5fe2-c1dd-47ab-8464-facefd253549-istio-podinfo\") pod \"e09c5fe2-c1dd-47ab-8464-facefd253549\" (UID: \"e09c5fe2-c1dd-47ab-8464-facefd253549\") "
Nov 02 12:50:14.328056 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:14.328034    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/597fdf69-8d19-46b2-b61f-bbe2ec731e61-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/597fdf69-8d19-46b2-b61f-bbe2ec731e61-istio-envoy\") pod \"597fdf69-8d19-46b2-b61f-bbe2ec731e61\" (UID: \"597fdf69-8d19-46b2-b61f-bbe2ec731e61\") "
Nov 02 12:50:14.328115 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:14.328097    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e09c5fe2-c1dd-47ab-8464-facefd253549-kube-api-access-dwnnv\" (UniqueName: \"kubernetes.io/projected/e09c5fe2-c1dd-47ab-8464-facefd253549-kube-api-access-dwnnv\") pod \"e09c5fe2-c1dd-47ab-8464-facefd253549\" (UID: \"e09c5fe2-c1dd-47ab-8464-facefd253549\") "
Nov 02 12:50:14.328155 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:14.328143    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/597fdf69-8d19-46b2-b61f-bbe2ec731e61-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/597fdf69-8d19-46b2-b61f-bbe2ec731e61-istiod-ca-cert\") pod \"597fdf69-8d19-46b2-b61f-bbe2ec731e61\" (UID: \"597fdf69-8d19-46b2-b61f-bbe2ec731e61\") "
Nov 02 12:50:14.329265 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:14.328452    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e09c5fe2-c1dd-47ab-8464-facefd253549-workload-socket" pod "e09c5fe2-c1dd-47ab-8464-facefd253549" (UID: "e09c5fe2-c1dd-47ab-8464-facefd253549"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:50:14.329265 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:14.328743    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e09c5fe2-c1dd-47ab-8464-facefd253549-istiod-ca-cert" pod "e09c5fe2-c1dd-47ab-8464-facefd253549" (UID: "e09c5fe2-c1dd-47ab-8464-facefd253549"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:50:14.329265 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:14.328838    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/597fdf69-8d19-46b2-b61f-bbe2ec731e61-istiod-ca-cert" pod "597fdf69-8d19-46b2-b61f-bbe2ec731e61" (UID: "597fdf69-8d19-46b2-b61f-bbe2ec731e61"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:50:14.329265 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:14.328847    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e09c5fe2-c1dd-47ab-8464-facefd253549-workload-certs" pod "e09c5fe2-c1dd-47ab-8464-facefd253549" (UID: "e09c5fe2-c1dd-47ab-8464-facefd253549"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:50:14.329265 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:14.329040    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e09c5fe2-c1dd-47ab-8464-facefd253549-credential-socket" pod "e09c5fe2-c1dd-47ab-8464-facefd253549" (UID: "e09c5fe2-c1dd-47ab-8464-facefd253549"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:50:14.329265 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:14.329112    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/597fdf69-8d19-46b2-b61f-bbe2ec731e61-istio-data" pod "597fdf69-8d19-46b2-b61f-bbe2ec731e61" (UID: "597fdf69-8d19-46b2-b61f-bbe2ec731e61"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:50:14.329705 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:14.329269    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/597fdf69-8d19-46b2-b61f-bbe2ec731e61-workload-certs" pod "597fdf69-8d19-46b2-b61f-bbe2ec731e61" (UID: "597fdf69-8d19-46b2-b61f-bbe2ec731e61"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:50:14.329705 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:14.329321    4559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e09c5fe2-c1dd-47ab-8464-facefd253549-istiod-ca-cert\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:50:14.329705 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:14.329342    4559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e09c5fe2-c1dd-47ab-8464-facefd253549-workload-certs\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:50:14.329705 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:14.329357    4559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/597fdf69-8d19-46b2-b61f-bbe2ec731e61-istio-data\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:50:14.329705 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:14.329372    4559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e09c5fe2-c1dd-47ab-8464-facefd253549-credential-socket\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:50:14.329978 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:14.329703    4559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/597fdf69-8d19-46b2-b61f-bbe2ec731e61-istiod-ca-cert\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:50:14.329978 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:14.329759    4559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e09c5fe2-c1dd-47ab-8464-facefd253549-workload-socket\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:50:14.329978 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:14.329471    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e09c5fe2-c1dd-47ab-8464-facefd253549-istio-data" pod "e09c5fe2-c1dd-47ab-8464-facefd253549" (UID: "e09c5fe2-c1dd-47ab-8464-facefd253549"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:50:14.329978 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:14.329687    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/597fdf69-8d19-46b2-b61f-bbe2ec731e61-credential-socket" pod "597fdf69-8d19-46b2-b61f-bbe2ec731e61" (UID: "597fdf69-8d19-46b2-b61f-bbe2ec731e61"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:50:14.331281 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:14.331257    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/597fdf69-8d19-46b2-b61f-bbe2ec731e61-workload-socket" pod "597fdf69-8d19-46b2-b61f-bbe2ec731e61" (UID: "597fdf69-8d19-46b2-b61f-bbe2ec731e61"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:50:14.332559 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:14.332534    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e09c5fe2-c1dd-47ab-8464-facefd253549-istio-envoy" pod "e09c5fe2-c1dd-47ab-8464-facefd253549" (UID: "e09c5fe2-c1dd-47ab-8464-facefd253549"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:50:14.332871 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:14.332839    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/597fdf69-8d19-46b2-b61f-bbe2ec731e61-kube-api-access-b98s2" pod "597fdf69-8d19-46b2-b61f-bbe2ec731e61" (UID: "597fdf69-8d19-46b2-b61f-bbe2ec731e61"). InnerVolumeSpecName "kube-api-access-b98s2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:50:14.333546 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:14.333522    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e09c5fe2-c1dd-47ab-8464-facefd253549-istio-token" pod "e09c5fe2-c1dd-47ab-8464-facefd253549" (UID: "e09c5fe2-c1dd-47ab-8464-facefd253549"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:50:14.333760 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:14.333740    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e09c5fe2-c1dd-47ab-8464-facefd253549-istio-podinfo" pod "e09c5fe2-c1dd-47ab-8464-facefd253549" (UID: "e09c5fe2-c1dd-47ab-8464-facefd253549"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 12:50:14.335056 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:14.335007    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/597fdf69-8d19-46b2-b61f-bbe2ec731e61-istio-envoy" pod "597fdf69-8d19-46b2-b61f-bbe2ec731e61" (UID: "597fdf69-8d19-46b2-b61f-bbe2ec731e61"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:50:14.335056 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:14.335015    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/597fdf69-8d19-46b2-b61f-bbe2ec731e61-istio-token" pod "597fdf69-8d19-46b2-b61f-bbe2ec731e61" (UID: "597fdf69-8d19-46b2-b61f-bbe2ec731e61"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:50:14.335987 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:14.335966    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/597fdf69-8d19-46b2-b61f-bbe2ec731e61-istio-podinfo" pod "597fdf69-8d19-46b2-b61f-bbe2ec731e61" (UID: "597fdf69-8d19-46b2-b61f-bbe2ec731e61"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 12:50:14.336426 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:14.336404    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e09c5fe2-c1dd-47ab-8464-facefd253549-kube-api-access-dwnnv" pod "e09c5fe2-c1dd-47ab-8464-facefd253549" (UID: "e09c5fe2-c1dd-47ab-8464-facefd253549"). InnerVolumeSpecName "kube-api-access-dwnnv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:50:14.431568 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:14.431492    4559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/597fdf69-8d19-46b2-b61f-bbe2ec731e61-istio-envoy\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:50:14.431568 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:14.431519    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dwnnv\" (UniqueName: \"kubernetes.io/projected/e09c5fe2-c1dd-47ab-8464-facefd253549-kube-api-access-dwnnv\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:50:14.431568 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:14.431529    4559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/597fdf69-8d19-46b2-b61f-bbe2ec731e61-workload-certs\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:50:14.431568 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:14.431538    4559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e09c5fe2-c1dd-47ab-8464-facefd253549-istio-data\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:50:14.431568 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:14.431545    4559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e09c5fe2-c1dd-47ab-8464-facefd253549-istio-envoy\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:50:14.431568 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:14.431553    4559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/597fdf69-8d19-46b2-b61f-bbe2ec731e61-workload-socket\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:50:14.431568 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:14.431561    4559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e09c5fe2-c1dd-47ab-8464-facefd253549-istio-token\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:50:14.431568 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:14.431569    4559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/597fdf69-8d19-46b2-b61f-bbe2ec731e61-istio-podinfo\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:50:14.431947 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:14.431579    4559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/597fdf69-8d19-46b2-b61f-bbe2ec731e61-istio-token\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:50:14.431947 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:14.431587    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b98s2\" (UniqueName: \"kubernetes.io/projected/597fdf69-8d19-46b2-b61f-bbe2ec731e61-kube-api-access-b98s2\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:50:14.431947 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:14.431595    4559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/597fdf69-8d19-46b2-b61f-bbe2ec731e61-credential-socket\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:50:14.431947 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:14.431606    4559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e09c5fe2-c1dd-47ab-8464-facefd253549-istio-podinfo\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:50:14.578722 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:14.578673    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9750c7-84d6d97dc4-cbh2m" event={"ID":"465410fd-fc85-469e-90fb-842446ce62e4","Type":"ContainerStarted","Data":"eb96c23257c4c6f7f1dea5166cb8c99a4ee005bd0fac9869e1a4dcd6ed4b1533"}
Nov 02 12:50:14.578916 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:14.578750    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9750c7-84d6d97dc4-cbh2m" event={"ID":"465410fd-fc85-469e-90fb-842446ce62e4","Type":"ContainerStarted","Data":"504d95d38bee30e6696443e5ced646f2c7a811aea41de9dd6def8365d0bdf18e"}
Nov 02 12:50:14.578916 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:14.578900    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9750c7-84d6d97dc4-cbh2m"
Nov 02 12:50:14.580158 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:14.580131    4559 generic.go:356] "Generic (PLEG): container finished" podID="597fdf69-8d19-46b2-b61f-bbe2ec731e61" containerID="d65388fb60ebe4470e7fdd8574df2e564aca716b71e203247453ff63961a0559" exitCode=0
Nov 02 12:50:14.580274 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:14.580187    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tz7qn"
Nov 02 12:50:14.580274 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:14.580212    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tz7qn" event={"ID":"597fdf69-8d19-46b2-b61f-bbe2ec731e61","Type":"ContainerDied","Data":"d65388fb60ebe4470e7fdd8574df2e564aca716b71e203247453ff63961a0559"}
Nov 02 12:50:14.580363 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:14.580267    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tz7qn" event={"ID":"597fdf69-8d19-46b2-b61f-bbe2ec731e61","Type":"ContainerDied","Data":"4235c5a2446f6e06b8e6b6ec683cb23887d6abbe707bb4bf60e549e05f72e639"}
Nov 02 12:50:14.580363 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:14.580316    4559 scope.go:122] "RemoveContainer" containerID="d65388fb60ebe4470e7fdd8574df2e564aca716b71e203247453ff63961a0559"
Nov 02 12:50:14.581724 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:14.581705    4559 generic.go:356] "Generic (PLEG): container finished" podID="e09c5fe2-c1dd-47ab-8464-facefd253549" containerID="01cb3f6d6d9046ecf876e009e0c30e94347a355c0df84d6ddcc2a15171f82c94" exitCode=0
Nov 02 12:50:14.581780 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:14.581757    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-66rd4"
Nov 02 12:50:14.581823 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:14.581772    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-66rd4" event={"ID":"e09c5fe2-c1dd-47ab-8464-facefd253549","Type":"ContainerDied","Data":"01cb3f6d6d9046ecf876e009e0c30e94347a355c0df84d6ddcc2a15171f82c94"}
Nov 02 12:50:14.581890 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:14.581837    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-66rd4" event={"ID":"e09c5fe2-c1dd-47ab-8464-facefd253549","Type":"ContainerDied","Data":"33a9b2bad3363c7295f82299f896d52d18589f35bc39f80de66de1b619fbd4c9"}
Nov 02 12:50:14.584439 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:14.584412    4559 generic.go:356] "Generic (PLEG): container finished" podID="eec5b49d-6cf5-4203-b386-9f9a5fa6c7fa" containerID="a35152e4368ed7dc8d754d70c001c11136a446cf09d66af582a531290627369a" exitCode=0
Nov 02 12:50:14.584581 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:14.584491    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x9hwg" event={"ID":"eec5b49d-6cf5-4203-b386-9f9a5fa6c7fa","Type":"ContainerDied","Data":"a35152e4368ed7dc8d754d70c001c11136a446cf09d66af582a531290627369a"}
Nov 02 12:50:14.595423 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:14.594962    4559 scope.go:122] "RemoveContainer" containerID="d65388fb60ebe4470e7fdd8574df2e564aca716b71e203247453ff63961a0559"
Nov 02 12:50:14.595423 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:50:14.595328    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d65388fb60ebe4470e7fdd8574df2e564aca716b71e203247453ff63961a0559\": container with ID starting with d65388fb60ebe4470e7fdd8574df2e564aca716b71e203247453ff63961a0559 not found: ID does not exist" containerID="d65388fb60ebe4470e7fdd8574df2e564aca716b71e203247453ff63961a0559"
Nov 02 12:50:14.595423 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:14.595362    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d65388fb60ebe4470e7fdd8574df2e564aca716b71e203247453ff63961a0559"} err="failed to get container status \"d65388fb60ebe4470e7fdd8574df2e564aca716b71e203247453ff63961a0559\": rpc error: code = NotFound desc = could not find container \"d65388fb60ebe4470e7fdd8574df2e564aca716b71e203247453ff63961a0559\": container with ID starting with d65388fb60ebe4470e7fdd8574df2e564aca716b71e203247453ff63961a0559 not found: ID does not exist"
Nov 02 12:50:14.595423 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:14.595394    4559 scope.go:122] "RemoveContainer" containerID="01cb3f6d6d9046ecf876e009e0c30e94347a355c0df84d6ddcc2a15171f82c94"
Nov 02 12:50:14.605047 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:14.605029    4559 scope.go:122] "RemoveContainer" containerID="01cb3f6d6d9046ecf876e009e0c30e94347a355c0df84d6ddcc2a15171f82c94"
Nov 02 12:50:14.605312 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:50:14.605292    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01cb3f6d6d9046ecf876e009e0c30e94347a355c0df84d6ddcc2a15171f82c94\": container with ID starting with 01cb3f6d6d9046ecf876e009e0c30e94347a355c0df84d6ddcc2a15171f82c94 not found: ID does not exist" containerID="01cb3f6d6d9046ecf876e009e0c30e94347a355c0df84d6ddcc2a15171f82c94"
Nov 02 12:50:14.605390 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:14.605320    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01cb3f6d6d9046ecf876e009e0c30e94347a355c0df84d6ddcc2a15171f82c94"} err="failed to get container status \"01cb3f6d6d9046ecf876e009e0c30e94347a355c0df84d6ddcc2a15171f82c94\": rpc error: code = NotFound desc = could not find container \"01cb3f6d6d9046ecf876e009e0c30e94347a355c0df84d6ddcc2a15171f82c94\": container with ID starting with 01cb3f6d6d9046ecf876e009e0c30e94347a355c0df84d6ddcc2a15171f82c94 not found: ID does not exist"
Nov 02 12:50:14.625998 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:14.625944    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9750c7-84d6d97dc4-cbh2m" podStartSLOduration=2.311642348 podStartE2EDuration="2.62592858s" totalImagesPullingTime="314.286232ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:50:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:50:14.623323101 +0000 UTC m=+1410.878722202" watchObservedRunningTime="2026-11-02 12:50:14.62592858 +0000 UTC m=+1410.881327676"
Nov 02 12:50:14.665519 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:14.665484    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-66rd4"]
Nov 02 12:50:14.669634 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:14.669597    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-66rd4"]
Nov 02 12:50:14.705354 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:14.705270    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tz7qn"]
Nov 02 12:50:14.710062 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:14.710036    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tz7qn"]
Nov 02 12:50:14.930728 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:14.930701    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x9hwg"
Nov 02 12:50:15.052469 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:15.052378    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eec5b49d-6cf5-4203-b386-9f9a5fa6c7fa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eec5b49d-6cf5-4203-b386-9f9a5fa6c7fa-istiod-ca-cert\") pod \"eec5b49d-6cf5-4203-b386-9f9a5fa6c7fa\" (UID: \"eec5b49d-6cf5-4203-b386-9f9a5fa6c7fa\") "
Nov 02 12:50:15.052469 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:15.052430    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eec5b49d-6cf5-4203-b386-9f9a5fa6c7fa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eec5b49d-6cf5-4203-b386-9f9a5fa6c7fa-istio-data\") pod \"eec5b49d-6cf5-4203-b386-9f9a5fa6c7fa\" (UID: \"eec5b49d-6cf5-4203-b386-9f9a5fa6c7fa\") "
Nov 02 12:50:15.052469 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:15.052458    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eec5b49d-6cf5-4203-b386-9f9a5fa6c7fa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eec5b49d-6cf5-4203-b386-9f9a5fa6c7fa-credential-socket\") pod \"eec5b49d-6cf5-4203-b386-9f9a5fa6c7fa\" (UID: \"eec5b49d-6cf5-4203-b386-9f9a5fa6c7fa\") "
Nov 02 12:50:15.052787 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:15.052529    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eec5b49d-6cf5-4203-b386-9f9a5fa6c7fa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eec5b49d-6cf5-4203-b386-9f9a5fa6c7fa-istio-envoy\") pod \"eec5b49d-6cf5-4203-b386-9f9a5fa6c7fa\" (UID: \"eec5b49d-6cf5-4203-b386-9f9a5fa6c7fa\") "
Nov 02 12:50:15.052787 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:15.052621    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eec5b49d-6cf5-4203-b386-9f9a5fa6c7fa-kube-api-access-dlrqm\" (UniqueName: \"kubernetes.io/projected/eec5b49d-6cf5-4203-b386-9f9a5fa6c7fa-kube-api-access-dlrqm\") pod \"eec5b49d-6cf5-4203-b386-9f9a5fa6c7fa\" (UID: \"eec5b49d-6cf5-4203-b386-9f9a5fa6c7fa\") "
Nov 02 12:50:15.052787 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:15.052718    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/eec5b49d-6cf5-4203-b386-9f9a5fa6c7fa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eec5b49d-6cf5-4203-b386-9f9a5fa6c7fa-istio-podinfo\") pod \"eec5b49d-6cf5-4203-b386-9f9a5fa6c7fa\" (UID: \"eec5b49d-6cf5-4203-b386-9f9a5fa6c7fa\") "
Nov 02 12:50:15.052944 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:15.052788    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eec5b49d-6cf5-4203-b386-9f9a5fa6c7fa-istio-data" pod "eec5b49d-6cf5-4203-b386-9f9a5fa6c7fa" (UID: "eec5b49d-6cf5-4203-b386-9f9a5fa6c7fa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:50:15.052944 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:15.052814    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eec5b49d-6cf5-4203-b386-9f9a5fa6c7fa-istiod-ca-cert" pod "eec5b49d-6cf5-4203-b386-9f9a5fa6c7fa" (UID: "eec5b49d-6cf5-4203-b386-9f9a5fa6c7fa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:50:15.052944 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:15.052845    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eec5b49d-6cf5-4203-b386-9f9a5fa6c7fa-credential-socket" pod "eec5b49d-6cf5-4203-b386-9f9a5fa6c7fa" (UID: "eec5b49d-6cf5-4203-b386-9f9a5fa6c7fa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:50:15.052944 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:15.052853    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eec5b49d-6cf5-4203-b386-9f9a5fa6c7fa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eec5b49d-6cf5-4203-b386-9f9a5fa6c7fa-workload-certs\") pod \"eec5b49d-6cf5-4203-b386-9f9a5fa6c7fa\" (UID: \"eec5b49d-6cf5-4203-b386-9f9a5fa6c7fa\") "
Nov 02 12:50:15.052944 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:15.052934    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eec5b49d-6cf5-4203-b386-9f9a5fa6c7fa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eec5b49d-6cf5-4203-b386-9f9a5fa6c7fa-workload-socket\") pod \"eec5b49d-6cf5-4203-b386-9f9a5fa6c7fa\" (UID: \"eec5b49d-6cf5-4203-b386-9f9a5fa6c7fa\") "
Nov 02 12:50:15.053191 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:15.053029    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eec5b49d-6cf5-4203-b386-9f9a5fa6c7fa-istio-token\" (UniqueName: \"kubernetes.io/projected/eec5b49d-6cf5-4203-b386-9f9a5fa6c7fa-istio-token\") pod \"eec5b49d-6cf5-4203-b386-9f9a5fa6c7fa\" (UID: \"eec5b49d-6cf5-4203-b386-9f9a5fa6c7fa\") "
Nov 02 12:50:15.053191 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:15.053079    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eec5b49d-6cf5-4203-b386-9f9a5fa6c7fa-workload-certs" pod "eec5b49d-6cf5-4203-b386-9f9a5fa6c7fa" (UID: "eec5b49d-6cf5-4203-b386-9f9a5fa6c7fa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:50:15.053191 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:15.053172    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eec5b49d-6cf5-4203-b386-9f9a5fa6c7fa-workload-socket" pod "eec5b49d-6cf5-4203-b386-9f9a5fa6c7fa" (UID: "eec5b49d-6cf5-4203-b386-9f9a5fa6c7fa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:50:15.054033 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:15.054008    4559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eec5b49d-6cf5-4203-b386-9f9a5fa6c7fa-workload-certs\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:50:15.054033 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:15.054034    4559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eec5b49d-6cf5-4203-b386-9f9a5fa6c7fa-workload-socket\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:50:15.054221 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:15.054043    4559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eec5b49d-6cf5-4203-b386-9f9a5fa6c7fa-istiod-ca-cert\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:50:15.054221 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:15.054055    4559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eec5b49d-6cf5-4203-b386-9f9a5fa6c7fa-istio-data\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:50:15.054221 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:15.054067    4559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eec5b49d-6cf5-4203-b386-9f9a5fa6c7fa-credential-socket\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:50:15.055051 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:15.055027    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/eec5b49d-6cf5-4203-b386-9f9a5fa6c7fa-istio-podinfo" pod "eec5b49d-6cf5-4203-b386-9f9a5fa6c7fa" (UID: "eec5b49d-6cf5-4203-b386-9f9a5fa6c7fa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 12:50:15.055211 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:15.055180    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eec5b49d-6cf5-4203-b386-9f9a5fa6c7fa-istio-token" pod "eec5b49d-6cf5-4203-b386-9f9a5fa6c7fa" (UID: "eec5b49d-6cf5-4203-b386-9f9a5fa6c7fa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:50:15.055310 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:15.055229    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eec5b49d-6cf5-4203-b386-9f9a5fa6c7fa-istio-envoy" pod "eec5b49d-6cf5-4203-b386-9f9a5fa6c7fa" (UID: "eec5b49d-6cf5-4203-b386-9f9a5fa6c7fa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:50:15.055358 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:15.055333    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eec5b49d-6cf5-4203-b386-9f9a5fa6c7fa-kube-api-access-dlrqm" pod "eec5b49d-6cf5-4203-b386-9f9a5fa6c7fa" (UID: "eec5b49d-6cf5-4203-b386-9f9a5fa6c7fa"). InnerVolumeSpecName "kube-api-access-dlrqm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:50:15.091885 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:50:15.091842    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqcmj" containerName="extractor"
Nov 02 12:50:15.155198 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:15.155166    4559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eec5b49d-6cf5-4203-b386-9f9a5fa6c7fa-istio-token\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:50:15.155198 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:15.155196    4559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eec5b49d-6cf5-4203-b386-9f9a5fa6c7fa-istio-envoy\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:50:15.155393 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:15.155206    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dlrqm\" (UniqueName: \"kubernetes.io/projected/eec5b49d-6cf5-4203-b386-9f9a5fa6c7fa-kube-api-access-dlrqm\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:50:15.155393 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:15.155215    4559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eec5b49d-6cf5-4203-b386-9f9a5fa6c7fa-istio-podinfo\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:50:15.593382 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:15.593348    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x9hwg"
Nov 02 12:50:15.593565 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:15.593384    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x9hwg" event={"ID":"eec5b49d-6cf5-4203-b386-9f9a5fa6c7fa","Type":"ContainerDied","Data":"97bc16107109223ef76bb78854322cf8b07d953a1da39288cb3ac0ad5bfded8d"}
Nov 02 12:50:15.593565 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:15.593451    4559 scope.go:122] "RemoveContainer" containerID="a35152e4368ed7dc8d754d70c001c11136a446cf09d66af582a531290627369a"
Nov 02 12:50:15.661022 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:15.660985    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x9hwg"]
Nov 02 12:50:15.673524 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:15.673492    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x9hwg"]
Nov 02 12:50:15.895527 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:15.895452    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="597fdf69-8d19-46b2-b61f-bbe2ec731e61" path="/var/lib/kubelet/pods/597fdf69-8d19-46b2-b61f-bbe2ec731e61/volumes"
Nov 02 12:50:15.896300 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:15.896276    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e09c5fe2-c1dd-47ab-8464-facefd253549" path="/var/lib/kubelet/pods/e09c5fe2-c1dd-47ab-8464-facefd253549/volumes"
Nov 02 12:50:15.896909 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:15.896891    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eec5b49d-6cf5-4203-b386-9f9a5fa6c7fa" path="/var/lib/kubelet/pods/eec5b49d-6cf5-4203-b386-9f9a5fa6c7fa/volumes"
Nov 02 12:50:15.910768 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:15.910729    4559 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-x9hwg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 12:50:15.910869 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:15.910791    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x9hwg" podUID="eec5b49d-6cf5-4203-b386-9f9a5fa6c7fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:50:16.911977 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:16.911936    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-eb24df-77b8d4b6bc-b7fx5" podUID="6b7aacab-55fe-482d-8e15-a8f847678e48" containerName="payload-processing" containerID="cri-o://6269a8abd8b22acd4ad898ef7b00f83a21595260625526b4ed2ce4843f31cf23" gracePeriod=25
Nov 02 12:50:16.927449 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:16.927416    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-eb24df-748477f8f5-67vfv" podUID="9f72904b-21b5-40f8-804d-cd747e649e02" containerName="payload-pre-processing" containerID="cri-o://a2766108e68f2f54be02ba0f63c6e1c596ebb23a22966cf318d61082a32c1b33" gracePeriod=25
Nov 02 12:50:17.205679 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:17.205655    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-eb24df-77b8d4b6bc-b7fx5"
Nov 02 12:50:17.242421 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:17.242397    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-eb24df-748477f8f5-67vfv"
Nov 02 12:50:17.286318 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:17.286281    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b7aacab-55fe-482d-8e15-a8f847678e48-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b7aacab-55fe-482d-8e15-a8f847678e48-plugins-config-volume\") pod \"6b7aacab-55fe-482d-8e15-a8f847678e48\" (UID: \"6b7aacab-55fe-482d-8e15-a8f847678e48\") "
Nov 02 12:50:17.286499 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:17.286402    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f72904b-21b5-40f8-804d-cd747e649e02-kube-api-access-smzn6\" (UniqueName: \"kubernetes.io/projected/9f72904b-21b5-40f8-804d-cd747e649e02-kube-api-access-smzn6\") pod \"9f72904b-21b5-40f8-804d-cd747e649e02\" (UID: \"9f72904b-21b5-40f8-804d-cd747e649e02\") "
Nov 02 12:50:17.286499 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:17.286469    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b7aacab-55fe-482d-8e15-a8f847678e48-kube-api-access-79cpp\" (UniqueName: \"kubernetes.io/projected/6b7aacab-55fe-482d-8e15-a8f847678e48-kube-api-access-79cpp\") pod \"6b7aacab-55fe-482d-8e15-a8f847678e48\" (UID: \"6b7aacab-55fe-482d-8e15-a8f847678e48\") "
Nov 02 12:50:17.286677 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:17.286536    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f72904b-21b5-40f8-804d-cd747e649e02-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f72904b-21b5-40f8-804d-cd747e649e02-plugins-config-volume\") pod \"9f72904b-21b5-40f8-804d-cd747e649e02\" (UID: \"9f72904b-21b5-40f8-804d-cd747e649e02\") "
Nov 02 12:50:17.287065 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:17.286795    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b7aacab-55fe-482d-8e15-a8f847678e48-plugins-config-volume" pod "6b7aacab-55fe-482d-8e15-a8f847678e48" (UID: "6b7aacab-55fe-482d-8e15-a8f847678e48"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:50:17.287180 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:17.287157    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f72904b-21b5-40f8-804d-cd747e649e02-plugins-config-volume" pod "9f72904b-21b5-40f8-804d-cd747e649e02" (UID: "9f72904b-21b5-40f8-804d-cd747e649e02"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:50:17.287297 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:17.287272    4559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f72904b-21b5-40f8-804d-cd747e649e02-plugins-config-volume\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:50:17.287297 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:17.287297    4559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b7aacab-55fe-482d-8e15-a8f847678e48-plugins-config-volume\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:50:17.289335 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:17.289309    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f72904b-21b5-40f8-804d-cd747e649e02-kube-api-access-smzn6" pod "9f72904b-21b5-40f8-804d-cd747e649e02" (UID: "9f72904b-21b5-40f8-804d-cd747e649e02"). InnerVolumeSpecName "kube-api-access-smzn6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:50:17.290039 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:17.289888    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b7aacab-55fe-482d-8e15-a8f847678e48-kube-api-access-79cpp" pod "6b7aacab-55fe-482d-8e15-a8f847678e48" (UID: "6b7aacab-55fe-482d-8e15-a8f847678e48"). InnerVolumeSpecName "kube-api-access-79cpp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:50:17.388597 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:17.388568    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-smzn6\" (UniqueName: \"kubernetes.io/projected/9f72904b-21b5-40f8-804d-cd747e649e02-kube-api-access-smzn6\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:50:17.388597 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:17.388592    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79cpp\" (UniqueName: \"kubernetes.io/projected/6b7aacab-55fe-482d-8e15-a8f847678e48-kube-api-access-79cpp\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:50:17.620092 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:17.620004    4559 generic.go:356] "Generic (PLEG): container finished" podID="6b7aacab-55fe-482d-8e15-a8f847678e48" containerID="6269a8abd8b22acd4ad898ef7b00f83a21595260625526b4ed2ce4843f31cf23" exitCode=0
Nov 02 12:50:17.620092 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:17.620063    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-eb24df-77b8d4b6bc-b7fx5"
Nov 02 12:50:17.620307 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:17.620091    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-eb24df-77b8d4b6bc-b7fx5" event={"ID":"6b7aacab-55fe-482d-8e15-a8f847678e48","Type":"ContainerDied","Data":"6269a8abd8b22acd4ad898ef7b00f83a21595260625526b4ed2ce4843f31cf23"}
Nov 02 12:50:17.620307 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:17.620182    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-eb24df-77b8d4b6bc-b7fx5" event={"ID":"6b7aacab-55fe-482d-8e15-a8f847678e48","Type":"ContainerDied","Data":"88272a1ac8069002b4d33897a05d369c2636f001b9aaa5d51377843e5d6b9ecb"}
Nov 02 12:50:17.620307 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:17.620210    4559 scope.go:122] "RemoveContainer" containerID="6269a8abd8b22acd4ad898ef7b00f83a21595260625526b4ed2ce4843f31cf23"
Nov 02 12:50:17.621788 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:17.621761    4559 generic.go:356] "Generic (PLEG): container finished" podID="9f72904b-21b5-40f8-804d-cd747e649e02" containerID="a2766108e68f2f54be02ba0f63c6e1c596ebb23a22966cf318d61082a32c1b33" exitCode=0
Nov 02 12:50:17.621924 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:17.621811    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-eb24df-748477f8f5-67vfv" event={"ID":"9f72904b-21b5-40f8-804d-cd747e649e02","Type":"ContainerDied","Data":"a2766108e68f2f54be02ba0f63c6e1c596ebb23a22966cf318d61082a32c1b33"}
Nov 02 12:50:17.621924 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:17.621819    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-eb24df-748477f8f5-67vfv"
Nov 02 12:50:17.621924 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:17.621850    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-eb24df-748477f8f5-67vfv" event={"ID":"9f72904b-21b5-40f8-804d-cd747e649e02","Type":"ContainerDied","Data":"dd8c60e5de2807b9df4eb4b5acc6450f37b7fba20c78715fbe2288a56360dab8"}
Nov 02 12:50:17.631569 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:17.631552    4559 scope.go:122] "RemoveContainer" containerID="6269a8abd8b22acd4ad898ef7b00f83a21595260625526b4ed2ce4843f31cf23"
Nov 02 12:50:17.631911 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:50:17.631885    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6269a8abd8b22acd4ad898ef7b00f83a21595260625526b4ed2ce4843f31cf23\": container with ID starting with 6269a8abd8b22acd4ad898ef7b00f83a21595260625526b4ed2ce4843f31cf23 not found: ID does not exist" containerID="6269a8abd8b22acd4ad898ef7b00f83a21595260625526b4ed2ce4843f31cf23"
Nov 02 12:50:17.631999 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:17.631919    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6269a8abd8b22acd4ad898ef7b00f83a21595260625526b4ed2ce4843f31cf23"} err="failed to get container status \"6269a8abd8b22acd4ad898ef7b00f83a21595260625526b4ed2ce4843f31cf23\": rpc error: code = NotFound desc = could not find container \"6269a8abd8b22acd4ad898ef7b00f83a21595260625526b4ed2ce4843f31cf23\": container with ID starting with 6269a8abd8b22acd4ad898ef7b00f83a21595260625526b4ed2ce4843f31cf23 not found: ID does not exist"
Nov 02 12:50:17.631999 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:17.631943    4559 scope.go:122] "RemoveContainer" containerID="a2766108e68f2f54be02ba0f63c6e1c596ebb23a22966cf318d61082a32c1b33"
Nov 02 12:50:17.641059 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:17.641041    4559 scope.go:122] "RemoveContainer" containerID="a2766108e68f2f54be02ba0f63c6e1c596ebb23a22966cf318d61082a32c1b33"
Nov 02 12:50:17.641328 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:50:17.641306    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2766108e68f2f54be02ba0f63c6e1c596ebb23a22966cf318d61082a32c1b33\": container with ID starting with a2766108e68f2f54be02ba0f63c6e1c596ebb23a22966cf318d61082a32c1b33 not found: ID does not exist" containerID="a2766108e68f2f54be02ba0f63c6e1c596ebb23a22966cf318d61082a32c1b33"
Nov 02 12:50:17.641379 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:17.641339    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2766108e68f2f54be02ba0f63c6e1c596ebb23a22966cf318d61082a32c1b33"} err="failed to get container status \"a2766108e68f2f54be02ba0f63c6e1c596ebb23a22966cf318d61082a32c1b33\": rpc error: code = NotFound desc = could not find container \"a2766108e68f2f54be02ba0f63c6e1c596ebb23a22966cf318d61082a32c1b33\": container with ID starting with a2766108e68f2f54be02ba0f63c6e1c596ebb23a22966cf318d61082a32c1b33 not found: ID does not exist"
Nov 02 12:50:17.674744 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:17.674716    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-eb24df-748477f8f5-67vfv"]
Nov 02 12:50:17.678061 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:17.678040    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-eb24df-748477f8f5-67vfv"]
Nov 02 12:50:17.714968 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:17.714942    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-eb24df-77b8d4b6bc-b7fx5"]
Nov 02 12:50:17.720143 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:17.720121    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-eb24df-77b8d4b6bc-b7fx5"]
Nov 02 12:50:17.894207 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:17.894135    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b7aacab-55fe-482d-8e15-a8f847678e48" path="/var/lib/kubelet/pods/6b7aacab-55fe-482d-8e15-a8f847678e48/volumes"
Nov 02 12:50:17.894581 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:17.894567    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f72904b-21b5-40f8-804d-cd747e649e02" path="/var/lib/kubelet/pods/9f72904b-21b5-40f8-804d-cd747e649e02/volumes"
Nov 02 12:50:19.514708 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:19.514666    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-d3a281-844b5468d7-hvsnh"
Nov 02 12:50:20.602289 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:20.602262    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9750c7-84d6d97dc4-cbh2m"
Nov 02 12:50:26.685868 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:26.685837    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-d3a281-6b8dcb978d-hcqxm"]
Nov 02 12:50:26.686475 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:26.686460    4559 cpu_manager.go:414] "containerMap: removing container" podUID="9f72904b-21b5-40f8-804d-cd747e649e02" containerName="payload-pre-processing"
Nov 02 12:50:26.686516 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:26.686477    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f72904b-21b5-40f8-804d-cd747e649e02" containerName="payload-pre-processing"
Nov 02 12:50:26.686516 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:26.686510    4559 cpu_manager.go:414] "containerMap: removing container" podUID="eec5b49d-6cf5-4203-b386-9f9a5fa6c7fa" containerName="istio-proxy"
Nov 02 12:50:26.686579 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:26.686516    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eec5b49d-6cf5-4203-b386-9f9a5fa6c7fa" containerName="istio-proxy"
Nov 02 12:50:26.686579 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:26.686541    4559 cpu_manager.go:414] "containerMap: removing container" podUID="e09c5fe2-c1dd-47ab-8464-facefd253549" containerName="istio-proxy"
Nov 02 12:50:26.686579 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:26.686547    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e09c5fe2-c1dd-47ab-8464-facefd253549" containerName="istio-proxy"
Nov 02 12:50:26.686579 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:26.686560    4559 cpu_manager.go:414] "containerMap: removing container" podUID="597fdf69-8d19-46b2-b61f-bbe2ec731e61" containerName="istio-proxy"
Nov 02 12:50:26.686579 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:26.686565    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="597fdf69-8d19-46b2-b61f-bbe2ec731e61" containerName="istio-proxy"
Nov 02 12:50:26.686579 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:26.686574    4559 cpu_manager.go:414] "containerMap: removing container" podUID="6b7aacab-55fe-482d-8e15-a8f847678e48" containerName="payload-processing"
Nov 02 12:50:26.686800 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:26.686581    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b7aacab-55fe-482d-8e15-a8f847678e48" containerName="payload-processing"
Nov 02 12:50:26.686800 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:26.686690    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f72904b-21b5-40f8-804d-cd747e649e02" containerName="payload-pre-processing"
Nov 02 12:50:26.686800 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:26.686703    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="eec5b49d-6cf5-4203-b386-9f9a5fa6c7fa" containerName="istio-proxy"
Nov 02 12:50:26.686800 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:26.686710    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="e09c5fe2-c1dd-47ab-8464-facefd253549" containerName="istio-proxy"
Nov 02 12:50:26.686800 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:26.686717    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="597fdf69-8d19-46b2-b61f-bbe2ec731e61" containerName="istio-proxy"
Nov 02 12:50:26.686800 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:26.686723    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b7aacab-55fe-482d-8e15-a8f847678e48" containerName="payload-processing"
Nov 02 12:50:26.691890 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:26.691871    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d3a281-6b8dcb978d-hcqxm"
Nov 02 12:50:26.707994 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:26.707974    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-d3a281-6b8dcb978d-hcqxm"]
Nov 02 12:50:26.722576 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:26.722547    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-d3a281-serving-cert\""
Nov 02 12:50:26.843394 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:26.843361    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7f14aa1-1b9b-47ff-ac46-2db84191079b-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-d3a281-6b8dcb978d-hcqxm\" (UID: \"e7f14aa1-1b9b-47ff-ac46-2db84191079b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d3a281-6b8dcb978d-hcqxm"
Nov 02 12:50:26.843550 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:26.843525    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e7f14aa1-1b9b-47ff-ac46-2db84191079b-maas-api-tls\") pod \"maas-api-e2e-praxis-b-d3a281-6b8dcb978d-hcqxm\" (UID: \"e7f14aa1-1b9b-47ff-ac46-2db84191079b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d3a281-6b8dcb978d-hcqxm"
Nov 02 12:50:26.843614 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:26.843578    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpp6p\" (UniqueName: \"kubernetes.io/projected/e7f14aa1-1b9b-47ff-ac46-2db84191079b-kube-api-access-jpp6p\") pod \"maas-api-e2e-praxis-b-d3a281-6b8dcb978d-hcqxm\" (UID: \"e7f14aa1-1b9b-47ff-ac46-2db84191079b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d3a281-6b8dcb978d-hcqxm"
Nov 02 12:50:26.843741 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:26.843726    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7f14aa1-1b9b-47ff-ac46-2db84191079b-metrics-certs\") pod \"maas-api-e2e-praxis-b-d3a281-6b8dcb978d-hcqxm\" (UID: \"e7f14aa1-1b9b-47ff-ac46-2db84191079b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d3a281-6b8dcb978d-hcqxm"
Nov 02 12:50:26.947711 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:26.947455    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e7f14aa1-1b9b-47ff-ac46-2db84191079b-maas-api-tls\") pod \"maas-api-e2e-praxis-b-d3a281-6b8dcb978d-hcqxm\" (UID: \"e7f14aa1-1b9b-47ff-ac46-2db84191079b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d3a281-6b8dcb978d-hcqxm"
Nov 02 12:50:26.947711 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:26.947530    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpp6p\" (UniqueName: \"kubernetes.io/projected/e7f14aa1-1b9b-47ff-ac46-2db84191079b-kube-api-access-jpp6p\") pod \"maas-api-e2e-praxis-b-d3a281-6b8dcb978d-hcqxm\" (UID: \"e7f14aa1-1b9b-47ff-ac46-2db84191079b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d3a281-6b8dcb978d-hcqxm"
Nov 02 12:50:26.947711 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:26.947664    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7f14aa1-1b9b-47ff-ac46-2db84191079b-metrics-certs\") pod \"maas-api-e2e-praxis-b-d3a281-6b8dcb978d-hcqxm\" (UID: \"e7f14aa1-1b9b-47ff-ac46-2db84191079b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d3a281-6b8dcb978d-hcqxm"
Nov 02 12:50:26.947996 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:26.947722    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7f14aa1-1b9b-47ff-ac46-2db84191079b-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-d3a281-6b8dcb978d-hcqxm\" (UID: \"e7f14aa1-1b9b-47ff-ac46-2db84191079b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d3a281-6b8dcb978d-hcqxm"
Nov 02 12:50:26.951668 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:26.948547    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7f14aa1-1b9b-47ff-ac46-2db84191079b-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-d3a281-6b8dcb978d-hcqxm\" (UID: \"e7f14aa1-1b9b-47ff-ac46-2db84191079b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d3a281-6b8dcb978d-hcqxm"
Nov 02 12:50:26.951668 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:50:26.948674    4559 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-b-d3a281-serving-cert: secret "maas-api-e2e-praxis-b-d3a281-serving-cert" not found
Nov 02 12:50:26.951668 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:50:26.948743    4559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e7f14aa1-1b9b-47ff-ac46-2db84191079b-maas-api-tls podName:e7f14aa1-1b9b-47ff-ac46-2db84191079b nodeName:}" failed. No retries permitted until 2026-11-02 12:50:27.448724365 +0000 UTC m=+1423.704123439 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e7f14aa1-1b9b-47ff-ac46-2db84191079b-maas-api-tls") pod "maas-api-e2e-praxis-b-d3a281-6b8dcb978d-hcqxm" (UID: "e7f14aa1-1b9b-47ff-ac46-2db84191079b") : secret "maas-api-e2e-praxis-b-d3a281-serving-cert" not found
Nov 02 12:50:26.985660 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:26.980229    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7f14aa1-1b9b-47ff-ac46-2db84191079b-metrics-certs\") pod \"maas-api-e2e-praxis-b-d3a281-6b8dcb978d-hcqxm\" (UID: \"e7f14aa1-1b9b-47ff-ac46-2db84191079b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d3a281-6b8dcb978d-hcqxm"
Nov 02 12:50:26.985660 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:26.981305    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpp6p\" (UniqueName: \"kubernetes.io/projected/e7f14aa1-1b9b-47ff-ac46-2db84191079b-kube-api-access-jpp6p\") pod \"maas-api-e2e-praxis-b-d3a281-6b8dcb978d-hcqxm\" (UID: \"e7f14aa1-1b9b-47ff-ac46-2db84191079b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d3a281-6b8dcb978d-hcqxm"
Nov 02 12:50:27.455798 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:27.455753    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e7f14aa1-1b9b-47ff-ac46-2db84191079b-maas-api-tls\") pod \"maas-api-e2e-praxis-b-d3a281-6b8dcb978d-hcqxm\" (UID: \"e7f14aa1-1b9b-47ff-ac46-2db84191079b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d3a281-6b8dcb978d-hcqxm"
Nov 02 12:50:27.458122 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:27.458105    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e7f14aa1-1b9b-47ff-ac46-2db84191079b-maas-api-tls\") pod \"maas-api-e2e-praxis-b-d3a281-6b8dcb978d-hcqxm\" (UID: \"e7f14aa1-1b9b-47ff-ac46-2db84191079b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d3a281-6b8dcb978d-hcqxm"
Nov 02 12:50:27.610583 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:27.610548    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d3a281-6b8dcb978d-hcqxm"
Nov 02 12:50:27.926864 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:50:27.926835    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7f14aa1_1b9b_47ff_ac46_2db84191079b.slice/crio-9e5022fb05270c5e966d8da63955e0bc0b902d56919e592ef89d1d6ac5f1604a WatchSource:0}: Error finding container 9e5022fb05270c5e966d8da63955e0bc0b902d56919e592ef89d1d6ac5f1604a: Status 404 returned error can't find the container with id 9e5022fb05270c5e966d8da63955e0bc0b902d56919e592ef89d1d6ac5f1604a
Nov 02 12:50:27.927936 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:27.927910    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-d3a281-6b8dcb978d-hcqxm"]
Nov 02 12:50:28.706352 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:28.706311    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d3a281-6b8dcb978d-hcqxm" event={"ID":"e7f14aa1-1b9b-47ff-ac46-2db84191079b","Type":"ContainerStarted","Data":"81175c8b270bc8b000dfd7e74444dd2c44e7c29a7979bd7a11ee8c31591e9ac6"}
Nov 02 12:50:28.706524 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:28.706378    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d3a281-6b8dcb978d-hcqxm" event={"ID":"e7f14aa1-1b9b-47ff-ac46-2db84191079b","Type":"ContainerStarted","Data":"9e5022fb05270c5e966d8da63955e0bc0b902d56919e592ef89d1d6ac5f1604a"}
Nov 02 12:50:28.706524 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:28.706410    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d3a281-6b8dcb978d-hcqxm"
Nov 02 12:50:28.758808 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:28.758754    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d3a281-6b8dcb978d-hcqxm" podStartSLOduration=2.396296201 podStartE2EDuration="2.758734588s" totalImagesPullingTime="362.438387ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:50:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:50:28.756740262 +0000 UTC m=+1425.012139358" watchObservedRunningTime="2026-11-02 12:50:28.758734588 +0000 UTC m=+1425.014133688"
Nov 02 12:50:31.918359 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:31.918324    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-eb24df-864494fcf-kzf6t"]
Nov 02 12:50:31.923825 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:31.923589    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-eb24df-864494fcf-kzf6t"]
Nov 02 12:50:31.923825 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:31.923688    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-eb24df-864494fcf-kzf6t"
Nov 02 12:50:31.939162 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:31.939134    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-eb24df\""
Nov 02 12:50:32.030067 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:32.030035    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-eb24df-dockercfg-d2gvt\""
Nov 02 12:50:32.088935 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:32.088895    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2glwc\" (UniqueName: \"kubernetes.io/projected/38679526-8764-40bb-a5c0-be287e2c6e5b-kube-api-access-2glwc\") pod \"payload-pre-processing-e2e-ipp-a-eb24df-864494fcf-kzf6t\" (UID: \"38679526-8764-40bb-a5c0-be287e2c6e5b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-eb24df-864494fcf-kzf6t"
Nov 02 12:50:32.089154 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:32.089134    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38679526-8764-40bb-a5c0-be287e2c6e5b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-eb24df-864494fcf-kzf6t\" (UID: \"38679526-8764-40bb-a5c0-be287e2c6e5b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-eb24df-864494fcf-kzf6t"
Nov 02 12:50:32.190038 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:32.189950    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2glwc\" (UniqueName: \"kubernetes.io/projected/38679526-8764-40bb-a5c0-be287e2c6e5b-kube-api-access-2glwc\") pod \"payload-pre-processing-e2e-ipp-a-eb24df-864494fcf-kzf6t\" (UID: \"38679526-8764-40bb-a5c0-be287e2c6e5b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-eb24df-864494fcf-kzf6t"
Nov 02 12:50:32.190221 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:32.190174    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38679526-8764-40bb-a5c0-be287e2c6e5b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-eb24df-864494fcf-kzf6t\" (UID: \"38679526-8764-40bb-a5c0-be287e2c6e5b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-eb24df-864494fcf-kzf6t"
Nov 02 12:50:32.190870 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:32.190850    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38679526-8764-40bb-a5c0-be287e2c6e5b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-eb24df-864494fcf-kzf6t\" (UID: \"38679526-8764-40bb-a5c0-be287e2c6e5b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-eb24df-864494fcf-kzf6t"
Nov 02 12:50:32.213563 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:32.213529    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2glwc\" (UniqueName: \"kubernetes.io/projected/38679526-8764-40bb-a5c0-be287e2c6e5b-kube-api-access-2glwc\") pod \"payload-pre-processing-e2e-ipp-a-eb24df-864494fcf-kzf6t\" (UID: \"38679526-8764-40bb-a5c0-be287e2c6e5b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-eb24df-864494fcf-kzf6t"
Nov 02 12:50:32.241784 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:32.241748    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-eb24df-864494fcf-kzf6t"
Nov 02 12:50:32.560418 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:50:32.560388    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38679526_8764_40bb_a5c0_be287e2c6e5b.slice/crio-6a8774e895f6c77a8f794d07e42dcb46d7a4b72377b91720c4c3fcfcbf7b9566 WatchSource:0}: Error finding container 6a8774e895f6c77a8f794d07e42dcb46d7a4b72377b91720c4c3fcfcbf7b9566: Status 404 returned error can't find the container with id 6a8774e895f6c77a8f794d07e42dcb46d7a4b72377b91720c4c3fcfcbf7b9566
Nov 02 12:50:32.562769 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:32.562741    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-eb24df-864494fcf-kzf6t"]
Nov 02 12:50:32.804935 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:32.804902    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-eb24df-864494fcf-kzf6t" event={"ID":"38679526-8764-40bb-a5c0-be287e2c6e5b","Type":"ContainerStarted","Data":"58c789e83735b6170c1b7ed603ece9861195ccf4eadf8b944732624640fadca3"}
Nov 02 12:50:32.805105 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:32.804953    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-eb24df-864494fcf-kzf6t" event={"ID":"38679526-8764-40bb-a5c0-be287e2c6e5b","Type":"ContainerStarted","Data":"6a8774e895f6c77a8f794d07e42dcb46d7a4b72377b91720c4c3fcfcbf7b9566"}
Nov 02 12:50:32.805105 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:32.805038    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-eb24df-864494fcf-kzf6t"
Nov 02 12:50:32.848378 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:32.848288    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-eb24df-864494fcf-kzf6t" podStartSLOduration=1.8482751739999999 podStartE2EDuration="1.848275174s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:50:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:50:32.847446964 +0000 UTC m=+1429.102846059" watchObservedRunningTime="2026-11-02 12:50:32.848275174 +0000 UTC m=+1429.103674266"
Nov 02 12:50:33.460682 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:33.460626    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"]
Nov 02 12:50:34.723344 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:34.723317    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d3a281-6b8dcb978d-hcqxm"
Nov 02 12:50:37.773320 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:37.773287    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-e93a4b-6947c6444f-8cc8j"]
Nov 02 12:50:37.810568 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:37.810524    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-e93a4b-bd984bc95-7fwhg"]
Nov 02 12:50:39.496718 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:39.493630    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-e93a4b-7697fd684f-gb8s9"]
Nov 02 12:50:39.496718 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:39.493894    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e93a4b-7697fd684f-gb8s9" podUID="7a673851-931d-4634-8b10-d26a2518552d" containerName="maas-api" containerID="cri-o://30fe7da74ba6e9084990cbc7fc3bc0d546f5df2580fdec2fd30b4691bcc05cb0" gracePeriod=30
Nov 02 12:50:40.139674 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:40.134470    4559 generic.go:356] "Generic (PLEG): container finished" podID="7a673851-931d-4634-8b10-d26a2518552d" containerID="30fe7da74ba6e9084990cbc7fc3bc0d546f5df2580fdec2fd30b4691bcc05cb0" exitCode=0
Nov 02 12:50:40.139674 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:40.134694    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e93a4b-7697fd684f-gb8s9" event={"ID":"7a673851-931d-4634-8b10-d26a2518552d","Type":"ContainerDied","Data":"30fe7da74ba6e9084990cbc7fc3bc0d546f5df2580fdec2fd30b4691bcc05cb0"}
Nov 02 12:50:40.154506 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:40.154476    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e93a4b-7697fd684f-gb8s9"
Nov 02 12:50:40.228106 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:40.228078    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7a673851-931d-4634-8b10-d26a2518552d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a673851-931d-4634-8b10-d26a2518552d-metrics-certs\") pod \"7a673851-931d-4634-8b10-d26a2518552d\" (UID: \"7a673851-931d-4634-8b10-d26a2518552d\") "
Nov 02 12:50:40.228408 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:40.228394    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a673851-931d-4634-8b10-d26a2518552d-kube-api-access-k9pqf\" (UniqueName: \"kubernetes.io/projected/7a673851-931d-4634-8b10-d26a2518552d-kube-api-access-k9pqf\") pod \"7a673851-931d-4634-8b10-d26a2518552d\" (UID: \"7a673851-931d-4634-8b10-d26a2518552d\") "
Nov 02 12:50:40.228521 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:40.228510    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7a673851-931d-4634-8b10-d26a2518552d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a673851-931d-4634-8b10-d26a2518552d-maas-api-tls\") pod \"7a673851-931d-4634-8b10-d26a2518552d\" (UID: \"7a673851-931d-4634-8b10-d26a2518552d\") "
Nov 02 12:50:40.228656 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:40.228612    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a673851-931d-4634-8b10-d26a2518552d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a673851-931d-4634-8b10-d26a2518552d-openshift-service-ca\") pod \"7a673851-931d-4634-8b10-d26a2518552d\" (UID: \"7a673851-931d-4634-8b10-d26a2518552d\") "
Nov 02 12:50:40.229580 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:40.229560    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a673851-931d-4634-8b10-d26a2518552d-openshift-service-ca" pod "7a673851-931d-4634-8b10-d26a2518552d" (UID: "7a673851-931d-4634-8b10-d26a2518552d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:50:40.232184 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:40.232160    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a673851-931d-4634-8b10-d26a2518552d-maas-api-tls" pod "7a673851-931d-4634-8b10-d26a2518552d" (UID: "7a673851-931d-4634-8b10-d26a2518552d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:50:40.233630 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:40.233581    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a673851-931d-4634-8b10-d26a2518552d-metrics-certs" pod "7a673851-931d-4634-8b10-d26a2518552d" (UID: "7a673851-931d-4634-8b10-d26a2518552d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:50:40.234687 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:40.234659    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a673851-931d-4634-8b10-d26a2518552d-kube-api-access-k9pqf" pod "7a673851-931d-4634-8b10-d26a2518552d" (UID: "7a673851-931d-4634-8b10-d26a2518552d"). InnerVolumeSpecName "kube-api-access-k9pqf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:50:40.333350 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:40.333295    4559 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a673851-931d-4634-8b10-d26a2518552d-metrics-certs\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:50:40.333350 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:40.333339    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9pqf\" (UniqueName: \"kubernetes.io/projected/7a673851-931d-4634-8b10-d26a2518552d-kube-api-access-k9pqf\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:50:40.333612 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:40.333355    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a673851-931d-4634-8b10-d26a2518552d-maas-api-tls\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:50:40.333612 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:40.333375    4559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a673851-931d-4634-8b10-d26a2518552d-openshift-service-ca\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:50:41.143573 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:41.143483    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e93a4b-7697fd684f-gb8s9"
Nov 02 12:50:41.143573 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:41.143510    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e93a4b-7697fd684f-gb8s9" event={"ID":"7a673851-931d-4634-8b10-d26a2518552d","Type":"ContainerDied","Data":"6f5e19279eba69dd9d9e1b232a655496fdf5bfe65447469b3abf53262fd8d784"}
Nov 02 12:50:41.144072 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:41.143578    4559 scope.go:122] "RemoveContainer" containerID="30fe7da74ba6e9084990cbc7fc3bc0d546f5df2580fdec2fd30b4691bcc05cb0"
Nov 02 12:50:41.199088 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:41.199057    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-e93a4b-7697fd684f-gb8s9"]
Nov 02 12:50:41.212100 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:41.212071    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-e93a4b-7697fd684f-gb8s9"]
Nov 02 12:50:41.896443 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:41.896402    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a673851-931d-4634-8b10-d26a2518552d" path="/var/lib/kubelet/pods/7a673851-931d-4634-8b10-d26a2518552d/volumes"
Nov 02 12:50:42.727857 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:42.727820    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e93a4b-7697fd684f-gb8s9" podUID="7a673851-931d-4634-8b10-d26a2518552d" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.152:8443/health\": context deadline exceeded"
Nov 02 12:50:42.774231 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:42.774191    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-e93a4b-6947c6444f-8cc8j" podUID="a7b7d167-6e3d-4632-822e-597f0578a3b0" containerName="payload-processing" containerID="cri-o://781a4b8347a3ba231d1cd5d969497d10c81ce681b569f179f1714c9225989539" gracePeriod=25
Nov 02 12:50:42.811250 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:42.811203    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e93a4b-bd984bc95-7fwhg" podUID="fbce1ffb-dfe6-47b1-bd72-d86787cc1a4b" containerName="payload-pre-processing" containerID="cri-o://a9bdff8f7e4e340e7050c063e9eed857152b2114f6d51bdfaef0643fc624f3ca" gracePeriod=25
Nov 02 12:50:43.109655 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:43.109606    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-e93a4b-6947c6444f-8cc8j"
Nov 02 12:50:43.146906 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:43.146878    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e93a4b-bd984bc95-7fwhg"
Nov 02 12:50:43.166696 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:43.163507    4559 generic.go:356] "Generic (PLEG): container finished" podID="fbce1ffb-dfe6-47b1-bd72-d86787cc1a4b" containerID="a9bdff8f7e4e340e7050c063e9eed857152b2114f6d51bdfaef0643fc624f3ca" exitCode=0
Nov 02 12:50:43.166696 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:43.163546    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e93a4b-bd984bc95-7fwhg" event={"ID":"fbce1ffb-dfe6-47b1-bd72-d86787cc1a4b","Type":"ContainerDied","Data":"a9bdff8f7e4e340e7050c063e9eed857152b2114f6d51bdfaef0643fc624f3ca"}
Nov 02 12:50:43.166696 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:43.163608    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e93a4b-bd984bc95-7fwhg" event={"ID":"fbce1ffb-dfe6-47b1-bd72-d86787cc1a4b","Type":"ContainerDied","Data":"022bbaed823e9b15b3f95eda0f43392fcf0b4a71ccbf5a136c6dc048a49ad71b"}
Nov 02 12:50:43.166696 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:43.163633    4559 scope.go:122] "RemoveContainer" containerID="a9bdff8f7e4e340e7050c063e9eed857152b2114f6d51bdfaef0643fc624f3ca"
Nov 02 12:50:43.166696 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:43.163559    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e93a4b-bd984bc95-7fwhg"
Nov 02 12:50:43.166696 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:43.165415    4559 generic.go:356] "Generic (PLEG): container finished" podID="a7b7d167-6e3d-4632-822e-597f0578a3b0" containerID="781a4b8347a3ba231d1cd5d969497d10c81ce681b569f179f1714c9225989539" exitCode=0
Nov 02 12:50:43.166696 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:43.165468    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-e93a4b-6947c6444f-8cc8j" event={"ID":"a7b7d167-6e3d-4632-822e-597f0578a3b0","Type":"ContainerDied","Data":"781a4b8347a3ba231d1cd5d969497d10c81ce681b569f179f1714c9225989539"}
Nov 02 12:50:43.166696 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:43.165497    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-e93a4b-6947c6444f-8cc8j"
Nov 02 12:50:43.166696 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:43.165505    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-e93a4b-6947c6444f-8cc8j" event={"ID":"a7b7d167-6e3d-4632-822e-597f0578a3b0","Type":"ContainerDied","Data":"8a4ab8fe54ea0ad650395630f2d8fe3e0faea0c5f7b8eed383d61d4625a8033e"}
Nov 02 12:50:43.174859 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:43.174832    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbce1ffb-dfe6-47b1-bd72-d86787cc1a4b-kube-api-access-gg5pg\" (UniqueName: \"kubernetes.io/projected/fbce1ffb-dfe6-47b1-bd72-d86787cc1a4b-kube-api-access-gg5pg\") pod \"fbce1ffb-dfe6-47b1-bd72-d86787cc1a4b\" (UID: \"fbce1ffb-dfe6-47b1-bd72-d86787cc1a4b\") "
Nov 02 12:50:43.175014 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:43.174879    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fbce1ffb-dfe6-47b1-bd72-d86787cc1a4b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbce1ffb-dfe6-47b1-bd72-d86787cc1a4b-plugins-config-volume\") pod \"fbce1ffb-dfe6-47b1-bd72-d86787cc1a4b\" (UID: \"fbce1ffb-dfe6-47b1-bd72-d86787cc1a4b\") "
Nov 02 12:50:43.175090 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:43.175032    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7b7d167-6e3d-4632-822e-597f0578a3b0-kube-api-access-5sx5v\" (UniqueName: \"kubernetes.io/projected/a7b7d167-6e3d-4632-822e-597f0578a3b0-kube-api-access-5sx5v\") pod \"a7b7d167-6e3d-4632-822e-597f0578a3b0\" (UID: \"a7b7d167-6e3d-4632-822e-597f0578a3b0\") "
Nov 02 12:50:43.175149 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:43.175127    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7b7d167-6e3d-4632-822e-597f0578a3b0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7b7d167-6e3d-4632-822e-597f0578a3b0-plugins-config-volume\") pod \"a7b7d167-6e3d-4632-822e-597f0578a3b0\" (UID: \"a7b7d167-6e3d-4632-822e-597f0578a3b0\") "
Nov 02 12:50:43.175348 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:43.175319    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbce1ffb-dfe6-47b1-bd72-d86787cc1a4b-plugins-config-volume" pod "fbce1ffb-dfe6-47b1-bd72-d86787cc1a4b" (UID: "fbce1ffb-dfe6-47b1-bd72-d86787cc1a4b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:50:43.175778 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:43.175751    4559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbce1ffb-dfe6-47b1-bd72-d86787cc1a4b-plugins-config-volume\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:50:43.175887 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:43.175849    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7b7d167-6e3d-4632-822e-597f0578a3b0-plugins-config-volume" pod "a7b7d167-6e3d-4632-822e-597f0578a3b0" (UID: "a7b7d167-6e3d-4632-822e-597f0578a3b0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:50:43.177188 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:43.177169    4559 scope.go:122] "RemoveContainer" containerID="a9bdff8f7e4e340e7050c063e9eed857152b2114f6d51bdfaef0643fc624f3ca"
Nov 02 12:50:43.177583 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:50:43.177558    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9bdff8f7e4e340e7050c063e9eed857152b2114f6d51bdfaef0643fc624f3ca\": container with ID starting with a9bdff8f7e4e340e7050c063e9eed857152b2114f6d51bdfaef0643fc624f3ca not found: ID does not exist" containerID="a9bdff8f7e4e340e7050c063e9eed857152b2114f6d51bdfaef0643fc624f3ca"
Nov 02 12:50:43.177875 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:43.177840    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9bdff8f7e4e340e7050c063e9eed857152b2114f6d51bdfaef0643fc624f3ca"} err="failed to get container status \"a9bdff8f7e4e340e7050c063e9eed857152b2114f6d51bdfaef0643fc624f3ca\": rpc error: code = NotFound desc = could not find container \"a9bdff8f7e4e340e7050c063e9eed857152b2114f6d51bdfaef0643fc624f3ca\": container with ID starting with a9bdff8f7e4e340e7050c063e9eed857152b2114f6d51bdfaef0643fc624f3ca not found: ID does not exist"
Nov 02 12:50:43.177976 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:43.177878    4559 scope.go:122] "RemoveContainer" containerID="781a4b8347a3ba231d1cd5d969497d10c81ce681b569f179f1714c9225989539"
Nov 02 12:50:43.177976 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:43.177779    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7b7d167-6e3d-4632-822e-597f0578a3b0-kube-api-access-5sx5v" pod "a7b7d167-6e3d-4632-822e-597f0578a3b0" (UID: "a7b7d167-6e3d-4632-822e-597f0578a3b0"). InnerVolumeSpecName "kube-api-access-5sx5v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:50:43.178057 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:43.177989    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbce1ffb-dfe6-47b1-bd72-d86787cc1a4b-kube-api-access-gg5pg" pod "fbce1ffb-dfe6-47b1-bd72-d86787cc1a4b" (UID: "fbce1ffb-dfe6-47b1-bd72-d86787cc1a4b"). InnerVolumeSpecName "kube-api-access-gg5pg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:50:43.190392 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:43.190371    4559 scope.go:122] "RemoveContainer" containerID="781a4b8347a3ba231d1cd5d969497d10c81ce681b569f179f1714c9225989539"
Nov 02 12:50:43.190716 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:50:43.190690    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"781a4b8347a3ba231d1cd5d969497d10c81ce681b569f179f1714c9225989539\": container with ID starting with 781a4b8347a3ba231d1cd5d969497d10c81ce681b569f179f1714c9225989539 not found: ID does not exist" containerID="781a4b8347a3ba231d1cd5d969497d10c81ce681b569f179f1714c9225989539"
Nov 02 12:50:43.190768 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:43.190729    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"781a4b8347a3ba231d1cd5d969497d10c81ce681b569f179f1714c9225989539"} err="failed to get container status \"781a4b8347a3ba231d1cd5d969497d10c81ce681b569f179f1714c9225989539\": rpc error: code = NotFound desc = could not find container \"781a4b8347a3ba231d1cd5d969497d10c81ce681b569f179f1714c9225989539\": container with ID starting with 781a4b8347a3ba231d1cd5d969497d10c81ce681b569f179f1714c9225989539 not found: ID does not exist"
Nov 02 12:50:43.277673 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:43.277569    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gg5pg\" (UniqueName: \"kubernetes.io/projected/fbce1ffb-dfe6-47b1-bd72-d86787cc1a4b-kube-api-access-gg5pg\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:50:43.277673 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:43.277596    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5sx5v\" (UniqueName: \"kubernetes.io/projected/a7b7d167-6e3d-4632-822e-597f0578a3b0-kube-api-access-5sx5v\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:50:43.277673 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:43.277608    4559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7b7d167-6e3d-4632-822e-597f0578a3b0-plugins-config-volume\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:50:43.277910 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:43.277845    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-pz8j5"]
Nov 02 12:50:43.278472 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:43.278458    4559 cpu_manager.go:414] "containerMap: removing container" podUID="fbce1ffb-dfe6-47b1-bd72-d86787cc1a4b" containerName="payload-pre-processing"
Nov 02 12:50:43.278514 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:43.278473    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbce1ffb-dfe6-47b1-bd72-d86787cc1a4b" containerName="payload-pre-processing"
Nov 02 12:50:43.278514 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:43.278499    4559 cpu_manager.go:414] "containerMap: removing container" podUID="7a673851-931d-4634-8b10-d26a2518552d" containerName="maas-api"
Nov 02 12:50:43.278514 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:43.278506    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a673851-931d-4634-8b10-d26a2518552d" containerName="maas-api"
Nov 02 12:50:43.278606 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:43.278528    4559 cpu_manager.go:414] "containerMap: removing container" podUID="a7b7d167-6e3d-4632-822e-597f0578a3b0" containerName="payload-processing"
Nov 02 12:50:43.278606 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:43.278534    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7b7d167-6e3d-4632-822e-597f0578a3b0" containerName="payload-processing"
Nov 02 12:50:43.278705 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:43.278634    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7b7d167-6e3d-4632-822e-597f0578a3b0" containerName="payload-processing"
Nov 02 12:50:43.278705 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:43.278659    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbce1ffb-dfe6-47b1-bd72-d86787cc1a4b" containerName="payload-pre-processing"
Nov 02 12:50:43.278705 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:43.278668    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a673851-931d-4634-8b10-d26a2518552d" containerName="maas-api"
Nov 02 12:50:43.283629 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:43.283612    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-pz8j5"
Nov 02 12:50:43.313584 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:43.313547    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-pz8j5"]
Nov 02 12:50:43.378701 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:43.378663    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/632e48df-f4f7-4e68-819a-593ffd6a76d6-credential-socket\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-pz8j5\" (UID: \"632e48df-f4f7-4e68-819a-593ffd6a76d6\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-pz8j5"
Nov 02 12:50:43.378863 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:43.378790    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/632e48df-f4f7-4e68-819a-593ffd6a76d6-istio-token\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-pz8j5\" (UID: \"632e48df-f4f7-4e68-819a-593ffd6a76d6\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-pz8j5"
Nov 02 12:50:43.378863 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:43.378856    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/632e48df-f4f7-4e68-819a-593ffd6a76d6-workload-socket\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-pz8j5\" (UID: \"632e48df-f4f7-4e68-819a-593ffd6a76d6\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-pz8j5"
Nov 02 12:50:43.378948 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:43.378922    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/632e48df-f4f7-4e68-819a-593ffd6a76d6-istio-podinfo\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-pz8j5\" (UID: \"632e48df-f4f7-4e68-819a-593ffd6a76d6\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-pz8j5"
Nov 02 12:50:43.378989 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:43.378960    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4bv2\" (UniqueName: \"kubernetes.io/projected/632e48df-f4f7-4e68-819a-593ffd6a76d6-kube-api-access-g4bv2\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-pz8j5\" (UID: \"632e48df-f4f7-4e68-819a-593ffd6a76d6\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-pz8j5"
Nov 02 12:50:43.378989 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:43.378985    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/632e48df-f4f7-4e68-819a-593ffd6a76d6-istio-envoy\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-pz8j5\" (UID: \"632e48df-f4f7-4e68-819a-593ffd6a76d6\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-pz8j5"
Nov 02 12:50:43.379111 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:43.379093    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/632e48df-f4f7-4e68-819a-593ffd6a76d6-workload-certs\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-pz8j5\" (UID: \"632e48df-f4f7-4e68-819a-593ffd6a76d6\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-pz8j5"
Nov 02 12:50:43.379173 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:43.379158    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/632e48df-f4f7-4e68-819a-593ffd6a76d6-istio-data\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-pz8j5\" (UID: \"632e48df-f4f7-4e68-819a-593ffd6a76d6\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-pz8j5"
Nov 02 12:50:43.379265 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:43.379249    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/632e48df-f4f7-4e68-819a-593ffd6a76d6-istiod-ca-cert\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-pz8j5\" (UID: \"632e48df-f4f7-4e68-819a-593ffd6a76d6\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-pz8j5"
Nov 02 12:50:43.483708 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:43.482428    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/632e48df-f4f7-4e68-819a-593ffd6a76d6-credential-socket\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-pz8j5\" (UID: \"632e48df-f4f7-4e68-819a-593ffd6a76d6\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-pz8j5"
Nov 02 12:50:43.483708 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:43.482546    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/632e48df-f4f7-4e68-819a-593ffd6a76d6-istio-token\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-pz8j5\" (UID: \"632e48df-f4f7-4e68-819a-593ffd6a76d6\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-pz8j5"
Nov 02 12:50:43.483708 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:43.482606    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/632e48df-f4f7-4e68-819a-593ffd6a76d6-workload-socket\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-pz8j5\" (UID: \"632e48df-f4f7-4e68-819a-593ffd6a76d6\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-pz8j5"
Nov 02 12:50:43.483708 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:43.482659    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/632e48df-f4f7-4e68-819a-593ffd6a76d6-istio-podinfo\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-pz8j5\" (UID: \"632e48df-f4f7-4e68-819a-593ffd6a76d6\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-pz8j5"
Nov 02 12:50:43.483708 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:43.482692    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4bv2\" (UniqueName: \"kubernetes.io/projected/632e48df-f4f7-4e68-819a-593ffd6a76d6-kube-api-access-g4bv2\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-pz8j5\" (UID: \"632e48df-f4f7-4e68-819a-593ffd6a76d6\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-pz8j5"
Nov 02 12:50:43.483708 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:43.482729    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/632e48df-f4f7-4e68-819a-593ffd6a76d6-istio-envoy\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-pz8j5\" (UID: \"632e48df-f4f7-4e68-819a-593ffd6a76d6\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-pz8j5"
Nov 02 12:50:43.483708 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:43.482782    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/632e48df-f4f7-4e68-819a-593ffd6a76d6-workload-certs\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-pz8j5\" (UID: \"632e48df-f4f7-4e68-819a-593ffd6a76d6\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-pz8j5"
Nov 02 12:50:43.483708 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:43.482840    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/632e48df-f4f7-4e68-819a-593ffd6a76d6-istio-data\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-pz8j5\" (UID: \"632e48df-f4f7-4e68-819a-593ffd6a76d6\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-pz8j5"
Nov 02 12:50:43.483708 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:43.482946    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/632e48df-f4f7-4e68-819a-593ffd6a76d6-istiod-ca-cert\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-pz8j5\" (UID: \"632e48df-f4f7-4e68-819a-593ffd6a76d6\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-pz8j5"
Nov 02 12:50:43.501863 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:43.484754    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/632e48df-f4f7-4e68-819a-593ffd6a76d6-credential-socket\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-pz8j5\" (UID: \"632e48df-f4f7-4e68-819a-593ffd6a76d6\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-pz8j5"
Nov 02 12:50:43.501863 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:43.485036    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/632e48df-f4f7-4e68-819a-593ffd6a76d6-workload-socket\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-pz8j5\" (UID: \"632e48df-f4f7-4e68-819a-593ffd6a76d6\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-pz8j5"
Nov 02 12:50:43.501863 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:43.485268    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/632e48df-f4f7-4e68-819a-593ffd6a76d6-workload-certs\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-pz8j5\" (UID: \"632e48df-f4f7-4e68-819a-593ffd6a76d6\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-pz8j5"
Nov 02 12:50:43.501863 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:43.486756    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/632e48df-f4f7-4e68-819a-593ffd6a76d6-istio-data\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-pz8j5\" (UID: \"632e48df-f4f7-4e68-819a-593ffd6a76d6\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-pz8j5"
Nov 02 12:50:43.501863 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:43.487381    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/632e48df-f4f7-4e68-819a-593ffd6a76d6-istiod-ca-cert\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-pz8j5\" (UID: \"632e48df-f4f7-4e68-819a-593ffd6a76d6\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-pz8j5"
Nov 02 12:50:43.501863 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:43.495764    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/632e48df-f4f7-4e68-819a-593ffd6a76d6-istio-podinfo\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-pz8j5\" (UID: \"632e48df-f4f7-4e68-819a-593ffd6a76d6\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-pz8j5"
Nov 02 12:50:43.501863 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:43.497591    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/632e48df-f4f7-4e68-819a-593ffd6a76d6-istio-envoy\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-pz8j5\" (UID: \"632e48df-f4f7-4e68-819a-593ffd6a76d6\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-pz8j5"
Nov 02 12:50:43.521130 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:43.521095    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/632e48df-f4f7-4e68-819a-593ffd6a76d6-istio-token\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-pz8j5\" (UID: \"632e48df-f4f7-4e68-819a-593ffd6a76d6\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-pz8j5"
Nov 02 12:50:43.521130 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:43.521111    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4bv2\" (UniqueName: \"kubernetes.io/projected/632e48df-f4f7-4e68-819a-593ffd6a76d6-kube-api-access-g4bv2\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-pz8j5\" (UID: \"632e48df-f4f7-4e68-819a-593ffd6a76d6\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-pz8j5"
Nov 02 12:50:43.550139 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:43.549898    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-e93a4b-bd984bc95-7fwhg"]
Nov 02 12:50:43.553375 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:43.553348    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-e93a4b-bd984bc95-7fwhg"]
Nov 02 12:50:43.588890 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:43.588853    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-e93a4b-6947c6444f-8cc8j"]
Nov 02 12:50:43.592980 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:43.592953    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-e93a4b-6947c6444f-8cc8j"]
Nov 02 12:50:43.600503 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:43.600480    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-pz8j5"
Nov 02 12:50:43.845662 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:43.843816    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-eb24df-864494fcf-kzf6t"
Nov 02 12:50:43.945918 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:43.945886    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7b7d167-6e3d-4632-822e-597f0578a3b0" path="/var/lib/kubelet/pods/a7b7d167-6e3d-4632-822e-597f0578a3b0/volumes"
Nov 02 12:50:43.946531 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:43.946509    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fbce1ffb-dfe6-47b1-bd72-d86787cc1a4b" path="/var/lib/kubelet/pods/fbce1ffb-dfe6-47b1-bd72-d86787cc1a4b/volumes"
Nov 02 12:50:43.962004 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:43.961966    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:50:43.962127 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:43.962053    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:50:43.962127 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:43.962096    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:50:43.965571 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:43.965551    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-pz8j5"]
Nov 02 12:50:44.175612 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:44.175565    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-pz8j5" event={"ID":"632e48df-f4f7-4e68-819a-593ffd6a76d6","Type":"ContainerStarted","Data":"ab4f6d47502dee62a1960429358f4fcf9877e1b30bb1ff781634de50a0b7adb0"}
Nov 02 12:50:44.175812 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:44.175628    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-pz8j5" event={"ID":"632e48df-f4f7-4e68-819a-593ffd6a76d6","Type":"ContainerStarted","Data":"50806eff80fe44f902e58ba4a621039514fbb842e9bda2164d1a20dc644fc5b3"}
Nov 02 12:50:44.224760 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:44.224704    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-pz8j5" podStartSLOduration=1.224689039 podStartE2EDuration="1.224689039s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:50:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:50:44.221704851 +0000 UTC m=+1440.477103947" watchObservedRunningTime="2026-11-02 12:50:44.224689039 +0000 UTC m=+1440.480088208"
Nov 02 12:50:44.606663 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:44.605732    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-pz8j5"
Nov 02 12:50:45.111805 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:50:45.111695    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqcmj" containerName="extractor"
Nov 02 12:50:45.209338 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:45.209293    4559 generic.go:356] "Generic (PLEG): container finished" podID="7666985a-ed93-432a-9bd8-c668a7bfd8f5" containerID="e1d6e40836826a764fb8d275885e3e19d0e23253fe2869d187becd49b38acd28" exitCode=0
Nov 02 12:50:45.209536 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:45.209340    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-s8hnh" event={"ID":"7666985a-ed93-432a-9bd8-c668a7bfd8f5","Type":"ContainerDied","Data":"e1d6e40836826a764fb8d275885e3e19d0e23253fe2869d187becd49b38acd28"}
Nov 02 12:50:45.209859 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:45.209841    4559 scope.go:122] "RemoveContainer" containerID="e1d6e40836826a764fb8d275885e3e19d0e23253fe2869d187becd49b38acd28"
Nov 02 12:50:45.211538 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:45.211507    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:50:45.211616 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:45.211578    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:50:45.211616 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:45.211612    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:50:45.609262 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:45.608925    4559 patch_prober.go:29] interesting pod/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-pz8j5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body=
Nov 02 12:50:45.609262 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:45.608986    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-pz8j5" podUID="632e48df-f4f7-4e68-819a-593ffd6a76d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused"
Nov 02 12:50:46.006663 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:46.004561    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-s8hnh"
Nov 02 12:50:46.265730 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:46.263126    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-s8hnh" event={"ID":"7666985a-ed93-432a-9bd8-c668a7bfd8f5","Type":"ContainerStarted","Data":"8ae9ec8c8d6a925274594b4addbedf5a237c297e29b3430b4c5992eaefe64e98"}
Nov 02 12:50:46.515449 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:46.511136    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-e93a4b-openshift-default-7f8c56bf9c-4pjxc"]
Nov 02 12:50:46.515449 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:46.511439    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-e93a4b-openshift-default-7f8c56bf9c-4pjxc" podUID="624a0494-692e-4864-810e-42ba5306a5f0" containerName="istio-proxy" containerID="cri-o://3d0e1a2a8726fba0000726f99720499ac2487a10d6b8d0553405a1f0ebe072ef" gracePeriod=30
Nov 02 12:50:46.661678 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:46.659031    4559 patch_prober.go:29] interesting pod/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-pz8j5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body=
Nov 02 12:50:46.661678 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:46.659099    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-pz8j5" podUID="632e48df-f4f7-4e68-819a-593ffd6a76d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused"
Nov 02 12:50:46.727488 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:46.727335    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-s8hnh"
Nov 02 12:50:46.727488 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:46.727395    4559 patch_prober.go:29] interesting pod/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-s8hnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body=
Nov 02 12:50:46.727488 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:46.727444    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-s8hnh" podUID="7666985a-ed93-432a-9bd8-c668a7bfd8f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused"
Nov 02 12:50:47.602117 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:47.602083    4559 patch_prober.go:29] interesting pod/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-pz8j5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body=
Nov 02 12:50:47.602764 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:47.602717    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-pz8j5" podUID="632e48df-f4f7-4e68-819a-593ffd6a76d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused"
Nov 02 12:50:47.726662 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:47.725941    4559 patch_prober.go:29] interesting pod/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-s8hnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body=
Nov 02 12:50:47.726662 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:47.726001    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-s8hnh" podUID="7666985a-ed93-432a-9bd8-c668a7bfd8f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused"
Nov 02 12:50:48.602032 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:48.601996    4559 patch_prober.go:29] interesting pod/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-pz8j5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body=
Nov 02 12:50:48.602367 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:48.602317    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-pz8j5" podUID="632e48df-f4f7-4e68-819a-593ffd6a76d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused"
Nov 02 12:50:48.739322 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:48.738755    4559 patch_prober.go:29] interesting pod/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-s8hnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body=
Nov 02 12:50:48.739322 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:48.738816    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-s8hnh" podUID="7666985a-ed93-432a-9bd8-c668a7bfd8f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused"
Nov 02 12:50:49.604840 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:49.604805    4559 patch_prober.go:29] interesting pod/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-pz8j5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body=
Nov 02 12:50:49.605453 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:49.605421    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-pz8j5" podUID="632e48df-f4f7-4e68-819a-593ffd6a76d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused"
Nov 02 12:50:49.674742 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:49.674709    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-d7801d-openshift-default-674cd67b5-xgb9d"]
Nov 02 12:50:49.682044 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:49.682021    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-d7801d-openshift-default-674cd67b5-xgb9d"
Nov 02 12:50:49.762657 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:49.756436    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-d7801d-openshift-default-dockercfg-mmw8m\""
Nov 02 12:50:49.763876 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:49.763851    4559 patch_prober.go:29] interesting pod/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-s8hnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body=
Nov 02 12:50:49.764063 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:49.764040    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-s8hnh" podUID="7666985a-ed93-432a-9bd8-c668a7bfd8f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused"
Nov 02 12:50:49.764452 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:49.764421    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03b6b83c-10cd-4876-b17a-96120c2317b7-workload-socket\") pod \"e2e-ipp-b-d7801d-openshift-default-674cd67b5-xgb9d\" (UID: \"03b6b83c-10cd-4876-b17a-96120c2317b7\") " pod="openshift-ingress/e2e-ipp-b-d7801d-openshift-default-674cd67b5-xgb9d"
Nov 02 12:50:49.764539 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:49.764497    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03b6b83c-10cd-4876-b17a-96120c2317b7-credential-socket\") pod \"e2e-ipp-b-d7801d-openshift-default-674cd67b5-xgb9d\" (UID: \"03b6b83c-10cd-4876-b17a-96120c2317b7\") " pod="openshift-ingress/e2e-ipp-b-d7801d-openshift-default-674cd67b5-xgb9d"
Nov 02 12:50:49.764539 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:49.764525    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03b6b83c-10cd-4876-b17a-96120c2317b7-istiod-ca-cert\") pod \"e2e-ipp-b-d7801d-openshift-default-674cd67b5-xgb9d\" (UID: \"03b6b83c-10cd-4876-b17a-96120c2317b7\") " pod="openshift-ingress/e2e-ipp-b-d7801d-openshift-default-674cd67b5-xgb9d"
Nov 02 12:50:49.764660 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:49.764628    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03b6b83c-10cd-4876-b17a-96120c2317b7-workload-certs\") pod \"e2e-ipp-b-d7801d-openshift-default-674cd67b5-xgb9d\" (UID: \"03b6b83c-10cd-4876-b17a-96120c2317b7\") " pod="openshift-ingress/e2e-ipp-b-d7801d-openshift-default-674cd67b5-xgb9d"
Nov 02 12:50:49.764721 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:49.764695    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03b6b83c-10cd-4876-b17a-96120c2317b7-istio-podinfo\") pod \"e2e-ipp-b-d7801d-openshift-default-674cd67b5-xgb9d\" (UID: \"03b6b83c-10cd-4876-b17a-96120c2317b7\") " pod="openshift-ingress/e2e-ipp-b-d7801d-openshift-default-674cd67b5-xgb9d"
Nov 02 12:50:49.764761 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:49.764750    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03b6b83c-10cd-4876-b17a-96120c2317b7-istio-data\") pod \"e2e-ipp-b-d7801d-openshift-default-674cd67b5-xgb9d\" (UID: \"03b6b83c-10cd-4876-b17a-96120c2317b7\") " pod="openshift-ingress/e2e-ipp-b-d7801d-openshift-default-674cd67b5-xgb9d"
Nov 02 12:50:49.764852 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:49.764837    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03b6b83c-10cd-4876-b17a-96120c2317b7-istio-token\") pod \"e2e-ipp-b-d7801d-openshift-default-674cd67b5-xgb9d\" (UID: \"03b6b83c-10cd-4876-b17a-96120c2317b7\") " pod="openshift-ingress/e2e-ipp-b-d7801d-openshift-default-674cd67b5-xgb9d"
Nov 02 12:50:49.764902 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:49.764870    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjs8r\" (UniqueName: \"kubernetes.io/projected/03b6b83c-10cd-4876-b17a-96120c2317b7-kube-api-access-rjs8r\") pod \"e2e-ipp-b-d7801d-openshift-default-674cd67b5-xgb9d\" (UID: \"03b6b83c-10cd-4876-b17a-96120c2317b7\") " pod="openshift-ingress/e2e-ipp-b-d7801d-openshift-default-674cd67b5-xgb9d"
Nov 02 12:50:49.764949 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:49.764898    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03b6b83c-10cd-4876-b17a-96120c2317b7-istio-envoy\") pod \"e2e-ipp-b-d7801d-openshift-default-674cd67b5-xgb9d\" (UID: \"03b6b83c-10cd-4876-b17a-96120c2317b7\") " pod="openshift-ingress/e2e-ipp-b-d7801d-openshift-default-674cd67b5-xgb9d"
Nov 02 12:50:49.766837 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:49.766817    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-d7801d-openshift-default-674cd67b5-xgb9d"]
Nov 02 12:50:50.067668 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:50.066602    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03b6b83c-10cd-4876-b17a-96120c2317b7-istio-token\") pod \"e2e-ipp-b-d7801d-openshift-default-674cd67b5-xgb9d\" (UID: \"03b6b83c-10cd-4876-b17a-96120c2317b7\") " pod="openshift-ingress/e2e-ipp-b-d7801d-openshift-default-674cd67b5-xgb9d"
Nov 02 12:50:50.067668 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:50.066680    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjs8r\" (UniqueName: \"kubernetes.io/projected/03b6b83c-10cd-4876-b17a-96120c2317b7-kube-api-access-rjs8r\") pod \"e2e-ipp-b-d7801d-openshift-default-674cd67b5-xgb9d\" (UID: \"03b6b83c-10cd-4876-b17a-96120c2317b7\") " pod="openshift-ingress/e2e-ipp-b-d7801d-openshift-default-674cd67b5-xgb9d"
Nov 02 12:50:50.067668 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:50.066809    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03b6b83c-10cd-4876-b17a-96120c2317b7-istio-envoy\") pod \"e2e-ipp-b-d7801d-openshift-default-674cd67b5-xgb9d\" (UID: \"03b6b83c-10cd-4876-b17a-96120c2317b7\") " pod="openshift-ingress/e2e-ipp-b-d7801d-openshift-default-674cd67b5-xgb9d"
Nov 02 12:50:50.067668 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:50.067123    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03b6b83c-10cd-4876-b17a-96120c2317b7-workload-socket\") pod \"e2e-ipp-b-d7801d-openshift-default-674cd67b5-xgb9d\" (UID: \"03b6b83c-10cd-4876-b17a-96120c2317b7\") " pod="openshift-ingress/e2e-ipp-b-d7801d-openshift-default-674cd67b5-xgb9d"
Nov 02 12:50:50.067668 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:50.067301    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03b6b83c-10cd-4876-b17a-96120c2317b7-credential-socket\") pod \"e2e-ipp-b-d7801d-openshift-default-674cd67b5-xgb9d\" (UID: \"03b6b83c-10cd-4876-b17a-96120c2317b7\") " pod="openshift-ingress/e2e-ipp-b-d7801d-openshift-default-674cd67b5-xgb9d"
Nov 02 12:50:50.067668 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:50.067430    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03b6b83c-10cd-4876-b17a-96120c2317b7-istiod-ca-cert\") pod \"e2e-ipp-b-d7801d-openshift-default-674cd67b5-xgb9d\" (UID: \"03b6b83c-10cd-4876-b17a-96120c2317b7\") " pod="openshift-ingress/e2e-ipp-b-d7801d-openshift-default-674cd67b5-xgb9d"
Nov 02 12:50:50.082187 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:50.068105    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03b6b83c-10cd-4876-b17a-96120c2317b7-workload-socket\") pod \"e2e-ipp-b-d7801d-openshift-default-674cd67b5-xgb9d\" (UID: \"03b6b83c-10cd-4876-b17a-96120c2317b7\") " pod="openshift-ingress/e2e-ipp-b-d7801d-openshift-default-674cd67b5-xgb9d"
Nov 02 12:50:50.082187 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:50.072157    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03b6b83c-10cd-4876-b17a-96120c2317b7-credential-socket\") pod \"e2e-ipp-b-d7801d-openshift-default-674cd67b5-xgb9d\" (UID: \"03b6b83c-10cd-4876-b17a-96120c2317b7\") " pod="openshift-ingress/e2e-ipp-b-d7801d-openshift-default-674cd67b5-xgb9d"
Nov 02 12:50:50.082187 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:50.073196    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03b6b83c-10cd-4876-b17a-96120c2317b7-istiod-ca-cert\") pod \"e2e-ipp-b-d7801d-openshift-default-674cd67b5-xgb9d\" (UID: \"03b6b83c-10cd-4876-b17a-96120c2317b7\") " pod="openshift-ingress/e2e-ipp-b-d7801d-openshift-default-674cd67b5-xgb9d"
Nov 02 12:50:50.082187 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:50.077027    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03b6b83c-10cd-4876-b17a-96120c2317b7-workload-certs\") pod \"e2e-ipp-b-d7801d-openshift-default-674cd67b5-xgb9d\" (UID: \"03b6b83c-10cd-4876-b17a-96120c2317b7\") " pod="openshift-ingress/e2e-ipp-b-d7801d-openshift-default-674cd67b5-xgb9d"
Nov 02 12:50:50.082187 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:50.077107    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03b6b83c-10cd-4876-b17a-96120c2317b7-istio-podinfo\") pod \"e2e-ipp-b-d7801d-openshift-default-674cd67b5-xgb9d\" (UID: \"03b6b83c-10cd-4876-b17a-96120c2317b7\") " pod="openshift-ingress/e2e-ipp-b-d7801d-openshift-default-674cd67b5-xgb9d"
Nov 02 12:50:50.082187 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:50.077189    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03b6b83c-10cd-4876-b17a-96120c2317b7-istio-data\") pod \"e2e-ipp-b-d7801d-openshift-default-674cd67b5-xgb9d\" (UID: \"03b6b83c-10cd-4876-b17a-96120c2317b7\") " pod="openshift-ingress/e2e-ipp-b-d7801d-openshift-default-674cd67b5-xgb9d"
Nov 02 12:50:50.127765 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:50.127726    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03b6b83c-10cd-4876-b17a-96120c2317b7-workload-certs\") pod \"e2e-ipp-b-d7801d-openshift-default-674cd67b5-xgb9d\" (UID: \"03b6b83c-10cd-4876-b17a-96120c2317b7\") " pod="openshift-ingress/e2e-ipp-b-d7801d-openshift-default-674cd67b5-xgb9d"
Nov 02 12:50:50.152036 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:50.152007    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03b6b83c-10cd-4876-b17a-96120c2317b7-istio-data\") pod \"e2e-ipp-b-d7801d-openshift-default-674cd67b5-xgb9d\" (UID: \"03b6b83c-10cd-4876-b17a-96120c2317b7\") " pod="openshift-ingress/e2e-ipp-b-d7801d-openshift-default-674cd67b5-xgb9d"
Nov 02 12:50:50.247082 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:50.245090    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03b6b83c-10cd-4876-b17a-96120c2317b7-istio-envoy\") pod \"e2e-ipp-b-d7801d-openshift-default-674cd67b5-xgb9d\" (UID: \"03b6b83c-10cd-4876-b17a-96120c2317b7\") " pod="openshift-ingress/e2e-ipp-b-d7801d-openshift-default-674cd67b5-xgb9d"
Nov 02 12:50:50.266134 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:50.265535    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjs8r\" (UniqueName: \"kubernetes.io/projected/03b6b83c-10cd-4876-b17a-96120c2317b7-kube-api-access-rjs8r\") pod \"e2e-ipp-b-d7801d-openshift-default-674cd67b5-xgb9d\" (UID: \"03b6b83c-10cd-4876-b17a-96120c2317b7\") " pod="openshift-ingress/e2e-ipp-b-d7801d-openshift-default-674cd67b5-xgb9d"
Nov 02 12:50:50.273935 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:50.273862    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03b6b83c-10cd-4876-b17a-96120c2317b7-istio-podinfo\") pod \"e2e-ipp-b-d7801d-openshift-default-674cd67b5-xgb9d\" (UID: \"03b6b83c-10cd-4876-b17a-96120c2317b7\") " pod="openshift-ingress/e2e-ipp-b-d7801d-openshift-default-674cd67b5-xgb9d"
Nov 02 12:50:50.287025 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:50.286993    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03b6b83c-10cd-4876-b17a-96120c2317b7-istio-token\") pod \"e2e-ipp-b-d7801d-openshift-default-674cd67b5-xgb9d\" (UID: \"03b6b83c-10cd-4876-b17a-96120c2317b7\") " pod="openshift-ingress/e2e-ipp-b-d7801d-openshift-default-674cd67b5-xgb9d"
Nov 02 12:50:50.439065 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:50.439021    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-d7801d-openshift-default-674cd67b5-xgb9d"
Nov 02 12:50:50.614769 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:50.614730    4559 patch_prober.go:29] interesting pod/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-pz8j5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body=
Nov 02 12:50:50.615163 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:50.614804    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-pz8j5" podUID="632e48df-f4f7-4e68-819a-593ffd6a76d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused"
Nov 02 12:50:50.724611 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:50.724528    4559 patch_prober.go:29] interesting pod/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-s8hnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body=
Nov 02 12:50:50.724611 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:50.724583    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-s8hnh" podUID="7666985a-ed93-432a-9bd8-c668a7bfd8f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused"
Nov 02 12:50:50.825609 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:50.825571    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-d7801d-openshift-default-674cd67b5-xgb9d"]
Nov 02 12:50:50.832659 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:50:50.828599    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03b6b83c_10cd_4876_b17a_96120c2317b7.slice/crio-2c512caa65bb6bb6e2021a95f4eaa40d0128dd350385ffa63dea79221720c621 WatchSource:0}: Error finding container 2c512caa65bb6bb6e2021a95f4eaa40d0128dd350385ffa63dea79221720c621: Status 404 returned error can't find the container with id 2c512caa65bb6bb6e2021a95f4eaa40d0128dd350385ffa63dea79221720c621
Nov 02 12:50:50.832659 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:50.832414    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:50:50.832659 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:50.832518    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:50:50.832659 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:50.832585    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:50:51.353005 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:51.352975    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d7801d-openshift-default-674cd67b5-xgb9d" event={"ID":"03b6b83c-10cd-4876-b17a-96120c2317b7","Type":"ContainerStarted","Data":"6224548b7e582bcc650188ee8bfdd325f69efc976323a76a62849fbafd0ecc29"}
Nov 02 12:50:51.353172 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:51.353036    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d7801d-openshift-default-674cd67b5-xgb9d" event={"ID":"03b6b83c-10cd-4876-b17a-96120c2317b7","Type":"ContainerStarted","Data":"2c512caa65bb6bb6e2021a95f4eaa40d0128dd350385ffa63dea79221720c621"}
Nov 02 12:50:51.440221 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:51.440186    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-d7801d-openshift-default-674cd67b5-xgb9d"
Nov 02 12:50:51.446310 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:51.446283    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-d7801d-openshift-default-674cd67b5-xgb9d"
Nov 02 12:50:51.505663 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:51.502326    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-d7801d-openshift-default-674cd67b5-xgb9d" podStartSLOduration=2.502305302 podStartE2EDuration="2.502305302s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:50:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:50:51.401973769 +0000 UTC m=+1447.657372870" watchObservedRunningTime="2026-11-02 12:50:51.502305302 +0000 UTC m=+1447.757704398"
Nov 02 12:50:51.628597 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:51.627092    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-pz8j5"
Nov 02 12:50:51.628597 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:51.628519    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-pz8j5"
Nov 02 12:50:51.631359 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:51.631309    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-pz8j5"
Nov 02 12:50:51.725981 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:51.725663    4559 patch_prober.go:29] interesting pod/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-s8hnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body=
Nov 02 12:50:51.726771 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:51.725739    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-s8hnh" podUID="7666985a-ed93-432a-9bd8-c668a7bfd8f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused"
Nov 02 12:50:52.012754 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:52.012725    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-e93a4b-openshift-default-7f8c56bf9c-4pjxc"
Nov 02 12:50:52.121367 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:52.121279    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/624a0494-692e-4864-810e-42ba5306a5f0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/624a0494-692e-4864-810e-42ba5306a5f0-istio-data\") pod \"624a0494-692e-4864-810e-42ba5306a5f0\" (UID: \"624a0494-692e-4864-810e-42ba5306a5f0\") "
Nov 02 12:50:52.121367 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:52.121338    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/624a0494-692e-4864-810e-42ba5306a5f0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/624a0494-692e-4864-810e-42ba5306a5f0-credential-socket\") pod \"624a0494-692e-4864-810e-42ba5306a5f0\" (UID: \"624a0494-692e-4864-810e-42ba5306a5f0\") "
Nov 02 12:50:52.121681 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:52.121504    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/624a0494-692e-4864-810e-42ba5306a5f0-kube-api-access-8jgtn\" (UniqueName: \"kubernetes.io/projected/624a0494-692e-4864-810e-42ba5306a5f0-kube-api-access-8jgtn\") pod \"624a0494-692e-4864-810e-42ba5306a5f0\" (UID: \"624a0494-692e-4864-810e-42ba5306a5f0\") "
Nov 02 12:50:52.121681 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:52.121550    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/624a0494-692e-4864-810e-42ba5306a5f0-istio-token\" (UniqueName: \"kubernetes.io/projected/624a0494-692e-4864-810e-42ba5306a5f0-istio-token\") pod \"624a0494-692e-4864-810e-42ba5306a5f0\" (UID: \"624a0494-692e-4864-810e-42ba5306a5f0\") "
Nov 02 12:50:52.121681 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:52.121579    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/624a0494-692e-4864-810e-42ba5306a5f0-istio-data" pod "624a0494-692e-4864-810e-42ba5306a5f0" (UID: "624a0494-692e-4864-810e-42ba5306a5f0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:50:52.121681 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:52.121675    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/624a0494-692e-4864-810e-42ba5306a5f0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/624a0494-692e-4864-810e-42ba5306a5f0-workload-socket\") pod \"624a0494-692e-4864-810e-42ba5306a5f0\" (UID: \"624a0494-692e-4864-810e-42ba5306a5f0\") "
Nov 02 12:50:52.121919 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:52.121733    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/624a0494-692e-4864-810e-42ba5306a5f0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/624a0494-692e-4864-810e-42ba5306a5f0-istio-podinfo\") pod \"624a0494-692e-4864-810e-42ba5306a5f0\" (UID: \"624a0494-692e-4864-810e-42ba5306a5f0\") "
Nov 02 12:50:52.121919 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:52.121807    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/624a0494-692e-4864-810e-42ba5306a5f0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/624a0494-692e-4864-810e-42ba5306a5f0-workload-certs\") pod \"624a0494-692e-4864-810e-42ba5306a5f0\" (UID: \"624a0494-692e-4864-810e-42ba5306a5f0\") "
Nov 02 12:50:52.121919 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:52.121855    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/624a0494-692e-4864-810e-42ba5306a5f0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/624a0494-692e-4864-810e-42ba5306a5f0-istio-envoy\") pod \"624a0494-692e-4864-810e-42ba5306a5f0\" (UID: \"624a0494-692e-4864-810e-42ba5306a5f0\") "
Nov 02 12:50:52.121919 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:52.121895    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/624a0494-692e-4864-810e-42ba5306a5f0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/624a0494-692e-4864-810e-42ba5306a5f0-istiod-ca-cert\") pod \"624a0494-692e-4864-810e-42ba5306a5f0\" (UID: \"624a0494-692e-4864-810e-42ba5306a5f0\") "
Nov 02 12:50:52.122255 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:52.122208    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/624a0494-692e-4864-810e-42ba5306a5f0-workload-certs" pod "624a0494-692e-4864-810e-42ba5306a5f0" (UID: "624a0494-692e-4864-810e-42ba5306a5f0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:50:52.122255 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:52.122231    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/624a0494-692e-4864-810e-42ba5306a5f0-workload-socket" pod "624a0494-692e-4864-810e-42ba5306a5f0" (UID: "624a0494-692e-4864-810e-42ba5306a5f0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:50:52.122555 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:52.122530    4559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/624a0494-692e-4864-810e-42ba5306a5f0-istio-data\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:50:52.122671 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:52.122562    4559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/624a0494-692e-4864-810e-42ba5306a5f0-workload-socket\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:50:52.122671 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:52.122538    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/624a0494-692e-4864-810e-42ba5306a5f0-credential-socket" pod "624a0494-692e-4864-810e-42ba5306a5f0" (UID: "624a0494-692e-4864-810e-42ba5306a5f0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:50:52.122671 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:52.122577    4559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/624a0494-692e-4864-810e-42ba5306a5f0-workload-certs\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:50:52.122671 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:52.122546    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/624a0494-692e-4864-810e-42ba5306a5f0-istiod-ca-cert" pod "624a0494-692e-4864-810e-42ba5306a5f0" (UID: "624a0494-692e-4864-810e-42ba5306a5f0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:50:52.124671 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:52.124609    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/624a0494-692e-4864-810e-42ba5306a5f0-istio-podinfo" pod "624a0494-692e-4864-810e-42ba5306a5f0" (UID: "624a0494-692e-4864-810e-42ba5306a5f0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 12:50:52.124778 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:52.124758    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/624a0494-692e-4864-810e-42ba5306a5f0-kube-api-access-8jgtn" pod "624a0494-692e-4864-810e-42ba5306a5f0" (UID: "624a0494-692e-4864-810e-42ba5306a5f0"). InnerVolumeSpecName "kube-api-access-8jgtn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:50:52.124899 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:52.124860    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/624a0494-692e-4864-810e-42ba5306a5f0-istio-envoy" pod "624a0494-692e-4864-810e-42ba5306a5f0" (UID: "624a0494-692e-4864-810e-42ba5306a5f0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:50:52.124999 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:52.124966    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/624a0494-692e-4864-810e-42ba5306a5f0-istio-token" pod "624a0494-692e-4864-810e-42ba5306a5f0" (UID: "624a0494-692e-4864-810e-42ba5306a5f0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:50:52.223802 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:52.223768    4559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/624a0494-692e-4864-810e-42ba5306a5f0-credential-socket\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:50:52.223802 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:52.223796    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8jgtn\" (UniqueName: \"kubernetes.io/projected/624a0494-692e-4864-810e-42ba5306a5f0-kube-api-access-8jgtn\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:50:52.223802 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:52.223807    4559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/624a0494-692e-4864-810e-42ba5306a5f0-istio-token\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:50:52.224048 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:52.223816    4559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/624a0494-692e-4864-810e-42ba5306a5f0-istio-podinfo\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:50:52.224048 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:52.223826    4559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/624a0494-692e-4864-810e-42ba5306a5f0-istio-envoy\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:50:52.224048 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:52.223834    4559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/624a0494-692e-4864-810e-42ba5306a5f0-istiod-ca-cert\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:50:52.361341 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:52.361311    4559 generic.go:356] "Generic (PLEG): container finished" podID="624a0494-692e-4864-810e-42ba5306a5f0" containerID="3d0e1a2a8726fba0000726f99720499ac2487a10d6b8d0553405a1f0ebe072ef" exitCode=0
Nov 02 12:50:52.361525 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:52.361359    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-e93a4b-openshift-default-7f8c56bf9c-4pjxc"
Nov 02 12:50:52.361525 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:52.361394    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-e93a4b-openshift-default-7f8c56bf9c-4pjxc" event={"ID":"624a0494-692e-4864-810e-42ba5306a5f0","Type":"ContainerDied","Data":"3d0e1a2a8726fba0000726f99720499ac2487a10d6b8d0553405a1f0ebe072ef"}
Nov 02 12:50:52.361525 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:52.361453    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-e93a4b-openshift-default-7f8c56bf9c-4pjxc" event={"ID":"624a0494-692e-4864-810e-42ba5306a5f0","Type":"ContainerDied","Data":"d5b53ecdf084c62a078df6fffe2b26241e0519c7bf1d79f91df2b2f423722f99"}
Nov 02 12:50:52.361525 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:52.361484    4559 scope.go:122] "RemoveContainer" containerID="3d0e1a2a8726fba0000726f99720499ac2487a10d6b8d0553405a1f0ebe072ef"
Nov 02 12:50:52.361915 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:52.361895    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-d7801d-openshift-default-674cd67b5-xgb9d"
Nov 02 12:50:52.363458 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:52.363440    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-d7801d-openshift-default-674cd67b5-xgb9d"
Nov 02 12:50:52.372425 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:52.372410    4559 scope.go:122] "RemoveContainer" containerID="55f6f8b469fdf1e3896bdc3ce5359712e1a572fe5b7d996b9744071c52c96126"
Nov 02 12:50:52.381940 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:52.381921    4559 scope.go:122] "RemoveContainer" containerID="3d0e1a2a8726fba0000726f99720499ac2487a10d6b8d0553405a1f0ebe072ef"
Nov 02 12:50:52.382194 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:50:52.382173    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d0e1a2a8726fba0000726f99720499ac2487a10d6b8d0553405a1f0ebe072ef\": container with ID starting with 3d0e1a2a8726fba0000726f99720499ac2487a10d6b8d0553405a1f0ebe072ef not found: ID does not exist" containerID="3d0e1a2a8726fba0000726f99720499ac2487a10d6b8d0553405a1f0ebe072ef"
Nov 02 12:50:52.382246 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:52.382200    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d0e1a2a8726fba0000726f99720499ac2487a10d6b8d0553405a1f0ebe072ef"} err="failed to get container status \"3d0e1a2a8726fba0000726f99720499ac2487a10d6b8d0553405a1f0ebe072ef\": rpc error: code = NotFound desc = could not find container \"3d0e1a2a8726fba0000726f99720499ac2487a10d6b8d0553405a1f0ebe072ef\": container with ID starting with 3d0e1a2a8726fba0000726f99720499ac2487a10d6b8d0553405a1f0ebe072ef not found: ID does not exist"
Nov 02 12:50:52.382246 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:52.382218    4559 scope.go:122] "RemoveContainer" containerID="55f6f8b469fdf1e3896bdc3ce5359712e1a572fe5b7d996b9744071c52c96126"
Nov 02 12:50:52.382457 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:50:52.382438    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55f6f8b469fdf1e3896bdc3ce5359712e1a572fe5b7d996b9744071c52c96126\": container with ID starting with 55f6f8b469fdf1e3896bdc3ce5359712e1a572fe5b7d996b9744071c52c96126 not found: ID does not exist" containerID="55f6f8b469fdf1e3896bdc3ce5359712e1a572fe5b7d996b9744071c52c96126"
Nov 02 12:50:52.382495 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:52.382465    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55f6f8b469fdf1e3896bdc3ce5359712e1a572fe5b7d996b9744071c52c96126"} err="failed to get container status \"55f6f8b469fdf1e3896bdc3ce5359712e1a572fe5b7d996b9744071c52c96126\": rpc error: code = NotFound desc = could not find container \"55f6f8b469fdf1e3896bdc3ce5359712e1a572fe5b7d996b9744071c52c96126\": container with ID starting with 55f6f8b469fdf1e3896bdc3ce5359712e1a572fe5b7d996b9744071c52c96126 not found: ID does not exist"
Nov 02 12:50:52.471621 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:52.471568    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-e93a4b-openshift-default-7f8c56bf9c-4pjxc"]
Nov 02 12:50:52.479836 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:52.479807    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-e93a4b-openshift-default-7f8c56bf9c-4pjxc"]
Nov 02 12:50:52.595211 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:52.595169    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-b271f0-6966d4c887-jthwp"]
Nov 02 12:50:52.630956 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:52.630883    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-b271f0-547b5469dc-42kll"]
Nov 02 12:50:52.746695 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:52.746608    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-s8hnh"
Nov 02 12:50:52.748269 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:52.748249    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-s8hnh"
Nov 02 12:50:52.749786 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:52.749761    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-s8hnh"
Nov 02 12:50:53.046773 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:53.046739    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-b271f0-578d584488-vwllx"]
Nov 02 12:50:53.047122 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:53.047086    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b271f0-578d584488-vwllx" podUID="0a0ab432-145b-45a0-8629-fa4dcb129694" containerName="maas-api" containerID="cri-o://f02fa12fe8defe52eda8006fefea34ffdb32973dc63f7f6bd287cc967efa68de" gracePeriod=30
Nov 02 12:50:53.426930 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:53.426592    4559 generic.go:356] "Generic (PLEG): container finished" podID="0a0ab432-145b-45a0-8629-fa4dcb129694" containerID="f02fa12fe8defe52eda8006fefea34ffdb32973dc63f7f6bd287cc967efa68de" exitCode=0
Nov 02 12:50:53.426930 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:53.426847    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b271f0-578d584488-vwllx" event={"ID":"0a0ab432-145b-45a0-8629-fa4dcb129694","Type":"ContainerDied","Data":"f02fa12fe8defe52eda8006fefea34ffdb32973dc63f7f6bd287cc967efa68de"}
Nov 02 12:50:53.486275 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:53.486247    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b271f0-578d584488-vwllx"
Nov 02 12:50:53.613673 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:53.568579    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0a0ab432-145b-45a0-8629-fa4dcb129694-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a0ab432-145b-45a0-8629-fa4dcb129694-metrics-certs\") pod \"0a0ab432-145b-45a0-8629-fa4dcb129694\" (UID: \"0a0ab432-145b-45a0-8629-fa4dcb129694\") "
Nov 02 12:50:53.613673 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:53.568861    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a0ab432-145b-45a0-8629-fa4dcb129694-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a0ab432-145b-45a0-8629-fa4dcb129694-openshift-service-ca\") pod \"0a0ab432-145b-45a0-8629-fa4dcb129694\" (UID: \"0a0ab432-145b-45a0-8629-fa4dcb129694\") "
Nov 02 12:50:53.613673 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:53.568928    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0a0ab432-145b-45a0-8629-fa4dcb129694-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a0ab432-145b-45a0-8629-fa4dcb129694-maas-api-tls\") pod \"0a0ab432-145b-45a0-8629-fa4dcb129694\" (UID: \"0a0ab432-145b-45a0-8629-fa4dcb129694\") "
Nov 02 12:50:53.613673 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:53.569378    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a0ab432-145b-45a0-8629-fa4dcb129694-kube-api-access-wp667\" (UniqueName: \"kubernetes.io/projected/0a0ab432-145b-45a0-8629-fa4dcb129694-kube-api-access-wp667\") pod \"0a0ab432-145b-45a0-8629-fa4dcb129694\" (UID: \"0a0ab432-145b-45a0-8629-fa4dcb129694\") "
Nov 02 12:50:53.613673 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:53.572179    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a0ab432-145b-45a0-8629-fa4dcb129694-openshift-service-ca" pod "0a0ab432-145b-45a0-8629-fa4dcb129694" (UID: "0a0ab432-145b-45a0-8629-fa4dcb129694"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:50:53.613673 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:53.587276    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a0ab432-145b-45a0-8629-fa4dcb129694-metrics-certs" pod "0a0ab432-145b-45a0-8629-fa4dcb129694" (UID: "0a0ab432-145b-45a0-8629-fa4dcb129694"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:50:53.674867 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:53.673430    4559 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a0ab432-145b-45a0-8629-fa4dcb129694-metrics-certs\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:50:53.678046 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:53.677906    4559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a0ab432-145b-45a0-8629-fa4dcb129694-openshift-service-ca\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:50:53.716026 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:53.715933    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a0ab432-145b-45a0-8629-fa4dcb129694-maas-api-tls" pod "0a0ab432-145b-45a0-8629-fa4dcb129694" (UID: "0a0ab432-145b-45a0-8629-fa4dcb129694"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:50:53.718053 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:53.718025    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a0ab432-145b-45a0-8629-fa4dcb129694-kube-api-access-wp667" pod "0a0ab432-145b-45a0-8629-fa4dcb129694" (UID: "0a0ab432-145b-45a0-8629-fa4dcb129694"). InnerVolumeSpecName "kube-api-access-wp667". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:50:53.779406 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:53.779365    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a0ab432-145b-45a0-8629-fa4dcb129694-maas-api-tls\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:50:53.779406 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:53.779405    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wp667\" (UniqueName: \"kubernetes.io/projected/0a0ab432-145b-45a0-8629-fa4dcb129694-kube-api-access-wp667\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:50:53.894243 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:53.894207    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="624a0494-692e-4864-810e-42ba5306a5f0" path="/var/lib/kubelet/pods/624a0494-692e-4864-810e-42ba5306a5f0/volumes"
Nov 02 12:50:54.436709 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:54.436682    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b271f0-578d584488-vwllx"
Nov 02 12:50:54.436709 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:54.436689    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b271f0-578d584488-vwllx" event={"ID":"0a0ab432-145b-45a0-8629-fa4dcb129694","Type":"ContainerDied","Data":"404c47666f9aa740bfcf0fa8aa2a402b9dd9c6881e8da316acce45b396f0a8e6"}
Nov 02 12:50:54.436975 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:54.436750    4559 scope.go:122] "RemoveContainer" containerID="f02fa12fe8defe52eda8006fefea34ffdb32973dc63f7f6bd287cc967efa68de"
Nov 02 12:50:54.493532 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:54.493505    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-b271f0-578d584488-vwllx"]
Nov 02 12:50:54.505860 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:54.505836    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-b271f0-578d584488-vwllx"]
Nov 02 12:50:55.706034 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:55.706001    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d7801d-867c88bd6c-2h9kh"]
Nov 02 12:50:55.706881 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:55.706859    4559 cpu_manager.go:414] "containerMap: removing container" podUID="624a0494-692e-4864-810e-42ba5306a5f0" containerName="istio-proxy"
Nov 02 12:50:55.706881 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:55.706881    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="624a0494-692e-4864-810e-42ba5306a5f0" containerName="istio-proxy"
Nov 02 12:50:55.707065 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:55.706899    4559 cpu_manager.go:414] "containerMap: removing container" podUID="624a0494-692e-4864-810e-42ba5306a5f0" containerName="istio-proxy"
Nov 02 12:50:55.707065 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:55.706907    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="624a0494-692e-4864-810e-42ba5306a5f0" containerName="istio-proxy"
Nov 02 12:50:55.707065 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:55.706956    4559 cpu_manager.go:414] "containerMap: removing container" podUID="0a0ab432-145b-45a0-8629-fa4dcb129694" containerName="maas-api"
Nov 02 12:50:55.707065 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:55.706964    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a0ab432-145b-45a0-8629-fa4dcb129694" containerName="maas-api"
Nov 02 12:50:55.707265 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:55.707108    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a0ab432-145b-45a0-8629-fa4dcb129694" containerName="maas-api"
Nov 02 12:50:55.707265 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:55.707128    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="624a0494-692e-4864-810e-42ba5306a5f0" containerName="istio-proxy"
Nov 02 12:50:55.707265 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:55.707138    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="624a0494-692e-4864-810e-42ba5306a5f0" containerName="istio-proxy"
Nov 02 12:50:55.713688 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:55.712332    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d7801d-867c88bd6c-2h9kh"
Nov 02 12:50:55.734512 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:55.734486    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d7801d-867c88bd6c-2h9kh"]
Nov 02 12:50:55.740851 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:55.740821    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-d7801d\""
Nov 02 12:50:55.740981 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:55.740961    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-d7801d-dockercfg-fmdjb\""
Nov 02 12:50:55.846710 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:55.843086    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfg9p\" (UniqueName: \"kubernetes.io/projected/8a3ff80c-ba06-4eef-8c21-3d14926ec509-kube-api-access-hfg9p\") pod \"payload-pre-processing-e2e-ipp-b-d7801d-867c88bd6c-2h9kh\" (UID: \"8a3ff80c-ba06-4eef-8c21-3d14926ec509\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d7801d-867c88bd6c-2h9kh"
Nov 02 12:50:55.846710 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:55.843385    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a3ff80c-ba06-4eef-8c21-3d14926ec509-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-d7801d-867c88bd6c-2h9kh\" (UID: \"8a3ff80c-ba06-4eef-8c21-3d14926ec509\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d7801d-867c88bd6c-2h9kh"
Nov 02 12:50:56.002712 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:55.955758    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfg9p\" (UniqueName: \"kubernetes.io/projected/8a3ff80c-ba06-4eef-8c21-3d14926ec509-kube-api-access-hfg9p\") pod \"payload-pre-processing-e2e-ipp-b-d7801d-867c88bd6c-2h9kh\" (UID: \"8a3ff80c-ba06-4eef-8c21-3d14926ec509\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d7801d-867c88bd6c-2h9kh"
Nov 02 12:50:56.002712 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:55.955938    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a3ff80c-ba06-4eef-8c21-3d14926ec509-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-d7801d-867c88bd6c-2h9kh\" (UID: \"8a3ff80c-ba06-4eef-8c21-3d14926ec509\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d7801d-867c88bd6c-2h9kh"
Nov 02 12:50:56.002712 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:55.956884    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a3ff80c-ba06-4eef-8c21-3d14926ec509-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-d7801d-867c88bd6c-2h9kh\" (UID: \"8a3ff80c-ba06-4eef-8c21-3d14926ec509\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d7801d-867c88bd6c-2h9kh"
Nov 02 12:50:56.033909 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:56.033875    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a0ab432-145b-45a0-8629-fa4dcb129694" path="/var/lib/kubelet/pods/0a0ab432-145b-45a0-8629-fa4dcb129694/volumes"
Nov 02 12:50:56.034218 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:56.033921    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfg9p\" (UniqueName: \"kubernetes.io/projected/8a3ff80c-ba06-4eef-8c21-3d14926ec509-kube-api-access-hfg9p\") pod \"payload-pre-processing-e2e-ipp-b-d7801d-867c88bd6c-2h9kh\" (UID: \"8a3ff80c-ba06-4eef-8c21-3d14926ec509\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d7801d-867c88bd6c-2h9kh"
Nov 02 12:50:56.034811 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:56.034791    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d7801d-c576f4dc9-2fddw"]
Nov 02 12:50:56.040785 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:56.040766    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d7801d-c576f4dc9-2fddw"]
Nov 02 12:50:56.040961 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:56.040863    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-d7801d-c576f4dc9-2fddw"
Nov 02 12:50:56.055862 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:56.049880    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d7801d-867c88bd6c-2h9kh"
Nov 02 12:50:56.161209 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:56.160813    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64tvk\" (UniqueName: \"kubernetes.io/projected/311b37ae-9f7a-4c54-9819-3a20fcbe18d9-kube-api-access-64tvk\") pod \"payload-processing-e2e-ipp-b-d7801d-c576f4dc9-2fddw\" (UID: \"311b37ae-9f7a-4c54-9819-3a20fcbe18d9\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d7801d-c576f4dc9-2fddw"
Nov 02 12:50:56.161209 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:56.161058    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/311b37ae-9f7a-4c54-9819-3a20fcbe18d9-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-d7801d-c576f4dc9-2fddw\" (UID: \"311b37ae-9f7a-4c54-9819-3a20fcbe18d9\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d7801d-c576f4dc9-2fddw"
Nov 02 12:50:56.262429 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:56.262351    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/311b37ae-9f7a-4c54-9819-3a20fcbe18d9-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-d7801d-c576f4dc9-2fddw\" (UID: \"311b37ae-9f7a-4c54-9819-3a20fcbe18d9\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d7801d-c576f4dc9-2fddw"
Nov 02 12:50:56.262591 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:56.262448    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64tvk\" (UniqueName: \"kubernetes.io/projected/311b37ae-9f7a-4c54-9819-3a20fcbe18d9-kube-api-access-64tvk\") pod \"payload-processing-e2e-ipp-b-d7801d-c576f4dc9-2fddw\" (UID: \"311b37ae-9f7a-4c54-9819-3a20fcbe18d9\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d7801d-c576f4dc9-2fddw"
Nov 02 12:50:56.263112 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:56.263088    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/311b37ae-9f7a-4c54-9819-3a20fcbe18d9-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-d7801d-c576f4dc9-2fddw\" (UID: \"311b37ae-9f7a-4c54-9819-3a20fcbe18d9\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d7801d-c576f4dc9-2fddw"
Nov 02 12:50:56.286199 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:56.286167    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64tvk\" (UniqueName: \"kubernetes.io/projected/311b37ae-9f7a-4c54-9819-3a20fcbe18d9-kube-api-access-64tvk\") pod \"payload-processing-e2e-ipp-b-d7801d-c576f4dc9-2fddw\" (UID: \"311b37ae-9f7a-4c54-9819-3a20fcbe18d9\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d7801d-c576f4dc9-2fddw"
Nov 02 12:50:56.409372 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:56.409333    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-d7801d-c576f4dc9-2fddw"
Nov 02 12:50:56.438829 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:50:56.438795    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a3ff80c_ba06_4eef_8c21_3d14926ec509.slice/crio-62eff538567627d152943bd22eb425001b275694ef0bcc64e8b94c0273d8f903 WatchSource:0}: Error finding container 62eff538567627d152943bd22eb425001b275694ef0bcc64e8b94c0273d8f903: Status 404 returned error can't find the container with id 62eff538567627d152943bd22eb425001b275694ef0bcc64e8b94c0273d8f903
Nov 02 12:50:56.439759 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:56.439734    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d7801d-867c88bd6c-2h9kh"]
Nov 02 12:50:56.451190 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:56.451162    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d7801d-867c88bd6c-2h9kh" event={"ID":"8a3ff80c-ba06-4eef-8c21-3d14926ec509","Type":"ContainerStarted","Data":"62eff538567627d152943bd22eb425001b275694ef0bcc64e8b94c0273d8f903"}
Nov 02 12:50:56.725132 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:50:56.725102    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod311b37ae_9f7a_4c54_9819_3a20fcbe18d9.slice/crio-0665567de0fd5cb2a20a564931d9a1b8ceeba03ac935cb6e08528ab331dfa54b WatchSource:0}: Error finding container 0665567de0fd5cb2a20a564931d9a1b8ceeba03ac935cb6e08528ab331dfa54b: Status 404 returned error can't find the container with id 0665567de0fd5cb2a20a564931d9a1b8ceeba03ac935cb6e08528ab331dfa54b
Nov 02 12:50:56.725882 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:56.725824    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d7801d-c576f4dc9-2fddw"]
Nov 02 12:50:57.459390 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:57.459353    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-d7801d-c576f4dc9-2fddw" event={"ID":"311b37ae-9f7a-4c54-9819-3a20fcbe18d9","Type":"ContainerStarted","Data":"cefb126845fb8462a521638cd20461659334a46d0876ad54a0e1b0e1fd51d791"}
Nov 02 12:50:57.459580 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:57.459541    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-d7801d-c576f4dc9-2fddw"
Nov 02 12:50:57.459652 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:57.459578    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-d7801d-c576f4dc9-2fddw" event={"ID":"311b37ae-9f7a-4c54-9819-3a20fcbe18d9","Type":"ContainerStarted","Data":"0665567de0fd5cb2a20a564931d9a1b8ceeba03ac935cb6e08528ab331dfa54b"}
Nov 02 12:50:57.461038 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:57.461014    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d7801d-867c88bd6c-2h9kh" event={"ID":"8a3ff80c-ba06-4eef-8c21-3d14926ec509","Type":"ContainerStarted","Data":"921aa3876819f809406c1e72a100565d57ac2b94902ed552e5663a6eef8bd5f3"}
Nov 02 12:50:57.461171 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:57.461101    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d7801d-867c88bd6c-2h9kh"
Nov 02 12:50:57.501782 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:57.501732    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-d7801d-c576f4dc9-2fddw" podStartSLOduration=2.5017152190000003 podStartE2EDuration="2.501715219s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:50:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:50:57.500460098 +0000 UTC m=+1453.755859195" watchObservedRunningTime="2026-11-02 12:50:57.501715219 +0000 UTC m=+1453.757114315"
Nov 02 12:50:57.542215 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:57.542155    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d7801d-867c88bd6c-2h9kh" podStartSLOduration=2.542140678 podStartE2EDuration="2.542140678s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:50:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:50:57.541065719 +0000 UTC m=+1453.796464814" watchObservedRunningTime="2026-11-02 12:50:57.542140678 +0000 UTC m=+1453.797539774"
Nov 02 12:50:57.596190 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:57.596148    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-b271f0-6966d4c887-jthwp" podUID="599fd828-375f-4169-9e81-cda2e36228a3" containerName="payload-processing" containerID="cri-o://4d640c29cff4a25c8c77dc34d5b4bc5a98b1ed24ab198fe9f07087be864c287a" gracePeriod=25
Nov 02 12:50:57.632186 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:57.632138    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b271f0-547b5469dc-42kll" podUID="32309c5a-744b-4225-b1c1-1b92704e4c2a" containerName="payload-pre-processing" containerID="cri-o://b213bfe5c5bc908ee1729837867ddfa08ae9f63fb7ec3f0acde24226cdfc9fc7" gracePeriod=25
Nov 02 12:50:57.929197 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:57.929168    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b271f0-547b5469dc-42kll"
Nov 02 12:50:57.944033 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:57.944003    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-b271f0-6966d4c887-jthwp"
Nov 02 12:50:58.040524 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:58.035400    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32309c5a-744b-4225-b1c1-1b92704e4c2a-kube-api-access-wplfm\" (UniqueName: \"kubernetes.io/projected/32309c5a-744b-4225-b1c1-1b92704e4c2a-kube-api-access-wplfm\") pod \"32309c5a-744b-4225-b1c1-1b92704e4c2a\" (UID: \"32309c5a-744b-4225-b1c1-1b92704e4c2a\") "
Nov 02 12:50:58.040524 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:58.035491    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/599fd828-375f-4169-9e81-cda2e36228a3-kube-api-access-4dwlm\" (UniqueName: \"kubernetes.io/projected/599fd828-375f-4169-9e81-cda2e36228a3-kube-api-access-4dwlm\") pod \"599fd828-375f-4169-9e81-cda2e36228a3\" (UID: \"599fd828-375f-4169-9e81-cda2e36228a3\") "
Nov 02 12:50:58.040524 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:58.036313    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32309c5a-744b-4225-b1c1-1b92704e4c2a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32309c5a-744b-4225-b1c1-1b92704e4c2a-plugins-config-volume\") pod \"32309c5a-744b-4225-b1c1-1b92704e4c2a\" (UID: \"32309c5a-744b-4225-b1c1-1b92704e4c2a\") "
Nov 02 12:50:58.040524 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:58.036404    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/599fd828-375f-4169-9e81-cda2e36228a3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/599fd828-375f-4169-9e81-cda2e36228a3-plugins-config-volume\") pod \"599fd828-375f-4169-9e81-cda2e36228a3\" (UID: \"599fd828-375f-4169-9e81-cda2e36228a3\") "
Nov 02 12:50:58.040524 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:58.037628    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32309c5a-744b-4225-b1c1-1b92704e4c2a-plugins-config-volume" pod "32309c5a-744b-4225-b1c1-1b92704e4c2a" (UID: "32309c5a-744b-4225-b1c1-1b92704e4c2a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:50:58.040524 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:58.040483    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/599fd828-375f-4169-9e81-cda2e36228a3-plugins-config-volume" pod "599fd828-375f-4169-9e81-cda2e36228a3" (UID: "599fd828-375f-4169-9e81-cda2e36228a3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:50:58.045763 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:58.045719    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/599fd828-375f-4169-9e81-cda2e36228a3-kube-api-access-4dwlm" pod "599fd828-375f-4169-9e81-cda2e36228a3" (UID: "599fd828-375f-4169-9e81-cda2e36228a3"). InnerVolumeSpecName "kube-api-access-4dwlm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:50:58.048241 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:58.048201    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32309c5a-744b-4225-b1c1-1b92704e4c2a-kube-api-access-wplfm" pod "32309c5a-744b-4225-b1c1-1b92704e4c2a" (UID: "32309c5a-744b-4225-b1c1-1b92704e4c2a"). InnerVolumeSpecName "kube-api-access-wplfm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:50:58.138259 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:58.138225    4559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/599fd828-375f-4169-9e81-cda2e36228a3-plugins-config-volume\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:50:58.138259 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:58.138258    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wplfm\" (UniqueName: \"kubernetes.io/projected/32309c5a-744b-4225-b1c1-1b92704e4c2a-kube-api-access-wplfm\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:50:58.138468 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:58.138268    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4dwlm\" (UniqueName: \"kubernetes.io/projected/599fd828-375f-4169-9e81-cda2e36228a3-kube-api-access-4dwlm\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:50:58.138468 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:58.138277    4559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32309c5a-744b-4225-b1c1-1b92704e4c2a-plugins-config-volume\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:50:58.468524 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:58.468492    4559 generic.go:356] "Generic (PLEG): container finished" podID="32309c5a-744b-4225-b1c1-1b92704e4c2a" containerID="b213bfe5c5bc908ee1729837867ddfa08ae9f63fb7ec3f0acde24226cdfc9fc7" exitCode=0
Nov 02 12:50:58.468757 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:58.468543    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b271f0-547b5469dc-42kll"
Nov 02 12:50:58.468757 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:58.468575    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b271f0-547b5469dc-42kll" event={"ID":"32309c5a-744b-4225-b1c1-1b92704e4c2a","Type":"ContainerDied","Data":"b213bfe5c5bc908ee1729837867ddfa08ae9f63fb7ec3f0acde24226cdfc9fc7"}
Nov 02 12:50:58.468757 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:58.468622    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b271f0-547b5469dc-42kll" event={"ID":"32309c5a-744b-4225-b1c1-1b92704e4c2a","Type":"ContainerDied","Data":"1154da98a1d1a1c9a55498d111c1e3d93d49623034832da51a3bb05929b53c1c"}
Nov 02 12:50:58.468757 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:58.468663    4559 scope.go:122] "RemoveContainer" containerID="b213bfe5c5bc908ee1729837867ddfa08ae9f63fb7ec3f0acde24226cdfc9fc7"
Nov 02 12:50:58.470318 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:58.470293    4559 generic.go:356] "Generic (PLEG): container finished" podID="599fd828-375f-4169-9e81-cda2e36228a3" containerID="4d640c29cff4a25c8c77dc34d5b4bc5a98b1ed24ab198fe9f07087be864c287a" exitCode=0
Nov 02 12:50:58.470435 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:58.470346    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-b271f0-6966d4c887-jthwp"
Nov 02 12:50:58.470435 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:58.470370    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-b271f0-6966d4c887-jthwp" event={"ID":"599fd828-375f-4169-9e81-cda2e36228a3","Type":"ContainerDied","Data":"4d640c29cff4a25c8c77dc34d5b4bc5a98b1ed24ab198fe9f07087be864c287a"}
Nov 02 12:50:58.470435 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:58.470423    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-b271f0-6966d4c887-jthwp" event={"ID":"599fd828-375f-4169-9e81-cda2e36228a3","Type":"ContainerDied","Data":"251d18f2f248c13737dc2ac258ebdca65bfa7baa997903ed1a4a097f94fc2021"}
Nov 02 12:50:58.481523 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:58.481503    4559 scope.go:122] "RemoveContainer" containerID="b213bfe5c5bc908ee1729837867ddfa08ae9f63fb7ec3f0acde24226cdfc9fc7"
Nov 02 12:50:58.481833 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:50:58.481814    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b213bfe5c5bc908ee1729837867ddfa08ae9f63fb7ec3f0acde24226cdfc9fc7\": container with ID starting with b213bfe5c5bc908ee1729837867ddfa08ae9f63fb7ec3f0acde24226cdfc9fc7 not found: ID does not exist" containerID="b213bfe5c5bc908ee1729837867ddfa08ae9f63fb7ec3f0acde24226cdfc9fc7"
Nov 02 12:50:58.481925 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:58.481843    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b213bfe5c5bc908ee1729837867ddfa08ae9f63fb7ec3f0acde24226cdfc9fc7"} err="failed to get container status \"b213bfe5c5bc908ee1729837867ddfa08ae9f63fb7ec3f0acde24226cdfc9fc7\": rpc error: code = NotFound desc = could not find container \"b213bfe5c5bc908ee1729837867ddfa08ae9f63fb7ec3f0acde24226cdfc9fc7\": container with ID starting with b213bfe5c5bc908ee1729837867ddfa08ae9f63fb7ec3f0acde24226cdfc9fc7 not found: ID does not exist"
Nov 02 12:50:58.481925 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:58.481865    4559 scope.go:122] "RemoveContainer" containerID="4d640c29cff4a25c8c77dc34d5b4bc5a98b1ed24ab198fe9f07087be864c287a"
Nov 02 12:50:58.491558 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:58.491535    4559 scope.go:122] "RemoveContainer" containerID="4d640c29cff4a25c8c77dc34d5b4bc5a98b1ed24ab198fe9f07087be864c287a"
Nov 02 12:50:58.491921 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:50:58.491901    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d640c29cff4a25c8c77dc34d5b4bc5a98b1ed24ab198fe9f07087be864c287a\": container with ID starting with 4d640c29cff4a25c8c77dc34d5b4bc5a98b1ed24ab198fe9f07087be864c287a not found: ID does not exist" containerID="4d640c29cff4a25c8c77dc34d5b4bc5a98b1ed24ab198fe9f07087be864c287a"
Nov 02 12:50:58.491980 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:58.491928    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d640c29cff4a25c8c77dc34d5b4bc5a98b1ed24ab198fe9f07087be864c287a"} err="failed to get container status \"4d640c29cff4a25c8c77dc34d5b4bc5a98b1ed24ab198fe9f07087be864c287a\": rpc error: code = NotFound desc = could not find container \"4d640c29cff4a25c8c77dc34d5b4bc5a98b1ed24ab198fe9f07087be864c287a\": container with ID starting with 4d640c29cff4a25c8c77dc34d5b4bc5a98b1ed24ab198fe9f07087be864c287a not found: ID does not exist"
Nov 02 12:50:58.523099 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:58.523074    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-b271f0-547b5469dc-42kll"]
Nov 02 12:50:58.535315 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:58.535287    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-b271f0-547b5469dc-42kll"]
Nov 02 12:50:58.570278 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:58.570224    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-b271f0-6966d4c887-jthwp"]
Nov 02 12:50:58.573540 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:58.573519    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-b271f0-6966d4c887-jthwp"]
Nov 02 12:50:59.893469 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:59.893435    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32309c5a-744b-4225-b1c1-1b92704e4c2a" path="/var/lib/kubelet/pods/32309c5a-744b-4225-b1c1-1b92704e4c2a/volumes"
Nov 02 12:50:59.894107 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:50:59.894085    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="599fd828-375f-4169-9e81-cda2e36228a3" path="/var/lib/kubelet/pods/599fd828-375f-4169-9e81-cda2e36228a3/volumes"
Nov 02 12:51:00.347526 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:00.347488    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"]
Nov 02 12:51:02.612535 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:02.612501    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-d6d314-767dbf49bc-55rvd"]
Nov 02 12:51:02.646354 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:02.646325    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-d6d314-77744649f9-rf95p"]
Nov 02 12:51:07.186311 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:07.186277    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-d6d314-67c7c65cdd-hb2gt"]
Nov 02 12:51:07.187011 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:07.186958    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d6d314-67c7c65cdd-hb2gt" podUID="af8db105-0f50-418c-b155-5bbef3014b00" containerName="maas-api" containerID="cri-o://babc7e55f3903f5c607674835f3a6b29d7e27f080634cbcb8505380b4249784e" gracePeriod=30
Nov 02 12:51:07.563501 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:07.563201    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d6d314-67c7c65cdd-hb2gt"
Nov 02 12:51:07.567124 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:07.567093    4559 generic.go:356] "Generic (PLEG): container finished" podID="af8db105-0f50-418c-b155-5bbef3014b00" containerID="babc7e55f3903f5c607674835f3a6b29d7e27f080634cbcb8505380b4249784e" exitCode=0
Nov 02 12:51:07.567288 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:07.567273    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d6d314-67c7c65cdd-hb2gt" event={"ID":"af8db105-0f50-418c-b155-5bbef3014b00","Type":"ContainerDied","Data":"babc7e55f3903f5c607674835f3a6b29d7e27f080634cbcb8505380b4249784e"}
Nov 02 12:51:07.567333 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:07.567314    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d6d314-67c7c65cdd-hb2gt" event={"ID":"af8db105-0f50-418c-b155-5bbef3014b00","Type":"ContainerDied","Data":"75dc2f512c0879650c76518d560668b5234feb85645f3f290f73a782988b4128"}
Nov 02 12:51:07.567381 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:07.567340    4559 scope.go:122] "RemoveContainer" containerID="babc7e55f3903f5c607674835f3a6b29d7e27f080634cbcb8505380b4249784e"
Nov 02 12:51:07.567500 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:07.567453    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d6d314-67c7c65cdd-hb2gt"
Nov 02 12:51:07.584893 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:07.583435    4559 scope.go:122] "RemoveContainer" containerID="babc7e55f3903f5c607674835f3a6b29d7e27f080634cbcb8505380b4249784e"
Nov 02 12:51:07.612152 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:51:07.587482    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"babc7e55f3903f5c607674835f3a6b29d7e27f080634cbcb8505380b4249784e\": container with ID starting with babc7e55f3903f5c607674835f3a6b29d7e27f080634cbcb8505380b4249784e not found: ID does not exist" containerID="babc7e55f3903f5c607674835f3a6b29d7e27f080634cbcb8505380b4249784e"
Nov 02 12:51:07.612152 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:07.587523    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"babc7e55f3903f5c607674835f3a6b29d7e27f080634cbcb8505380b4249784e"} err="failed to get container status \"babc7e55f3903f5c607674835f3a6b29d7e27f080634cbcb8505380b4249784e\": rpc error: code = NotFound desc = could not find container \"babc7e55f3903f5c607674835f3a6b29d7e27f080634cbcb8505380b4249784e\": container with ID starting with babc7e55f3903f5c607674835f3a6b29d7e27f080634cbcb8505380b4249784e not found: ID does not exist"
Nov 02 12:51:07.656663 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:07.653440    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d6d314-77744649f9-rf95p" podUID="d504a5ab-c0f3-4f82-8385-9c11624edc60" containerName="payload-pre-processing" containerID="cri-o://0c96e44c6b992653b29a3e361bf2ce1f071b8cc824e80a7d4239a5936491e2c3" gracePeriod=25
Nov 02 12:51:07.656663 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:07.654057    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-d6d314-767dbf49bc-55rvd" podUID="ced18836-7a8d-4913-bcd3-24d3d75592d6" containerName="payload-processing" containerID="cri-o://bfc46b31db75ba10f0296415d523502e2cf188f0e88ecb8fceb85858bc6d31bb" gracePeriod=25
Nov 02 12:51:07.778944 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:07.756709    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/af8db105-0f50-418c-b155-5bbef3014b00-metrics-certs\" (UniqueName: \"kubernetes.io/secret/af8db105-0f50-418c-b155-5bbef3014b00-metrics-certs\") pod \"af8db105-0f50-418c-b155-5bbef3014b00\" (UID: \"af8db105-0f50-418c-b155-5bbef3014b00\") "
Nov 02 12:51:07.778944 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:07.756813    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af8db105-0f50-418c-b155-5bbef3014b00-kube-api-access-f9rx4\" (UniqueName: \"kubernetes.io/projected/af8db105-0f50-418c-b155-5bbef3014b00-kube-api-access-f9rx4\") pod \"af8db105-0f50-418c-b155-5bbef3014b00\" (UID: \"af8db105-0f50-418c-b155-5bbef3014b00\") "
Nov 02 12:51:07.778944 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:07.756861    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af8db105-0f50-418c-b155-5bbef3014b00-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/af8db105-0f50-418c-b155-5bbef3014b00-openshift-service-ca\") pod \"af8db105-0f50-418c-b155-5bbef3014b00\" (UID: \"af8db105-0f50-418c-b155-5bbef3014b00\") "
Nov 02 12:51:07.778944 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:07.756990    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/af8db105-0f50-418c-b155-5bbef3014b00-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af8db105-0f50-418c-b155-5bbef3014b00-maas-api-tls\") pod \"af8db105-0f50-418c-b155-5bbef3014b00\" (UID: \"af8db105-0f50-418c-b155-5bbef3014b00\") "
Nov 02 12:51:07.778944 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:07.777568    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af8db105-0f50-418c-b155-5bbef3014b00-openshift-service-ca" pod "af8db105-0f50-418c-b155-5bbef3014b00" (UID: "af8db105-0f50-418c-b155-5bbef3014b00"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:51:07.800002 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:07.799960    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af8db105-0f50-418c-b155-5bbef3014b00-maas-api-tls" pod "af8db105-0f50-418c-b155-5bbef3014b00" (UID: "af8db105-0f50-418c-b155-5bbef3014b00"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:51:07.802129 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:07.802082    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af8db105-0f50-418c-b155-5bbef3014b00-metrics-certs" pod "af8db105-0f50-418c-b155-5bbef3014b00" (UID: "af8db105-0f50-418c-b155-5bbef3014b00"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:51:07.805761 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:07.802756    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af8db105-0f50-418c-b155-5bbef3014b00-kube-api-access-f9rx4" pod "af8db105-0f50-418c-b155-5bbef3014b00" (UID: "af8db105-0f50-418c-b155-5bbef3014b00"). InnerVolumeSpecName "kube-api-access-f9rx4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:51:07.858968 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:07.858941    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af8db105-0f50-418c-b155-5bbef3014b00-maas-api-tls\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:51:07.858968 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:07.858966    4559 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af8db105-0f50-418c-b155-5bbef3014b00-metrics-certs\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:51:07.859145 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:07.858976    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f9rx4\" (UniqueName: \"kubernetes.io/projected/af8db105-0f50-418c-b155-5bbef3014b00-kube-api-access-f9rx4\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:51:07.859145 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:07.858985    4559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/af8db105-0f50-418c-b155-5bbef3014b00-openshift-service-ca\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:51:07.919454 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:07.919412    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-d6d314-67c7c65cdd-hb2gt"]
Nov 02 12:51:07.929946 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:07.929917    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-d6d314-67c7c65cdd-hb2gt"]
Nov 02 12:51:08.113198 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:08.113169    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-d6d314-767dbf49bc-55rvd"
Nov 02 12:51:08.179339 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:08.179303    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d6d314-77744649f9-rf95p"
Nov 02 12:51:08.249667 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:08.249604    4559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-729108-6898dcb756-ksksg_52cf3408-1bb5-49aa-9a7e-2b70df75ccf7/payload-processing/0.log"
Nov 02 12:51:08.272741 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:08.272715    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ced18836-7a8d-4913-bcd3-24d3d75592d6-kube-api-access-c678c\" (UniqueName: \"kubernetes.io/projected/ced18836-7a8d-4913-bcd3-24d3d75592d6-kube-api-access-c678c\") pod \"ced18836-7a8d-4913-bcd3-24d3d75592d6\" (UID: \"ced18836-7a8d-4913-bcd3-24d3d75592d6\") "
Nov 02 12:51:08.272875 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:08.272853    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d504a5ab-c0f3-4f82-8385-9c11624edc60-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d504a5ab-c0f3-4f82-8385-9c11624edc60-plugins-config-volume\") pod \"d504a5ab-c0f3-4f82-8385-9c11624edc60\" (UID: \"d504a5ab-c0f3-4f82-8385-9c11624edc60\") "
Nov 02 12:51:08.272952 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:08.272938    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d504a5ab-c0f3-4f82-8385-9c11624edc60-kube-api-access-4dpf9\" (UniqueName: \"kubernetes.io/projected/d504a5ab-c0f3-4f82-8385-9c11624edc60-kube-api-access-4dpf9\") pod \"d504a5ab-c0f3-4f82-8385-9c11624edc60\" (UID: \"d504a5ab-c0f3-4f82-8385-9c11624edc60\") "
Nov 02 12:51:08.273047 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:08.273031    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ced18836-7a8d-4913-bcd3-24d3d75592d6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ced18836-7a8d-4913-bcd3-24d3d75592d6-plugins-config-volume\") pod \"ced18836-7a8d-4913-bcd3-24d3d75592d6\" (UID: \"ced18836-7a8d-4913-bcd3-24d3d75592d6\") "
Nov 02 12:51:08.273257 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:08.273227    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d504a5ab-c0f3-4f82-8385-9c11624edc60-plugins-config-volume" pod "d504a5ab-c0f3-4f82-8385-9c11624edc60" (UID: "d504a5ab-c0f3-4f82-8385-9c11624edc60"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:51:08.273518 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:08.273500    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ced18836-7a8d-4913-bcd3-24d3d75592d6-plugins-config-volume" pod "ced18836-7a8d-4913-bcd3-24d3d75592d6" (UID: "ced18836-7a8d-4913-bcd3-24d3d75592d6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:51:08.273727 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:08.273709    4559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d504a5ab-c0f3-4f82-8385-9c11624edc60-plugins-config-volume\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:51:08.273800 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:08.273732    4559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ced18836-7a8d-4913-bcd3-24d3d75592d6-plugins-config-volume\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:51:08.275117 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:08.275091    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ced18836-7a8d-4913-bcd3-24d3d75592d6-kube-api-access-c678c" pod "ced18836-7a8d-4913-bcd3-24d3d75592d6" (UID: "ced18836-7a8d-4913-bcd3-24d3d75592d6"). InnerVolumeSpecName "kube-api-access-c678c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:51:08.275556 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:08.275537    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d504a5ab-c0f3-4f82-8385-9c11624edc60-kube-api-access-4dpf9" pod "d504a5ab-c0f3-4f82-8385-9c11624edc60" (UID: "d504a5ab-c0f3-4f82-8385-9c11624edc60"). InnerVolumeSpecName "kube-api-access-4dpf9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:51:08.375102 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:08.375018    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c678c\" (UniqueName: \"kubernetes.io/projected/ced18836-7a8d-4913-bcd3-24d3d75592d6-kube-api-access-c678c\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:51:08.375102 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:08.375051    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4dpf9\" (UniqueName: \"kubernetes.io/projected/d504a5ab-c0f3-4f82-8385-9c11624edc60-kube-api-access-4dpf9\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:51:08.470058 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:08.470031    4559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-r6rdp_2fef2bd6-5793-4c34-a390-37c907bcfd19/payload-processing/0.log"
Nov 02 12:51:08.473822 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:08.473795    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-d7801d-c576f4dc9-2fddw"
Nov 02 12:51:08.473976 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:08.473895    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d7801d-867c88bd6c-2h9kh"
Nov 02 12:51:08.576879 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:08.576841    4559 generic.go:356] "Generic (PLEG): container finished" podID="d504a5ab-c0f3-4f82-8385-9c11624edc60" containerID="0c96e44c6b992653b29a3e361bf2ce1f071b8cc824e80a7d4239a5936491e2c3" exitCode=0
Nov 02 12:51:08.577065 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:08.576911    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d6d314-77744649f9-rf95p"
Nov 02 12:51:08.577065 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:08.576985    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d6d314-77744649f9-rf95p" event={"ID":"d504a5ab-c0f3-4f82-8385-9c11624edc60","Type":"ContainerDied","Data":"0c96e44c6b992653b29a3e361bf2ce1f071b8cc824e80a7d4239a5936491e2c3"}
Nov 02 12:51:08.577371 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:08.577346    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d6d314-77744649f9-rf95p" event={"ID":"d504a5ab-c0f3-4f82-8385-9c11624edc60","Type":"ContainerDied","Data":"c3ac599317f9b9ddc3e038a2fe6da52796c8e239ef047d3f4e794ace0af7022e"}
Nov 02 12:51:08.577724 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:08.577397    4559 scope.go:122] "RemoveContainer" containerID="0c96e44c6b992653b29a3e361bf2ce1f071b8cc824e80a7d4239a5936491e2c3"
Nov 02 12:51:08.580996 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:08.580971    4559 generic.go:356] "Generic (PLEG): container finished" podID="ced18836-7a8d-4913-bcd3-24d3d75592d6" containerID="bfc46b31db75ba10f0296415d523502e2cf188f0e88ecb8fceb85858bc6d31bb" exitCode=0
Nov 02 12:51:08.581121 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:08.581076    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-d6d314-767dbf49bc-55rvd" event={"ID":"ced18836-7a8d-4913-bcd3-24d3d75592d6","Type":"ContainerDied","Data":"bfc46b31db75ba10f0296415d523502e2cf188f0e88ecb8fceb85858bc6d31bb"}
Nov 02 12:51:08.581121 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:08.581118    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-d6d314-767dbf49bc-55rvd" event={"ID":"ced18836-7a8d-4913-bcd3-24d3d75592d6","Type":"ContainerDied","Data":"43738b648eb7131273d931068f0e496d960c982d5cb32f1c7827aabe19a922a2"}
Nov 02 12:51:08.581291 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:08.581204    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-d6d314-767dbf49bc-55rvd"
Nov 02 12:51:08.598773 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:08.593390    4559 scope.go:122] "RemoveContainer" containerID="0c96e44c6b992653b29a3e361bf2ce1f071b8cc824e80a7d4239a5936491e2c3"
Nov 02 12:51:08.598773 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:51:08.594048    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c96e44c6b992653b29a3e361bf2ce1f071b8cc824e80a7d4239a5936491e2c3\": container with ID starting with 0c96e44c6b992653b29a3e361bf2ce1f071b8cc824e80a7d4239a5936491e2c3 not found: ID does not exist" containerID="0c96e44c6b992653b29a3e361bf2ce1f071b8cc824e80a7d4239a5936491e2c3"
Nov 02 12:51:08.598773 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:08.594088    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c96e44c6b992653b29a3e361bf2ce1f071b8cc824e80a7d4239a5936491e2c3"} err="failed to get container status \"0c96e44c6b992653b29a3e361bf2ce1f071b8cc824e80a7d4239a5936491e2c3\": rpc error: code = NotFound desc = could not find container \"0c96e44c6b992653b29a3e361bf2ce1f071b8cc824e80a7d4239a5936491e2c3\": container with ID starting with 0c96e44c6b992653b29a3e361bf2ce1f071b8cc824e80a7d4239a5936491e2c3 not found: ID does not exist"
Nov 02 12:51:08.598773 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:08.594112    4559 scope.go:122] "RemoveContainer" containerID="bfc46b31db75ba10f0296415d523502e2cf188f0e88ecb8fceb85858bc6d31bb"
Nov 02 12:51:08.604064 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:08.604036    4559 scope.go:122] "RemoveContainer" containerID="bfc46b31db75ba10f0296415d523502e2cf188f0e88ecb8fceb85858bc6d31bb"
Nov 02 12:51:08.604338 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:51:08.604316    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bfc46b31db75ba10f0296415d523502e2cf188f0e88ecb8fceb85858bc6d31bb\": container with ID starting with bfc46b31db75ba10f0296415d523502e2cf188f0e88ecb8fceb85858bc6d31bb not found: ID does not exist" containerID="bfc46b31db75ba10f0296415d523502e2cf188f0e88ecb8fceb85858bc6d31bb"
Nov 02 12:51:08.604417 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:08.604356    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfc46b31db75ba10f0296415d523502e2cf188f0e88ecb8fceb85858bc6d31bb"} err="failed to get container status \"bfc46b31db75ba10f0296415d523502e2cf188f0e88ecb8fceb85858bc6d31bb\": rpc error: code = NotFound desc = could not find container \"bfc46b31db75ba10f0296415d523502e2cf188f0e88ecb8fceb85858bc6d31bb\": container with ID starting with bfc46b31db75ba10f0296415d523502e2cf188f0e88ecb8fceb85858bc6d31bb not found: ID does not exist"
Nov 02 12:51:08.650977 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:08.650915    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-d6d314-77744649f9-rf95p"]
Nov 02 12:51:08.658462 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:08.658435    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-d6d314-77744649f9-rf95p"]
Nov 02 12:51:08.692032 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:08.692002    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-d6d314-767dbf49bc-55rvd"]
Nov 02 12:51:08.699050 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:08.699023    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-d6d314-767dbf49bc-55rvd"]
Nov 02 12:51:08.707122 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:08.707094    4559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-r6rdp_2fef2bd6-5793-4c34-a390-37c907bcfd19/payload-processing/0.log"
Nov 02 12:51:08.904140 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:08.904069    4559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-299e74-78c848cbc6-xk4wx_9d93eaf1-b398-4b86-9156-845a73b24aa9/payload-processing/0.log"
Nov 02 12:51:09.093190 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:09.093162    4559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-299e74-78c848cbc6-xk4wx_9d93eaf1-b398-4b86-9156-845a73b24aa9/payload-processing/0.log"
Nov 02 12:51:09.894211 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:09.894181    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af8db105-0f50-418c-b155-5bbef3014b00" path="/var/lib/kubelet/pods/af8db105-0f50-418c-b155-5bbef3014b00/volumes"
Nov 02 12:51:09.894595 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:09.894580    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ced18836-7a8d-4913-bcd3-24d3d75592d6" path="/var/lib/kubelet/pods/ced18836-7a8d-4913-bcd3-24d3d75592d6/volumes"
Nov 02 12:51:09.894974 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:09.894961    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d504a5ab-c0f3-4f82-8385-9c11624edc60" path="/var/lib/kubelet/pods/d504a5ab-c0f3-4f82-8385-9c11624edc60/volumes"
Nov 02 12:51:10.657789 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:10.657753    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-d6d314-openshift-default-5b99b69565-2kkgt"]
Nov 02 12:51:10.658071 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:10.658023    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-d6d314-openshift-default-5b99b69565-2kkgt" podUID="4ac6508c-24db-40f7-a14c-0964dac7768b" containerName="istio-proxy" containerID="cri-o://7cfb9e4cf80abe147a36dfb196732604cd012e5a6b624875c3b2451d3f18a047" gracePeriod=30
Nov 02 12:51:12.654172 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:12.654138    4559 patch_prober.go:29] interesting pod/e2e-praxis-a-d6d314-openshift-default-5b99b69565-2kkgt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 12:51:12.654549 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:12.654200    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-d6d314-openshift-default-5b99b69565-2kkgt" podUID="4ac6508c-24db-40f7-a14c-0964dac7768b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 12:51:14.711041 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:14.711009    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-d7801d-6ffffb74dc-prdc6"]
Nov 02 12:51:14.711664 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:14.711634    4559 cpu_manager.go:414] "containerMap: removing container" podUID="ced18836-7a8d-4913-bcd3-24d3d75592d6" containerName="payload-processing"
Nov 02 12:51:14.711735 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:14.711666    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ced18836-7a8d-4913-bcd3-24d3d75592d6" containerName="payload-processing"
Nov 02 12:51:14.711735 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:14.711679    4559 cpu_manager.go:414] "containerMap: removing container" podUID="599fd828-375f-4169-9e81-cda2e36228a3" containerName="payload-processing"
Nov 02 12:51:14.711735 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:14.711684    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="599fd828-375f-4169-9e81-cda2e36228a3" containerName="payload-processing"
Nov 02 12:51:14.711735 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:14.711694    4559 cpu_manager.go:414] "containerMap: removing container" podUID="d504a5ab-c0f3-4f82-8385-9c11624edc60" containerName="payload-pre-processing"
Nov 02 12:51:14.711735 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:14.711701    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d504a5ab-c0f3-4f82-8385-9c11624edc60" containerName="payload-pre-processing"
Nov 02 12:51:14.711735 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:14.711708    4559 cpu_manager.go:414] "containerMap: removing container" podUID="af8db105-0f50-418c-b155-5bbef3014b00" containerName="maas-api"
Nov 02 12:51:14.711735 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:14.711713    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af8db105-0f50-418c-b155-5bbef3014b00" containerName="maas-api"
Nov 02 12:51:14.711982 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:14.711748    4559 cpu_manager.go:414] "containerMap: removing container" podUID="32309c5a-744b-4225-b1c1-1b92704e4c2a" containerName="payload-pre-processing"
Nov 02 12:51:14.711982 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:14.711753    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32309c5a-744b-4225-b1c1-1b92704e4c2a" containerName="payload-pre-processing"
Nov 02 12:51:14.711982 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:14.711851    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="32309c5a-744b-4225-b1c1-1b92704e4c2a" containerName="payload-pre-processing"
Nov 02 12:51:14.711982 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:14.711863    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="ced18836-7a8d-4913-bcd3-24d3d75592d6" containerName="payload-processing"
Nov 02 12:51:14.711982 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:14.711871    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="599fd828-375f-4169-9e81-cda2e36228a3" containerName="payload-processing"
Nov 02 12:51:14.711982 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:14.711878    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="d504a5ab-c0f3-4f82-8385-9c11624edc60" containerName="payload-pre-processing"
Nov 02 12:51:14.711982 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:14.711883    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="af8db105-0f50-418c-b155-5bbef3014b00" containerName="maas-api"
Nov 02 12:51:14.716883 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:14.716865    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d7801d-6ffffb74dc-prdc6"
Nov 02 12:51:14.738764 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:14.738737    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-d7801d-6ffffb74dc-prdc6"]
Nov 02 12:51:14.740930 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:14.740898    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-d7801d-serving-cert\""
Nov 02 12:51:14.881294 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:14.881260    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9479aa3-35be-4c23-95e6-efc7d46583cd-metrics-certs\") pod \"maas-api-e2e-ipp-b-d7801d-6ffffb74dc-prdc6\" (UID: \"c9479aa3-35be-4c23-95e6-efc7d46583cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d7801d-6ffffb74dc-prdc6"
Nov 02 12:51:14.881463 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:14.881331    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9479aa3-35be-4c23-95e6-efc7d46583cd-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-d7801d-6ffffb74dc-prdc6\" (UID: \"c9479aa3-35be-4c23-95e6-efc7d46583cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d7801d-6ffffb74dc-prdc6"
Nov 02 12:51:14.881508 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:14.881469    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8xtp\" (UniqueName: \"kubernetes.io/projected/c9479aa3-35be-4c23-95e6-efc7d46583cd-kube-api-access-b8xtp\") pod \"maas-api-e2e-ipp-b-d7801d-6ffffb74dc-prdc6\" (UID: \"c9479aa3-35be-4c23-95e6-efc7d46583cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d7801d-6ffffb74dc-prdc6"
Nov 02 12:51:14.881552 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:14.881530    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9479aa3-35be-4c23-95e6-efc7d46583cd-maas-api-tls\") pod \"maas-api-e2e-ipp-b-d7801d-6ffffb74dc-prdc6\" (UID: \"c9479aa3-35be-4c23-95e6-efc7d46583cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d7801d-6ffffb74dc-prdc6"
Nov 02 12:51:14.999013 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:14.998919    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9479aa3-35be-4c23-95e6-efc7d46583cd-metrics-certs\") pod \"maas-api-e2e-ipp-b-d7801d-6ffffb74dc-prdc6\" (UID: \"c9479aa3-35be-4c23-95e6-efc7d46583cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d7801d-6ffffb74dc-prdc6"
Nov 02 12:51:15.021812 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:15.021778    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9479aa3-35be-4c23-95e6-efc7d46583cd-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-d7801d-6ffffb74dc-prdc6\" (UID: \"c9479aa3-35be-4c23-95e6-efc7d46583cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d7801d-6ffffb74dc-prdc6"
Nov 02 12:51:15.022099 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:15.022082    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8xtp\" (UniqueName: \"kubernetes.io/projected/c9479aa3-35be-4c23-95e6-efc7d46583cd-kube-api-access-b8xtp\") pod \"maas-api-e2e-ipp-b-d7801d-6ffffb74dc-prdc6\" (UID: \"c9479aa3-35be-4c23-95e6-efc7d46583cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d7801d-6ffffb74dc-prdc6"
Nov 02 12:51:15.022248 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:15.022235    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9479aa3-35be-4c23-95e6-efc7d46583cd-maas-api-tls\") pod \"maas-api-e2e-ipp-b-d7801d-6ffffb74dc-prdc6\" (UID: \"c9479aa3-35be-4c23-95e6-efc7d46583cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d7801d-6ffffb74dc-prdc6"
Nov 02 12:51:15.023088 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:15.014411    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9479aa3-35be-4c23-95e6-efc7d46583cd-metrics-certs\") pod \"maas-api-e2e-ipp-b-d7801d-6ffffb74dc-prdc6\" (UID: \"c9479aa3-35be-4c23-95e6-efc7d46583cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d7801d-6ffffb74dc-prdc6"
Nov 02 12:51:15.037866 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:15.023969    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9479aa3-35be-4c23-95e6-efc7d46583cd-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-d7801d-6ffffb74dc-prdc6\" (UID: \"c9479aa3-35be-4c23-95e6-efc7d46583cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d7801d-6ffffb74dc-prdc6"
Nov 02 12:51:15.061579 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:15.061492    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9479aa3-35be-4c23-95e6-efc7d46583cd-maas-api-tls\") pod \"maas-api-e2e-ipp-b-d7801d-6ffffb74dc-prdc6\" (UID: \"c9479aa3-35be-4c23-95e6-efc7d46583cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d7801d-6ffffb74dc-prdc6"
Nov 02 12:51:15.063016 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:15.062949    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8xtp\" (UniqueName: \"kubernetes.io/projected/c9479aa3-35be-4c23-95e6-efc7d46583cd-kube-api-access-b8xtp\") pod \"maas-api-e2e-ipp-b-d7801d-6ffffb74dc-prdc6\" (UID: \"c9479aa3-35be-4c23-95e6-efc7d46583cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d7801d-6ffffb74dc-prdc6"
Nov 02 12:51:15.098714 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:51:15.098675    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqcmj" containerName="extractor"
Nov 02 12:51:15.333370 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:15.333280    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d7801d-6ffffb74dc-prdc6"
Nov 02 12:51:15.648676 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:51:15.648622    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9479aa3_35be_4c23_95e6_efc7d46583cd.slice/crio-a5363013607a4b7d89876ff7b1329a2d80a182ff6693c8ebf17d44615cc1b998 WatchSource:0}: Error finding container a5363013607a4b7d89876ff7b1329a2d80a182ff6693c8ebf17d44615cc1b998: Status 404 returned error can't find the container with id a5363013607a4b7d89876ff7b1329a2d80a182ff6693c8ebf17d44615cc1b998
Nov 02 12:51:15.650421 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:15.650397    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-d7801d-6ffffb74dc-prdc6"]
Nov 02 12:51:16.058984 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:16.058955    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-d6d314-openshift-default-5b99b69565-2kkgt"
Nov 02 12:51:16.241104 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:16.241029    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ac6508c-24db-40f7-a14c-0964dac7768b-kube-api-access-t2qfg\" (UniqueName: \"kubernetes.io/projected/4ac6508c-24db-40f7-a14c-0964dac7768b-kube-api-access-t2qfg\") pod \"4ac6508c-24db-40f7-a14c-0964dac7768b\" (UID: \"4ac6508c-24db-40f7-a14c-0964dac7768b\") "
Nov 02 12:51:16.241104 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:16.241085    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ac6508c-24db-40f7-a14c-0964dac7768b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ac6508c-24db-40f7-a14c-0964dac7768b-workload-socket\") pod \"4ac6508c-24db-40f7-a14c-0964dac7768b\" (UID: \"4ac6508c-24db-40f7-a14c-0964dac7768b\") "
Nov 02 12:51:16.241354 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:16.241150    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4ac6508c-24db-40f7-a14c-0964dac7768b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ac6508c-24db-40f7-a14c-0964dac7768b-istio-podinfo\") pod \"4ac6508c-24db-40f7-a14c-0964dac7768b\" (UID: \"4ac6508c-24db-40f7-a14c-0964dac7768b\") "
Nov 02 12:51:16.241354 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:16.241186    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ac6508c-24db-40f7-a14c-0964dac7768b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ac6508c-24db-40f7-a14c-0964dac7768b-workload-certs\") pod \"4ac6508c-24db-40f7-a14c-0964dac7768b\" (UID: \"4ac6508c-24db-40f7-a14c-0964dac7768b\") "
Nov 02 12:51:16.241354 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:16.241243    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ac6508c-24db-40f7-a14c-0964dac7768b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ac6508c-24db-40f7-a14c-0964dac7768b-istio-data\") pod \"4ac6508c-24db-40f7-a14c-0964dac7768b\" (UID: \"4ac6508c-24db-40f7-a14c-0964dac7768b\") "
Nov 02 12:51:16.241512 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:16.241411    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ac6508c-24db-40f7-a14c-0964dac7768b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ac6508c-24db-40f7-a14c-0964dac7768b-istio-envoy\") pod \"4ac6508c-24db-40f7-a14c-0964dac7768b\" (UID: \"4ac6508c-24db-40f7-a14c-0964dac7768b\") "
Nov 02 12:51:16.241512 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:16.241456    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ac6508c-24db-40f7-a14c-0964dac7768b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ac6508c-24db-40f7-a14c-0964dac7768b-istiod-ca-cert\") pod \"4ac6508c-24db-40f7-a14c-0964dac7768b\" (UID: \"4ac6508c-24db-40f7-a14c-0964dac7768b\") "
Nov 02 12:51:16.241512 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:16.241450    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ac6508c-24db-40f7-a14c-0964dac7768b-workload-socket" pod "4ac6508c-24db-40f7-a14c-0964dac7768b" (UID: "4ac6508c-24db-40f7-a14c-0964dac7768b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:51:16.241724 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:16.241534    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ac6508c-24db-40f7-a14c-0964dac7768b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ac6508c-24db-40f7-a14c-0964dac7768b-credential-socket\") pod \"4ac6508c-24db-40f7-a14c-0964dac7768b\" (UID: \"4ac6508c-24db-40f7-a14c-0964dac7768b\") "
Nov 02 12:51:16.241724 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:16.241533    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ac6508c-24db-40f7-a14c-0964dac7768b-workload-certs" pod "4ac6508c-24db-40f7-a14c-0964dac7768b" (UID: "4ac6508c-24db-40f7-a14c-0964dac7768b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:51:16.241724 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:16.241538    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ac6508c-24db-40f7-a14c-0964dac7768b-istio-data" pod "4ac6508c-24db-40f7-a14c-0964dac7768b" (UID: "4ac6508c-24db-40f7-a14c-0964dac7768b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:51:16.241724 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:16.241593    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ac6508c-24db-40f7-a14c-0964dac7768b-istio-token\" (UniqueName: \"kubernetes.io/projected/4ac6508c-24db-40f7-a14c-0964dac7768b-istio-token\") pod \"4ac6508c-24db-40f7-a14c-0964dac7768b\" (UID: \"4ac6508c-24db-40f7-a14c-0964dac7768b\") "
Nov 02 12:51:16.241920 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:16.241772    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ac6508c-24db-40f7-a14c-0964dac7768b-credential-socket" pod "4ac6508c-24db-40f7-a14c-0964dac7768b" (UID: "4ac6508c-24db-40f7-a14c-0964dac7768b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:51:16.241920 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:16.241888    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ac6508c-24db-40f7-a14c-0964dac7768b-istiod-ca-cert" pod "4ac6508c-24db-40f7-a14c-0964dac7768b" (UID: "4ac6508c-24db-40f7-a14c-0964dac7768b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:51:16.242261 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:16.242241    4559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ac6508c-24db-40f7-a14c-0964dac7768b-istio-data\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:51:16.242332 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:16.242268    4559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ac6508c-24db-40f7-a14c-0964dac7768b-istiod-ca-cert\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:51:16.242332 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:16.242284    4559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ac6508c-24db-40f7-a14c-0964dac7768b-credential-socket\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:51:16.242332 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:16.242299    4559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ac6508c-24db-40f7-a14c-0964dac7768b-workload-socket\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:51:16.242332 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:16.242314    4559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ac6508c-24db-40f7-a14c-0964dac7768b-workload-certs\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:51:16.243437 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:16.243405    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ac6508c-24db-40f7-a14c-0964dac7768b-istio-envoy" pod "4ac6508c-24db-40f7-a14c-0964dac7768b" (UID: "4ac6508c-24db-40f7-a14c-0964dac7768b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:51:16.243537 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:16.243495    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4ac6508c-24db-40f7-a14c-0964dac7768b-istio-podinfo" pod "4ac6508c-24db-40f7-a14c-0964dac7768b" (UID: "4ac6508c-24db-40f7-a14c-0964dac7768b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 12:51:16.243762 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:16.243742    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ac6508c-24db-40f7-a14c-0964dac7768b-kube-api-access-t2qfg" pod "4ac6508c-24db-40f7-a14c-0964dac7768b" (UID: "4ac6508c-24db-40f7-a14c-0964dac7768b"). InnerVolumeSpecName "kube-api-access-t2qfg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:51:16.243838 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:16.243820    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ac6508c-24db-40f7-a14c-0964dac7768b-istio-token" pod "4ac6508c-24db-40f7-a14c-0964dac7768b" (UID: "4ac6508c-24db-40f7-a14c-0964dac7768b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:51:16.343780 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:16.343746    4559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ac6508c-24db-40f7-a14c-0964dac7768b-istio-token\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:51:16.343780 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:16.343775    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t2qfg\" (UniqueName: \"kubernetes.io/projected/4ac6508c-24db-40f7-a14c-0964dac7768b-kube-api-access-t2qfg\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:51:16.343780 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:16.343785    4559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ac6508c-24db-40f7-a14c-0964dac7768b-istio-podinfo\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:51:16.344002 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:16.343794    4559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ac6508c-24db-40f7-a14c-0964dac7768b-istio-envoy\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:51:16.641467 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:16.641387    4559 generic.go:356] "Generic (PLEG): container finished" podID="4ac6508c-24db-40f7-a14c-0964dac7768b" containerID="7cfb9e4cf80abe147a36dfb196732604cd012e5a6b624875c3b2451d3f18a047" exitCode=0
Nov 02 12:51:16.641467 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:16.641438    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-d6d314-openshift-default-5b99b69565-2kkgt"
Nov 02 12:51:16.641745 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:16.641462    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-d6d314-openshift-default-5b99b69565-2kkgt" event={"ID":"4ac6508c-24db-40f7-a14c-0964dac7768b","Type":"ContainerDied","Data":"7cfb9e4cf80abe147a36dfb196732604cd012e5a6b624875c3b2451d3f18a047"}
Nov 02 12:51:16.641745 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:16.641520    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-d6d314-openshift-default-5b99b69565-2kkgt" event={"ID":"4ac6508c-24db-40f7-a14c-0964dac7768b","Type":"ContainerDied","Data":"028afb01e37fa5cd0f88b256ac66eb54fc7c15aef627bda2d9dfff456f109e1e"}
Nov 02 12:51:16.641745 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:16.641551    4559 scope.go:122] "RemoveContainer" containerID="7cfb9e4cf80abe147a36dfb196732604cd012e5a6b624875c3b2451d3f18a047"
Nov 02 12:51:16.648411 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:16.648381    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d7801d-6ffffb74dc-prdc6" event={"ID":"c9479aa3-35be-4c23-95e6-efc7d46583cd","Type":"ContainerStarted","Data":"b835a2c0f7b3c90e3593fab5b8274c6c5f0cced9ae7ffa0c162a52eb26d57eab"}
Nov 02 12:51:16.648531 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:16.648437    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d7801d-6ffffb74dc-prdc6" event={"ID":"c9479aa3-35be-4c23-95e6-efc7d46583cd","Type":"ContainerStarted","Data":"a5363013607a4b7d89876ff7b1329a2d80a182ff6693c8ebf17d44615cc1b998"}
Nov 02 12:51:16.648531 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:16.648482    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d7801d-6ffffb74dc-prdc6"
Nov 02 12:51:16.654777 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:16.654751    4559 scope.go:122] "RemoveContainer" containerID="41768043411c52609ed00008bc4c1517796cd57983acc45534761dacdf6cf1d0"
Nov 02 12:51:16.664190 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:16.664171    4559 scope.go:122] "RemoveContainer" containerID="7cfb9e4cf80abe147a36dfb196732604cd012e5a6b624875c3b2451d3f18a047"
Nov 02 12:51:16.664474 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:51:16.664452    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7cfb9e4cf80abe147a36dfb196732604cd012e5a6b624875c3b2451d3f18a047\": container with ID starting with 7cfb9e4cf80abe147a36dfb196732604cd012e5a6b624875c3b2451d3f18a047 not found: ID does not exist" containerID="7cfb9e4cf80abe147a36dfb196732604cd012e5a6b624875c3b2451d3f18a047"
Nov 02 12:51:16.664543 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:16.664488    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7cfb9e4cf80abe147a36dfb196732604cd012e5a6b624875c3b2451d3f18a047"} err="failed to get container status \"7cfb9e4cf80abe147a36dfb196732604cd012e5a6b624875c3b2451d3f18a047\": rpc error: code = NotFound desc = could not find container \"7cfb9e4cf80abe147a36dfb196732604cd012e5a6b624875c3b2451d3f18a047\": container with ID starting with 7cfb9e4cf80abe147a36dfb196732604cd012e5a6b624875c3b2451d3f18a047 not found: ID does not exist"
Nov 02 12:51:16.664543 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:16.664512    4559 scope.go:122] "RemoveContainer" containerID="41768043411c52609ed00008bc4c1517796cd57983acc45534761dacdf6cf1d0"
Nov 02 12:51:16.664810 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:51:16.664792    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41768043411c52609ed00008bc4c1517796cd57983acc45534761dacdf6cf1d0\": container with ID starting with 41768043411c52609ed00008bc4c1517796cd57983acc45534761dacdf6cf1d0 not found: ID does not exist" containerID="41768043411c52609ed00008bc4c1517796cd57983acc45534761dacdf6cf1d0"
Nov 02 12:51:16.664863 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:16.664817    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41768043411c52609ed00008bc4c1517796cd57983acc45534761dacdf6cf1d0"} err="failed to get container status \"41768043411c52609ed00008bc4c1517796cd57983acc45534761dacdf6cf1d0\": rpc error: code = NotFound desc = could not find container \"41768043411c52609ed00008bc4c1517796cd57983acc45534761dacdf6cf1d0\": container with ID starting with 41768043411c52609ed00008bc4c1517796cd57983acc45534761dacdf6cf1d0 not found: ID does not exist"
Nov 02 12:51:16.716020 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:16.715973    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d7801d-6ffffb74dc-prdc6" podStartSLOduration=2.286870009 podStartE2EDuration="2.71595372s" totalImagesPullingTime="429.083711ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:51:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:51:16.713711041 +0000 UTC m=+1472.969110136" watchObservedRunningTime="2026-11-02 12:51:16.71595372 +0000 UTC m=+1472.971352816"
Nov 02 12:51:16.759073 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:16.759036    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-d6d314-openshift-default-5b99b69565-2kkgt"]
Nov 02 12:51:16.763469 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:16.763446    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-d6d314-openshift-default-5b99b69565-2kkgt"]
Nov 02 12:51:17.767649 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:17.767600    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-ng469"]
Nov 02 12:51:17.768125 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:17.767827    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-ng469" podUID="4991a41e-4031-4dec-b038-7a10b5da7ac5" containerName="istio-proxy" containerID="cri-o://f16fb49d3e7d64d9351e4fb622963b0c0540e3032bb6967ba10c7af8d5ed6ddb" gracePeriod=30
Nov 02 12:51:17.894530 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:17.894500    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ac6508c-24db-40f7-a14c-0964dac7768b" path="/var/lib/kubelet/pods/4ac6508c-24db-40f7-a14c-0964dac7768b/volumes"
Nov 02 12:51:18.406393 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:18.406362    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-d3a281-844b5468d7-hvsnh"]
Nov 02 12:51:22.661609 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:22.661581    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d7801d-6ffffb74dc-prdc6"
Nov 02 12:51:23.053602 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:23.053574    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-ng469"
Nov 02 12:51:23.162405 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:23.162370    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4991a41e-4031-4dec-b038-7a10b5da7ac5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4991a41e-4031-4dec-b038-7a10b5da7ac5-istio-envoy\") pod \"4991a41e-4031-4dec-b038-7a10b5da7ac5\" (UID: \"4991a41e-4031-4dec-b038-7a10b5da7ac5\") "
Nov 02 12:51:23.162571 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:23.162524    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4991a41e-4031-4dec-b038-7a10b5da7ac5-istio-token\" (UniqueName: \"kubernetes.io/projected/4991a41e-4031-4dec-b038-7a10b5da7ac5-istio-token\") pod \"4991a41e-4031-4dec-b038-7a10b5da7ac5\" (UID: \"4991a41e-4031-4dec-b038-7a10b5da7ac5\") "
Nov 02 12:51:23.162615 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:23.162595    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4991a41e-4031-4dec-b038-7a10b5da7ac5-kube-api-access-ssvvp\" (UniqueName: \"kubernetes.io/projected/4991a41e-4031-4dec-b038-7a10b5da7ac5-kube-api-access-ssvvp\") pod \"4991a41e-4031-4dec-b038-7a10b5da7ac5\" (UID: \"4991a41e-4031-4dec-b038-7a10b5da7ac5\") "
Nov 02 12:51:23.162680 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:23.162649    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4991a41e-4031-4dec-b038-7a10b5da7ac5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4991a41e-4031-4dec-b038-7a10b5da7ac5-workload-certs\") pod \"4991a41e-4031-4dec-b038-7a10b5da7ac5\" (UID: \"4991a41e-4031-4dec-b038-7a10b5da7ac5\") "
Nov 02 12:51:23.162738 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:23.162711    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4991a41e-4031-4dec-b038-7a10b5da7ac5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4991a41e-4031-4dec-b038-7a10b5da7ac5-istio-podinfo\") pod \"4991a41e-4031-4dec-b038-7a10b5da7ac5\" (UID: \"4991a41e-4031-4dec-b038-7a10b5da7ac5\") "
Nov 02 12:51:23.162788 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:23.162760    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4991a41e-4031-4dec-b038-7a10b5da7ac5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4991a41e-4031-4dec-b038-7a10b5da7ac5-workload-socket\") pod \"4991a41e-4031-4dec-b038-7a10b5da7ac5\" (UID: \"4991a41e-4031-4dec-b038-7a10b5da7ac5\") "
Nov 02 12:51:23.162849 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:23.162797    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4991a41e-4031-4dec-b038-7a10b5da7ac5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4991a41e-4031-4dec-b038-7a10b5da7ac5-istiod-ca-cert\") pod \"4991a41e-4031-4dec-b038-7a10b5da7ac5\" (UID: \"4991a41e-4031-4dec-b038-7a10b5da7ac5\") "
Nov 02 12:51:23.162967 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:23.162878    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4991a41e-4031-4dec-b038-7a10b5da7ac5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4991a41e-4031-4dec-b038-7a10b5da7ac5-istio-data\") pod \"4991a41e-4031-4dec-b038-7a10b5da7ac5\" (UID: \"4991a41e-4031-4dec-b038-7a10b5da7ac5\") "
Nov 02 12:51:23.162967 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:23.162936    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4991a41e-4031-4dec-b038-7a10b5da7ac5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4991a41e-4031-4dec-b038-7a10b5da7ac5-credential-socket\") pod \"4991a41e-4031-4dec-b038-7a10b5da7ac5\" (UID: \"4991a41e-4031-4dec-b038-7a10b5da7ac5\") "
Nov 02 12:51:23.163078 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:23.162983    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4991a41e-4031-4dec-b038-7a10b5da7ac5-workload-certs" pod "4991a41e-4031-4dec-b038-7a10b5da7ac5" (UID: "4991a41e-4031-4dec-b038-7a10b5da7ac5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:51:23.163648 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:23.163369    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4991a41e-4031-4dec-b038-7a10b5da7ac5-istiod-ca-cert" pod "4991a41e-4031-4dec-b038-7a10b5da7ac5" (UID: "4991a41e-4031-4dec-b038-7a10b5da7ac5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:51:23.163648 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:23.163487    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4991a41e-4031-4dec-b038-7a10b5da7ac5-istio-data" pod "4991a41e-4031-4dec-b038-7a10b5da7ac5" (UID: "4991a41e-4031-4dec-b038-7a10b5da7ac5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:51:23.163648 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:23.163573    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4991a41e-4031-4dec-b038-7a10b5da7ac5-workload-socket" pod "4991a41e-4031-4dec-b038-7a10b5da7ac5" (UID: "4991a41e-4031-4dec-b038-7a10b5da7ac5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:51:23.163648 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:23.163594    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4991a41e-4031-4dec-b038-7a10b5da7ac5-credential-socket" pod "4991a41e-4031-4dec-b038-7a10b5da7ac5" (UID: "4991a41e-4031-4dec-b038-7a10b5da7ac5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:51:23.163648 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:23.163620    4559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4991a41e-4031-4dec-b038-7a10b5da7ac5-workload-certs\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:51:23.163899 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:23.163657    4559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4991a41e-4031-4dec-b038-7a10b5da7ac5-istiod-ca-cert\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:51:23.163899 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:23.163671    4559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4991a41e-4031-4dec-b038-7a10b5da7ac5-istio-data\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:51:23.164892 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:23.164861    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4991a41e-4031-4dec-b038-7a10b5da7ac5-istio-token" pod "4991a41e-4031-4dec-b038-7a10b5da7ac5" (UID: "4991a41e-4031-4dec-b038-7a10b5da7ac5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:51:23.165076 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:23.165049    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4991a41e-4031-4dec-b038-7a10b5da7ac5-kube-api-access-ssvvp" pod "4991a41e-4031-4dec-b038-7a10b5da7ac5" (UID: "4991a41e-4031-4dec-b038-7a10b5da7ac5"). InnerVolumeSpecName "kube-api-access-ssvvp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:51:23.165179 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:23.165105    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4991a41e-4031-4dec-b038-7a10b5da7ac5-istio-envoy" pod "4991a41e-4031-4dec-b038-7a10b5da7ac5" (UID: "4991a41e-4031-4dec-b038-7a10b5da7ac5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:51:23.165324 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:23.165303    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4991a41e-4031-4dec-b038-7a10b5da7ac5-istio-podinfo" pod "4991a41e-4031-4dec-b038-7a10b5da7ac5" (UID: "4991a41e-4031-4dec-b038-7a10b5da7ac5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 12:51:23.264561 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:23.264483    4559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4991a41e-4031-4dec-b038-7a10b5da7ac5-credential-socket\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:51:23.264561 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:23.264510    4559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4991a41e-4031-4dec-b038-7a10b5da7ac5-istio-envoy\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:51:23.264561 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:23.264523    4559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4991a41e-4031-4dec-b038-7a10b5da7ac5-istio-token\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:51:23.264561 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:23.264535    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ssvvp\" (UniqueName: \"kubernetes.io/projected/4991a41e-4031-4dec-b038-7a10b5da7ac5-kube-api-access-ssvvp\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:51:23.264561 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:23.264548    4559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4991a41e-4031-4dec-b038-7a10b5da7ac5-istio-podinfo\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:51:23.264561 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:23.264557    4559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4991a41e-4031-4dec-b038-7a10b5da7ac5-workload-socket\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:51:23.407418 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:23.407374    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-d3a281-844b5468d7-hvsnh" podUID="19ffc8e8-e659-4879-b936-29bd33dfc5a9" containerName="payload-processing" containerID="cri-o://33673c7c6198e753102c1109a80d842d17201f57b4508af3fc7b708c54654279" gracePeriod=25
Nov 02 12:51:23.680789 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:23.680761    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-d3a281-844b5468d7-hvsnh"
Nov 02 12:51:23.731829 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:23.731795    4559 generic.go:356] "Generic (PLEG): container finished" podID="4991a41e-4031-4dec-b038-7a10b5da7ac5" containerID="f16fb49d3e7d64d9351e4fb622963b0c0540e3032bb6967ba10c7af8d5ed6ddb" exitCode=0
Nov 02 12:51:23.732057 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:23.731843    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-ng469"
Nov 02 12:51:23.732057 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:23.731880    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-ng469" event={"ID":"4991a41e-4031-4dec-b038-7a10b5da7ac5","Type":"ContainerDied","Data":"f16fb49d3e7d64d9351e4fb622963b0c0540e3032bb6967ba10c7af8d5ed6ddb"}
Nov 02 12:51:23.732057 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:23.731941    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-ng469" event={"ID":"4991a41e-4031-4dec-b038-7a10b5da7ac5","Type":"ContainerDied","Data":"074afa5afc393ed865400dfd05bd84e3b1ca0c688e26791746f62deefc953ca3"}
Nov 02 12:51:23.732057 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:23.731971    4559 scope.go:122] "RemoveContainer" containerID="f16fb49d3e7d64d9351e4fb622963b0c0540e3032bb6967ba10c7af8d5ed6ddb"
Nov 02 12:51:23.736688 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:23.733807    4559 generic.go:356] "Generic (PLEG): container finished" podID="19ffc8e8-e659-4879-b936-29bd33dfc5a9" containerID="33673c7c6198e753102c1109a80d842d17201f57b4508af3fc7b708c54654279" exitCode=0
Nov 02 12:51:23.736688 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:23.733841    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-d3a281-844b5468d7-hvsnh" event={"ID":"19ffc8e8-e659-4879-b936-29bd33dfc5a9","Type":"ContainerDied","Data":"33673c7c6198e753102c1109a80d842d17201f57b4508af3fc7b708c54654279"}
Nov 02 12:51:23.736688 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:23.733902    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-d3a281-844b5468d7-hvsnh" event={"ID":"19ffc8e8-e659-4879-b936-29bd33dfc5a9","Type":"ContainerDied","Data":"a1564eb0e74a1fd8fb7cfb744aad5a34a8961cdb030c775f26ffbcce4756c3ed"}
Nov 02 12:51:23.736688 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:23.733999    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-d3a281-844b5468d7-hvsnh"
Nov 02 12:51:23.746918 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:23.746899    4559 scope.go:122] "RemoveContainer" containerID="f16fb49d3e7d64d9351e4fb622963b0c0540e3032bb6967ba10c7af8d5ed6ddb"
Nov 02 12:51:23.747225 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:51:23.747205    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f16fb49d3e7d64d9351e4fb622963b0c0540e3032bb6967ba10c7af8d5ed6ddb\": container with ID starting with f16fb49d3e7d64d9351e4fb622963b0c0540e3032bb6967ba10c7af8d5ed6ddb not found: ID does not exist" containerID="f16fb49d3e7d64d9351e4fb622963b0c0540e3032bb6967ba10c7af8d5ed6ddb"
Nov 02 12:51:23.747297 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:23.747233    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f16fb49d3e7d64d9351e4fb622963b0c0540e3032bb6967ba10c7af8d5ed6ddb"} err="failed to get container status \"f16fb49d3e7d64d9351e4fb622963b0c0540e3032bb6967ba10c7af8d5ed6ddb\": rpc error: code = NotFound desc = could not find container \"f16fb49d3e7d64d9351e4fb622963b0c0540e3032bb6967ba10c7af8d5ed6ddb\": container with ID starting with f16fb49d3e7d64d9351e4fb622963b0c0540e3032bb6967ba10c7af8d5ed6ddb not found: ID does not exist"
Nov 02 12:51:23.747297 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:23.747251    4559 scope.go:122] "RemoveContainer" containerID="33673c7c6198e753102c1109a80d842d17201f57b4508af3fc7b708c54654279"
Nov 02 12:51:23.757745 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:23.757724    4559 scope.go:122] "RemoveContainer" containerID="33673c7c6198e753102c1109a80d842d17201f57b4508af3fc7b708c54654279"
Nov 02 12:51:23.758025 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:51:23.758000    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33673c7c6198e753102c1109a80d842d17201f57b4508af3fc7b708c54654279\": container with ID starting with 33673c7c6198e753102c1109a80d842d17201f57b4508af3fc7b708c54654279 not found: ID does not exist" containerID="33673c7c6198e753102c1109a80d842d17201f57b4508af3fc7b708c54654279"
Nov 02 12:51:23.758117 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:23.758033    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33673c7c6198e753102c1109a80d842d17201f57b4508af3fc7b708c54654279"} err="failed to get container status \"33673c7c6198e753102c1109a80d842d17201f57b4508af3fc7b708c54654279\": rpc error: code = NotFound desc = could not find container \"33673c7c6198e753102c1109a80d842d17201f57b4508af3fc7b708c54654279\": container with ID starting with 33673c7c6198e753102c1109a80d842d17201f57b4508af3fc7b708c54654279 not found: ID does not exist"
Nov 02 12:51:23.771980 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:23.771954    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19ffc8e8-e659-4879-b936-29bd33dfc5a9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19ffc8e8-e659-4879-b936-29bd33dfc5a9-plugins-config-volume\") pod \"19ffc8e8-e659-4879-b936-29bd33dfc5a9\" (UID: \"19ffc8e8-e659-4879-b936-29bd33dfc5a9\") "
Nov 02 12:51:23.772156 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:23.772012    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19ffc8e8-e659-4879-b936-29bd33dfc5a9-kube-api-access-njp6m\" (UniqueName: \"kubernetes.io/projected/19ffc8e8-e659-4879-b936-29bd33dfc5a9-kube-api-access-njp6m\") pod \"19ffc8e8-e659-4879-b936-29bd33dfc5a9\" (UID: \"19ffc8e8-e659-4879-b936-29bd33dfc5a9\") "
Nov 02 12:51:23.772416 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:23.772390    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19ffc8e8-e659-4879-b936-29bd33dfc5a9-plugins-config-volume" pod "19ffc8e8-e659-4879-b936-29bd33dfc5a9" (UID: "19ffc8e8-e659-4879-b936-29bd33dfc5a9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:51:23.772624 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:23.772604    4559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19ffc8e8-e659-4879-b936-29bd33dfc5a9-plugins-config-volume\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:51:23.774970 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:23.774933    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19ffc8e8-e659-4879-b936-29bd33dfc5a9-kube-api-access-njp6m" pod "19ffc8e8-e659-4879-b936-29bd33dfc5a9" (UID: "19ffc8e8-e659-4879-b936-29bd33dfc5a9"). InnerVolumeSpecName "kube-api-access-njp6m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:51:23.805779 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:23.805751    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-ng469"]
Nov 02 12:51:23.816064 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:23.816040    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-ng469"]
Nov 02 12:51:23.873562 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:23.873535    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-njp6m\" (UniqueName: \"kubernetes.io/projected/19ffc8e8-e659-4879-b936-29bd33dfc5a9-kube-api-access-njp6m\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:51:23.893193 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:23.893165    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4991a41e-4031-4dec-b038-7a10b5da7ac5" path="/var/lib/kubelet/pods/4991a41e-4031-4dec-b038-7a10b5da7ac5/volumes"
Nov 02 12:51:24.076490 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:24.076413    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-d3a281-844b5468d7-hvsnh"]
Nov 02 12:51:24.080403 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:24.080377    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-d3a281-844b5468d7-hvsnh"]
Nov 02 12:51:25.218034 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:25.217942    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-b271f0-openshift-default-5544549cdf-q9c6j"]
Nov 02 12:51:25.218797 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:25.218761    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-b271f0-openshift-default-5544549cdf-q9c6j" podUID="e5435e4b-b8a8-45b9-a90a-bca4b5a65506" containerName="istio-proxy" containerID="cri-o://860a818cf8db5e8a6f7f52c2740074327c73bf66f4c85bb806b694f683003b7a" gracePeriod=30
Nov 02 12:51:25.893501 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:25.893470    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19ffc8e8-e659-4879-b936-29bd33dfc5a9" path="/var/lib/kubelet/pods/19ffc8e8-e659-4879-b936-29bd33dfc5a9/volumes"
Nov 02 12:51:28.431665 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:28.431105    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-wmzzn"]
Nov 02 12:51:28.432193 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:28.432060    4559 cpu_manager.go:414] "containerMap: removing container" podUID="4ac6508c-24db-40f7-a14c-0964dac7768b" containerName="istio-proxy"
Nov 02 12:51:28.432193 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:28.432082    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ac6508c-24db-40f7-a14c-0964dac7768b" containerName="istio-proxy"
Nov 02 12:51:28.432193 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:28.432122    4559 cpu_manager.go:414] "containerMap: removing container" podUID="19ffc8e8-e659-4879-b936-29bd33dfc5a9" containerName="payload-processing"
Nov 02 12:51:28.432193 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:28.432131    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19ffc8e8-e659-4879-b936-29bd33dfc5a9" containerName="payload-processing"
Nov 02 12:51:28.432193 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:28.432165    4559 cpu_manager.go:414] "containerMap: removing container" podUID="4991a41e-4031-4dec-b038-7a10b5da7ac5" containerName="istio-proxy"
Nov 02 12:51:28.432193 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:28.432172    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4991a41e-4031-4dec-b038-7a10b5da7ac5" containerName="istio-proxy"
Nov 02 12:51:28.432544 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:28.432339    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ac6508c-24db-40f7-a14c-0964dac7768b" containerName="istio-proxy"
Nov 02 12:51:28.432544 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:28.432353    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="4991a41e-4031-4dec-b038-7a10b5da7ac5" containerName="istio-proxy"
Nov 02 12:51:28.432544 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:28.432376    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="19ffc8e8-e659-4879-b936-29bd33dfc5a9" containerName="payload-processing"
Nov 02 12:51:28.456665 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:28.452692    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-wmzzn"
Nov 02 12:51:28.471675 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:28.469674    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-wmzzn"]
Nov 02 12:51:28.557223 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:28.554123    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1722c3b-b864-43f0-b5cb-94d96526858a-istio-podinfo\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-wmzzn\" (UID: \"e1722c3b-b864-43f0-b5cb-94d96526858a\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-wmzzn"
Nov 02 12:51:28.557223 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:28.554169    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1722c3b-b864-43f0-b5cb-94d96526858a-workload-certs\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-wmzzn\" (UID: \"e1722c3b-b864-43f0-b5cb-94d96526858a\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-wmzzn"
Nov 02 12:51:28.557223 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:28.554219    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1722c3b-b864-43f0-b5cb-94d96526858a-istio-envoy\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-wmzzn\" (UID: \"e1722c3b-b864-43f0-b5cb-94d96526858a\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-wmzzn"
Nov 02 12:51:28.557223 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:28.554245    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1722c3b-b864-43f0-b5cb-94d96526858a-istiod-ca-cert\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-wmzzn\" (UID: \"e1722c3b-b864-43f0-b5cb-94d96526858a\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-wmzzn"
Nov 02 12:51:28.557223 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:28.554291    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1722c3b-b864-43f0-b5cb-94d96526858a-istio-token\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-wmzzn\" (UID: \"e1722c3b-b864-43f0-b5cb-94d96526858a\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-wmzzn"
Nov 02 12:51:28.570699 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:28.569026    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1722c3b-b864-43f0-b5cb-94d96526858a-workload-socket\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-wmzzn\" (UID: \"e1722c3b-b864-43f0-b5cb-94d96526858a\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-wmzzn"
Nov 02 12:51:28.570699 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:28.569078    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1722c3b-b864-43f0-b5cb-94d96526858a-credential-socket\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-wmzzn\" (UID: \"e1722c3b-b864-43f0-b5cb-94d96526858a\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-wmzzn"
Nov 02 12:51:28.570699 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:28.569130    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1722c3b-b864-43f0-b5cb-94d96526858a-istio-data\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-wmzzn\" (UID: \"e1722c3b-b864-43f0-b5cb-94d96526858a\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-wmzzn"
Nov 02 12:51:28.575413 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:28.575377    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5m4c4\" (UniqueName: \"kubernetes.io/projected/e1722c3b-b864-43f0-b5cb-94d96526858a-kube-api-access-5m4c4\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-wmzzn\" (UID: \"e1722c3b-b864-43f0-b5cb-94d96526858a\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-wmzzn"
Nov 02 12:51:28.711717 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:28.710832    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1722c3b-b864-43f0-b5cb-94d96526858a-istio-envoy\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-wmzzn\" (UID: \"e1722c3b-b864-43f0-b5cb-94d96526858a\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-wmzzn"
Nov 02 12:51:28.711717 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:28.710896    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1722c3b-b864-43f0-b5cb-94d96526858a-istiod-ca-cert\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-wmzzn\" (UID: \"e1722c3b-b864-43f0-b5cb-94d96526858a\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-wmzzn"
Nov 02 12:51:28.711717 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:28.710944    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1722c3b-b864-43f0-b5cb-94d96526858a-istio-token\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-wmzzn\" (UID: \"e1722c3b-b864-43f0-b5cb-94d96526858a\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-wmzzn"
Nov 02 12:51:28.711717 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:28.711158    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1722c3b-b864-43f0-b5cb-94d96526858a-workload-socket\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-wmzzn\" (UID: \"e1722c3b-b864-43f0-b5cb-94d96526858a\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-wmzzn"
Nov 02 12:51:28.711717 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:28.711189    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1722c3b-b864-43f0-b5cb-94d96526858a-credential-socket\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-wmzzn\" (UID: \"e1722c3b-b864-43f0-b5cb-94d96526858a\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-wmzzn"
Nov 02 12:51:28.711717 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:28.711235    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1722c3b-b864-43f0-b5cb-94d96526858a-istio-data\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-wmzzn\" (UID: \"e1722c3b-b864-43f0-b5cb-94d96526858a\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-wmzzn"
Nov 02 12:51:28.711717 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:28.711287    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5m4c4\" (UniqueName: \"kubernetes.io/projected/e1722c3b-b864-43f0-b5cb-94d96526858a-kube-api-access-5m4c4\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-wmzzn\" (UID: \"e1722c3b-b864-43f0-b5cb-94d96526858a\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-wmzzn"
Nov 02 12:51:28.711717 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:28.711326    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1722c3b-b864-43f0-b5cb-94d96526858a-istio-podinfo\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-wmzzn\" (UID: \"e1722c3b-b864-43f0-b5cb-94d96526858a\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-wmzzn"
Nov 02 12:51:28.711717 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:28.711352    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1722c3b-b864-43f0-b5cb-94d96526858a-workload-certs\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-wmzzn\" (UID: \"e1722c3b-b864-43f0-b5cb-94d96526858a\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-wmzzn"
Nov 02 12:51:28.740699 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:28.733999    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1722c3b-b864-43f0-b5cb-94d96526858a-workload-certs\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-wmzzn\" (UID: \"e1722c3b-b864-43f0-b5cb-94d96526858a\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-wmzzn"
Nov 02 12:51:28.740699 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:28.735019    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1722c3b-b864-43f0-b5cb-94d96526858a-credential-socket\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-wmzzn\" (UID: \"e1722c3b-b864-43f0-b5cb-94d96526858a\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-wmzzn"
Nov 02 12:51:28.740699 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:28.735715    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1722c3b-b864-43f0-b5cb-94d96526858a-istiod-ca-cert\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-wmzzn\" (UID: \"e1722c3b-b864-43f0-b5cb-94d96526858a\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-wmzzn"
Nov 02 12:51:28.740699 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:28.736262    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1722c3b-b864-43f0-b5cb-94d96526858a-workload-socket\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-wmzzn\" (UID: \"e1722c3b-b864-43f0-b5cb-94d96526858a\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-wmzzn"
Nov 02 12:51:28.740699 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:28.736776    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1722c3b-b864-43f0-b5cb-94d96526858a-istio-data\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-wmzzn\" (UID: \"e1722c3b-b864-43f0-b5cb-94d96526858a\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-wmzzn"
Nov 02 12:51:28.771521 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:28.771176    4559 patch_prober.go:29] interesting pod/e2e-praxis-a-b271f0-openshift-default-5544549cdf-q9c6j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 12:51:28.771521 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:28.771228    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-b271f0-openshift-default-5544549cdf-q9c6j" podUID="e5435e4b-b8a8-45b9-a90a-bca4b5a65506" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 12:51:28.818908 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:28.791133    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1722c3b-b864-43f0-b5cb-94d96526858a-istio-envoy\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-wmzzn\" (UID: \"e1722c3b-b864-43f0-b5cb-94d96526858a\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-wmzzn"
Nov 02 12:51:28.818908 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:28.798229    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1722c3b-b864-43f0-b5cb-94d96526858a-istio-token\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-wmzzn\" (UID: \"e1722c3b-b864-43f0-b5cb-94d96526858a\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-wmzzn"
Nov 02 12:51:28.844667 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:28.832328    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1722c3b-b864-43f0-b5cb-94d96526858a-istio-podinfo\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-wmzzn\" (UID: \"e1722c3b-b864-43f0-b5cb-94d96526858a\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-wmzzn"
Nov 02 12:51:28.851662 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:28.846444    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5m4c4\" (UniqueName: \"kubernetes.io/projected/e1722c3b-b864-43f0-b5cb-94d96526858a-kube-api-access-5m4c4\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-wmzzn\" (UID: \"e1722c3b-b864-43f0-b5cb-94d96526858a\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-wmzzn"
Nov 02 12:51:28.878591 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:28.878541    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-wmzzn"
Nov 02 12:51:29.369046 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:51:29.369013    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1722c3b_b864_43f0_b5cb_94d96526858a.slice/crio-7a9cb93e1f80ca38158863dfe60e1224ab8dab5a0b3affadca87b7419d682bec WatchSource:0}: Error finding container 7a9cb93e1f80ca38158863dfe60e1224ab8dab5a0b3affadca87b7419d682bec: Status 404 returned error can't find the container with id 7a9cb93e1f80ca38158863dfe60e1224ab8dab5a0b3affadca87b7419d682bec
Nov 02 12:51:29.369666 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:29.369625    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-wmzzn"]
Nov 02 12:51:29.371832 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:29.371797    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:51:29.371949 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:29.371879    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:51:29.371949 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:29.371925    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:51:29.827460 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:29.827421    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-wmzzn" event={"ID":"e1722c3b-b864-43f0-b5cb-94d96526858a","Type":"ContainerStarted","Data":"dde63fb43d91e0b37562756a515e89239f38a5241c38c45719959620fd8b6186"}
Nov 02 12:51:29.827940 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:29.827483    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-wmzzn" event={"ID":"e1722c3b-b864-43f0-b5cb-94d96526858a","Type":"ContainerStarted","Data":"7a9cb93e1f80ca38158863dfe60e1224ab8dab5a0b3affadca87b7419d682bec"}
Nov 02 12:51:29.879350 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:29.879316    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-wmzzn"
Nov 02 12:51:29.919269 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:29.919212    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-wmzzn" podStartSLOduration=1.919193431 podStartE2EDuration="1.919193431s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:51:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:51:29.91867059 +0000 UTC m=+1486.174069682" watchObservedRunningTime="2026-11-02 12:51:29.919193431 +0000 UTC m=+1486.174592527"
Nov 02 12:51:30.748716 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:30.748175    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-b271f0-openshift-default-5544549cdf-q9c6j"
Nov 02 12:51:30.848147 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:30.848110    4559 generic.go:356] "Generic (PLEG): container finished" podID="e5435e4b-b8a8-45b9-a90a-bca4b5a65506" containerID="860a818cf8db5e8a6f7f52c2740074327c73bf66f4c85bb806b694f683003b7a" exitCode=0
Nov 02 12:51:30.849126 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:30.848755    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-b271f0-openshift-default-5544549cdf-q9c6j"
Nov 02 12:51:30.849126 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:30.848819    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-b271f0-openshift-default-5544549cdf-q9c6j" event={"ID":"e5435e4b-b8a8-45b9-a90a-bca4b5a65506","Type":"ContainerDied","Data":"860a818cf8db5e8a6f7f52c2740074327c73bf66f4c85bb806b694f683003b7a"}
Nov 02 12:51:30.849126 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:30.848884    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-b271f0-openshift-default-5544549cdf-q9c6j" event={"ID":"e5435e4b-b8a8-45b9-a90a-bca4b5a65506","Type":"ContainerDied","Data":"e58195cf96c3b3195e8ca0b8f8e7fa291484081a895f1b316b37dc5f9d435950"}
Nov 02 12:51:30.849126 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:30.848954    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e5435e4b-b8a8-45b9-a90a-bca4b5a65506-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5435e4b-b8a8-45b9-a90a-bca4b5a65506-istio-podinfo\") pod \"e5435e4b-b8a8-45b9-a90a-bca4b5a65506\" (UID: \"e5435e4b-b8a8-45b9-a90a-bca4b5a65506\") "
Nov 02 12:51:30.849126 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:30.849004    4559 scope.go:122] "RemoveContainer" containerID="860a818cf8db5e8a6f7f52c2740074327c73bf66f4c85bb806b694f683003b7a"
Nov 02 12:51:30.849478 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:30.849435    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5435e4b-b8a8-45b9-a90a-bca4b5a65506-kube-api-access-htbqv\" (UniqueName: \"kubernetes.io/projected/e5435e4b-b8a8-45b9-a90a-bca4b5a65506-kube-api-access-htbqv\") pod \"e5435e4b-b8a8-45b9-a90a-bca4b5a65506\" (UID: \"e5435e4b-b8a8-45b9-a90a-bca4b5a65506\") "
Nov 02 12:51:30.849542 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:30.849496    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5435e4b-b8a8-45b9-a90a-bca4b5a65506-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5435e4b-b8a8-45b9-a90a-bca4b5a65506-istiod-ca-cert\") pod \"e5435e4b-b8a8-45b9-a90a-bca4b5a65506\" (UID: \"e5435e4b-b8a8-45b9-a90a-bca4b5a65506\") "
Nov 02 12:51:30.849599 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:30.849536    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5435e4b-b8a8-45b9-a90a-bca4b5a65506-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5435e4b-b8a8-45b9-a90a-bca4b5a65506-workload-socket\") pod \"e5435e4b-b8a8-45b9-a90a-bca4b5a65506\" (UID: \"e5435e4b-b8a8-45b9-a90a-bca4b5a65506\") "
Nov 02 12:51:30.849599 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:30.849571    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5435e4b-b8a8-45b9-a90a-bca4b5a65506-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5435e4b-b8a8-45b9-a90a-bca4b5a65506-workload-certs\") pod \"e5435e4b-b8a8-45b9-a90a-bca4b5a65506\" (UID: \"e5435e4b-b8a8-45b9-a90a-bca4b5a65506\") "
Nov 02 12:51:30.849933 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:30.849745    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5435e4b-b8a8-45b9-a90a-bca4b5a65506-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5435e4b-b8a8-45b9-a90a-bca4b5a65506-istio-envoy\") pod \"e5435e4b-b8a8-45b9-a90a-bca4b5a65506\" (UID: \"e5435e4b-b8a8-45b9-a90a-bca4b5a65506\") "
Nov 02 12:51:30.849933 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:30.849806    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5435e4b-b8a8-45b9-a90a-bca4b5a65506-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5435e4b-b8a8-45b9-a90a-bca4b5a65506-istio-data\") pod \"e5435e4b-b8a8-45b9-a90a-bca4b5a65506\" (UID: \"e5435e4b-b8a8-45b9-a90a-bca4b5a65506\") "
Nov 02 12:51:30.849933 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:30.849846    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5435e4b-b8a8-45b9-a90a-bca4b5a65506-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5435e4b-b8a8-45b9-a90a-bca4b5a65506-credential-socket\") pod \"e5435e4b-b8a8-45b9-a90a-bca4b5a65506\" (UID: \"e5435e4b-b8a8-45b9-a90a-bca4b5a65506\") "
Nov 02 12:51:30.850139 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:30.849958    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5435e4b-b8a8-45b9-a90a-bca4b5a65506-istio-token\" (UniqueName: \"kubernetes.io/projected/e5435e4b-b8a8-45b9-a90a-bca4b5a65506-istio-token\") pod \"e5435e4b-b8a8-45b9-a90a-bca4b5a65506\" (UID: \"e5435e4b-b8a8-45b9-a90a-bca4b5a65506\") "
Nov 02 12:51:30.850768 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:30.849955    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5435e4b-b8a8-45b9-a90a-bca4b5a65506-istiod-ca-cert" pod "e5435e4b-b8a8-45b9-a90a-bca4b5a65506" (UID: "e5435e4b-b8a8-45b9-a90a-bca4b5a65506"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:51:30.850768 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:30.850224    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5435e4b-b8a8-45b9-a90a-bca4b5a65506-credential-socket" pod "e5435e4b-b8a8-45b9-a90a-bca4b5a65506" (UID: "e5435e4b-b8a8-45b9-a90a-bca4b5a65506"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:51:30.850768 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:30.850221    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5435e4b-b8a8-45b9-a90a-bca4b5a65506-workload-certs" pod "e5435e4b-b8a8-45b9-a90a-bca4b5a65506" (UID: "e5435e4b-b8a8-45b9-a90a-bca4b5a65506"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:51:30.850768 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:30.850247    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5435e4b-b8a8-45b9-a90a-bca4b5a65506-workload-socket" pod "e5435e4b-b8a8-45b9-a90a-bca4b5a65506" (UID: "e5435e4b-b8a8-45b9-a90a-bca4b5a65506"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:51:30.850768 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:30.850408    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5435e4b-b8a8-45b9-a90a-bca4b5a65506-istio-data" pod "e5435e4b-b8a8-45b9-a90a-bca4b5a65506" (UID: "e5435e4b-b8a8-45b9-a90a-bca4b5a65506"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:51:30.850768 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:30.850487    4559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5435e4b-b8a8-45b9-a90a-bca4b5a65506-istio-data\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:51:30.850768 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:30.850510    4559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5435e4b-b8a8-45b9-a90a-bca4b5a65506-credential-socket\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:51:30.850768 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:30.850525    4559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5435e4b-b8a8-45b9-a90a-bca4b5a65506-istiod-ca-cert\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:51:30.850768 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:30.850538    4559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5435e4b-b8a8-45b9-a90a-bca4b5a65506-workload-socket\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:51:30.850768 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:30.850550    4559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5435e4b-b8a8-45b9-a90a-bca4b5a65506-workload-certs\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:51:30.852480 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:30.852457    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5435e4b-b8a8-45b9-a90a-bca4b5a65506-istio-envoy" pod "e5435e4b-b8a8-45b9-a90a-bca4b5a65506" (UID: "e5435e4b-b8a8-45b9-a90a-bca4b5a65506"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:51:30.853241 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:30.853219    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5435e4b-b8a8-45b9-a90a-bca4b5a65506-istio-token" pod "e5435e4b-b8a8-45b9-a90a-bca4b5a65506" (UID: "e5435e4b-b8a8-45b9-a90a-bca4b5a65506"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:51:30.853315 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:30.853282    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e5435e4b-b8a8-45b9-a90a-bca4b5a65506-istio-podinfo" pod "e5435e4b-b8a8-45b9-a90a-bca4b5a65506" (UID: "e5435e4b-b8a8-45b9-a90a-bca4b5a65506"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 12:51:30.853907 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:30.853887    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5435e4b-b8a8-45b9-a90a-bca4b5a65506-kube-api-access-htbqv" pod "e5435e4b-b8a8-45b9-a90a-bca4b5a65506" (UID: "e5435e4b-b8a8-45b9-a90a-bca4b5a65506"). InnerVolumeSpecName "kube-api-access-htbqv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:51:30.880657 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:30.880583    4559 patch_prober.go:29] interesting pod/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-wmzzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body=
Nov 02 12:51:30.880657 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:30.880595    4559 scope.go:122] "RemoveContainer" containerID="b79c65999f4b4d8153087bcfb6f1e7caf141c10c574e6be52b7157b40c272036"
Nov 02 12:51:30.880817 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:30.880655    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-wmzzn" podUID="e1722c3b-b864-43f0-b5cb-94d96526858a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused"
Nov 02 12:51:30.936539 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:30.936421    4559 scope.go:122] "RemoveContainer" containerID="860a818cf8db5e8a6f7f52c2740074327c73bf66f4c85bb806b694f683003b7a"
Nov 02 12:51:30.936990 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:51:30.936855    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"860a818cf8db5e8a6f7f52c2740074327c73bf66f4c85bb806b694f683003b7a\": container with ID starting with 860a818cf8db5e8a6f7f52c2740074327c73bf66f4c85bb806b694f683003b7a not found: ID does not exist" containerID="860a818cf8db5e8a6f7f52c2740074327c73bf66f4c85bb806b694f683003b7a"
Nov 02 12:51:30.936990 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:30.936891    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"860a818cf8db5e8a6f7f52c2740074327c73bf66f4c85bb806b694f683003b7a"} err="failed to get container status \"860a818cf8db5e8a6f7f52c2740074327c73bf66f4c85bb806b694f683003b7a\": rpc error: code = NotFound desc = could not find container \"860a818cf8db5e8a6f7f52c2740074327c73bf66f4c85bb806b694f683003b7a\": container with ID starting with 860a818cf8db5e8a6f7f52c2740074327c73bf66f4c85bb806b694f683003b7a not found: ID does not exist"
Nov 02 12:51:30.936990 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:30.936915    4559 scope.go:122] "RemoveContainer" containerID="b79c65999f4b4d8153087bcfb6f1e7caf141c10c574e6be52b7157b40c272036"
Nov 02 12:51:30.937246 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:51:30.937214    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b79c65999f4b4d8153087bcfb6f1e7caf141c10c574e6be52b7157b40c272036\": container with ID starting with b79c65999f4b4d8153087bcfb6f1e7caf141c10c574e6be52b7157b40c272036 not found: ID does not exist" containerID="b79c65999f4b4d8153087bcfb6f1e7caf141c10c574e6be52b7157b40c272036"
Nov 02 12:51:30.937309 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:30.937259    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b79c65999f4b4d8153087bcfb6f1e7caf141c10c574e6be52b7157b40c272036"} err="failed to get container status \"b79c65999f4b4d8153087bcfb6f1e7caf141c10c574e6be52b7157b40c272036\": rpc error: code = NotFound desc = could not find container \"b79c65999f4b4d8153087bcfb6f1e7caf141c10c574e6be52b7157b40c272036\": container with ID starting with b79c65999f4b4d8153087bcfb6f1e7caf141c10c574e6be52b7157b40c272036 not found: ID does not exist"
Nov 02 12:51:30.951927 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:30.951721    4559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5435e4b-b8a8-45b9-a90a-bca4b5a65506-istio-token\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:51:30.951927 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:30.951744    4559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5435e4b-b8a8-45b9-a90a-bca4b5a65506-istio-podinfo\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:51:30.951927 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:30.951753    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-htbqv\" (UniqueName: \"kubernetes.io/projected/e5435e4b-b8a8-45b9-a90a-bca4b5a65506-kube-api-access-htbqv\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:51:30.951927 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:30.951762    4559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5435e4b-b8a8-45b9-a90a-bca4b5a65506-istio-envoy\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:51:31.232726 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:31.231658    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-b271f0-openshift-default-5544549cdf-q9c6j"]
Nov 02 12:51:31.241399 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:31.241333    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-b271f0-openshift-default-5544549cdf-q9c6j"]
Nov 02 12:51:31.879693 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:31.879629    4559 patch_prober.go:29] interesting pod/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-wmzzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body=
Nov 02 12:51:31.880171 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:31.879709    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-wmzzn" podUID="e1722c3b-b864-43f0-b5cb-94d96526858a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused"
Nov 02 12:51:31.938824 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:31.938792    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5435e4b-b8a8-45b9-a90a-bca4b5a65506" path="/var/lib/kubelet/pods/e5435e4b-b8a8-45b9-a90a-bca4b5a65506/volumes"
Nov 02 12:51:32.879360 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:32.879323    4559 patch_prober.go:29] interesting pod/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-wmzzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body=
Nov 02 12:51:32.879514 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:32.879387    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-wmzzn" podUID="e1722c3b-b864-43f0-b5cb-94d96526858a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused"
Nov 02 12:51:33.555812 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:33.555775    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-9750c7-5df98f984c-mplnn"]
Nov 02 12:51:33.880240 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:33.880144    4559 patch_prober.go:29] interesting pod/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-wmzzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body=
Nov 02 12:51:33.880240 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:33.880207    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-wmzzn" podUID="e1722c3b-b864-43f0-b5cb-94d96526858a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused"
Nov 02 12:51:34.879379 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:34.879347    4559 patch_prober.go:29] interesting pod/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-wmzzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body=
Nov 02 12:51:34.879777 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:34.879397    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-wmzzn" podUID="e1722c3b-b864-43f0-b5cb-94d96526858a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused"
Nov 02 12:51:35.418894 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:35.418843    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-d3a281-6b8dcb978d-hcqxm"]
Nov 02 12:51:35.419157 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:35.419111    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d3a281-6b8dcb978d-hcqxm" podUID="e7f14aa1-1b9b-47ff-ac46-2db84191079b" containerName="maas-api" containerID="cri-o://81175c8b270bc8b000dfd7e74444dd2c44e7c29a7979bd7a11ee8c31591e9ac6" gracePeriod=30
Nov 02 12:51:35.954015 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:35.950066    4559 patch_prober.go:29] interesting pod/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-wmzzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body=
Nov 02 12:51:35.954015 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:35.950141    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-wmzzn" podUID="e1722c3b-b864-43f0-b5cb-94d96526858a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused"
Nov 02 12:51:36.034277 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:36.030880    4559 generic.go:356] "Generic (PLEG): container finished" podID="e7f14aa1-1b9b-47ff-ac46-2db84191079b" containerID="81175c8b270bc8b000dfd7e74444dd2c44e7c29a7979bd7a11ee8c31591e9ac6" exitCode=0
Nov 02 12:51:36.034277 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:36.033960    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d3a281-6b8dcb978d-hcqxm" event={"ID":"e7f14aa1-1b9b-47ff-ac46-2db84191079b","Type":"ContainerDied","Data":"81175c8b270bc8b000dfd7e74444dd2c44e7c29a7979bd7a11ee8c31591e9ac6"}
Nov 02 12:51:36.034277 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:36.034024    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d3a281-6b8dcb978d-hcqxm" event={"ID":"e7f14aa1-1b9b-47ff-ac46-2db84191079b","Type":"ContainerDied","Data":"9e5022fb05270c5e966d8da63955e0bc0b902d56919e592ef89d1d6ac5f1604a"}
Nov 02 12:51:36.034277 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:36.034050    4559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9e5022fb05270c5e966d8da63955e0bc0b902d56919e592ef89d1d6ac5f1604a"
Nov 02 12:51:36.036855 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:36.036819    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d3a281-6b8dcb978d-hcqxm"
Nov 02 12:51:36.081457 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:36.081358    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7f14aa1-1b9b-47ff-ac46-2db84191079b-kube-api-access-jpp6p\" (UniqueName: \"kubernetes.io/projected/e7f14aa1-1b9b-47ff-ac46-2db84191079b-kube-api-access-jpp6p\") pod \"e7f14aa1-1b9b-47ff-ac46-2db84191079b\" (UID: \"e7f14aa1-1b9b-47ff-ac46-2db84191079b\") "
Nov 02 12:51:36.081457 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:36.081449    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7f14aa1-1b9b-47ff-ac46-2db84191079b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7f14aa1-1b9b-47ff-ac46-2db84191079b-openshift-service-ca\") pod \"e7f14aa1-1b9b-47ff-ac46-2db84191079b\" (UID: \"e7f14aa1-1b9b-47ff-ac46-2db84191079b\") "
Nov 02 12:51:36.081729 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:36.081522    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e7f14aa1-1b9b-47ff-ac46-2db84191079b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7f14aa1-1b9b-47ff-ac46-2db84191079b-metrics-certs\") pod \"e7f14aa1-1b9b-47ff-ac46-2db84191079b\" (UID: \"e7f14aa1-1b9b-47ff-ac46-2db84191079b\") "
Nov 02 12:51:36.081784 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:36.081723    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e7f14aa1-1b9b-47ff-ac46-2db84191079b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e7f14aa1-1b9b-47ff-ac46-2db84191079b-maas-api-tls\") pod \"e7f14aa1-1b9b-47ff-ac46-2db84191079b\" (UID: \"e7f14aa1-1b9b-47ff-ac46-2db84191079b\") "
Nov 02 12:51:36.081882 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:36.081851    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7f14aa1-1b9b-47ff-ac46-2db84191079b-openshift-service-ca" pod "e7f14aa1-1b9b-47ff-ac46-2db84191079b" (UID: "e7f14aa1-1b9b-47ff-ac46-2db84191079b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:51:36.082321 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:36.082296    4559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7f14aa1-1b9b-47ff-ac46-2db84191079b-openshift-service-ca\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:51:36.083922 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:36.083896    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7f14aa1-1b9b-47ff-ac46-2db84191079b-kube-api-access-jpp6p" pod "e7f14aa1-1b9b-47ff-ac46-2db84191079b" (UID: "e7f14aa1-1b9b-47ff-ac46-2db84191079b"). InnerVolumeSpecName "kube-api-access-jpp6p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:51:36.084034 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:36.084010    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7f14aa1-1b9b-47ff-ac46-2db84191079b-maas-api-tls" pod "e7f14aa1-1b9b-47ff-ac46-2db84191079b" (UID: "e7f14aa1-1b9b-47ff-ac46-2db84191079b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:51:36.084702 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:36.084671    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7f14aa1-1b9b-47ff-ac46-2db84191079b-metrics-certs" pod "e7f14aa1-1b9b-47ff-ac46-2db84191079b" (UID: "e7f14aa1-1b9b-47ff-ac46-2db84191079b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:51:36.184098 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:36.184050    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e7f14aa1-1b9b-47ff-ac46-2db84191079b-maas-api-tls\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:51:36.184098 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:36.184094    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jpp6p\" (UniqueName: \"kubernetes.io/projected/e7f14aa1-1b9b-47ff-ac46-2db84191079b-kube-api-access-jpp6p\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:51:36.184098 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:36.184110    4559 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7f14aa1-1b9b-47ff-ac46-2db84191079b-metrics-certs\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:51:36.883905 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:36.883814    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-wmzzn"
Nov 02 12:51:36.884321 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:36.884292    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-wmzzn"
Nov 02 12:51:36.885314 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:36.885293    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-wmzzn"
Nov 02 12:51:37.044926 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:37.044889    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d3a281-6b8dcb978d-hcqxm"
Nov 02 12:51:37.100618 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:37.100586    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-d3a281-6b8dcb978d-hcqxm"]
Nov 02 12:51:37.112726 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:37.106888    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-d3a281-6b8dcb978d-hcqxm"]
Nov 02 12:51:37.895791 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:37.895749    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7f14aa1-1b9b-47ff-ac46-2db84191079b" path="/var/lib/kubelet/pods/e7f14aa1-1b9b-47ff-ac46-2db84191079b/volumes"
Nov 02 12:51:38.361717 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:38.360200    4559 patch_prober.go:29] interesting pod/e2e-ipp-b-d7801d-openshift-default-674cd67b5-xgb9d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": read tcp 10.133.0.2:58856->10.133.0.161:15021: read: connection reset by peer" start-of-body=
Nov 02 12:51:38.361717 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:38.360264    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-d7801d-openshift-default-674cd67b5-xgb9d" podUID="03b6b83c-10cd-4876-b17a-96120c2317b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": read tcp 10.133.0.2:58856->10.133.0.161:15021: read: connection reset by peer"
Nov 02 12:51:38.556631 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:38.556529    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-9750c7-5df98f984c-mplnn" podUID="19089e8c-1e56-4d5a-9697-3c3477bcd803" containerName="payload-processing" containerID="cri-o://47dbc90563b222dc0e8db6fc0241a7cdf0917bcf2ebfd47b358cc1a739fc0efd" gracePeriod=25
Nov 02 12:51:38.828804 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:38.828776    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-9750c7-5df98f984c-mplnn"
Nov 02 12:51:38.937536 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:38.937444    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19089e8c-1e56-4d5a-9697-3c3477bcd803-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19089e8c-1e56-4d5a-9697-3c3477bcd803-plugins-config-volume\") pod \"19089e8c-1e56-4d5a-9697-3c3477bcd803\" (UID: \"19089e8c-1e56-4d5a-9697-3c3477bcd803\") "
Nov 02 12:51:38.937745 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:38.937595    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19089e8c-1e56-4d5a-9697-3c3477bcd803-kube-api-access-tlnng\" (UniqueName: \"kubernetes.io/projected/19089e8c-1e56-4d5a-9697-3c3477bcd803-kube-api-access-tlnng\") pod \"19089e8c-1e56-4d5a-9697-3c3477bcd803\" (UID: \"19089e8c-1e56-4d5a-9697-3c3477bcd803\") "
Nov 02 12:51:38.937957 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:38.937925    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19089e8c-1e56-4d5a-9697-3c3477bcd803-plugins-config-volume" pod "19089e8c-1e56-4d5a-9697-3c3477bcd803" (UID: "19089e8c-1e56-4d5a-9697-3c3477bcd803"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:51:38.938085 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:38.938039    4559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19089e8c-1e56-4d5a-9697-3c3477bcd803-plugins-config-volume\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:51:38.940591 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:38.940563    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19089e8c-1e56-4d5a-9697-3c3477bcd803-kube-api-access-tlnng" pod "19089e8c-1e56-4d5a-9697-3c3477bcd803" (UID: "19089e8c-1e56-4d5a-9697-3c3477bcd803"). InnerVolumeSpecName "kube-api-access-tlnng". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:51:39.039993 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:39.039943    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tlnng\" (UniqueName: \"kubernetes.io/projected/19089e8c-1e56-4d5a-9697-3c3477bcd803-kube-api-access-tlnng\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:51:39.075121 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:39.075091    4559 generic.go:356] "Generic (PLEG): container finished" podID="03b6b83c-10cd-4876-b17a-96120c2317b7" containerID="6224548b7e582bcc650188ee8bfdd325f69efc976323a76a62849fbafd0ecc29" exitCode=0
Nov 02 12:51:39.075285 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:39.075162    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d7801d-openshift-default-674cd67b5-xgb9d" event={"ID":"03b6b83c-10cd-4876-b17a-96120c2317b7","Type":"ContainerDied","Data":"6224548b7e582bcc650188ee8bfdd325f69efc976323a76a62849fbafd0ecc29"}
Nov 02 12:51:39.075655 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:39.075623    4559 scope.go:122] "RemoveContainer" containerID="6224548b7e582bcc650188ee8bfdd325f69efc976323a76a62849fbafd0ecc29"
Nov 02 12:51:39.076757 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:39.076732    4559 generic.go:356] "Generic (PLEG): container finished" podID="19089e8c-1e56-4d5a-9697-3c3477bcd803" containerID="47dbc90563b222dc0e8db6fc0241a7cdf0917bcf2ebfd47b358cc1a739fc0efd" exitCode=0
Nov 02 12:51:39.076874 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:39.076791    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-9750c7-5df98f984c-mplnn"
Nov 02 12:51:39.076874 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:39.076810    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-9750c7-5df98f984c-mplnn" event={"ID":"19089e8c-1e56-4d5a-9697-3c3477bcd803","Type":"ContainerDied","Data":"47dbc90563b222dc0e8db6fc0241a7cdf0917bcf2ebfd47b358cc1a739fc0efd"}
Nov 02 12:51:39.076874 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:39.076864    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-9750c7-5df98f984c-mplnn" event={"ID":"19089e8c-1e56-4d5a-9697-3c3477bcd803","Type":"ContainerDied","Data":"3876c6a7b8caaa2c21f2fa94207721f6459d4318f822df26329d7d0180b96da0"}
Nov 02 12:51:39.076982 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:39.076887    4559 scope.go:122] "RemoveContainer" containerID="47dbc90563b222dc0e8db6fc0241a7cdf0917bcf2ebfd47b358cc1a739fc0efd"
Nov 02 12:51:39.077312 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:39.077278    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:51:39.077444 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:39.077386    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:51:39.077499 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:39.077441    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:51:39.091134 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:39.091110    4559 scope.go:122] "RemoveContainer" containerID="47dbc90563b222dc0e8db6fc0241a7cdf0917bcf2ebfd47b358cc1a739fc0efd"
Nov 02 12:51:39.091403 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:51:39.091380    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47dbc90563b222dc0e8db6fc0241a7cdf0917bcf2ebfd47b358cc1a739fc0efd\": container with ID starting with 47dbc90563b222dc0e8db6fc0241a7cdf0917bcf2ebfd47b358cc1a739fc0efd not found: ID does not exist" containerID="47dbc90563b222dc0e8db6fc0241a7cdf0917bcf2ebfd47b358cc1a739fc0efd"
Nov 02 12:51:39.091487 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:39.091413    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47dbc90563b222dc0e8db6fc0241a7cdf0917bcf2ebfd47b358cc1a739fc0efd"} err="failed to get container status \"47dbc90563b222dc0e8db6fc0241a7cdf0917bcf2ebfd47b358cc1a739fc0efd\": rpc error: code = NotFound desc = could not find container \"47dbc90563b222dc0e8db6fc0241a7cdf0917bcf2ebfd47b358cc1a739fc0efd\": container with ID starting with 47dbc90563b222dc0e8db6fc0241a7cdf0917bcf2ebfd47b358cc1a739fc0efd not found: ID does not exist"
Nov 02 12:51:39.164016 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:39.163980    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-9750c7-5df98f984c-mplnn"]
Nov 02 12:51:39.167776 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:39.167752    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-9750c7-5df98f984c-mplnn"]
Nov 02 12:51:39.587691 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:39.587604    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d7801d-c576f4dc9-2fddw"]
Nov 02 12:51:39.623893 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:39.623861    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d7801d-867c88bd6c-2h9kh"]
Nov 02 12:51:39.916467 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:39.916393    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19089e8c-1e56-4d5a-9697-3c3477bcd803" path="/var/lib/kubelet/pods/19089e8c-1e56-4d5a-9697-3c3477bcd803/volumes"
Nov 02 12:51:40.155018 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:40.154973    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d7801d-openshift-default-674cd67b5-xgb9d" event={"ID":"03b6b83c-10cd-4876-b17a-96120c2317b7","Type":"ContainerStarted","Data":"ee7f13ca3c8c1367922192eb56747411b40607e8ed457ce506472100e803ad05"}
Nov 02 12:51:40.439463 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:40.439417    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-d7801d-openshift-default-674cd67b5-xgb9d"
Nov 02 12:51:40.439892 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:40.439551    4559 patch_prober.go:29] interesting pod/e2e-ipp-b-d7801d-openshift-default-674cd67b5-xgb9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" start-of-body=
Nov 02 12:51:40.439892 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:40.439597    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d7801d-openshift-default-674cd67b5-xgb9d" podUID="03b6b83c-10cd-4876-b17a-96120c2317b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused"
Nov 02 12:51:41.440204 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:41.440165    4559 patch_prober.go:29] interesting pod/e2e-ipp-b-d7801d-openshift-default-674cd67b5-xgb9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" start-of-body=
Nov 02 12:51:41.440610 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:41.440222    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d7801d-openshift-default-674cd67b5-xgb9d" podUID="03b6b83c-10cd-4876-b17a-96120c2317b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused"
Nov 02 12:51:42.457457 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:42.457419    4559 patch_prober.go:29] interesting pod/e2e-ipp-b-d7801d-openshift-default-674cd67b5-xgb9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" start-of-body=
Nov 02 12:51:42.458329 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:42.458295    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d7801d-openshift-default-674cd67b5-xgb9d" podUID="03b6b83c-10cd-4876-b17a-96120c2317b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused"
Nov 02 12:51:43.442653 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:43.442605    4559 patch_prober.go:29] interesting pod/e2e-ipp-b-d7801d-openshift-default-674cd67b5-xgb9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" start-of-body=
Nov 02 12:51:43.442975 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:43.442929    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d7801d-openshift-default-674cd67b5-xgb9d" podUID="03b6b83c-10cd-4876-b17a-96120c2317b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused"
Nov 02 12:51:44.440178 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:44.440137    4559 patch_prober.go:29] interesting pod/e2e-ipp-b-d7801d-openshift-default-674cd67b5-xgb9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" start-of-body=
Nov 02 12:51:44.440563 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:44.440196    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d7801d-openshift-default-674cd67b5-xgb9d" podUID="03b6b83c-10cd-4876-b17a-96120c2317b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused"
Nov 02 12:51:44.604766 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:44.604706    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-d7801d-c576f4dc9-2fddw" podUID="311b37ae-9f7a-4c54-9819-3a20fcbe18d9" containerName="payload-processing" containerID="cri-o://cefb126845fb8462a521638cd20461659334a46d0876ad54a0e1b0e1fd51d791" gracePeriod=25
Nov 02 12:51:44.624712 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:44.624631    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d7801d-867c88bd6c-2h9kh" podUID="8a3ff80c-ba06-4eef-8c21-3d14926ec509" containerName="payload-pre-processing" containerID="cri-o://921aa3876819f809406c1e72a100565d57ac2b94902ed552e5663a6eef8bd5f3" gracePeriod=25
Nov 02 12:51:45.023650 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:45.022546    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-fwt4g"]
Nov 02 12:51:45.023650 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:45.023452    4559 cpu_manager.go:414] "containerMap: removing container" podUID="e5435e4b-b8a8-45b9-a90a-bca4b5a65506" containerName="istio-proxy"
Nov 02 12:51:45.023650 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:45.023467    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5435e4b-b8a8-45b9-a90a-bca4b5a65506" containerName="istio-proxy"
Nov 02 12:51:45.023650 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:45.023489    4559 cpu_manager.go:414] "containerMap: removing container" podUID="e5435e4b-b8a8-45b9-a90a-bca4b5a65506" containerName="istio-proxy"
Nov 02 12:51:45.023650 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:45.023497    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5435e4b-b8a8-45b9-a90a-bca4b5a65506" containerName="istio-proxy"
Nov 02 12:51:45.023650 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:45.023521    4559 cpu_manager.go:414] "containerMap: removing container" podUID="19089e8c-1e56-4d5a-9697-3c3477bcd803" containerName="payload-processing"
Nov 02 12:51:45.023650 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:45.023529    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19089e8c-1e56-4d5a-9697-3c3477bcd803" containerName="payload-processing"
Nov 02 12:51:45.023650 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:45.023556    4559 cpu_manager.go:414] "containerMap: removing container" podUID="4ac6508c-24db-40f7-a14c-0964dac7768b" containerName="istio-proxy"
Nov 02 12:51:45.023650 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:45.023563    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ac6508c-24db-40f7-a14c-0964dac7768b" containerName="istio-proxy"
Nov 02 12:51:45.023650 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:45.023584    4559 cpu_manager.go:414] "containerMap: removing container" podUID="e7f14aa1-1b9b-47ff-ac46-2db84191079b" containerName="maas-api"
Nov 02 12:51:45.023650 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:45.023591    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7f14aa1-1b9b-47ff-ac46-2db84191079b" containerName="maas-api"
Nov 02 12:51:45.026264 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:45.024943    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7f14aa1-1b9b-47ff-ac46-2db84191079b" containerName="maas-api"
Nov 02 12:51:45.026264 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:45.024972    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5435e4b-b8a8-45b9-a90a-bca4b5a65506" containerName="istio-proxy"
Nov 02 12:51:45.026264 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:45.024982    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ac6508c-24db-40f7-a14c-0964dac7768b" containerName="istio-proxy"
Nov 02 12:51:45.026264 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:45.024990    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5435e4b-b8a8-45b9-a90a-bca4b5a65506" containerName="istio-proxy"
Nov 02 12:51:45.026264 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:45.025003    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="19089e8c-1e56-4d5a-9697-3c3477bcd803" containerName="payload-processing"
Nov 02 12:51:45.029951 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:45.029558    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-52bzl"]
Nov 02 12:51:45.030820 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:45.029689    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-fwt4g"
Nov 02 12:51:45.040133 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:45.039984    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-fwt4g"]
Nov 02 12:51:45.040254 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:45.040217    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-52bzl"
Nov 02 12:51:45.040675 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:45.040362    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-52bzl"]
Nov 02 12:51:45.066284 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:45.066255    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-d7801d-c576f4dc9-2fddw"
Nov 02 12:51:45.120385 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:51:45.120349    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqcmj" containerName="extractor"
Nov 02 12:51:45.132378 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:45.132353    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d7801d-867c88bd6c-2h9kh"
Nov 02 12:51:45.199043 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:45.183060    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/311b37ae-9f7a-4c54-9819-3a20fcbe18d9-kube-api-access-64tvk\" (UniqueName: \"kubernetes.io/projected/311b37ae-9f7a-4c54-9819-3a20fcbe18d9-kube-api-access-64tvk\") pod \"311b37ae-9f7a-4c54-9819-3a20fcbe18d9\" (UID: \"311b37ae-9f7a-4c54-9819-3a20fcbe18d9\") "
Nov 02 12:51:45.199043 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:45.183200    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/311b37ae-9f7a-4c54-9819-3a20fcbe18d9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/311b37ae-9f7a-4c54-9819-3a20fcbe18d9-plugins-config-volume\") pod \"311b37ae-9f7a-4c54-9819-3a20fcbe18d9\" (UID: \"311b37ae-9f7a-4c54-9819-3a20fcbe18d9\") "
Nov 02 12:51:45.199043 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:45.183513    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/160f0d66-4e21-49eb-9c25-b467bdb56f41-credential-socket\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-52bzl\" (UID: \"160f0d66-4e21-49eb-9c25-b467bdb56f41\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-52bzl"
Nov 02 12:51:45.199043 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:45.183586    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91783d5b-f538-43a0-a9d8-29f7270cadcd-istio-token\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-fwt4g\" (UID: \"91783d5b-f538-43a0-a9d8-29f7270cadcd\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-fwt4g"
Nov 02 12:51:45.199043 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:45.183620    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91783d5b-f538-43a0-a9d8-29f7270cadcd-istio-data\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-fwt4g\" (UID: \"91783d5b-f538-43a0-a9d8-29f7270cadcd\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-fwt4g"
Nov 02 12:51:45.199479 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:45.199269    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/311b37ae-9f7a-4c54-9819-3a20fcbe18d9-plugins-config-volume" pod "311b37ae-9f7a-4c54-9819-3a20fcbe18d9" (UID: "311b37ae-9f7a-4c54-9819-3a20fcbe18d9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:51:45.199847 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:45.199816    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91783d5b-f538-43a0-a9d8-29f7270cadcd-istio-envoy\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-fwt4g\" (UID: \"91783d5b-f538-43a0-a9d8-29f7270cadcd\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-fwt4g"
Nov 02 12:51:45.199979 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:45.199876    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/160f0d66-4e21-49eb-9c25-b467bdb56f41-istio-token\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-52bzl\" (UID: \"160f0d66-4e21-49eb-9c25-b467bdb56f41\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-52bzl"
Nov 02 12:51:45.199979 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:45.199945    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91783d5b-f538-43a0-a9d8-29f7270cadcd-workload-certs\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-fwt4g\" (UID: \"91783d5b-f538-43a0-a9d8-29f7270cadcd\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-fwt4g"
Nov 02 12:51:45.200103 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:45.199994    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/160f0d66-4e21-49eb-9c25-b467bdb56f41-istio-envoy\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-52bzl\" (UID: \"160f0d66-4e21-49eb-9c25-b467bdb56f41\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-52bzl"
Nov 02 12:51:45.200103 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:45.200031    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/160f0d66-4e21-49eb-9c25-b467bdb56f41-workload-socket\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-52bzl\" (UID: \"160f0d66-4e21-49eb-9c25-b467bdb56f41\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-52bzl"
Nov 02 12:51:45.200103 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:45.200061    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/160f0d66-4e21-49eb-9c25-b467bdb56f41-istio-data\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-52bzl\" (UID: \"160f0d66-4e21-49eb-9c25-b467bdb56f41\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-52bzl"
Nov 02 12:51:45.200277 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:45.200100    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91783d5b-f538-43a0-a9d8-29f7270cadcd-istio-podinfo\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-fwt4g\" (UID: \"91783d5b-f538-43a0-a9d8-29f7270cadcd\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-fwt4g"
Nov 02 12:51:45.200277 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:45.200141    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91783d5b-f538-43a0-a9d8-29f7270cadcd-credential-socket\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-fwt4g\" (UID: \"91783d5b-f538-43a0-a9d8-29f7270cadcd\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-fwt4g"
Nov 02 12:51:45.200394 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:45.200259    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9shkk\" (UniqueName: \"kubernetes.io/projected/91783d5b-f538-43a0-a9d8-29f7270cadcd-kube-api-access-9shkk\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-fwt4g\" (UID: \"91783d5b-f538-43a0-a9d8-29f7270cadcd\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-fwt4g"
Nov 02 12:51:45.200459 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:45.200397    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91783d5b-f538-43a0-a9d8-29f7270cadcd-workload-socket\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-fwt4g\" (UID: \"91783d5b-f538-43a0-a9d8-29f7270cadcd\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-fwt4g"
Nov 02 12:51:45.200459 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:45.200438    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91783d5b-f538-43a0-a9d8-29f7270cadcd-istiod-ca-cert\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-fwt4g\" (UID: \"91783d5b-f538-43a0-a9d8-29f7270cadcd\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-fwt4g"
Nov 02 12:51:45.200875 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:45.200577    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/160f0d66-4e21-49eb-9c25-b467bdb56f41-workload-certs\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-52bzl\" (UID: \"160f0d66-4e21-49eb-9c25-b467bdb56f41\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-52bzl"
Nov 02 12:51:45.200875 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:45.200618    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttrdh\" (UniqueName: \"kubernetes.io/projected/160f0d66-4e21-49eb-9c25-b467bdb56f41-kube-api-access-ttrdh\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-52bzl\" (UID: \"160f0d66-4e21-49eb-9c25-b467bdb56f41\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-52bzl"
Nov 02 12:51:45.200875 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:45.200697    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/160f0d66-4e21-49eb-9c25-b467bdb56f41-istio-podinfo\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-52bzl\" (UID: \"160f0d66-4e21-49eb-9c25-b467bdb56f41\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-52bzl"
Nov 02 12:51:45.200875 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:45.200734    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/160f0d66-4e21-49eb-9c25-b467bdb56f41-istiod-ca-cert\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-52bzl\" (UID: \"160f0d66-4e21-49eb-9c25-b467bdb56f41\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-52bzl"
Nov 02 12:51:45.200875 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:45.200855    4559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/311b37ae-9f7a-4c54-9819-3a20fcbe18d9-plugins-config-volume\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:51:45.202771 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:45.202742    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/311b37ae-9f7a-4c54-9819-3a20fcbe18d9-kube-api-access-64tvk" pod "311b37ae-9f7a-4c54-9819-3a20fcbe18d9" (UID: "311b37ae-9f7a-4c54-9819-3a20fcbe18d9"). InnerVolumeSpecName "kube-api-access-64tvk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:51:45.205987 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:45.205967    4559 generic.go:356] "Generic (PLEG): container finished" podID="311b37ae-9f7a-4c54-9819-3a20fcbe18d9" containerID="cefb126845fb8462a521638cd20461659334a46d0876ad54a0e1b0e1fd51d791" exitCode=0
Nov 02 12:51:45.206076 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:45.206026    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-d7801d-c576f4dc9-2fddw"
Nov 02 12:51:45.206121 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:45.206094    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-d7801d-c576f4dc9-2fddw" event={"ID":"311b37ae-9f7a-4c54-9819-3a20fcbe18d9","Type":"ContainerDied","Data":"cefb126845fb8462a521638cd20461659334a46d0876ad54a0e1b0e1fd51d791"}
Nov 02 12:51:45.206157 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:45.206130    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-d7801d-c576f4dc9-2fddw" event={"ID":"311b37ae-9f7a-4c54-9819-3a20fcbe18d9","Type":"ContainerDied","Data":"0665567de0fd5cb2a20a564931d9a1b8ceeba03ac935cb6e08528ab331dfa54b"}
Nov 02 12:51:45.206157 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:45.206151    4559 scope.go:122] "RemoveContainer" containerID="cefb126845fb8462a521638cd20461659334a46d0876ad54a0e1b0e1fd51d791"
Nov 02 12:51:45.207598 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:45.207580    4559 generic.go:356] "Generic (PLEG): container finished" podID="8a3ff80c-ba06-4eef-8c21-3d14926ec509" containerID="921aa3876819f809406c1e72a100565d57ac2b94902ed552e5663a6eef8bd5f3" exitCode=0
Nov 02 12:51:45.207676 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:45.207662    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d7801d-867c88bd6c-2h9kh" event={"ID":"8a3ff80c-ba06-4eef-8c21-3d14926ec509","Type":"ContainerDied","Data":"921aa3876819f809406c1e72a100565d57ac2b94902ed552e5663a6eef8bd5f3"}
Nov 02 12:51:45.207728 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:45.207692    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d7801d-867c88bd6c-2h9kh" event={"ID":"8a3ff80c-ba06-4eef-8c21-3d14926ec509","Type":"ContainerDied","Data":"62eff538567627d152943bd22eb425001b275694ef0bcc64e8b94c0273d8f903"}
Nov 02 12:51:45.207775 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:45.207739    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d7801d-867c88bd6c-2h9kh"
Nov 02 12:51:45.216375 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:45.216356    4559 scope.go:122] "RemoveContainer" containerID="cefb126845fb8462a521638cd20461659334a46d0876ad54a0e1b0e1fd51d791"
Nov 02 12:51:45.216717 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:51:45.216694    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cefb126845fb8462a521638cd20461659334a46d0876ad54a0e1b0e1fd51d791\": container with ID starting with cefb126845fb8462a521638cd20461659334a46d0876ad54a0e1b0e1fd51d791 not found: ID does not exist" containerID="cefb126845fb8462a521638cd20461659334a46d0876ad54a0e1b0e1fd51d791"
Nov 02 12:51:45.216781 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:45.216729    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cefb126845fb8462a521638cd20461659334a46d0876ad54a0e1b0e1fd51d791"} err="failed to get container status \"cefb126845fb8462a521638cd20461659334a46d0876ad54a0e1b0e1fd51d791\": rpc error: code = NotFound desc = could not find container \"cefb126845fb8462a521638cd20461659334a46d0876ad54a0e1b0e1fd51d791\": container with ID starting with cefb126845fb8462a521638cd20461659334a46d0876ad54a0e1b0e1fd51d791 not found: ID does not exist"
Nov 02 12:51:45.216781 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:45.216752    4559 scope.go:122] "RemoveContainer" containerID="921aa3876819f809406c1e72a100565d57ac2b94902ed552e5663a6eef8bd5f3"
Nov 02 12:51:45.228515 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:45.228497    4559 scope.go:122] "RemoveContainer" containerID="921aa3876819f809406c1e72a100565d57ac2b94902ed552e5663a6eef8bd5f3"
Nov 02 12:51:45.228889 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:51:45.228859    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"921aa3876819f809406c1e72a100565d57ac2b94902ed552e5663a6eef8bd5f3\": container with ID starting with 921aa3876819f809406c1e72a100565d57ac2b94902ed552e5663a6eef8bd5f3 not found: ID does not exist" containerID="921aa3876819f809406c1e72a100565d57ac2b94902ed552e5663a6eef8bd5f3"
Nov 02 12:51:45.228989 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:45.228897    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"921aa3876819f809406c1e72a100565d57ac2b94902ed552e5663a6eef8bd5f3"} err="failed to get container status \"921aa3876819f809406c1e72a100565d57ac2b94902ed552e5663a6eef8bd5f3\": rpc error: code = NotFound desc = could not find container \"921aa3876819f809406c1e72a100565d57ac2b94902ed552e5663a6eef8bd5f3\": container with ID starting with 921aa3876819f809406c1e72a100565d57ac2b94902ed552e5663a6eef8bd5f3 not found: ID does not exist"
Nov 02 12:51:45.302394 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:45.302309    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a3ff80c-ba06-4eef-8c21-3d14926ec509-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a3ff80c-ba06-4eef-8c21-3d14926ec509-plugins-config-volume\") pod \"8a3ff80c-ba06-4eef-8c21-3d14926ec509\" (UID: \"8a3ff80c-ba06-4eef-8c21-3d14926ec509\") "
Nov 02 12:51:45.302579 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:45.302404    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a3ff80c-ba06-4eef-8c21-3d14926ec509-kube-api-access-hfg9p\" (UniqueName: \"kubernetes.io/projected/8a3ff80c-ba06-4eef-8c21-3d14926ec509-kube-api-access-hfg9p\") pod \"8a3ff80c-ba06-4eef-8c21-3d14926ec509\" (UID: \"8a3ff80c-ba06-4eef-8c21-3d14926ec509\") "
Nov 02 12:51:45.302756 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:45.302733    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91783d5b-f538-43a0-a9d8-29f7270cadcd-istio-token\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-fwt4g\" (UID: \"91783d5b-f538-43a0-a9d8-29f7270cadcd\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-fwt4g"
Nov 02 12:51:45.302852 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:45.302777    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91783d5b-f538-43a0-a9d8-29f7270cadcd-istio-data\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-fwt4g\" (UID: \"91783d5b-f538-43a0-a9d8-29f7270cadcd\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-fwt4g"
Nov 02 12:51:45.302852 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:45.302804    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a3ff80c-ba06-4eef-8c21-3d14926ec509-plugins-config-volume" pod "8a3ff80c-ba06-4eef-8c21-3d14926ec509" (UID: "8a3ff80c-ba06-4eef-8c21-3d14926ec509"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:51:45.302852 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:45.302824    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91783d5b-f538-43a0-a9d8-29f7270cadcd-istio-envoy\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-fwt4g\" (UID: \"91783d5b-f538-43a0-a9d8-29f7270cadcd\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-fwt4g"
Nov 02 12:51:45.303034 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:45.302866    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/160f0d66-4e21-49eb-9c25-b467bdb56f41-istio-token\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-52bzl\" (UID: \"160f0d66-4e21-49eb-9c25-b467bdb56f41\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-52bzl"
Nov 02 12:51:45.303034 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:45.302929    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91783d5b-f538-43a0-a9d8-29f7270cadcd-workload-certs\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-fwt4g\" (UID: \"91783d5b-f538-43a0-a9d8-29f7270cadcd\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-fwt4g"
Nov 02 12:51:45.303034 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:45.302975    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/160f0d66-4e21-49eb-9c25-b467bdb56f41-istio-envoy\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-52bzl\" (UID: \"160f0d66-4e21-49eb-9c25-b467bdb56f41\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-52bzl"
Nov 02 12:51:45.303034 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:45.303009    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/160f0d66-4e21-49eb-9c25-b467bdb56f41-workload-socket\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-52bzl\" (UID: \"160f0d66-4e21-49eb-9c25-b467bdb56f41\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-52bzl"
Nov 02 12:51:45.303250 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:45.303051    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/160f0d66-4e21-49eb-9c25-b467bdb56f41-istio-data\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-52bzl\" (UID: \"160f0d66-4e21-49eb-9c25-b467bdb56f41\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-52bzl"
Nov 02 12:51:45.303250 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:45.303089    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91783d5b-f538-43a0-a9d8-29f7270cadcd-istio-podinfo\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-fwt4g\" (UID: \"91783d5b-f538-43a0-a9d8-29f7270cadcd\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-fwt4g"
Nov 02 12:51:45.303250 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:45.303136    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91783d5b-f538-43a0-a9d8-29f7270cadcd-credential-socket\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-fwt4g\" (UID: \"91783d5b-f538-43a0-a9d8-29f7270cadcd\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-fwt4g"
Nov 02 12:51:45.303250 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:45.303207    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9shkk\" (UniqueName: \"kubernetes.io/projected/91783d5b-f538-43a0-a9d8-29f7270cadcd-kube-api-access-9shkk\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-fwt4g\" (UID: \"91783d5b-f538-43a0-a9d8-29f7270cadcd\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-fwt4g"
Nov 02 12:51:45.303556 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:45.303279    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91783d5b-f538-43a0-a9d8-29f7270cadcd-workload-socket\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-fwt4g\" (UID: \"91783d5b-f538-43a0-a9d8-29f7270cadcd\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-fwt4g"
Nov 02 12:51:45.303556 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:45.303295    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91783d5b-f538-43a0-a9d8-29f7270cadcd-istio-data\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-fwt4g\" (UID: \"91783d5b-f538-43a0-a9d8-29f7270cadcd\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-fwt4g"
Nov 02 12:51:45.303556 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:45.303317    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91783d5b-f538-43a0-a9d8-29f7270cadcd-istiod-ca-cert\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-fwt4g\" (UID: \"91783d5b-f538-43a0-a9d8-29f7270cadcd\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-fwt4g"
Nov 02 12:51:45.303556 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:45.303381    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/160f0d66-4e21-49eb-9c25-b467bdb56f41-workload-certs\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-52bzl\" (UID: \"160f0d66-4e21-49eb-9c25-b467bdb56f41\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-52bzl"
Nov 02 12:51:45.303556 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:45.303418    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttrdh\" (UniqueName: \"kubernetes.io/projected/160f0d66-4e21-49eb-9c25-b467bdb56f41-kube-api-access-ttrdh\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-52bzl\" (UID: \"160f0d66-4e21-49eb-9c25-b467bdb56f41\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-52bzl"
Nov 02 12:51:45.303556 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:45.303478    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/160f0d66-4e21-49eb-9c25-b467bdb56f41-istio-podinfo\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-52bzl\" (UID: \"160f0d66-4e21-49eb-9c25-b467bdb56f41\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-52bzl"
Nov 02 12:51:45.303556 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:45.303515    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/160f0d66-4e21-49eb-9c25-b467bdb56f41-istiod-ca-cert\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-52bzl\" (UID: \"160f0d66-4e21-49eb-9c25-b467bdb56f41\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-52bzl"
Nov 02 12:51:45.303848 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:45.303589    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/160f0d66-4e21-49eb-9c25-b467bdb56f41-credential-socket\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-52bzl\" (UID: \"160f0d66-4e21-49eb-9c25-b467bdb56f41\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-52bzl"
Nov 02 12:51:45.303848 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:45.303718    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-64tvk\" (UniqueName: \"kubernetes.io/projected/311b37ae-9f7a-4c54-9819-3a20fcbe18d9-kube-api-access-64tvk\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:51:45.303848 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:45.303742    4559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a3ff80c-ba06-4eef-8c21-3d14926ec509-plugins-config-volume\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:51:45.304007 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:45.303932    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/160f0d66-4e21-49eb-9c25-b467bdb56f41-workload-socket\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-52bzl\" (UID: \"160f0d66-4e21-49eb-9c25-b467bdb56f41\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-52bzl"
Nov 02 12:51:45.304068 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:45.304016    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/160f0d66-4e21-49eb-9c25-b467bdb56f41-credential-socket\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-52bzl\" (UID: \"160f0d66-4e21-49eb-9c25-b467bdb56f41\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-52bzl"
Nov 02 12:51:45.304124 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:45.304088    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91783d5b-f538-43a0-a9d8-29f7270cadcd-workload-certs\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-fwt4g\" (UID: \"91783d5b-f538-43a0-a9d8-29f7270cadcd\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-fwt4g"
Nov 02 12:51:45.304799 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:45.304771    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/160f0d66-4e21-49eb-9c25-b467bdb56f41-istiod-ca-cert\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-52bzl\" (UID: \"160f0d66-4e21-49eb-9c25-b467bdb56f41\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-52bzl"
Nov 02 12:51:45.304928 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:45.304827    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91783d5b-f538-43a0-a9d8-29f7270cadcd-istiod-ca-cert\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-fwt4g\" (UID: \"91783d5b-f538-43a0-a9d8-29f7270cadcd\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-fwt4g"
Nov 02 12:51:45.305061 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:45.305039    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/160f0d66-4e21-49eb-9c25-b467bdb56f41-workload-certs\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-52bzl\" (UID: \"160f0d66-4e21-49eb-9c25-b467bdb56f41\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-52bzl"
Nov 02 12:51:45.305498 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:45.305473    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91783d5b-f538-43a0-a9d8-29f7270cadcd-workload-socket\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-fwt4g\" (UID: \"91783d5b-f538-43a0-a9d8-29f7270cadcd\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-fwt4g"
Nov 02 12:51:45.305618 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:45.305594    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91783d5b-f538-43a0-a9d8-29f7270cadcd-credential-socket\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-fwt4g\" (UID: \"91783d5b-f538-43a0-a9d8-29f7270cadcd\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-fwt4g"
Nov 02 12:51:45.305877 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:45.305856    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91783d5b-f538-43a0-a9d8-29f7270cadcd-istio-envoy\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-fwt4g\" (UID: \"91783d5b-f538-43a0-a9d8-29f7270cadcd\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-fwt4g"
Nov 02 12:51:45.305996 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:45.305856    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/160f0d66-4e21-49eb-9c25-b467bdb56f41-istio-data\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-52bzl\" (UID: \"160f0d66-4e21-49eb-9c25-b467bdb56f41\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-52bzl"
Nov 02 12:51:45.306076 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:45.306054    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/160f0d66-4e21-49eb-9c25-b467bdb56f41-istio-envoy\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-52bzl\" (UID: \"160f0d66-4e21-49eb-9c25-b467bdb56f41\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-52bzl"
Nov 02 12:51:45.306769 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:45.306743    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a3ff80c-ba06-4eef-8c21-3d14926ec509-kube-api-access-hfg9p" pod "8a3ff80c-ba06-4eef-8c21-3d14926ec509" (UID: "8a3ff80c-ba06-4eef-8c21-3d14926ec509"). InnerVolumeSpecName "kube-api-access-hfg9p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:51:45.307705 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:45.307678    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91783d5b-f538-43a0-a9d8-29f7270cadcd-istio-podinfo\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-fwt4g\" (UID: \"91783d5b-f538-43a0-a9d8-29f7270cadcd\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-fwt4g"
Nov 02 12:51:45.307805 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:45.307780    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/160f0d66-4e21-49eb-9c25-b467bdb56f41-istio-podinfo\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-52bzl\" (UID: \"160f0d66-4e21-49eb-9c25-b467bdb56f41\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-52bzl"
Nov 02 12:51:45.316261 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:45.316241    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d7801d-c576f4dc9-2fddw"]
Nov 02 12:51:45.322047 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:45.322024    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d7801d-c576f4dc9-2fddw"]
Nov 02 12:51:45.327300 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:45.327270    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91783d5b-f538-43a0-a9d8-29f7270cadcd-istio-token\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-fwt4g\" (UID: \"91783d5b-f538-43a0-a9d8-29f7270cadcd\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-fwt4g"
Nov 02 12:51:45.327300 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:45.327287    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9shkk\" (UniqueName: \"kubernetes.io/projected/91783d5b-f538-43a0-a9d8-29f7270cadcd-kube-api-access-9shkk\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-fwt4g\" (UID: \"91783d5b-f538-43a0-a9d8-29f7270cadcd\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-fwt4g"
Nov 02 12:51:45.327629 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:45.327607    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttrdh\" (UniqueName: \"kubernetes.io/projected/160f0d66-4e21-49eb-9c25-b467bdb56f41-kube-api-access-ttrdh\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-52bzl\" (UID: \"160f0d66-4e21-49eb-9c25-b467bdb56f41\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-52bzl"
Nov 02 12:51:45.327735 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:45.327614    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/160f0d66-4e21-49eb-9c25-b467bdb56f41-istio-token\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-52bzl\" (UID: \"160f0d66-4e21-49eb-9c25-b467bdb56f41\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-52bzl"
Nov 02 12:51:45.405127 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:45.405085    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hfg9p\" (UniqueName: \"kubernetes.io/projected/8a3ff80c-ba06-4eef-8c21-3d14926ec509-kube-api-access-hfg9p\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:51:45.421808 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:45.421767    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-fwt4g"
Nov 02 12:51:45.430757 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:45.430726    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-52bzl"
Nov 02 12:51:45.572135 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:45.572107    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d7801d-867c88bd6c-2h9kh"]
Nov 02 12:51:45.576450 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:45.576422    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d7801d-867c88bd6c-2h9kh"]
Nov 02 12:51:45.726355 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:45.726321    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-d7801d-openshift-default-674cd67b5-xgb9d"
Nov 02 12:51:45.727224 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:45.727203    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-d7801d-openshift-default-674cd67b5-xgb9d"
Nov 02 12:51:45.728572 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:45.728537    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-d7801d-openshift-default-674cd67b5-xgb9d"
Nov 02 12:51:45.753058 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:51:45.753031    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91783d5b_f538_43a0_a9d8_29f7270cadcd.slice/crio-3dee1d8dc53ff0b6ef36cdf323543a7e9df177d3ecc508f30cbf450919087c2e WatchSource:0}: Error finding container 3dee1d8dc53ff0b6ef36cdf323543a7e9df177d3ecc508f30cbf450919087c2e: Status 404 returned error can't find the container with id 3dee1d8dc53ff0b6ef36cdf323543a7e9df177d3ecc508f30cbf450919087c2e
Nov 02 12:51:45.754446 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:45.754424    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-fwt4g"]
Nov 02 12:51:45.757806 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:45.757771    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:51:45.757898 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:45.757843    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:51:45.757898 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:45.757874    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:51:45.758796 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:45.758775    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-52bzl"]
Nov 02 12:51:45.758946 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:45.758917    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:51:45.759004 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:45.758990    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:51:45.759041 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:45.759030    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:51:45.907005 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:45.906961    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="311b37ae-9f7a-4c54-9819-3a20fcbe18d9" path="/var/lib/kubelet/pods/311b37ae-9f7a-4c54-9819-3a20fcbe18d9/volumes"
Nov 02 12:51:45.907839 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:45.907821    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a3ff80c-ba06-4eef-8c21-3d14926ec509" path="/var/lib/kubelet/pods/8a3ff80c-ba06-4eef-8c21-3d14926ec509/volumes"
Nov 02 12:51:46.074304 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:46.074219    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-9750c7-84d6d97dc4-cbh2m"]
Nov 02 12:51:46.074509 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:46.074480    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9750c7-84d6d97dc4-cbh2m" podUID="465410fd-fc85-469e-90fb-842446ce62e4" containerName="maas-api" containerID="cri-o://eb96c23257c4c6f7f1dea5166cb8c99a4ee005bd0fac9869e1a4dcd6ed4b1533" gracePeriod=30
Nov 02 12:51:46.217353 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:46.217315    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-52bzl" event={"ID":"160f0d66-4e21-49eb-9c25-b467bdb56f41","Type":"ContainerStarted","Data":"a56f8505504b37a1ad5c87e4fd9da9d9f95274c306e46273d0c503e6576f210e"}
Nov 02 12:51:46.217554 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:46.217395    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-52bzl" event={"ID":"160f0d66-4e21-49eb-9c25-b467bdb56f41","Type":"ContainerStarted","Data":"e476de3840eab8389aa0c3a0e72bb7e77c3fb5039fd0b23669fbb0407b5ad0d2"}
Nov 02 12:51:46.220024 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:46.219994    4559 generic.go:356] "Generic (PLEG): container finished" podID="465410fd-fc85-469e-90fb-842446ce62e4" containerID="eb96c23257c4c6f7f1dea5166cb8c99a4ee005bd0fac9869e1a4dcd6ed4b1533" exitCode=0
Nov 02 12:51:46.220194 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:46.220091    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9750c7-84d6d97dc4-cbh2m" event={"ID":"465410fd-fc85-469e-90fb-842446ce62e4","Type":"ContainerDied","Data":"eb96c23257c4c6f7f1dea5166cb8c99a4ee005bd0fac9869e1a4dcd6ed4b1533"}
Nov 02 12:51:46.222123 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:46.222073    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-fwt4g" event={"ID":"91783d5b-f538-43a0-a9d8-29f7270cadcd","Type":"ContainerStarted","Data":"f75d8cc15890dd03513350f317fa6efc86f60028e165eacc7c22765c06af76ad"}
Nov 02 12:51:46.222123 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:46.222125    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-fwt4g" event={"ID":"91783d5b-f538-43a0-a9d8-29f7270cadcd","Type":"ContainerStarted","Data":"3dee1d8dc53ff0b6ef36cdf323543a7e9df177d3ecc508f30cbf450919087c2e"}
Nov 02 12:51:46.269806 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:46.265925    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-52bzl" podStartSLOduration=2.265905888 podStartE2EDuration="2.265905888s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:51:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:51:46.265118291 +0000 UTC m=+1502.520517387" watchObservedRunningTime="2026-11-02 12:51:46.265905888 +0000 UTC m=+1502.521304986"
Nov 02 12:51:46.426659 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:46.426315    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-fwt4g"
Nov 02 12:51:46.426659 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:46.426520    4559 patch_prober.go:29] interesting pod/e2e-worker-w3-068756-openshift-default-84f69c9cf5-fwt4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" start-of-body=
Nov 02 12:51:46.426659 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:46.426577    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-fwt4g" podUID="91783d5b-f538-43a0-a9d8-29f7270cadcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused"
Nov 02 12:51:46.432884 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:46.432844    4559 patch_prober.go:29] interesting pod/e2e-worker-w3-068756-openshift-default-84f69c9cf5-52bzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" start-of-body=
Nov 02 12:51:46.433214 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:46.433176    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-52bzl" podUID="160f0d66-4e21-49eb-9c25-b467bdb56f41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused"
Nov 02 12:51:46.433397 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:46.433371    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-52bzl"
Nov 02 12:51:46.648931 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:46.647953    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9750c7-84d6d97dc4-cbh2m"
Nov 02 12:51:46.722169 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:46.722083    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/465410fd-fc85-469e-90fb-842446ce62e4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/465410fd-fc85-469e-90fb-842446ce62e4-metrics-certs\") pod \"465410fd-fc85-469e-90fb-842446ce62e4\" (UID: \"465410fd-fc85-469e-90fb-842446ce62e4\") "
Nov 02 12:51:46.725660 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:46.722412    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/465410fd-fc85-469e-90fb-842446ce62e4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/465410fd-fc85-469e-90fb-842446ce62e4-openshift-service-ca\") pod \"465410fd-fc85-469e-90fb-842446ce62e4\" (UID: \"465410fd-fc85-469e-90fb-842446ce62e4\") "
Nov 02 12:51:46.726050 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:46.726026    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/465410fd-fc85-469e-90fb-842446ce62e4-kube-api-access-bcl9c\" (UniqueName: \"kubernetes.io/projected/465410fd-fc85-469e-90fb-842446ce62e4-kube-api-access-bcl9c\") pod \"465410fd-fc85-469e-90fb-842446ce62e4\" (UID: \"465410fd-fc85-469e-90fb-842446ce62e4\") "
Nov 02 12:51:46.726401 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:46.726381    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/465410fd-fc85-469e-90fb-842446ce62e4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/465410fd-fc85-469e-90fb-842446ce62e4-maas-api-tls\") pod \"465410fd-fc85-469e-90fb-842446ce62e4\" (UID: \"465410fd-fc85-469e-90fb-842446ce62e4\") "
Nov 02 12:51:46.749685 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:46.738718    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-fwt4g" podStartSLOduration=2.738698329 podStartE2EDuration="2.738698329s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:51:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:51:46.371724554 +0000 UTC m=+1502.627123648" watchObservedRunningTime="2026-11-02 12:51:46.738698329 +0000 UTC m=+1502.994097422"
Nov 02 12:51:46.758046 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:46.753862    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/465410fd-fc85-469e-90fb-842446ce62e4-openshift-service-ca" pod "465410fd-fc85-469e-90fb-842446ce62e4" (UID: "465410fd-fc85-469e-90fb-842446ce62e4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:51:46.783340 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:46.783292    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/465410fd-fc85-469e-90fb-842446ce62e4-metrics-certs" pod "465410fd-fc85-469e-90fb-842446ce62e4" (UID: "465410fd-fc85-469e-90fb-842446ce62e4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:51:46.784299 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:46.784258    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/465410fd-fc85-469e-90fb-842446ce62e4-maas-api-tls" pod "465410fd-fc85-469e-90fb-842446ce62e4" (UID: "465410fd-fc85-469e-90fb-842446ce62e4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:51:46.784483 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:46.784465    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/465410fd-fc85-469e-90fb-842446ce62e4-kube-api-access-bcl9c" pod "465410fd-fc85-469e-90fb-842446ce62e4" (UID: "465410fd-fc85-469e-90fb-842446ce62e4"). InnerVolumeSpecName "kube-api-access-bcl9c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:51:46.831876 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:46.831743    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bcl9c\" (UniqueName: \"kubernetes.io/projected/465410fd-fc85-469e-90fb-842446ce62e4-kube-api-access-bcl9c\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:51:46.831876 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:46.831790    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/465410fd-fc85-469e-90fb-842446ce62e4-maas-api-tls\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:51:46.831876 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:46.831804    4559 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/465410fd-fc85-469e-90fb-842446ce62e4-metrics-certs\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:51:46.831876 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:46.831818    4559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/465410fd-fc85-469e-90fb-842446ce62e4-openshift-service-ca\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:51:47.239753 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:47.239724    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9750c7-84d6d97dc4-cbh2m"
Nov 02 12:51:47.239939 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:47.239752    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9750c7-84d6d97dc4-cbh2m" event={"ID":"465410fd-fc85-469e-90fb-842446ce62e4","Type":"ContainerDied","Data":"504d95d38bee30e6696443e5ced646f2c7a811aea41de9dd6def8365d0bdf18e"}
Nov 02 12:51:47.239939 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:47.239816    4559 scope.go:122] "RemoveContainer" containerID="eb96c23257c4c6f7f1dea5166cb8c99a4ee005bd0fac9869e1a4dcd6ed4b1533"
Nov 02 12:51:47.331759 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:47.331703    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-9750c7-84d6d97dc4-cbh2m"]
Nov 02 12:51:47.337914 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:47.337884    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-9750c7-84d6d97dc4-cbh2m"]
Nov 02 12:51:47.425425 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:47.424987    4559 patch_prober.go:29] interesting pod/e2e-worker-w3-068756-openshift-default-84f69c9cf5-fwt4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" start-of-body=
Nov 02 12:51:47.425425 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:47.425049    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-fwt4g" podUID="91783d5b-f538-43a0-a9d8-29f7270cadcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused"
Nov 02 12:51:47.433057 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:47.433010    4559 patch_prober.go:29] interesting pod/e2e-worker-w3-068756-openshift-default-84f69c9cf5-52bzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" start-of-body=
Nov 02 12:51:47.433245 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:47.433086    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-52bzl" podUID="160f0d66-4e21-49eb-9c25-b467bdb56f41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused"
Nov 02 12:51:47.930298 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:47.930258    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="465410fd-fc85-469e-90fb-842446ce62e4" path="/var/lib/kubelet/pods/465410fd-fc85-469e-90fb-842446ce62e4/volumes"
Nov 02 12:51:48.427706 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:48.426988    4559 patch_prober.go:29] interesting pod/e2e-worker-w3-068756-openshift-default-84f69c9cf5-fwt4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" start-of-body=
Nov 02 12:51:48.427706 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:48.427049    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-fwt4g" podUID="91783d5b-f538-43a0-a9d8-29f7270cadcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused"
Nov 02 12:51:48.436812 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:48.436685    4559 patch_prober.go:29] interesting pod/e2e-worker-w3-068756-openshift-default-84f69c9cf5-52bzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" start-of-body=
Nov 02 12:51:48.436812 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:48.436751    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-52bzl" podUID="160f0d66-4e21-49eb-9c25-b467bdb56f41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused"
Nov 02 12:51:49.427685 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:49.427050    4559 patch_prober.go:29] interesting pod/e2e-worker-w3-068756-openshift-default-84f69c9cf5-fwt4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" start-of-body=
Nov 02 12:51:49.427685 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:49.427117    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-fwt4g" podUID="91783d5b-f538-43a0-a9d8-29f7270cadcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused"
Nov 02 12:51:49.439680 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:49.434962    4559 patch_prober.go:29] interesting pod/e2e-worker-w3-068756-openshift-default-84f69c9cf5-52bzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" start-of-body=
Nov 02 12:51:49.439680 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:49.435017    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-52bzl" podUID="160f0d66-4e21-49eb-9c25-b467bdb56f41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused"
Nov 02 12:51:50.423132 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:50.423091    4559 patch_prober.go:29] interesting pod/e2e-worker-w3-068756-openshift-default-84f69c9cf5-fwt4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" start-of-body=
Nov 02 12:51:50.423444 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:50.423410    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-fwt4g" podUID="91783d5b-f538-43a0-a9d8-29f7270cadcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused"
Nov 02 12:51:50.432071 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:50.432036    4559 patch_prober.go:29] interesting pod/e2e-worker-w3-068756-openshift-default-84f69c9cf5-52bzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" start-of-body=
Nov 02 12:51:50.432612 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:50.432583    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-52bzl" podUID="160f0d66-4e21-49eb-9c25-b467bdb56f41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused"
Nov 02 12:51:51.423662 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:51.423052    4559 patch_prober.go:29] interesting pod/e2e-worker-w3-068756-openshift-default-84f69c9cf5-fwt4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" start-of-body=
Nov 02 12:51:51.423662 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:51.423118    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-fwt4g" podUID="91783d5b-f538-43a0-a9d8-29f7270cadcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused"
Nov 02 12:51:51.432209 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:51.432032    4559 patch_prober.go:29] interesting pod/e2e-worker-w3-068756-openshift-default-84f69c9cf5-52bzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" start-of-body=
Nov 02 12:51:51.432209 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:51.432094    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-52bzl" podUID="160f0d66-4e21-49eb-9c25-b467bdb56f41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused"
Nov 02 12:51:52.422969 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:52.422920    4559 patch_prober.go:29] interesting pod/e2e-worker-w3-068756-openshift-default-84f69c9cf5-fwt4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" start-of-body=
Nov 02 12:51:52.423153 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:52.422994    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-fwt4g" podUID="91783d5b-f538-43a0-a9d8-29f7270cadcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused"
Nov 02 12:51:52.431223 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:52.431195    4559 patch_prober.go:29] interesting pod/e2e-worker-w3-068756-openshift-default-84f69c9cf5-52bzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" start-of-body=
Nov 02 12:51:52.431347 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:52.431253    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-52bzl" podUID="160f0d66-4e21-49eb-9c25-b467bdb56f41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused"
Nov 02 12:51:53.422460 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:53.422418    4559 patch_prober.go:29] interesting pod/e2e-worker-w3-068756-openshift-default-84f69c9cf5-fwt4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" start-of-body=
Nov 02 12:51:53.422890 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:53.422496    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-fwt4g" podUID="91783d5b-f538-43a0-a9d8-29f7270cadcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused"
Nov 02 12:51:53.973381 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:53.972735    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-52bzl"
Nov 02 12:51:53.973381 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:53.973184    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-52bzl"
Nov 02 12:51:53.974023 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:53.974003    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-52bzl"
Nov 02 12:51:54.427156 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:54.427130    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-fwt4g"
Nov 02 12:51:54.427587 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:54.427570    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-fwt4g"
Nov 02 12:51:54.428231 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:54.428215    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-fwt4g"
Nov 02 12:51:57.019037 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:57.018999    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"]
Nov 02 12:51:58.328666 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:58.322660    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-s8hnh"]
Nov 02 12:51:58.328666 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:58.322964    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-s8hnh" podUID="7666985a-ed93-432a-9bd8-c668a7bfd8f5" containerName="istio-proxy" containerID="cri-o://8ae9ec8c8d6a925274594b4addbedf5a237c297e29b3430b4c5992eaefe64e98" gracePeriod=30
Nov 02 12:51:58.443440 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:58.443402    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-pz8j5"]
Nov 02 12:51:58.460664 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:58.454883    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-pz8j5" podUID="632e48df-f4f7-4e68-819a-593ffd6a76d6" containerName="istio-proxy" containerID="cri-o://ab4f6d47502dee62a1960429358f4fcf9877e1b30bb1ff781634de50a0b7adb0" gracePeriod=30
Nov 02 12:51:58.494086 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:58.492709    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-wmzzn"]
Nov 02 12:51:58.494086 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:58.492983    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-wmzzn" podUID="e1722c3b-b864-43f0-b5cb-94d96526858a" containerName="istio-proxy" containerID="cri-o://dde63fb43d91e0b37562756a515e89239f38a5241c38c45719959620fd8b6186" gracePeriod=30
Nov 02 12:51:58.501829 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:58.498886    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d7801d-5f9f5c649f-9frfl"]
Nov 02 12:51:58.508216 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:58.507578    4559 cpu_manager.go:414] "containerMap: removing container" podUID="8a3ff80c-ba06-4eef-8c21-3d14926ec509" containerName="payload-pre-processing"
Nov 02 12:51:58.508216 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:58.507604    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a3ff80c-ba06-4eef-8c21-3d14926ec509" containerName="payload-pre-processing"
Nov 02 12:51:58.508216 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:58.507666    4559 cpu_manager.go:414] "containerMap: removing container" podUID="465410fd-fc85-469e-90fb-842446ce62e4" containerName="maas-api"
Nov 02 12:51:58.508216 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:58.507675    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="465410fd-fc85-469e-90fb-842446ce62e4" containerName="maas-api"
Nov 02 12:51:58.508216 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:58.507772    4559 cpu_manager.go:414] "containerMap: removing container" podUID="311b37ae-9f7a-4c54-9819-3a20fcbe18d9" containerName="payload-processing"
Nov 02 12:51:58.508216 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:58.507784    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="311b37ae-9f7a-4c54-9819-3a20fcbe18d9" containerName="payload-processing"
Nov 02 12:51:58.508216 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:58.507952    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="311b37ae-9f7a-4c54-9819-3a20fcbe18d9" containerName="payload-processing"
Nov 02 12:51:58.508216 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:58.507970    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a3ff80c-ba06-4eef-8c21-3d14926ec509" containerName="payload-pre-processing"
Nov 02 12:51:58.508216 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:58.507984    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="465410fd-fc85-469e-90fb-842446ce62e4" containerName="maas-api"
Nov 02 12:51:58.701310 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:58.667398    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d7801d-5f9f5c649f-9frfl"]
Nov 02 12:51:58.701310 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:58.667528    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d7801d-5f9f5c649f-9frfl"
Nov 02 12:51:58.701310 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:58.689576    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-d7801d\""
Nov 02 12:51:58.701310 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:58.689747    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-d7801d-dockercfg-472hp\""
Nov 02 12:51:58.728351 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:58.728235    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3e81e99-3871-429a-b725-37cf024fbce1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-d7801d-5f9f5c649f-9frfl\" (UID: \"d3e81e99-3871-429a-b725-37cf024fbce1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d7801d-5f9f5c649f-9frfl"
Nov 02 12:51:58.728540 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:58.728396    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89hgr\" (UniqueName: \"kubernetes.io/projected/d3e81e99-3871-429a-b725-37cf024fbce1-kube-api-access-89hgr\") pod \"payload-pre-processing-e2e-ipp-b-d7801d-5f9f5c649f-9frfl\" (UID: \"d3e81e99-3871-429a-b725-37cf024fbce1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d7801d-5f9f5c649f-9frfl"
Nov 02 12:51:58.829495 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:58.829434    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89hgr\" (UniqueName: \"kubernetes.io/projected/d3e81e99-3871-429a-b725-37cf024fbce1-kube-api-access-89hgr\") pod \"payload-pre-processing-e2e-ipp-b-d7801d-5f9f5c649f-9frfl\" (UID: \"d3e81e99-3871-429a-b725-37cf024fbce1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d7801d-5f9f5c649f-9frfl"
Nov 02 12:51:58.829762 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:58.829722    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3e81e99-3871-429a-b725-37cf024fbce1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-d7801d-5f9f5c649f-9frfl\" (UID: \"d3e81e99-3871-429a-b725-37cf024fbce1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d7801d-5f9f5c649f-9frfl"
Nov 02 12:51:58.830741 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:58.830709    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3e81e99-3871-429a-b725-37cf024fbce1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-d7801d-5f9f5c649f-9frfl\" (UID: \"d3e81e99-3871-429a-b725-37cf024fbce1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d7801d-5f9f5c649f-9frfl"
Nov 02 12:51:58.872098 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:58.872063    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-89hgr\" (UniqueName: \"kubernetes.io/projected/d3e81e99-3871-429a-b725-37cf024fbce1-kube-api-access-89hgr\") pod \"payload-pre-processing-e2e-ipp-b-d7801d-5f9f5c649f-9frfl\" (UID: \"d3e81e99-3871-429a-b725-37cf024fbce1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d7801d-5f9f5c649f-9frfl"
Nov 02 12:51:58.931006 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:58.930965    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d7801d-678658877b-bwjnd"]
Nov 02 12:51:58.973556 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:58.970998    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-d7801d-678658877b-bwjnd"
Nov 02 12:51:58.982447 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:58.982410    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d7801d-678658877b-bwjnd"]
Nov 02 12:51:59.033424 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:59.033392    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d7801d-5f9f5c649f-9frfl"
Nov 02 12:51:59.174673 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:59.173461    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvtlf\" (UniqueName: \"kubernetes.io/projected/997a3029-5f17-4a07-be79-0d8f7aa2e4b2-kube-api-access-mvtlf\") pod \"payload-processing-e2e-ipp-b-d7801d-678658877b-bwjnd\" (UID: \"997a3029-5f17-4a07-be79-0d8f7aa2e4b2\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d7801d-678658877b-bwjnd"
Nov 02 12:51:59.174673 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:59.173577    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/997a3029-5f17-4a07-be79-0d8f7aa2e4b2-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-d7801d-678658877b-bwjnd\" (UID: \"997a3029-5f17-4a07-be79-0d8f7aa2e4b2\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d7801d-678658877b-bwjnd"
Nov 02 12:51:59.320299 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:59.301228    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvtlf\" (UniqueName: \"kubernetes.io/projected/997a3029-5f17-4a07-be79-0d8f7aa2e4b2-kube-api-access-mvtlf\") pod \"payload-processing-e2e-ipp-b-d7801d-678658877b-bwjnd\" (UID: \"997a3029-5f17-4a07-be79-0d8f7aa2e4b2\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d7801d-678658877b-bwjnd"
Nov 02 12:51:59.323042 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:59.323006    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/997a3029-5f17-4a07-be79-0d8f7aa2e4b2-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-d7801d-678658877b-bwjnd\" (UID: \"997a3029-5f17-4a07-be79-0d8f7aa2e4b2\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d7801d-678658877b-bwjnd"
Nov 02 12:51:59.370516 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:59.370479    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/997a3029-5f17-4a07-be79-0d8f7aa2e4b2-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-d7801d-678658877b-bwjnd\" (UID: \"997a3029-5f17-4a07-be79-0d8f7aa2e4b2\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d7801d-678658877b-bwjnd"
Nov 02 12:51:59.398968 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:59.393451    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvtlf\" (UniqueName: \"kubernetes.io/projected/997a3029-5f17-4a07-be79-0d8f7aa2e4b2-kube-api-access-mvtlf\") pod \"payload-processing-e2e-ipp-b-d7801d-678658877b-bwjnd\" (UID: \"997a3029-5f17-4a07-be79-0d8f7aa2e4b2\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d7801d-678658877b-bwjnd"
Nov 02 12:51:59.579443 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:51:59.446898    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-d7801d-678658877b-bwjnd"
Nov 02 12:52:00.540753 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:52:00.540722    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod997a3029_5f17_4a07_be79_0d8f7aa2e4b2.slice/crio-f136fbf0202915e31efb8a07957fee1c4a08517909f41eb41e82e8955e891131 WatchSource:0}: Error finding container f136fbf0202915e31efb8a07957fee1c4a08517909f41eb41e82e8955e891131: Status 404 returned error can't find the container with id f136fbf0202915e31efb8a07957fee1c4a08517909f41eb41e82e8955e891131
Nov 02 12:52:00.542074 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:00.542048    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d7801d-678658877b-bwjnd"]
Nov 02 12:52:00.603351 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:52:00.603317    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3e81e99_3871_429a_b725_37cf024fbce1.slice/crio-5efa1a6cdc27f33118213655ddff3bee2beab03568a6778d767455e6d979e225 WatchSource:0}: Error finding container 5efa1a6cdc27f33118213655ddff3bee2beab03568a6778d767455e6d979e225: Status 404 returned error can't find the container with id 5efa1a6cdc27f33118213655ddff3bee2beab03568a6778d767455e6d979e225
Nov 02 12:52:00.603944 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:00.603917    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d7801d-5f9f5c649f-9frfl"]
Nov 02 12:52:00.673001 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:00.672970    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-d7801d-678658877b-bwjnd" event={"ID":"997a3029-5f17-4a07-be79-0d8f7aa2e4b2","Type":"ContainerStarted","Data":"5c30e4e48efdac5cb874cea03d931c7dae2f79f13dda0c7f96228b86830431e1"}
Nov 02 12:52:00.673130 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:00.673031    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-d7801d-678658877b-bwjnd" event={"ID":"997a3029-5f17-4a07-be79-0d8f7aa2e4b2","Type":"ContainerStarted","Data":"f136fbf0202915e31efb8a07957fee1c4a08517909f41eb41e82e8955e891131"}
Nov 02 12:52:00.673130 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:00.673061    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-d7801d-678658877b-bwjnd"
Nov 02 12:52:00.674724 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:00.674703    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d7801d-5f9f5c649f-9frfl" event={"ID":"d3e81e99-3871-429a-b725-37cf024fbce1","Type":"ContainerStarted","Data":"5efa1a6cdc27f33118213655ddff3bee2beab03568a6778d767455e6d979e225"}
Nov 02 12:52:00.716772 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:00.716721    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-d7801d-678658877b-bwjnd" podStartSLOduration=2.716707229 podStartE2EDuration="2.716707229s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:51:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:52:00.714499441 +0000 UTC m=+1516.969898535" watchObservedRunningTime="2026-11-02 12:52:00.716707229 +0000 UTC m=+1516.972106324"
Nov 02 12:52:01.683854 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:01.683816    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d7801d-5f9f5c649f-9frfl" event={"ID":"d3e81e99-3871-429a-b725-37cf024fbce1","Type":"ContainerStarted","Data":"98637dd2dea79b341d14b5c84b86de69922c1148d7757c5834a8cae749092fa3"}
Nov 02 12:52:01.684383 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:01.684166    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d7801d-5f9f5c649f-9frfl"
Nov 02 12:52:01.726284 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:01.726212    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d7801d-5f9f5c649f-9frfl" podStartSLOduration=3.726198117 podStartE2EDuration="3.726198117s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:51:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:52:01.725069489 +0000 UTC m=+1517.980468586" watchObservedRunningTime="2026-11-02 12:52:01.726198117 +0000 UTC m=+1517.981597212"
Nov 02 12:52:03.707412 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:03.707373    4559 generic.go:356] "Generic (PLEG): container finished" podID="632e48df-f4f7-4e68-819a-593ffd6a76d6" containerID="ab4f6d47502dee62a1960429358f4fcf9877e1b30bb1ff781634de50a0b7adb0" exitCode=0
Nov 02 12:52:03.707879 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:03.707613    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-pz8j5" event={"ID":"632e48df-f4f7-4e68-819a-593ffd6a76d6","Type":"ContainerDied","Data":"ab4f6d47502dee62a1960429358f4fcf9877e1b30bb1ff781634de50a0b7adb0"}
Nov 02 12:52:03.710201 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:03.710170    4559 generic.go:356] "Generic (PLEG): container finished" podID="7666985a-ed93-432a-9bd8-c668a7bfd8f5" containerID="8ae9ec8c8d6a925274594b4addbedf5a237c297e29b3430b4c5992eaefe64e98" exitCode=0
Nov 02 12:52:03.710351 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:03.710283    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-s8hnh" event={"ID":"7666985a-ed93-432a-9bd8-c668a7bfd8f5","Type":"ContainerDied","Data":"8ae9ec8c8d6a925274594b4addbedf5a237c297e29b3430b4c5992eaefe64e98"}
Nov 02 12:52:03.710351 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:03.710334    4559 scope.go:122] "RemoveContainer" containerID="e1d6e40836826a764fb8d275885e3e19d0e23253fe2869d187becd49b38acd28"
Nov 02 12:52:03.712593 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:03.712471    4559 generic.go:356] "Generic (PLEG): container finished" podID="e1722c3b-b864-43f0-b5cb-94d96526858a" containerID="dde63fb43d91e0b37562756a515e89239f38a5241c38c45719959620fd8b6186" exitCode=0
Nov 02 12:52:03.712593 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:03.712521    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-wmzzn" event={"ID":"e1722c3b-b864-43f0-b5cb-94d96526858a","Type":"ContainerDied","Data":"dde63fb43d91e0b37562756a515e89239f38a5241c38c45719959620fd8b6186"}
Nov 02 12:52:03.807366 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:03.807338    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-pz8j5"
Nov 02 12:52:03.818276 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:03.818251    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-wmzzn"
Nov 02 12:52:03.841109 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:03.841087    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-s8hnh"
Nov 02 12:52:03.949880 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:03.949843    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1722c3b-b864-43f0-b5cb-94d96526858a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1722c3b-b864-43f0-b5cb-94d96526858a-workload-certs\") pod \"e1722c3b-b864-43f0-b5cb-94d96526858a\" (UID: \"e1722c3b-b864-43f0-b5cb-94d96526858a\") "
Nov 02 12:52:03.949880 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:03.949892    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7666985a-ed93-432a-9bd8-c668a7bfd8f5-istio-token\" (UniqueName: \"kubernetes.io/projected/7666985a-ed93-432a-9bd8-c668a7bfd8f5-istio-token\") pod \"7666985a-ed93-432a-9bd8-c668a7bfd8f5\" (UID: \"7666985a-ed93-432a-9bd8-c668a7bfd8f5\") "
Nov 02 12:52:03.950145 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:03.949947    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/632e48df-f4f7-4e68-819a-593ffd6a76d6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/632e48df-f4f7-4e68-819a-593ffd6a76d6-credential-socket\") pod \"632e48df-f4f7-4e68-819a-593ffd6a76d6\" (UID: \"632e48df-f4f7-4e68-819a-593ffd6a76d6\") "
Nov 02 12:52:03.950145 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:03.949981    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7666985a-ed93-432a-9bd8-c668a7bfd8f5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7666985a-ed93-432a-9bd8-c668a7bfd8f5-credential-socket\") pod \"7666985a-ed93-432a-9bd8-c668a7bfd8f5\" (UID: \"7666985a-ed93-432a-9bd8-c668a7bfd8f5\") "
Nov 02 12:52:03.950145 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:03.950013    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/632e48df-f4f7-4e68-819a-593ffd6a76d6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/632e48df-f4f7-4e68-819a-593ffd6a76d6-istio-envoy\") pod \"632e48df-f4f7-4e68-819a-593ffd6a76d6\" (UID: \"632e48df-f4f7-4e68-819a-593ffd6a76d6\") "
Nov 02 12:52:03.950145 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:03.950064    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7666985a-ed93-432a-9bd8-c668a7bfd8f5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7666985a-ed93-432a-9bd8-c668a7bfd8f5-workload-socket\") pod \"7666985a-ed93-432a-9bd8-c668a7bfd8f5\" (UID: \"7666985a-ed93-432a-9bd8-c668a7bfd8f5\") "
Nov 02 12:52:03.950145 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:03.950116    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/632e48df-f4f7-4e68-819a-593ffd6a76d6-istio-token\" (UniqueName: \"kubernetes.io/projected/632e48df-f4f7-4e68-819a-593ffd6a76d6-istio-token\") pod \"632e48df-f4f7-4e68-819a-593ffd6a76d6\" (UID: \"632e48df-f4f7-4e68-819a-593ffd6a76d6\") "
Nov 02 12:52:03.950418 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:03.950160    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/632e48df-f4f7-4e68-819a-593ffd6a76d6-kube-api-access-g4bv2\" (UniqueName: \"kubernetes.io/projected/632e48df-f4f7-4e68-819a-593ffd6a76d6-kube-api-access-g4bv2\") pod \"632e48df-f4f7-4e68-819a-593ffd6a76d6\" (UID: \"632e48df-f4f7-4e68-819a-593ffd6a76d6\") "
Nov 02 12:52:03.950418 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:03.950180    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1722c3b-b864-43f0-b5cb-94d96526858a-workload-certs" pod "e1722c3b-b864-43f0-b5cb-94d96526858a" (UID: "e1722c3b-b864-43f0-b5cb-94d96526858a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:52:03.950418 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:03.950213    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1722c3b-b864-43f0-b5cb-94d96526858a-istio-token\" (UniqueName: \"kubernetes.io/projected/e1722c3b-b864-43f0-b5cb-94d96526858a-istio-token\") pod \"e1722c3b-b864-43f0-b5cb-94d96526858a\" (UID: \"e1722c3b-b864-43f0-b5cb-94d96526858a\") "
Nov 02 12:52:03.950418 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:03.950250    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e1722c3b-b864-43f0-b5cb-94d96526858a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1722c3b-b864-43f0-b5cb-94d96526858a-istio-podinfo\") pod \"e1722c3b-b864-43f0-b5cb-94d96526858a\" (UID: \"e1722c3b-b864-43f0-b5cb-94d96526858a\") "
Nov 02 12:52:03.950418 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:03.950326    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7666985a-ed93-432a-9bd8-c668a7bfd8f5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7666985a-ed93-432a-9bd8-c668a7bfd8f5-istiod-ca-cert\") pod \"7666985a-ed93-432a-9bd8-c668a7bfd8f5\" (UID: \"7666985a-ed93-432a-9bd8-c668a7bfd8f5\") "
Nov 02 12:52:03.950418 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:03.950359    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1722c3b-b864-43f0-b5cb-94d96526858a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1722c3b-b864-43f0-b5cb-94d96526858a-istio-data\") pod \"e1722c3b-b864-43f0-b5cb-94d96526858a\" (UID: \"e1722c3b-b864-43f0-b5cb-94d96526858a\") "
Nov 02 12:52:03.950418 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:03.950392    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7666985a-ed93-432a-9bd8-c668a7bfd8f5-workload-socket" pod "7666985a-ed93-432a-9bd8-c668a7bfd8f5" (UID: "7666985a-ed93-432a-9bd8-c668a7bfd8f5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:52:03.950418 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:03.950397    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1722c3b-b864-43f0-b5cb-94d96526858a-kube-api-access-5m4c4\" (UniqueName: \"kubernetes.io/projected/e1722c3b-b864-43f0-b5cb-94d96526858a-kube-api-access-5m4c4\") pod \"e1722c3b-b864-43f0-b5cb-94d96526858a\" (UID: \"e1722c3b-b864-43f0-b5cb-94d96526858a\") "
Nov 02 12:52:03.950860 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:03.950436    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7666985a-ed93-432a-9bd8-c668a7bfd8f5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7666985a-ed93-432a-9bd8-c668a7bfd8f5-istio-data\") pod \"7666985a-ed93-432a-9bd8-c668a7bfd8f5\" (UID: \"7666985a-ed93-432a-9bd8-c668a7bfd8f5\") "
Nov 02 12:52:03.950860 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:03.950470    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7666985a-ed93-432a-9bd8-c668a7bfd8f5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7666985a-ed93-432a-9bd8-c668a7bfd8f5-workload-certs\") pod \"7666985a-ed93-432a-9bd8-c668a7bfd8f5\" (UID: \"7666985a-ed93-432a-9bd8-c668a7bfd8f5\") "
Nov 02 12:52:03.950860 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:03.950497    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/632e48df-f4f7-4e68-819a-593ffd6a76d6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/632e48df-f4f7-4e68-819a-593ffd6a76d6-workload-certs\") pod \"632e48df-f4f7-4e68-819a-593ffd6a76d6\" (UID: \"632e48df-f4f7-4e68-819a-593ffd6a76d6\") "
Nov 02 12:52:03.950860 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:03.950553    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7666985a-ed93-432a-9bd8-c668a7bfd8f5-kube-api-access-j4m4q\" (UniqueName: \"kubernetes.io/projected/7666985a-ed93-432a-9bd8-c668a7bfd8f5-kube-api-access-j4m4q\") pod \"7666985a-ed93-432a-9bd8-c668a7bfd8f5\" (UID: \"7666985a-ed93-432a-9bd8-c668a7bfd8f5\") "
Nov 02 12:52:03.950860 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:03.950581    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/632e48df-f4f7-4e68-819a-593ffd6a76d6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/632e48df-f4f7-4e68-819a-593ffd6a76d6-istiod-ca-cert\") pod \"632e48df-f4f7-4e68-819a-593ffd6a76d6\" (UID: \"632e48df-f4f7-4e68-819a-593ffd6a76d6\") "
Nov 02 12:52:03.950860 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:03.950668    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/632e48df-f4f7-4e68-819a-593ffd6a76d6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/632e48df-f4f7-4e68-819a-593ffd6a76d6-istio-podinfo\") pod \"632e48df-f4f7-4e68-819a-593ffd6a76d6\" (UID: \"632e48df-f4f7-4e68-819a-593ffd6a76d6\") "
Nov 02 12:52:03.950860 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:03.950723    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7666985a-ed93-432a-9bd8-c668a7bfd8f5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7666985a-ed93-432a-9bd8-c668a7bfd8f5-istio-podinfo\") pod \"7666985a-ed93-432a-9bd8-c668a7bfd8f5\" (UID: \"7666985a-ed93-432a-9bd8-c668a7bfd8f5\") "
Nov 02 12:52:03.950860 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:03.950771    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7666985a-ed93-432a-9bd8-c668a7bfd8f5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7666985a-ed93-432a-9bd8-c668a7bfd8f5-istio-envoy\") pod \"7666985a-ed93-432a-9bd8-c668a7bfd8f5\" (UID: \"7666985a-ed93-432a-9bd8-c668a7bfd8f5\") "
Nov 02 12:52:03.950860 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:03.950797    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/632e48df-f4f7-4e68-819a-593ffd6a76d6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/632e48df-f4f7-4e68-819a-593ffd6a76d6-workload-socket\") pod \"632e48df-f4f7-4e68-819a-593ffd6a76d6\" (UID: \"632e48df-f4f7-4e68-819a-593ffd6a76d6\") "
Nov 02 12:52:03.950860 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:03.950801    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7666985a-ed93-432a-9bd8-c668a7bfd8f5-istiod-ca-cert" pod "7666985a-ed93-432a-9bd8-c668a7bfd8f5" (UID: "7666985a-ed93-432a-9bd8-c668a7bfd8f5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:52:03.950860 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:03.950837    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1722c3b-b864-43f0-b5cb-94d96526858a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1722c3b-b864-43f0-b5cb-94d96526858a-workload-socket\") pod \"e1722c3b-b864-43f0-b5cb-94d96526858a\" (UID: \"e1722c3b-b864-43f0-b5cb-94d96526858a\") "
Nov 02 12:52:03.951419 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:03.950875    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1722c3b-b864-43f0-b5cb-94d96526858a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1722c3b-b864-43f0-b5cb-94d96526858a-credential-socket\") pod \"e1722c3b-b864-43f0-b5cb-94d96526858a\" (UID: \"e1722c3b-b864-43f0-b5cb-94d96526858a\") "
Nov 02 12:52:03.951419 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:03.950902    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/632e48df-f4f7-4e68-819a-593ffd6a76d6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/632e48df-f4f7-4e68-819a-593ffd6a76d6-istio-data\") pod \"632e48df-f4f7-4e68-819a-593ffd6a76d6\" (UID: \"632e48df-f4f7-4e68-819a-593ffd6a76d6\") "
Nov 02 12:52:03.951419 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:03.950929    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7666985a-ed93-432a-9bd8-c668a7bfd8f5-credential-socket" pod "7666985a-ed93-432a-9bd8-c668a7bfd8f5" (UID: "7666985a-ed93-432a-9bd8-c668a7bfd8f5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:52:03.951419 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:03.950947    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1722c3b-b864-43f0-b5cb-94d96526858a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1722c3b-b864-43f0-b5cb-94d96526858a-istio-envoy\") pod \"e1722c3b-b864-43f0-b5cb-94d96526858a\" (UID: \"e1722c3b-b864-43f0-b5cb-94d96526858a\") "
Nov 02 12:52:03.951419 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:03.950990    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e1722c3b-b864-43f0-b5cb-94d96526858a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1722c3b-b864-43f0-b5cb-94d96526858a-istiod-ca-cert\") pod \"e1722c3b-b864-43f0-b5cb-94d96526858a\" (UID: \"e1722c3b-b864-43f0-b5cb-94d96526858a\") "
Nov 02 12:52:03.951688 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:03.951542    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1722c3b-b864-43f0-b5cb-94d96526858a-istio-data" pod "e1722c3b-b864-43f0-b5cb-94d96526858a" (UID: "e1722c3b-b864-43f0-b5cb-94d96526858a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:52:03.951688 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:03.951673    4559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1722c3b-b864-43f0-b5cb-94d96526858a-workload-certs\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:52:03.951806 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:03.951693    4559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7666985a-ed93-432a-9bd8-c668a7bfd8f5-credential-socket\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:52:03.951806 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:03.951707    4559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7666985a-ed93-432a-9bd8-c668a7bfd8f5-workload-socket\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:52:03.951806 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:03.951755    4559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7666985a-ed93-432a-9bd8-c668a7bfd8f5-istiod-ca-cert\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:52:03.951806 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:03.951769    4559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1722c3b-b864-43f0-b5cb-94d96526858a-istio-data\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:52:03.952007 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:03.951860    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1722c3b-b864-43f0-b5cb-94d96526858a-istiod-ca-cert" pod "e1722c3b-b864-43f0-b5cb-94d96526858a" (UID: "e1722c3b-b864-43f0-b5cb-94d96526858a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:52:03.952304 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:03.952279    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1722c3b-b864-43f0-b5cb-94d96526858a-credential-socket" pod "e1722c3b-b864-43f0-b5cb-94d96526858a" (UID: "e1722c3b-b864-43f0-b5cb-94d96526858a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:52:03.952492 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:03.952469    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/632e48df-f4f7-4e68-819a-593ffd6a76d6-workload-socket" pod "632e48df-f4f7-4e68-819a-593ffd6a76d6" (UID: "632e48df-f4f7-4e68-819a-593ffd6a76d6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:52:03.952705 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:03.952681    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1722c3b-b864-43f0-b5cb-94d96526858a-workload-socket" pod "e1722c3b-b864-43f0-b5cb-94d96526858a" (UID: "e1722c3b-b864-43f0-b5cb-94d96526858a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:52:03.952892 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:03.952868    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/632e48df-f4f7-4e68-819a-593ffd6a76d6-istio-data" pod "632e48df-f4f7-4e68-819a-593ffd6a76d6" (UID: "632e48df-f4f7-4e68-819a-593ffd6a76d6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:52:03.953413 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:03.953386    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/632e48df-f4f7-4e68-819a-593ffd6a76d6-credential-socket" pod "632e48df-f4f7-4e68-819a-593ffd6a76d6" (UID: "632e48df-f4f7-4e68-819a-593ffd6a76d6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:52:03.953498 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:03.953410    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/632e48df-f4f7-4e68-819a-593ffd6a76d6-workload-certs" pod "632e48df-f4f7-4e68-819a-593ffd6a76d6" (UID: "632e48df-f4f7-4e68-819a-593ffd6a76d6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:52:03.953677 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:03.953611    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7666985a-ed93-432a-9bd8-c668a7bfd8f5-workload-certs" pod "7666985a-ed93-432a-9bd8-c668a7bfd8f5" (UID: "7666985a-ed93-432a-9bd8-c668a7bfd8f5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:52:03.953880 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:03.953856    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/632e48df-f4f7-4e68-819a-593ffd6a76d6-istiod-ca-cert" pod "632e48df-f4f7-4e68-819a-593ffd6a76d6" (UID: "632e48df-f4f7-4e68-819a-593ffd6a76d6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:52:03.954474 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:03.954447    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7666985a-ed93-432a-9bd8-c668a7bfd8f5-istio-data" pod "7666985a-ed93-432a-9bd8-c668a7bfd8f5" (UID: "7666985a-ed93-432a-9bd8-c668a7bfd8f5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:52:03.957239 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:03.956488    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7666985a-ed93-432a-9bd8-c668a7bfd8f5-istio-envoy" pod "7666985a-ed93-432a-9bd8-c668a7bfd8f5" (UID: "7666985a-ed93-432a-9bd8-c668a7bfd8f5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:52:03.957239 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:03.956502    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/632e48df-f4f7-4e68-819a-593ffd6a76d6-istio-envoy" pod "632e48df-f4f7-4e68-819a-593ffd6a76d6" (UID: "632e48df-f4f7-4e68-819a-593ffd6a76d6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:52:03.957239 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:03.956785    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1722c3b-b864-43f0-b5cb-94d96526858a-istio-envoy" pod "e1722c3b-b864-43f0-b5cb-94d96526858a" (UID: "e1722c3b-b864-43f0-b5cb-94d96526858a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:52:03.959617 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:03.958976    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7666985a-ed93-432a-9bd8-c668a7bfd8f5-istio-token" pod "7666985a-ed93-432a-9bd8-c668a7bfd8f5" (UID: "7666985a-ed93-432a-9bd8-c668a7bfd8f5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:52:03.961208 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:03.960565    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7666985a-ed93-432a-9bd8-c668a7bfd8f5-kube-api-access-j4m4q" pod "7666985a-ed93-432a-9bd8-c668a7bfd8f5" (UID: "7666985a-ed93-432a-9bd8-c668a7bfd8f5"). InnerVolumeSpecName "kube-api-access-j4m4q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:52:03.961208 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:03.961103    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/632e48df-f4f7-4e68-819a-593ffd6a76d6-istio-podinfo" pod "632e48df-f4f7-4e68-819a-593ffd6a76d6" (UID: "632e48df-f4f7-4e68-819a-593ffd6a76d6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 12:52:03.961902 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:03.961868    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e1722c3b-b864-43f0-b5cb-94d96526858a-istio-podinfo" pod "e1722c3b-b864-43f0-b5cb-94d96526858a" (UID: "e1722c3b-b864-43f0-b5cb-94d96526858a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 12:52:03.962125 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:03.962089    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1722c3b-b864-43f0-b5cb-94d96526858a-kube-api-access-5m4c4" pod "e1722c3b-b864-43f0-b5cb-94d96526858a" (UID: "e1722c3b-b864-43f0-b5cb-94d96526858a"). InnerVolumeSpecName "kube-api-access-5m4c4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:52:03.962408 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:03.962353    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1722c3b-b864-43f0-b5cb-94d96526858a-istio-token" pod "e1722c3b-b864-43f0-b5cb-94d96526858a" (UID: "e1722c3b-b864-43f0-b5cb-94d96526858a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:52:03.962705 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:03.962681    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/632e48df-f4f7-4e68-819a-593ffd6a76d6-istio-token" pod "632e48df-f4f7-4e68-819a-593ffd6a76d6" (UID: "632e48df-f4f7-4e68-819a-593ffd6a76d6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:52:03.962838 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:03.962814    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7666985a-ed93-432a-9bd8-c668a7bfd8f5-istio-podinfo" pod "7666985a-ed93-432a-9bd8-c668a7bfd8f5" (UID: "7666985a-ed93-432a-9bd8-c668a7bfd8f5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 12:52:03.964020 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:03.963995    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/632e48df-f4f7-4e68-819a-593ffd6a76d6-kube-api-access-g4bv2" pod "632e48df-f4f7-4e68-819a-593ffd6a76d6" (UID: "632e48df-f4f7-4e68-819a-593ffd6a76d6"). InnerVolumeSpecName "kube-api-access-g4bv2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:52:04.053531 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:04.053457    4559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1722c3b-b864-43f0-b5cb-94d96526858a-istio-token\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:52:04.053531 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:04.053484    4559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1722c3b-b864-43f0-b5cb-94d96526858a-istio-podinfo\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:52:04.053531 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:04.053494    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5m4c4\" (UniqueName: \"kubernetes.io/projected/e1722c3b-b864-43f0-b5cb-94d96526858a-kube-api-access-5m4c4\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:52:04.053531 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:04.053503    4559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7666985a-ed93-432a-9bd8-c668a7bfd8f5-istio-data\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:52:04.053531 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:04.053512    4559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7666985a-ed93-432a-9bd8-c668a7bfd8f5-workload-certs\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:52:04.053531 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:04.053523    4559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/632e48df-f4f7-4e68-819a-593ffd6a76d6-workload-certs\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:52:04.053531 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:04.053530    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4m4q\" (UniqueName: \"kubernetes.io/projected/7666985a-ed93-432a-9bd8-c668a7bfd8f5-kube-api-access-j4m4q\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:52:04.053957 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:04.053539    4559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/632e48df-f4f7-4e68-819a-593ffd6a76d6-istiod-ca-cert\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:52:04.053957 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:04.053548    4559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/632e48df-f4f7-4e68-819a-593ffd6a76d6-istio-podinfo\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:52:04.053957 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:04.053556    4559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7666985a-ed93-432a-9bd8-c668a7bfd8f5-istio-podinfo\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:52:04.053957 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:04.053564    4559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7666985a-ed93-432a-9bd8-c668a7bfd8f5-istio-envoy\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:52:04.053957 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:04.053572    4559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/632e48df-f4f7-4e68-819a-593ffd6a76d6-workload-socket\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:52:04.053957 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:04.053580    4559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1722c3b-b864-43f0-b5cb-94d96526858a-workload-socket\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:52:04.053957 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:04.053590    4559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1722c3b-b864-43f0-b5cb-94d96526858a-credential-socket\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:52:04.053957 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:04.053598    4559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/632e48df-f4f7-4e68-819a-593ffd6a76d6-istio-data\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:52:04.053957 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:04.053608    4559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1722c3b-b864-43f0-b5cb-94d96526858a-istio-envoy\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:52:04.053957 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:04.053615    4559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1722c3b-b864-43f0-b5cb-94d96526858a-istiod-ca-cert\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:52:04.053957 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:04.053634    4559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7666985a-ed93-432a-9bd8-c668a7bfd8f5-istio-token\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:52:04.053957 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:04.053675    4559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/632e48df-f4f7-4e68-819a-593ffd6a76d6-credential-socket\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:52:04.053957 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:04.053684    4559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/632e48df-f4f7-4e68-819a-593ffd6a76d6-istio-envoy\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:52:04.053957 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:04.053692    4559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/632e48df-f4f7-4e68-819a-593ffd6a76d6-istio-token\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:52:04.053957 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:04.053701    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g4bv2\" (UniqueName: \"kubernetes.io/projected/632e48df-f4f7-4e68-819a-593ffd6a76d6-kube-api-access-g4bv2\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:52:04.720693 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:04.720666    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-s8hnh"
Nov 02 12:52:04.721168 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:04.720687    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-s8hnh" event={"ID":"7666985a-ed93-432a-9bd8-c668a7bfd8f5","Type":"ContainerDied","Data":"59d5eca8ec2c4160076dd57f8d29c2f552b11a7873c3699425034e618f19394a"}
Nov 02 12:52:04.721168 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:04.720762    4559 scope.go:122] "RemoveContainer" containerID="8ae9ec8c8d6a925274594b4addbedf5a237c297e29b3430b4c5992eaefe64e98"
Nov 02 12:52:04.722268 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:04.722243    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-wmzzn"
Nov 02 12:52:04.722349 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:04.722275    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-wmzzn" event={"ID":"e1722c3b-b864-43f0-b5cb-94d96526858a","Type":"ContainerDied","Data":"7a9cb93e1f80ca38158863dfe60e1224ab8dab5a0b3affadca87b7419d682bec"}
Nov 02 12:52:04.728901 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:04.728873    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-pz8j5"
Nov 02 12:52:04.728988 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:04.728930    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-pz8j5" event={"ID":"632e48df-f4f7-4e68-819a-593ffd6a76d6","Type":"ContainerDied","Data":"50806eff80fe44f902e58ba4a621039514fbb842e9bda2164d1a20dc644fc5b3"}
Nov 02 12:52:04.731003 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:04.730981    4559 scope.go:122] "RemoveContainer" containerID="dde63fb43d91e0b37562756a515e89239f38a5241c38c45719959620fd8b6186"
Nov 02 12:52:04.741198 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:04.741183    4559 scope.go:122] "RemoveContainer" containerID="ab4f6d47502dee62a1960429358f4fcf9877e1b30bb1ff781634de50a0b7adb0"
Nov 02 12:52:04.788497 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:04.788464    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-s8hnh"]
Nov 02 12:52:04.801374 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:04.801337    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-s8hnh"]
Nov 02 12:52:04.838800 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:04.838763    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-pz8j5"]
Nov 02 12:52:04.843660 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:04.843612    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-pz8j5"]
Nov 02 12:52:04.881564 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:04.881536    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-wmzzn"]
Nov 02 12:52:04.888565 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:04.888535    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-wmzzn"]
Nov 02 12:52:05.894498 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:05.894466    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="632e48df-f4f7-4e68-819a-593ffd6a76d6" path="/var/lib/kubelet/pods/632e48df-f4f7-4e68-819a-593ffd6a76d6/volumes"
Nov 02 12:52:05.894968 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:05.894952    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7666985a-ed93-432a-9bd8-c668a7bfd8f5" path="/var/lib/kubelet/pods/7666985a-ed93-432a-9bd8-c668a7bfd8f5/volumes"
Nov 02 12:52:05.895438 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:05.895423    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e1722c3b-b864-43f0-b5cb-94d96526858a" path="/var/lib/kubelet/pods/e1722c3b-b864-43f0-b5cb-94d96526858a/volumes"
Nov 02 12:52:07.518130 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:07.518095    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-729108-6898dcb756-ksksg"]
Nov 02 12:52:07.518609 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:07.518296    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-729108-6898dcb756-ksksg" podUID="52cf3408-1bb5-49aa-9a7e-2b70df75ccf7" containerName="payload-processing" containerID="cri-o://aa8210b26914c5e0c1210e5ef099bb12595e70eba8ae998884b5ccc654a3005f" gracePeriod=30
Nov 02 12:52:07.537746 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:07.537718    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-729108-7f68cf4d79-8clxs"]
Nov 02 12:52:07.537925 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:07.537905    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-729108-7f68cf4d79-8clxs" podUID="e3df8de3-6c80-4695-8f07-0ec0873a7b9f" containerName="payload-pre-processing" containerID="cri-o://ebb88c03701127796f08227a9123f1afe36b16c9401ad355c735721390214c54" gracePeriod=30
Nov 02 12:52:07.766924 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:07.766883    4559 generic.go:356] "Generic (PLEG): container finished" podID="52cf3408-1bb5-49aa-9a7e-2b70df75ccf7" containerID="aa8210b26914c5e0c1210e5ef099bb12595e70eba8ae998884b5ccc654a3005f" exitCode=0
Nov 02 12:52:07.767102 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:07.767040    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-729108-6898dcb756-ksksg" event={"ID":"52cf3408-1bb5-49aa-9a7e-2b70df75ccf7","Type":"ContainerDied","Data":"aa8210b26914c5e0c1210e5ef099bb12595e70eba8ae998884b5ccc654a3005f"}
Nov 02 12:52:07.770124 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:07.770055    4559 generic.go:356] "Generic (PLEG): container finished" podID="e3df8de3-6c80-4695-8f07-0ec0873a7b9f" containerID="ebb88c03701127796f08227a9123f1afe36b16c9401ad355c735721390214c54" exitCode=0
Nov 02 12:52:07.770263 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:07.770220    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-729108-7f68cf4d79-8clxs" event={"ID":"e3df8de3-6c80-4695-8f07-0ec0873a7b9f","Type":"ContainerDied","Data":"ebb88c03701127796f08227a9123f1afe36b16c9401ad355c735721390214c54"}
Nov 02 12:52:07.826263 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:07.826234    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-729108-6898dcb756-ksksg"
Nov 02 12:52:07.975708 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:07.949744    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52cf3408-1bb5-49aa-9a7e-2b70df75ccf7-kube-api-access-6fcht\" (UniqueName: \"kubernetes.io/projected/52cf3408-1bb5-49aa-9a7e-2b70df75ccf7-kube-api-access-6fcht\") pod \"52cf3408-1bb5-49aa-9a7e-2b70df75ccf7\" (UID: \"52cf3408-1bb5-49aa-9a7e-2b70df75ccf7\") "
Nov 02 12:52:07.975708 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:07.950648    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52cf3408-1bb5-49aa-9a7e-2b70df75ccf7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52cf3408-1bb5-49aa-9a7e-2b70df75ccf7-plugins-config-volume\") pod \"52cf3408-1bb5-49aa-9a7e-2b70df75ccf7\" (UID: \"52cf3408-1bb5-49aa-9a7e-2b70df75ccf7\") "
Nov 02 12:52:07.975708 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:07.954293    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52cf3408-1bb5-49aa-9a7e-2b70df75ccf7-plugins-config-volume" pod "52cf3408-1bb5-49aa-9a7e-2b70df75ccf7" (UID: "52cf3408-1bb5-49aa-9a7e-2b70df75ccf7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:52:08.030738 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:08.028696    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52cf3408-1bb5-49aa-9a7e-2b70df75ccf7-kube-api-access-6fcht" pod "52cf3408-1bb5-49aa-9a7e-2b70df75ccf7" (UID: "52cf3408-1bb5-49aa-9a7e-2b70df75ccf7"). InnerVolumeSpecName "kube-api-access-6fcht". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:52:08.030738 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:08.028801    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-729108-7f68cf4d79-8clxs"
Nov 02 12:52:08.051950 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:08.051922    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3df8de3-6c80-4695-8f07-0ec0873a7b9f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3df8de3-6c80-4695-8f07-0ec0873a7b9f-plugins-config-volume\") pod \"e3df8de3-6c80-4695-8f07-0ec0873a7b9f\" (UID: \"e3df8de3-6c80-4695-8f07-0ec0873a7b9f\") "
Nov 02 12:52:08.052274 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:08.052250    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3df8de3-6c80-4695-8f07-0ec0873a7b9f-kube-api-access-kxj96\" (UniqueName: \"kubernetes.io/projected/e3df8de3-6c80-4695-8f07-0ec0873a7b9f-kube-api-access-kxj96\") pod \"e3df8de3-6c80-4695-8f07-0ec0873a7b9f\" (UID: \"e3df8de3-6c80-4695-8f07-0ec0873a7b9f\") "
Nov 02 12:52:08.052470 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:08.052334    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3df8de3-6c80-4695-8f07-0ec0873a7b9f-plugins-config-volume" pod "e3df8de3-6c80-4695-8f07-0ec0873a7b9f" (UID: "e3df8de3-6c80-4695-8f07-0ec0873a7b9f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:52:08.052917 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:08.052895    4559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52cf3408-1bb5-49aa-9a7e-2b70df75ccf7-plugins-config-volume\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:52:08.053000 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:08.052919    4559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3df8de3-6c80-4695-8f07-0ec0873a7b9f-plugins-config-volume\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:52:08.053000 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:08.052932    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6fcht\" (UniqueName: \"kubernetes.io/projected/52cf3408-1bb5-49aa-9a7e-2b70df75ccf7-kube-api-access-6fcht\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:52:08.055201 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:08.055178    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3df8de3-6c80-4695-8f07-0ec0873a7b9f-kube-api-access-kxj96" pod "e3df8de3-6c80-4695-8f07-0ec0873a7b9f" (UID: "e3df8de3-6c80-4695-8f07-0ec0873a7b9f"). InnerVolumeSpecName "kube-api-access-kxj96". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:52:08.153880 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:08.153846    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kxj96\" (UniqueName: \"kubernetes.io/projected/e3df8de3-6c80-4695-8f07-0ec0873a7b9f-kube-api-access-kxj96\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:52:08.777432 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:08.777400    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-729108-6898dcb756-ksksg"
Nov 02 12:52:08.777933 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:08.777430    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-729108-6898dcb756-ksksg" event={"ID":"52cf3408-1bb5-49aa-9a7e-2b70df75ccf7","Type":"ContainerDied","Data":"ec947d6ee9da31c323a9c2538e4788b65eae8001063fcc5e8bec5c4fbdb8ce66"}
Nov 02 12:52:08.777933 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:08.777523    4559 scope.go:122] "RemoveContainer" containerID="aa8210b26914c5e0c1210e5ef099bb12595e70eba8ae998884b5ccc654a3005f"
Nov 02 12:52:08.779033 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:08.779007    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-729108-7f68cf4d79-8clxs" event={"ID":"e3df8de3-6c80-4695-8f07-0ec0873a7b9f","Type":"ContainerDied","Data":"c2b054e027eeb4f3f64d8defabd6cef85fcffeaff00bc46984f61587ec3b22ef"}
Nov 02 12:52:08.779033 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:08.779023    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-729108-7f68cf4d79-8clxs"
Nov 02 12:52:08.788564 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:08.788538    4559 scope.go:122] "RemoveContainer" containerID="ebb88c03701127796f08227a9123f1afe36b16c9401ad355c735721390214c54"
Nov 02 12:52:08.831937 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:08.831900    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-729108-7f68cf4d79-8clxs"]
Nov 02 12:52:08.844591 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:08.844558    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-729108-7f68cf4d79-8clxs"]
Nov 02 12:52:08.881361 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:08.881326    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-729108-6898dcb756-ksksg"]
Nov 02 12:52:08.884033 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:08.884007    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-729108-6898dcb756-ksksg"]
Nov 02 12:52:09.929148 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:09.921409    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52cf3408-1bb5-49aa-9a7e-2b70df75ccf7" path="/var/lib/kubelet/pods/52cf3408-1bb5-49aa-9a7e-2b70df75ccf7/volumes"
Nov 02 12:52:09.929148 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:09.922505    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3df8de3-6c80-4695-8f07-0ec0873a7b9f" path="/var/lib/kubelet/pods/e3df8de3-6c80-4695-8f07-0ec0873a7b9f/volumes"
Nov 02 12:52:11.686044 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:11.686015    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-d7801d-678658877b-bwjnd"
Nov 02 12:52:12.692045 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:12.692015    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d7801d-5f9f5c649f-9frfl"
Nov 02 12:52:15.087578 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:52:15.087540    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqcmj" containerName="extractor"
Nov 02 12:52:19.260856 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:19.260302    4559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-r6rdp_2fef2bd6-5793-4c34-a390-37c907bcfd19/payload-processing/0.log"
Nov 02 12:52:19.484597 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:19.484551    4559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-r6rdp_2fef2bd6-5793-4c34-a390-37c907bcfd19/payload-processing/0.log"
Nov 02 12:52:19.708556 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:19.708529    4559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-d7801d-678658877b-bwjnd_997a3029-5f17-4a07-be79-0d8f7aa2e4b2/payload-processing/0.log"
Nov 02 12:52:20.611838 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:20.611769    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-61e341/distinct-w1-61e341-kserve-787cd566f4-cjj8s" podUID="6886ce28-5800-402d-b893-f92109e289a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:52:20.639600 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:20.639555    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-61e341/distinct-2-w1-61e341-kserve-954cf956f-n7jmg" podUID="c5e1a5f8-97b4-469d-be9a-eae7ca995283" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:52:33.002917 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:33.002880    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-vtpz8"]
Nov 02 12:52:33.003500 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:33.003484    4559 cpu_manager.go:414] "containerMap: removing container" podUID="7666985a-ed93-432a-9bd8-c668a7bfd8f5" containerName="istio-proxy"
Nov 02 12:52:33.003551 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:33.003501    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7666985a-ed93-432a-9bd8-c668a7bfd8f5" containerName="istio-proxy"
Nov 02 12:52:33.003551 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:33.003516    4559 cpu_manager.go:414] "containerMap: removing container" podUID="52cf3408-1bb5-49aa-9a7e-2b70df75ccf7" containerName="payload-processing"
Nov 02 12:52:33.003551 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:33.003522    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52cf3408-1bb5-49aa-9a7e-2b70df75ccf7" containerName="payload-processing"
Nov 02 12:52:33.003551 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:33.003533    4559 cpu_manager.go:414] "containerMap: removing container" podUID="632e48df-f4f7-4e68-819a-593ffd6a76d6" containerName="istio-proxy"
Nov 02 12:52:33.003551 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:33.003539    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="632e48df-f4f7-4e68-819a-593ffd6a76d6" containerName="istio-proxy"
Nov 02 12:52:33.003766 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:33.003551    4559 cpu_manager.go:414] "containerMap: removing container" podUID="7666985a-ed93-432a-9bd8-c668a7bfd8f5" containerName="istio-proxy"
Nov 02 12:52:33.003766 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:33.003559    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7666985a-ed93-432a-9bd8-c668a7bfd8f5" containerName="istio-proxy"
Nov 02 12:52:33.003766 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:33.003567    4559 cpu_manager.go:414] "containerMap: removing container" podUID="e1722c3b-b864-43f0-b5cb-94d96526858a" containerName="istio-proxy"
Nov 02 12:52:33.003766 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:33.003574    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1722c3b-b864-43f0-b5cb-94d96526858a" containerName="istio-proxy"
Nov 02 12:52:33.003766 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:33.003608    4559 cpu_manager.go:414] "containerMap: removing container" podUID="e3df8de3-6c80-4695-8f07-0ec0873a7b9f" containerName="payload-pre-processing"
Nov 02 12:52:33.003766 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:33.003614    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3df8de3-6c80-4695-8f07-0ec0873a7b9f" containerName="payload-pre-processing"
Nov 02 12:52:33.003766 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:33.003763    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3df8de3-6c80-4695-8f07-0ec0873a7b9f" containerName="payload-pre-processing"
Nov 02 12:52:33.004043 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:33.003779    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="7666985a-ed93-432a-9bd8-c668a7bfd8f5" containerName="istio-proxy"
Nov 02 12:52:33.004043 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:33.003789    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="52cf3408-1bb5-49aa-9a7e-2b70df75ccf7" containerName="payload-processing"
Nov 02 12:52:33.004043 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:33.003798    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="632e48df-f4f7-4e68-819a-593ffd6a76d6" containerName="istio-proxy"
Nov 02 12:52:33.004043 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:33.003809    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="7666985a-ed93-432a-9bd8-c668a7bfd8f5" containerName="istio-proxy"
Nov 02 12:52:33.004043 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:33.003819    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1722c3b-b864-43f0-b5cb-94d96526858a" containerName="istio-proxy"
Nov 02 12:52:33.008993 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:33.008973    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-vtpz8"
Nov 02 12:52:33.032008 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:33.025298    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-mdchl"]
Nov 02 12:52:33.032008 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:33.030326    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-mdchl"
Nov 02 12:52:33.038005 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:33.037977    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-vtpz8"]
Nov 02 12:52:33.057499 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:33.053394    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-mdchl"]
Nov 02 12:52:33.132770 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:33.132721    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb1e97a1-f680-4922-920c-ee0a3468cb1f-istio-data\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-mdchl\" (UID: \"bb1e97a1-f680-4922-920c-ee0a3468cb1f\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-mdchl"
Nov 02 12:52:33.132964 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:33.132788    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb1e97a1-f680-4922-920c-ee0a3468cb1f-istiod-ca-cert\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-mdchl\" (UID: \"bb1e97a1-f680-4922-920c-ee0a3468cb1f\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-mdchl"
Nov 02 12:52:33.132964 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:33.132896    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7f18bc2-015d-41da-a666-912b2e250e2a-credential-socket\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-vtpz8\" (UID: \"d7f18bc2-015d-41da-a666-912b2e250e2a\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-vtpz8"
Nov 02 12:52:33.132964 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:33.132928    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7f18bc2-015d-41da-a666-912b2e250e2a-istio-data\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-vtpz8\" (UID: \"d7f18bc2-015d-41da-a666-912b2e250e2a\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-vtpz8"
Nov 02 12:52:33.132964 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:33.132947    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb1e97a1-f680-4922-920c-ee0a3468cb1f-istio-envoy\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-mdchl\" (UID: \"bb1e97a1-f680-4922-920c-ee0a3468cb1f\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-mdchl"
Nov 02 12:52:33.133179 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:33.133029    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fcmdk\" (UniqueName: \"kubernetes.io/projected/bb1e97a1-f680-4922-920c-ee0a3468cb1f-kube-api-access-fcmdk\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-mdchl\" (UID: \"bb1e97a1-f680-4922-920c-ee0a3468cb1f\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-mdchl"
Nov 02 12:52:33.133179 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:33.133124    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7f18bc2-015d-41da-a666-912b2e250e2a-istio-envoy\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-vtpz8\" (UID: \"d7f18bc2-015d-41da-a666-912b2e250e2a\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-vtpz8"
Nov 02 12:52:33.133179 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:33.133143    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb1e97a1-f680-4922-920c-ee0a3468cb1f-istio-token\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-mdchl\" (UID: \"bb1e97a1-f680-4922-920c-ee0a3468cb1f\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-mdchl"
Nov 02 12:52:33.133179 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:33.133167    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7f18bc2-015d-41da-a666-912b2e250e2a-istio-podinfo\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-vtpz8\" (UID: \"d7f18bc2-015d-41da-a666-912b2e250e2a\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-vtpz8"
Nov 02 12:52:33.133339 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:33.133271    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7f18bc2-015d-41da-a666-912b2e250e2a-istio-token\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-vtpz8\" (UID: \"d7f18bc2-015d-41da-a666-912b2e250e2a\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-vtpz8"
Nov 02 12:52:33.133339 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:33.133298    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb1e97a1-f680-4922-920c-ee0a3468cb1f-credential-socket\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-mdchl\" (UID: \"bb1e97a1-f680-4922-920c-ee0a3468cb1f\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-mdchl"
Nov 02 12:52:33.133339 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:33.133330    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkcq6\" (UniqueName: \"kubernetes.io/projected/d7f18bc2-015d-41da-a666-912b2e250e2a-kube-api-access-dkcq6\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-vtpz8\" (UID: \"d7f18bc2-015d-41da-a666-912b2e250e2a\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-vtpz8"
Nov 02 12:52:33.133476 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:33.133345    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb1e97a1-f680-4922-920c-ee0a3468cb1f-workload-certs\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-mdchl\" (UID: \"bb1e97a1-f680-4922-920c-ee0a3468cb1f\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-mdchl"
Nov 02 12:52:33.133476 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:33.133409    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7f18bc2-015d-41da-a666-912b2e250e2a-istiod-ca-cert\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-vtpz8\" (UID: \"d7f18bc2-015d-41da-a666-912b2e250e2a\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-vtpz8"
Nov 02 12:52:33.133706 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:33.133685    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7f18bc2-015d-41da-a666-912b2e250e2a-workload-socket\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-vtpz8\" (UID: \"d7f18bc2-015d-41da-a666-912b2e250e2a\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-vtpz8"
Nov 02 12:52:33.133800 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:33.133713    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb1e97a1-f680-4922-920c-ee0a3468cb1f-istio-podinfo\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-mdchl\" (UID: \"bb1e97a1-f680-4922-920c-ee0a3468cb1f\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-mdchl"
Nov 02 12:52:33.133800 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:33.133745    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb1e97a1-f680-4922-920c-ee0a3468cb1f-workload-socket\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-mdchl\" (UID: \"bb1e97a1-f680-4922-920c-ee0a3468cb1f\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-mdchl"
Nov 02 12:52:33.133907 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:33.133873    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7f18bc2-015d-41da-a666-912b2e250e2a-workload-certs\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-vtpz8\" (UID: \"d7f18bc2-015d-41da-a666-912b2e250e2a\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-vtpz8"
Nov 02 12:52:33.235125 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:33.235091    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkcq6\" (UniqueName: \"kubernetes.io/projected/d7f18bc2-015d-41da-a666-912b2e250e2a-kube-api-access-dkcq6\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-vtpz8\" (UID: \"d7f18bc2-015d-41da-a666-912b2e250e2a\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-vtpz8"
Nov 02 12:52:33.235125 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:33.235126    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb1e97a1-f680-4922-920c-ee0a3468cb1f-workload-certs\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-mdchl\" (UID: \"bb1e97a1-f680-4922-920c-ee0a3468cb1f\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-mdchl"
Nov 02 12:52:33.235372 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:33.235145    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7f18bc2-015d-41da-a666-912b2e250e2a-istiod-ca-cert\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-vtpz8\" (UID: \"d7f18bc2-015d-41da-a666-912b2e250e2a\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-vtpz8"
Nov 02 12:52:33.235372 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:33.235183    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7f18bc2-015d-41da-a666-912b2e250e2a-workload-socket\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-vtpz8\" (UID: \"d7f18bc2-015d-41da-a666-912b2e250e2a\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-vtpz8"
Nov 02 12:52:33.235372 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:33.235208    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb1e97a1-f680-4922-920c-ee0a3468cb1f-istio-podinfo\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-mdchl\" (UID: \"bb1e97a1-f680-4922-920c-ee0a3468cb1f\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-mdchl"
Nov 02 12:52:33.235372 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:33.235252    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb1e97a1-f680-4922-920c-ee0a3468cb1f-workload-socket\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-mdchl\" (UID: \"bb1e97a1-f680-4922-920c-ee0a3468cb1f\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-mdchl"
Nov 02 12:52:33.235372 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:33.235322    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7f18bc2-015d-41da-a666-912b2e250e2a-workload-certs\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-vtpz8\" (UID: \"d7f18bc2-015d-41da-a666-912b2e250e2a\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-vtpz8"
Nov 02 12:52:33.235682 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:33.235395    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb1e97a1-f680-4922-920c-ee0a3468cb1f-istio-data\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-mdchl\" (UID: \"bb1e97a1-f680-4922-920c-ee0a3468cb1f\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-mdchl"
Nov 02 12:52:33.235682 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:33.235456    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb1e97a1-f680-4922-920c-ee0a3468cb1f-istiod-ca-cert\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-mdchl\" (UID: \"bb1e97a1-f680-4922-920c-ee0a3468cb1f\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-mdchl"
Nov 02 12:52:33.235682 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:33.235528    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7f18bc2-015d-41da-a666-912b2e250e2a-credential-socket\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-vtpz8\" (UID: \"d7f18bc2-015d-41da-a666-912b2e250e2a\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-vtpz8"
Nov 02 12:52:33.235682 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:33.235558    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7f18bc2-015d-41da-a666-912b2e250e2a-istio-data\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-vtpz8\" (UID: \"d7f18bc2-015d-41da-a666-912b2e250e2a\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-vtpz8"
Nov 02 12:52:33.235682 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:33.235581    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb1e97a1-f680-4922-920c-ee0a3468cb1f-istio-envoy\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-mdchl\" (UID: \"bb1e97a1-f680-4922-920c-ee0a3468cb1f\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-mdchl"
Nov 02 12:52:33.235682 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:33.235631    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcmdk\" (UniqueName: \"kubernetes.io/projected/bb1e97a1-f680-4922-920c-ee0a3468cb1f-kube-api-access-fcmdk\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-mdchl\" (UID: \"bb1e97a1-f680-4922-920c-ee0a3468cb1f\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-mdchl"
Nov 02 12:52:33.235682 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:33.235679    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7f18bc2-015d-41da-a666-912b2e250e2a-istio-envoy\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-vtpz8\" (UID: \"d7f18bc2-015d-41da-a666-912b2e250e2a\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-vtpz8"
Nov 02 12:52:33.236081 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:33.235689    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb1e97a1-f680-4922-920c-ee0a3468cb1f-workload-certs\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-mdchl\" (UID: \"bb1e97a1-f680-4922-920c-ee0a3468cb1f\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-mdchl"
Nov 02 12:52:33.236081 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:33.235703    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb1e97a1-f680-4922-920c-ee0a3468cb1f-istio-token\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-mdchl\" (UID: \"bb1e97a1-f680-4922-920c-ee0a3468cb1f\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-mdchl"
Nov 02 12:52:33.236081 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:33.235708    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb1e97a1-f680-4922-920c-ee0a3468cb1f-workload-socket\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-mdchl\" (UID: \"bb1e97a1-f680-4922-920c-ee0a3468cb1f\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-mdchl"
Nov 02 12:52:33.236081 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:33.235728    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7f18bc2-015d-41da-a666-912b2e250e2a-workload-socket\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-vtpz8\" (UID: \"d7f18bc2-015d-41da-a666-912b2e250e2a\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-vtpz8"
Nov 02 12:52:33.236081 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:33.235749    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7f18bc2-015d-41da-a666-912b2e250e2a-istio-podinfo\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-vtpz8\" (UID: \"d7f18bc2-015d-41da-a666-912b2e250e2a\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-vtpz8"
Nov 02 12:52:33.236081 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:33.235760    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7f18bc2-015d-41da-a666-912b2e250e2a-workload-certs\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-vtpz8\" (UID: \"d7f18bc2-015d-41da-a666-912b2e250e2a\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-vtpz8"
Nov 02 12:52:33.236081 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:33.235840    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7f18bc2-015d-41da-a666-912b2e250e2a-istio-token\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-vtpz8\" (UID: \"d7f18bc2-015d-41da-a666-912b2e250e2a\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-vtpz8"
Nov 02 12:52:33.236081 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:33.235882    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb1e97a1-f680-4922-920c-ee0a3468cb1f-credential-socket\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-mdchl\" (UID: \"bb1e97a1-f680-4922-920c-ee0a3468cb1f\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-mdchl"
Nov 02 12:52:33.236081 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:33.236007    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7f18bc2-015d-41da-a666-912b2e250e2a-istio-data\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-vtpz8\" (UID: \"d7f18bc2-015d-41da-a666-912b2e250e2a\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-vtpz8"
Nov 02 12:52:33.236081 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:33.236011    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7f18bc2-015d-41da-a666-912b2e250e2a-istiod-ca-cert\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-vtpz8\" (UID: \"d7f18bc2-015d-41da-a666-912b2e250e2a\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-vtpz8"
Nov 02 12:52:33.236653 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:33.236197    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb1e97a1-f680-4922-920c-ee0a3468cb1f-credential-socket\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-mdchl\" (UID: \"bb1e97a1-f680-4922-920c-ee0a3468cb1f\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-mdchl"
Nov 02 12:52:33.236653 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:33.236262    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7f18bc2-015d-41da-a666-912b2e250e2a-credential-socket\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-vtpz8\" (UID: \"d7f18bc2-015d-41da-a666-912b2e250e2a\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-vtpz8"
Nov 02 12:52:33.236653 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:33.236437    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb1e97a1-f680-4922-920c-ee0a3468cb1f-istiod-ca-cert\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-mdchl\" (UID: \"bb1e97a1-f680-4922-920c-ee0a3468cb1f\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-mdchl"
Nov 02 12:52:33.236653 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:33.236505    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb1e97a1-f680-4922-920c-ee0a3468cb1f-istio-data\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-mdchl\" (UID: \"bb1e97a1-f680-4922-920c-ee0a3468cb1f\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-mdchl"
Nov 02 12:52:33.238039 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:33.238015    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb1e97a1-f680-4922-920c-ee0a3468cb1f-istio-podinfo\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-mdchl\" (UID: \"bb1e97a1-f680-4922-920c-ee0a3468cb1f\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-mdchl"
Nov 02 12:52:33.238438 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:33.238414    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb1e97a1-f680-4922-920c-ee0a3468cb1f-istio-envoy\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-mdchl\" (UID: \"bb1e97a1-f680-4922-920c-ee0a3468cb1f\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-mdchl"
Nov 02 12:52:33.238548 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:33.238490    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7f18bc2-015d-41da-a666-912b2e250e2a-istio-envoy\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-vtpz8\" (UID: \"d7f18bc2-015d-41da-a666-912b2e250e2a\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-vtpz8"
Nov 02 12:52:33.238596 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:33.238571    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7f18bc2-015d-41da-a666-912b2e250e2a-istio-podinfo\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-vtpz8\" (UID: \"d7f18bc2-015d-41da-a666-912b2e250e2a\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-vtpz8"
Nov 02 12:52:33.259330 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:33.259272    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb1e97a1-f680-4922-920c-ee0a3468cb1f-istio-token\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-mdchl\" (UID: \"bb1e97a1-f680-4922-920c-ee0a3468cb1f\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-mdchl"
Nov 02 12:52:33.259524 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:33.259500    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7f18bc2-015d-41da-a666-912b2e250e2a-istio-token\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-vtpz8\" (UID: \"d7f18bc2-015d-41da-a666-912b2e250e2a\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-vtpz8"
Nov 02 12:52:33.259567 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:33.259510    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkcq6\" (UniqueName: \"kubernetes.io/projected/d7f18bc2-015d-41da-a666-912b2e250e2a-kube-api-access-dkcq6\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-vtpz8\" (UID: \"d7f18bc2-015d-41da-a666-912b2e250e2a\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-vtpz8"
Nov 02 12:52:33.259620 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:33.259603    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fcmdk\" (UniqueName: \"kubernetes.io/projected/bb1e97a1-f680-4922-920c-ee0a3468cb1f-kube-api-access-fcmdk\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-mdchl\" (UID: \"bb1e97a1-f680-4922-920c-ee0a3468cb1f\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-mdchl"
Nov 02 12:52:33.324927 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:33.324895    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-vtpz8"
Nov 02 12:52:33.351370 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:33.351344    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-mdchl"
Nov 02 12:52:33.648312 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:52:33.648283    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7f18bc2_015d_41da_a666_912b2e250e2a.slice/crio-1fc462bf450a25de16439d40abc49c09bba0bc08d2b35592a47a07b56068abd9 WatchSource:0}: Error finding container 1fc462bf450a25de16439d40abc49c09bba0bc08d2b35592a47a07b56068abd9: Status 404 returned error can't find the container with id 1fc462bf450a25de16439d40abc49c09bba0bc08d2b35592a47a07b56068abd9
Nov 02 12:52:33.648629 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:33.648608    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-vtpz8"]
Nov 02 12:52:33.650738 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:33.650707    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:52:33.650816 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:33.650770    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:52:33.650816 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:33.650801    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:52:33.688012 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:52:33.687987    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb1e97a1_f680_4922_920c_ee0a3468cb1f.slice/crio-83c2158a2a6cac15ee3ff57eaddd3b9775ca87d24749ec97cb83a8ae39a02334 WatchSource:0}: Error finding container 83c2158a2a6cac15ee3ff57eaddd3b9775ca87d24749ec97cb83a8ae39a02334: Status 404 returned error can't find the container with id 83c2158a2a6cac15ee3ff57eaddd3b9775ca87d24749ec97cb83a8ae39a02334
Nov 02 12:52:33.688267 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:33.688244    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-mdchl"]
Nov 02 12:52:33.690768 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:33.690739    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:52:33.690862 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:33.690807    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:52:33.690862 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:33.690837    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:52:34.079590 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:34.079551    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-mdchl" event={"ID":"bb1e97a1-f680-4922-920c-ee0a3468cb1f","Type":"ContainerStarted","Data":"3e0c9b0bc6a05cd6cd0b8e306b2739f4742aaabf4c9a9bd3666c4584dfce9b96"}
Nov 02 12:52:34.080091 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:34.079611    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-mdchl" event={"ID":"bb1e97a1-f680-4922-920c-ee0a3468cb1f","Type":"ContainerStarted","Data":"83c2158a2a6cac15ee3ff57eaddd3b9775ca87d24749ec97cb83a8ae39a02334"}
Nov 02 12:52:34.081406 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:34.081373    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-vtpz8" event={"ID":"d7f18bc2-015d-41da-a666-912b2e250e2a","Type":"ContainerStarted","Data":"a49513b40f2e0a5da91fdbeeac25ddfa386465768ff6828922520c816389cb94"}
Nov 02 12:52:34.081532 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:34.081424    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-vtpz8" event={"ID":"d7f18bc2-015d-41da-a666-912b2e250e2a","Type":"ContainerStarted","Data":"1fc462bf450a25de16439d40abc49c09bba0bc08d2b35592a47a07b56068abd9"}
Nov 02 12:52:34.142337 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:34.142276    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-mdchl" podStartSLOduration=2.142258209 podStartE2EDuration="2.142258209s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:52:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:52:34.13855769 +0000 UTC m=+1550.393956786" watchObservedRunningTime="2026-11-02 12:52:34.142258209 +0000 UTC m=+1550.397657304"
Nov 02 12:52:34.185438 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:34.185370    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-vtpz8" podStartSLOduration=2.185350318 podStartE2EDuration="2.185350318s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:52:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:52:34.18404391 +0000 UTC m=+1550.439443005" watchObservedRunningTime="2026-11-02 12:52:34.185350318 +0000 UTC m=+1550.440749416"
Nov 02 12:52:34.327746 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:34.327704    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-vtpz8"
Nov 02 12:52:34.328632 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:34.328598    4559 patch_prober.go:29] interesting pod/e2e-worker-w2-95ee26-openshift-default-66679d8f57-vtpz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" start-of-body=
Nov 02 12:52:34.328849 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:34.328711    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-vtpz8" podUID="d7f18bc2-015d-41da-a666-912b2e250e2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused"
Nov 02 12:52:34.352287 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:34.352196    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-mdchl"
Nov 02 12:52:34.353847 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:34.353819    4559 patch_prober.go:29] interesting pod/e2e-worker-w2-95ee26-openshift-default-66679d8f57-mdchl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" start-of-body=
Nov 02 12:52:34.353986 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:34.353886    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-mdchl" podUID="bb1e97a1-f680-4922-920c-ee0a3468cb1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused"
Nov 02 12:52:34.540449 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:34.540414    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-eb24df-864494fcf-kzf6t"]
Nov 02 12:52:34.540632 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:34.540603    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-eb24df-864494fcf-kzf6t" podUID="38679526-8764-40bb-a5c0-be287e2c6e5b" containerName="payload-pre-processing" containerID="cri-o://58c789e83735b6170c1b7ed603ece9861195ccf4eadf8b944732624640fadca3" gracePeriod=30
Nov 02 12:52:35.157911 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:35.157874    4559 generic.go:356] "Generic (PLEG): container finished" podID="38679526-8764-40bb-a5c0-be287e2c6e5b" containerID="58c789e83735b6170c1b7ed603ece9861195ccf4eadf8b944732624640fadca3" exitCode=0
Nov 02 12:52:35.158288 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:35.157945    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-eb24df-864494fcf-kzf6t" event={"ID":"38679526-8764-40bb-a5c0-be287e2c6e5b","Type":"ContainerDied","Data":"58c789e83735b6170c1b7ed603ece9861195ccf4eadf8b944732624640fadca3"}
Nov 02 12:52:35.158288 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:35.158023    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-eb24df-864494fcf-kzf6t" event={"ID":"38679526-8764-40bb-a5c0-be287e2c6e5b","Type":"ContainerDied","Data":"6a8774e895f6c77a8f794d07e42dcb46d7a4b72377b91720c4c3fcfcbf7b9566"}
Nov 02 12:52:35.158288 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:35.158040    4559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6a8774e895f6c77a8f794d07e42dcb46d7a4b72377b91720c4c3fcfcbf7b9566"
Nov 02 12:52:35.160033 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:35.160006    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-eb24df-864494fcf-kzf6t"
Nov 02 12:52:35.310725 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:35.310489    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38679526-8764-40bb-a5c0-be287e2c6e5b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38679526-8764-40bb-a5c0-be287e2c6e5b-plugins-config-volume\") pod \"38679526-8764-40bb-a5c0-be287e2c6e5b\" (UID: \"38679526-8764-40bb-a5c0-be287e2c6e5b\") "
Nov 02 12:52:35.310725 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:35.310719    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38679526-8764-40bb-a5c0-be287e2c6e5b-kube-api-access-2glwc\" (UniqueName: \"kubernetes.io/projected/38679526-8764-40bb-a5c0-be287e2c6e5b-kube-api-access-2glwc\") pod \"38679526-8764-40bb-a5c0-be287e2c6e5b\" (UID: \"38679526-8764-40bb-a5c0-be287e2c6e5b\") "
Nov 02 12:52:35.338668 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:35.327928    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38679526-8764-40bb-a5c0-be287e2c6e5b-plugins-config-volume" pod "38679526-8764-40bb-a5c0-be287e2c6e5b" (UID: "38679526-8764-40bb-a5c0-be287e2c6e5b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:52:35.353881 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:35.353735    4559 patch_prober.go:29] interesting pod/e2e-worker-w2-95ee26-openshift-default-66679d8f57-mdchl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" start-of-body=
Nov 02 12:52:35.353881 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:35.353797    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-mdchl" podUID="bb1e97a1-f680-4922-920c-ee0a3468cb1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused"
Nov 02 12:52:35.357663 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:35.354673    4559 patch_prober.go:29] interesting pod/e2e-worker-w2-95ee26-openshift-default-66679d8f57-vtpz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" start-of-body=
Nov 02 12:52:35.357663 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:35.354813    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-vtpz8" podUID="d7f18bc2-015d-41da-a666-912b2e250e2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused"
Nov 02 12:52:35.357663 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:35.356040    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38679526-8764-40bb-a5c0-be287e2c6e5b-kube-api-access-2glwc" pod "38679526-8764-40bb-a5c0-be287e2c6e5b" (UID: "38679526-8764-40bb-a5c0-be287e2c6e5b"). InnerVolumeSpecName "kube-api-access-2glwc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:52:35.429663 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:35.419011    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2glwc\" (UniqueName: \"kubernetes.io/projected/38679526-8764-40bb-a5c0-be287e2c6e5b-kube-api-access-2glwc\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:52:35.429663 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:35.419059    4559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38679526-8764-40bb-a5c0-be287e2c6e5b-plugins-config-volume\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:52:36.164349 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:36.164310    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-eb24df-864494fcf-kzf6t"
Nov 02 12:52:36.243301 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:36.243178    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-eb24df-864494fcf-kzf6t"]
Nov 02 12:52:36.250706 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:36.250672    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-eb24df-864494fcf-kzf6t"]
Nov 02 12:52:36.328110 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:36.328059    4559 patch_prober.go:29] interesting pod/e2e-worker-w2-95ee26-openshift-default-66679d8f57-vtpz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" start-of-body=
Nov 02 12:52:36.328327 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:36.328128    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-vtpz8" podUID="d7f18bc2-015d-41da-a666-912b2e250e2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused"
Nov 02 12:52:36.352555 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:36.352525    4559 patch_prober.go:29] interesting pod/e2e-worker-w2-95ee26-openshift-default-66679d8f57-mdchl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" start-of-body=
Nov 02 12:52:36.352713 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:36.352573    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-mdchl" podUID="bb1e97a1-f680-4922-920c-ee0a3468cb1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused"
Nov 02 12:52:36.865657 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:36.865605    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-299e74-78c848cbc6-xk4wx"]
Nov 02 12:52:36.866030 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:36.866000    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-299e74-78c848cbc6-xk4wx" podUID="9d93eaf1-b398-4b86-9156-845a73b24aa9" containerName="payload-processing" containerID="cri-o://712feb6225923324880ddeea02e69d552b0dd7f3428ae50d233012ccf9f4f478" gracePeriod=30
Nov 02 12:52:36.931662 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:36.909757    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-299e74-f6d59fd87-5s5pn"]
Nov 02 12:52:36.937421 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:36.937099    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-299e74-f6d59fd87-5s5pn" podUID="6f136bff-02b6-41b6-9bee-cf5626c366dc" containerName="payload-pre-processing" containerID="cri-o://fa3e160d1374c182c1eef451397023b8a8a89f3dc776de7c5cd24574d1130d86" gracePeriod=30
Nov 02 12:52:37.343669 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:37.343241    4559 patch_prober.go:29] interesting pod/e2e-worker-w2-95ee26-openshift-default-66679d8f57-vtpz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" start-of-body=
Nov 02 12:52:37.343669 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:37.343296    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-vtpz8" podUID="d7f18bc2-015d-41da-a666-912b2e250e2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused"
Nov 02 12:52:37.377665 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:37.377397    4559 patch_prober.go:29] interesting pod/e2e-worker-w2-95ee26-openshift-default-66679d8f57-mdchl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" start-of-body=
Nov 02 12:52:37.377665 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:37.377457    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-mdchl" podUID="bb1e97a1-f680-4922-920c-ee0a3468cb1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused"
Nov 02 12:52:37.460074 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:37.456918    4559 generic.go:356] "Generic (PLEG): container finished" podID="6f136bff-02b6-41b6-9bee-cf5626c366dc" containerID="fa3e160d1374c182c1eef451397023b8a8a89f3dc776de7c5cd24574d1130d86" exitCode=0
Nov 02 12:52:37.460074 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:37.457033    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-299e74-f6d59fd87-5s5pn" event={"ID":"6f136bff-02b6-41b6-9bee-cf5626c366dc","Type":"ContainerDied","Data":"fa3e160d1374c182c1eef451397023b8a8a89f3dc776de7c5cd24574d1130d86"}
Nov 02 12:52:37.460501 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:37.460457    4559 generic.go:356] "Generic (PLEG): container finished" podID="9d93eaf1-b398-4b86-9156-845a73b24aa9" containerID="712feb6225923324880ddeea02e69d552b0dd7f3428ae50d233012ccf9f4f478" exitCode=0
Nov 02 12:52:37.460864 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:37.460531    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-299e74-78c848cbc6-xk4wx" event={"ID":"9d93eaf1-b398-4b86-9156-845a73b24aa9","Type":"ContainerDied","Data":"712feb6225923324880ddeea02e69d552b0dd7f3428ae50d233012ccf9f4f478"}
Nov 02 12:52:37.573526 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:37.573489    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-299e74-78c848cbc6-xk4wx"
Nov 02 12:52:37.663274 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:37.663245    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d93eaf1-b398-4b86-9156-845a73b24aa9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d93eaf1-b398-4b86-9156-845a73b24aa9-plugins-config-volume\") pod \"9d93eaf1-b398-4b86-9156-845a73b24aa9\" (UID: \"9d93eaf1-b398-4b86-9156-845a73b24aa9\") "
Nov 02 12:52:37.663443 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:37.663332    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d93eaf1-b398-4b86-9156-845a73b24aa9-kube-api-access-qggst\" (UniqueName: \"kubernetes.io/projected/9d93eaf1-b398-4b86-9156-845a73b24aa9-kube-api-access-qggst\") pod \"9d93eaf1-b398-4b86-9156-845a73b24aa9\" (UID: \"9d93eaf1-b398-4b86-9156-845a73b24aa9\") "
Nov 02 12:52:37.663875 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:37.663853    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d93eaf1-b398-4b86-9156-845a73b24aa9-plugins-config-volume" pod "9d93eaf1-b398-4b86-9156-845a73b24aa9" (UID: "9d93eaf1-b398-4b86-9156-845a73b24aa9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:52:37.665883 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:37.665859    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d93eaf1-b398-4b86-9156-845a73b24aa9-kube-api-access-qggst" pod "9d93eaf1-b398-4b86-9156-845a73b24aa9" (UID: "9d93eaf1-b398-4b86-9156-845a73b24aa9"). InnerVolumeSpecName "kube-api-access-qggst". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:52:37.722698 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:37.720428    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-299e74-f6d59fd87-5s5pn"
Nov 02 12:52:37.768068 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:37.765940    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f136bff-02b6-41b6-9bee-cf5626c366dc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f136bff-02b6-41b6-9bee-cf5626c366dc-plugins-config-volume\") pod \"6f136bff-02b6-41b6-9bee-cf5626c366dc\" (UID: \"6f136bff-02b6-41b6-9bee-cf5626c366dc\") "
Nov 02 12:52:37.768068 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:37.766090    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f136bff-02b6-41b6-9bee-cf5626c366dc-kube-api-access-cq8jq\" (UniqueName: \"kubernetes.io/projected/6f136bff-02b6-41b6-9bee-cf5626c366dc-kube-api-access-cq8jq\") pod \"6f136bff-02b6-41b6-9bee-cf5626c366dc\" (UID: \"6f136bff-02b6-41b6-9bee-cf5626c366dc\") "
Nov 02 12:52:37.768068 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:37.766691    4559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d93eaf1-b398-4b86-9156-845a73b24aa9-plugins-config-volume\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:52:37.768068 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:37.766719    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qggst\" (UniqueName: \"kubernetes.io/projected/9d93eaf1-b398-4b86-9156-845a73b24aa9-kube-api-access-qggst\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:52:37.768068 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:37.768006    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f136bff-02b6-41b6-9bee-cf5626c366dc-plugins-config-volume" pod "6f136bff-02b6-41b6-9bee-cf5626c366dc" (UID: "6f136bff-02b6-41b6-9bee-cf5626c366dc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:52:37.782690 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:37.776192    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f136bff-02b6-41b6-9bee-cf5626c366dc-kube-api-access-cq8jq" pod "6f136bff-02b6-41b6-9bee-cf5626c366dc" (UID: "6f136bff-02b6-41b6-9bee-cf5626c366dc"). InnerVolumeSpecName "kube-api-access-cq8jq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:52:37.867578 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:37.867503    4559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f136bff-02b6-41b6-9bee-cf5626c366dc-plugins-config-volume\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:52:37.867769 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:37.867696    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cq8jq\" (UniqueName: \"kubernetes.io/projected/6f136bff-02b6-41b6-9bee-cf5626c366dc-kube-api-access-cq8jq\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:52:37.938607 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:37.938572    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38679526-8764-40bb-a5c0-be287e2c6e5b" path="/var/lib/kubelet/pods/38679526-8764-40bb-a5c0-be287e2c6e5b/volumes"
Nov 02 12:52:38.333453 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:38.332963    4559 patch_prober.go:29] interesting pod/e2e-worker-w2-95ee26-openshift-default-66679d8f57-vtpz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" start-of-body=
Nov 02 12:52:38.333453 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:38.333031    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-vtpz8" podUID="d7f18bc2-015d-41da-a666-912b2e250e2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused"
Nov 02 12:52:38.352961 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:38.352929    4559 patch_prober.go:29] interesting pod/e2e-worker-w2-95ee26-openshift-default-66679d8f57-mdchl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" start-of-body=
Nov 02 12:52:38.353321 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:38.352980    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-mdchl" podUID="bb1e97a1-f680-4922-920c-ee0a3468cb1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused"
Nov 02 12:52:38.467700 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:38.467665    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-299e74-f6d59fd87-5s5pn" event={"ID":"6f136bff-02b6-41b6-9bee-cf5626c366dc","Type":"ContainerDied","Data":"8109b794553a14c836500c097089f3c44064ff8f52ea4203d1f59e67820a6179"}
Nov 02 12:52:38.467909 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:38.467728    4559 scope.go:122] "RemoveContainer" containerID="fa3e160d1374c182c1eef451397023b8a8a89f3dc776de7c5cd24574d1130d86"
Nov 02 12:52:38.467909 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:38.467672    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-299e74-f6d59fd87-5s5pn"
Nov 02 12:52:38.469782 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:38.469761    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-299e74-78c848cbc6-xk4wx" event={"ID":"9d93eaf1-b398-4b86-9156-845a73b24aa9","Type":"ContainerDied","Data":"03d1db20dd219134dc90c6f8dfab1e7def962c3742fd7da32709a27ea20c5fdc"}
Nov 02 12:52:38.469963 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:38.469826    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-299e74-78c848cbc6-xk4wx"
Nov 02 12:52:38.478931 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:38.478913    4559 scope.go:122] "RemoveContainer" containerID="712feb6225923324880ddeea02e69d552b0dd7f3428ae50d233012ccf9f4f478"
Nov 02 12:52:38.561114 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:38.559883    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-299e74-f6d59fd87-5s5pn"]
Nov 02 12:52:38.571082 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:38.571047    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-299e74-f6d59fd87-5s5pn"]
Nov 02 12:52:38.633806 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:38.633705    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-299e74-78c848cbc6-xk4wx"]
Nov 02 12:52:38.638290 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:38.638208    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-299e74-78c848cbc6-xk4wx"]
Nov 02 12:52:38.741978 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:38.741938    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"]
Nov 02 12:52:39.332662 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:39.327047    4559 patch_prober.go:29] interesting pod/e2e-worker-w2-95ee26-openshift-default-66679d8f57-vtpz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" start-of-body=
Nov 02 12:52:39.332662 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:39.327116    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-vtpz8" podUID="d7f18bc2-015d-41da-a666-912b2e250e2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused"
Nov 02 12:52:39.352438 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:39.352398    4559 patch_prober.go:29] interesting pod/e2e-worker-w2-95ee26-openshift-default-66679d8f57-mdchl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" start-of-body=
Nov 02 12:52:39.352601 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:39.352459    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-mdchl" podUID="bb1e97a1-f680-4922-920c-ee0a3468cb1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused"
Nov 02 12:52:39.915769 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:39.915726    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f136bff-02b6-41b6-9bee-cf5626c366dc" path="/var/lib/kubelet/pods/6f136bff-02b6-41b6-9bee-cf5626c366dc/volumes"
Nov 02 12:52:39.919676 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:39.916353    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d93eaf1-b398-4b86-9156-845a73b24aa9" path="/var/lib/kubelet/pods/9d93eaf1-b398-4b86-9156-845a73b24aa9/volumes"
Nov 02 12:52:40.333110 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:40.333069    4559 patch_prober.go:29] interesting pod/e2e-worker-w2-95ee26-openshift-default-66679d8f57-vtpz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" start-of-body=
Nov 02 12:52:40.333459 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:40.333418    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-vtpz8" podUID="d7f18bc2-015d-41da-a666-912b2e250e2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused"
Nov 02 12:52:40.354108 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:40.354067    4559 patch_prober.go:29] interesting pod/e2e-worker-w2-95ee26-openshift-default-66679d8f57-mdchl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" start-of-body=
Nov 02 12:52:40.354398 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:40.354362    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-mdchl" podUID="bb1e97a1-f680-4922-920c-ee0a3468cb1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused"
Nov 02 12:52:41.008791 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:41.008739    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843978/distinct-w4-843978-kserve-6fcf588858-4nrwx" podUID="82e5d483-e95b-43e6-9135-af960823427a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Nov 02 12:52:41.046348 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:41.046300    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-61e341/premium-w1-61e341-kserve-746477b99-645g6" podUID="02ff21d3-5a37-493a-ab95-8e4784c29123" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:52:41.349077 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:41.348979    4559 patch_prober.go:29] interesting pod/e2e-worker-w2-95ee26-openshift-default-66679d8f57-vtpz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" start-of-body=
Nov 02 12:52:41.364762 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:41.364717    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-vtpz8" podUID="d7f18bc2-015d-41da-a666-912b2e250e2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused"
Nov 02 12:52:41.365379 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:41.352985    4559 patch_prober.go:29] interesting pod/e2e-worker-w2-95ee26-openshift-default-66679d8f57-mdchl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" start-of-body=
Nov 02 12:52:41.365574 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:41.365550    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-mdchl" podUID="bb1e97a1-f680-4922-920c-ee0a3468cb1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused"
Nov 02 12:52:41.654875 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:41.654767    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-61e341/distinct-w1-61e341-kserve-787cd566f4-cjj8s" podUID="6886ce28-5800-402d-b893-f92109e289a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:52:42.329695 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:42.327308    4559 patch_prober.go:29] interesting pod/e2e-worker-w2-95ee26-openshift-default-66679d8f57-vtpz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" start-of-body=
Nov 02 12:52:42.329695 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:42.327383    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-vtpz8" podUID="d7f18bc2-015d-41da-a666-912b2e250e2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused"
Nov 02 12:52:42.353563 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:42.353525    4559 patch_prober.go:29] interesting pod/e2e-worker-w2-95ee26-openshift-default-66679d8f57-mdchl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" start-of-body=
Nov 02 12:52:42.353896 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:42.353866    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-mdchl" podUID="bb1e97a1-f680-4922-920c-ee0a3468cb1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused"
Nov 02 12:52:44.366836 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:44.366796    4559 patch_prober.go:29] interesting pod/e2e-worker-w2-95ee26-openshift-default-66679d8f57-vtpz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.170:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 12:52:44.367396 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:44.366866    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-vtpz8" podUID="d7f18bc2-015d-41da-a666-912b2e250e2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.170:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:52:44.393808 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:44.393764    4559 patch_prober.go:29] interesting pod/e2e-worker-w2-95ee26-openshift-default-66679d8f57-mdchl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 12:52:44.394001 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:44.393842    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-mdchl" podUID="bb1e97a1-f680-4922-920c-ee0a3468cb1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:52:44.445578 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:44.445546    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-vtpz8"
Nov 02 12:52:44.446011 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:44.445983    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-vtpz8"
Nov 02 12:52:44.524489 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:44.523667    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-vtpz8"
Nov 02 12:52:44.698825 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:44.698795    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-mdchl"
Nov 02 12:52:44.699170 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:44.699147    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-mdchl"
Nov 02 12:52:44.700546 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:44.700528    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-mdchl"
Nov 02 12:52:45.086225 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:52:45.086147    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqcmj" containerName="extractor"
Nov 02 12:52:45.538190 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:45.538155    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-eb24df-openshift-default-67f48b7f65-fg527"]
Nov 02 12:52:45.538718 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:45.538382    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-eb24df-openshift-default-67f48b7f65-fg527" podUID="f19dde18-978d-4894-83d3-5c39478c866c" containerName="istio-proxy" containerID="cri-o://cbb8fc2b4dd1bab56485740516f2c420f0785c1f1bd528be5a84f6ac0fd9ccb6" gracePeriod=30
Nov 02 12:52:50.833166 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:50.833139    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-eb24df-openshift-default-67f48b7f65-fg527"
Nov 02 12:52:50.938622 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:50.938583    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f19dde18-978d-4894-83d3-5c39478c866c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f19dde18-978d-4894-83d3-5c39478c866c-istio-data\") pod \"f19dde18-978d-4894-83d3-5c39478c866c\" (UID: \"f19dde18-978d-4894-83d3-5c39478c866c\") "
Nov 02 12:52:50.938856 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:50.938675    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f19dde18-978d-4894-83d3-5c39478c866c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f19dde18-978d-4894-83d3-5c39478c866c-istiod-ca-cert\") pod \"f19dde18-978d-4894-83d3-5c39478c866c\" (UID: \"f19dde18-978d-4894-83d3-5c39478c866c\") "
Nov 02 12:52:50.938856 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:50.938703    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f19dde18-978d-4894-83d3-5c39478c866c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f19dde18-978d-4894-83d3-5c39478c866c-workload-certs\") pod \"f19dde18-978d-4894-83d3-5c39478c866c\" (UID: \"f19dde18-978d-4894-83d3-5c39478c866c\") "
Nov 02 12:52:50.938856 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:50.938802    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f19dde18-978d-4894-83d3-5c39478c866c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f19dde18-978d-4894-83d3-5c39478c866c-istio-podinfo\") pod \"f19dde18-978d-4894-83d3-5c39478c866c\" (UID: \"f19dde18-978d-4894-83d3-5c39478c866c\") "
Nov 02 12:52:50.939022 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:50.938865    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f19dde18-978d-4894-83d3-5c39478c866c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f19dde18-978d-4894-83d3-5c39478c866c-istio-envoy\") pod \"f19dde18-978d-4894-83d3-5c39478c866c\" (UID: \"f19dde18-978d-4894-83d3-5c39478c866c\") "
Nov 02 12:52:50.939022 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:50.938904    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f19dde18-978d-4894-83d3-5c39478c866c-istio-token\" (UniqueName: \"kubernetes.io/projected/f19dde18-978d-4894-83d3-5c39478c866c-istio-token\") pod \"f19dde18-978d-4894-83d3-5c39478c866c\" (UID: \"f19dde18-978d-4894-83d3-5c39478c866c\") "
Nov 02 12:52:50.939022 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:50.938935    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f19dde18-978d-4894-83d3-5c39478c866c-kube-api-access-ct826\" (UniqueName: \"kubernetes.io/projected/f19dde18-978d-4894-83d3-5c39478c866c-kube-api-access-ct826\") pod \"f19dde18-978d-4894-83d3-5c39478c866c\" (UID: \"f19dde18-978d-4894-83d3-5c39478c866c\") "
Nov 02 12:52:50.939022 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:50.938932    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f19dde18-978d-4894-83d3-5c39478c866c-istio-data" pod "f19dde18-978d-4894-83d3-5c39478c866c" (UID: "f19dde18-978d-4894-83d3-5c39478c866c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:52:50.939221 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:50.939054    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f19dde18-978d-4894-83d3-5c39478c866c-workload-certs" pod "f19dde18-978d-4894-83d3-5c39478c866c" (UID: "f19dde18-978d-4894-83d3-5c39478c866c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:52:50.939221 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:50.939089    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f19dde18-978d-4894-83d3-5c39478c866c-istiod-ca-cert" pod "f19dde18-978d-4894-83d3-5c39478c866c" (UID: "f19dde18-978d-4894-83d3-5c39478c866c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:52:50.939221 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:50.939105    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f19dde18-978d-4894-83d3-5c39478c866c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f19dde18-978d-4894-83d3-5c39478c866c-credential-socket\") pod \"f19dde18-978d-4894-83d3-5c39478c866c\" (UID: \"f19dde18-978d-4894-83d3-5c39478c866c\") "
Nov 02 12:52:50.939221 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:50.939157    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f19dde18-978d-4894-83d3-5c39478c866c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f19dde18-978d-4894-83d3-5c39478c866c-workload-socket\") pod \"f19dde18-978d-4894-83d3-5c39478c866c\" (UID: \"f19dde18-978d-4894-83d3-5c39478c866c\") "
Nov 02 12:52:50.939506 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:50.939482    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f19dde18-978d-4894-83d3-5c39478c866c-credential-socket" pod "f19dde18-978d-4894-83d3-5c39478c866c" (UID: "f19dde18-978d-4894-83d3-5c39478c866c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:52:50.939699 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:50.939674    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f19dde18-978d-4894-83d3-5c39478c866c-workload-socket" pod "f19dde18-978d-4894-83d3-5c39478c866c" (UID: "f19dde18-978d-4894-83d3-5c39478c866c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:52:50.939787 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:50.939696    4559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f19dde18-978d-4894-83d3-5c39478c866c-credential-socket\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:52:50.939787 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:50.939712    4559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f19dde18-978d-4894-83d3-5c39478c866c-istio-data\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:52:50.939787 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:50.939721    4559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f19dde18-978d-4894-83d3-5c39478c866c-istiod-ca-cert\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:52:50.939787 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:50.939729    4559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f19dde18-978d-4894-83d3-5c39478c866c-workload-certs\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:52:50.941205 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:50.941174    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f19dde18-978d-4894-83d3-5c39478c866c-istio-envoy" pod "f19dde18-978d-4894-83d3-5c39478c866c" (UID: "f19dde18-978d-4894-83d3-5c39478c866c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:52:50.941307 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:50.941223    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f19dde18-978d-4894-83d3-5c39478c866c-istio-podinfo" pod "f19dde18-978d-4894-83d3-5c39478c866c" (UID: "f19dde18-978d-4894-83d3-5c39478c866c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 12:52:50.941368 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:50.941320    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f19dde18-978d-4894-83d3-5c39478c866c-istio-token" pod "f19dde18-978d-4894-83d3-5c39478c866c" (UID: "f19dde18-978d-4894-83d3-5c39478c866c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:52:50.941368 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:50.941343    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f19dde18-978d-4894-83d3-5c39478c866c-kube-api-access-ct826" pod "f19dde18-978d-4894-83d3-5c39478c866c" (UID: "f19dde18-978d-4894-83d3-5c39478c866c"). InnerVolumeSpecName "kube-api-access-ct826". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:52:51.040922 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:51.040887    4559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f19dde18-978d-4894-83d3-5c39478c866c-workload-socket\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:52:51.040922 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:51.040916    4559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f19dde18-978d-4894-83d3-5c39478c866c-istio-podinfo\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:52:51.040922 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:51.040926    4559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f19dde18-978d-4894-83d3-5c39478c866c-istio-envoy\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:52:51.041176 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:51.040935    4559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f19dde18-978d-4894-83d3-5c39478c866c-istio-token\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:52:51.041176 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:51.040944    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ct826\" (UniqueName: \"kubernetes.io/projected/f19dde18-978d-4894-83d3-5c39478c866c-kube-api-access-ct826\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:52:51.623012 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:51.622977    4559 generic.go:356] "Generic (PLEG): container finished" podID="f19dde18-978d-4894-83d3-5c39478c866c" containerID="cbb8fc2b4dd1bab56485740516f2c420f0785c1f1bd528be5a84f6ac0fd9ccb6" exitCode=0
Nov 02 12:52:51.623192 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:51.623025    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-eb24df-openshift-default-67f48b7f65-fg527"
Nov 02 12:52:51.623192 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:51.623055    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-eb24df-openshift-default-67f48b7f65-fg527" event={"ID":"f19dde18-978d-4894-83d3-5c39478c866c","Type":"ContainerDied","Data":"cbb8fc2b4dd1bab56485740516f2c420f0785c1f1bd528be5a84f6ac0fd9ccb6"}
Nov 02 12:52:51.623192 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:51.623110    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-eb24df-openshift-default-67f48b7f65-fg527" event={"ID":"f19dde18-978d-4894-83d3-5c39478c866c","Type":"ContainerDied","Data":"b309cc72a8fe1c0e05c37031a9c221b26153803e374eb337ffd236842c32a3aa"}
Nov 02 12:52:51.623192 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:51.623135    4559 scope.go:122] "RemoveContainer" containerID="cbb8fc2b4dd1bab56485740516f2c420f0785c1f1bd528be5a84f6ac0fd9ccb6"
Nov 02 12:52:51.633768 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:51.633750    4559 scope.go:122] "RemoveContainer" containerID="6ec22c25fd91edf8dac1aa8536d95431284d95d3c6d0009924cbe8851f1f9949"
Nov 02 12:52:51.645373 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:51.645339    4559 scope.go:122] "RemoveContainer" containerID="cbb8fc2b4dd1bab56485740516f2c420f0785c1f1bd528be5a84f6ac0fd9ccb6"
Nov 02 12:52:51.646808 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:52:51.646782    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cbb8fc2b4dd1bab56485740516f2c420f0785c1f1bd528be5a84f6ac0fd9ccb6\": container with ID starting with cbb8fc2b4dd1bab56485740516f2c420f0785c1f1bd528be5a84f6ac0fd9ccb6 not found: ID does not exist" containerID="cbb8fc2b4dd1bab56485740516f2c420f0785c1f1bd528be5a84f6ac0fd9ccb6"
Nov 02 12:52:51.646918 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:51.646817    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbb8fc2b4dd1bab56485740516f2c420f0785c1f1bd528be5a84f6ac0fd9ccb6"} err="failed to get container status \"cbb8fc2b4dd1bab56485740516f2c420f0785c1f1bd528be5a84f6ac0fd9ccb6\": rpc error: code = NotFound desc = could not find container \"cbb8fc2b4dd1bab56485740516f2c420f0785c1f1bd528be5a84f6ac0fd9ccb6\": container with ID starting with cbb8fc2b4dd1bab56485740516f2c420f0785c1f1bd528be5a84f6ac0fd9ccb6 not found: ID does not exist"
Nov 02 12:52:51.646918 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:51.646835    4559 scope.go:122] "RemoveContainer" containerID="6ec22c25fd91edf8dac1aa8536d95431284d95d3c6d0009924cbe8851f1f9949"
Nov 02 12:52:51.647150 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:52:51.647126    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ec22c25fd91edf8dac1aa8536d95431284d95d3c6d0009924cbe8851f1f9949\": container with ID starting with 6ec22c25fd91edf8dac1aa8536d95431284d95d3c6d0009924cbe8851f1f9949 not found: ID does not exist" containerID="6ec22c25fd91edf8dac1aa8536d95431284d95d3c6d0009924cbe8851f1f9949"
Nov 02 12:52:51.647201 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:51.647161    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ec22c25fd91edf8dac1aa8536d95431284d95d3c6d0009924cbe8851f1f9949"} err="failed to get container status \"6ec22c25fd91edf8dac1aa8536d95431284d95d3c6d0009924cbe8851f1f9949\": rpc error: code = NotFound desc = could not find container \"6ec22c25fd91edf8dac1aa8536d95431284d95d3c6d0009924cbe8851f1f9949\": container with ID starting with 6ec22c25fd91edf8dac1aa8536d95431284d95d3c6d0009924cbe8851f1f9949 not found: ID does not exist"
Nov 02 12:52:51.749663 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:51.742710    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-eb24df-openshift-default-67f48b7f65-fg527"]
Nov 02 12:52:51.756956 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:51.756915    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-eb24df-openshift-default-67f48b7f65-fg527"]
Nov 02 12:52:51.894085 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:51.893993    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f19dde18-978d-4894-83d3-5c39478c866c" path="/var/lib/kubelet/pods/f19dde18-978d-4894-83d3-5c39478c866c/volumes"
Nov 02 12:52:52.042701 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:52.042663    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-znwwt"]
Nov 02 12:52:52.043464 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:52.043448    4559 cpu_manager.go:414] "containerMap: removing container" podUID="f19dde18-978d-4894-83d3-5c39478c866c" containerName="istio-proxy"
Nov 02 12:52:52.043509 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:52.043467    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f19dde18-978d-4894-83d3-5c39478c866c" containerName="istio-proxy"
Nov 02 12:52:52.043509 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:52.043505    4559 cpu_manager.go:414] "containerMap: removing container" podUID="38679526-8764-40bb-a5c0-be287e2c6e5b" containerName="payload-pre-processing"
Nov 02 12:52:52.043576 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:52.043513    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38679526-8764-40bb-a5c0-be287e2c6e5b" containerName="payload-pre-processing"
Nov 02 12:52:52.043576 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:52.043531    4559 cpu_manager.go:414] "containerMap: removing container" podUID="9d93eaf1-b398-4b86-9156-845a73b24aa9" containerName="payload-processing"
Nov 02 12:52:52.043576 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:52.043539    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d93eaf1-b398-4b86-9156-845a73b24aa9" containerName="payload-processing"
Nov 02 12:52:52.043576 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:52.043565    4559 cpu_manager.go:414] "containerMap: removing container" podUID="6f136bff-02b6-41b6-9bee-cf5626c366dc" containerName="payload-pre-processing"
Nov 02 12:52:52.043576 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:52.043572    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f136bff-02b6-41b6-9bee-cf5626c366dc" containerName="payload-pre-processing"
Nov 02 12:52:52.043783 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:52.043715    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="f19dde18-978d-4894-83d3-5c39478c866c" containerName="istio-proxy"
Nov 02 12:52:52.043783 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:52.043729    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f136bff-02b6-41b6-9bee-cf5626c366dc" containerName="payload-pre-processing"
Nov 02 12:52:52.043783 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:52.043744    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="f19dde18-978d-4894-83d3-5c39478c866c" containerName="istio-proxy"
Nov 02 12:52:52.043783 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:52.043760    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="38679526-8764-40bb-a5c0-be287e2c6e5b" containerName="payload-pre-processing"
Nov 02 12:52:52.043783 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:52.043770    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d93eaf1-b398-4b86-9156-845a73b24aa9" containerName="payload-processing"
Nov 02 12:52:52.048986 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:52.048961    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-znwwt"
Nov 02 12:52:52.057109 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:52.057078    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-7qxgl"]
Nov 02 12:52:52.057661 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:52.057631    4559 cpu_manager.go:414] "containerMap: removing container" podUID="f19dde18-978d-4894-83d3-5c39478c866c" containerName="istio-proxy"
Nov 02 12:52:52.057739 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:52.057663    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f19dde18-978d-4894-83d3-5c39478c866c" containerName="istio-proxy"
Nov 02 12:52:52.061030 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:52.061011    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-7qxgl"
Nov 02 12:52:52.076255 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:52.076207    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-znwwt"]
Nov 02 12:52:52.096911 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:52.096874    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-7qxgl"]
Nov 02 12:52:52.172544 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:52.172510    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53afdb1f-5ab4-4106-bccd-9ce04d9ced37-istiod-ca-cert\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-znwwt\" (UID: \"53afdb1f-5ab4-4106-bccd-9ce04d9ced37\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-znwwt"
Nov 02 12:52:52.172759 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:52.172553    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53afdb1f-5ab4-4106-bccd-9ce04d9ced37-istio-envoy\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-znwwt\" (UID: \"53afdb1f-5ab4-4106-bccd-9ce04d9ced37\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-znwwt"
Nov 02 12:52:52.172759 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:52.172725    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13f8bac2-10c1-40aa-a968-3a9b64a6676f-istio-podinfo\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-7qxgl\" (UID: \"13f8bac2-10c1-40aa-a968-3a9b64a6676f\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-7qxgl"
Nov 02 12:52:52.172896 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:52.172796    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13f8bac2-10c1-40aa-a968-3a9b64a6676f-istio-data\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-7qxgl\" (UID: \"13f8bac2-10c1-40aa-a968-3a9b64a6676f\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-7qxgl"
Nov 02 12:52:52.172896 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:52.172866    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13f8bac2-10c1-40aa-a968-3a9b64a6676f-workload-socket\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-7qxgl\" (UID: \"13f8bac2-10c1-40aa-a968-3a9b64a6676f\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-7qxgl"
Nov 02 12:52:52.172995 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:52.172905    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13f8bac2-10c1-40aa-a968-3a9b64a6676f-credential-socket\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-7qxgl\" (UID: \"13f8bac2-10c1-40aa-a968-3a9b64a6676f\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-7qxgl"
Nov 02 12:52:52.173031 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:52.172994    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53afdb1f-5ab4-4106-bccd-9ce04d9ced37-istio-data\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-znwwt\" (UID: \"53afdb1f-5ab4-4106-bccd-9ce04d9ced37\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-znwwt"
Nov 02 12:52:52.173071 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:52.173029    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53afdb1f-5ab4-4106-bccd-9ce04d9ced37-istio-podinfo\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-znwwt\" (UID: \"53afdb1f-5ab4-4106-bccd-9ce04d9ced37\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-znwwt"
Nov 02 12:52:52.173071 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:52.173061    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13f8bac2-10c1-40aa-a968-3a9b64a6676f-istio-envoy\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-7qxgl\" (UID: \"13f8bac2-10c1-40aa-a968-3a9b64a6676f\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-7qxgl"
Nov 02 12:52:52.173170 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:52.173095    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53afdb1f-5ab4-4106-bccd-9ce04d9ced37-istio-token\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-znwwt\" (UID: \"53afdb1f-5ab4-4106-bccd-9ce04d9ced37\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-znwwt"
Nov 02 12:52:52.173213 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:52.173191    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13f8bac2-10c1-40aa-a968-3a9b64a6676f-workload-certs\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-7qxgl\" (UID: \"13f8bac2-10c1-40aa-a968-3a9b64a6676f\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-7qxgl"
Nov 02 12:52:52.173254 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:52.173221    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13f8bac2-10c1-40aa-a968-3a9b64a6676f-istio-token\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-7qxgl\" (UID: \"13f8bac2-10c1-40aa-a968-3a9b64a6676f\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-7qxgl"
Nov 02 12:52:52.173254 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:52.173242    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53afdb1f-5ab4-4106-bccd-9ce04d9ced37-credential-socket\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-znwwt\" (UID: \"53afdb1f-5ab4-4106-bccd-9ce04d9ced37\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-znwwt"
Nov 02 12:52:52.173356 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:52.173331    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53afdb1f-5ab4-4106-bccd-9ce04d9ced37-workload-certs\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-znwwt\" (UID: \"53afdb1f-5ab4-4106-bccd-9ce04d9ced37\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-znwwt"
Nov 02 12:52:52.173405 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:52.173353    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53afdb1f-5ab4-4106-bccd-9ce04d9ced37-workload-socket\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-znwwt\" (UID: \"53afdb1f-5ab4-4106-bccd-9ce04d9ced37\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-znwwt"
Nov 02 12:52:52.173444 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:52.173434    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13f8bac2-10c1-40aa-a968-3a9b64a6676f-istiod-ca-cert\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-7qxgl\" (UID: \"13f8bac2-10c1-40aa-a968-3a9b64a6676f\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-7qxgl"
Nov 02 12:52:52.173478 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:52.173454    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkbjj\" (UniqueName: \"kubernetes.io/projected/53afdb1f-5ab4-4106-bccd-9ce04d9ced37-kube-api-access-fkbjj\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-znwwt\" (UID: \"53afdb1f-5ab4-4106-bccd-9ce04d9ced37\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-znwwt"
Nov 02 12:52:52.173525 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:52.173498    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkfb6\" (UniqueName: \"kubernetes.io/projected/13f8bac2-10c1-40aa-a968-3a9b64a6676f-kube-api-access-bkfb6\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-7qxgl\" (UID: \"13f8bac2-10c1-40aa-a968-3a9b64a6676f\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-7qxgl"
Nov 02 12:52:52.275254 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:52.275221    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53afdb1f-5ab4-4106-bccd-9ce04d9ced37-istiod-ca-cert\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-znwwt\" (UID: \"53afdb1f-5ab4-4106-bccd-9ce04d9ced37\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-znwwt"
Nov 02 12:52:52.275254 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:52.275258    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53afdb1f-5ab4-4106-bccd-9ce04d9ced37-istio-envoy\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-znwwt\" (UID: \"53afdb1f-5ab4-4106-bccd-9ce04d9ced37\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-znwwt"
Nov 02 12:52:52.275526 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:52.275289    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13f8bac2-10c1-40aa-a968-3a9b64a6676f-istio-podinfo\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-7qxgl\" (UID: \"13f8bac2-10c1-40aa-a968-3a9b64a6676f\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-7qxgl"
Nov 02 12:52:52.275526 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:52.275334    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13f8bac2-10c1-40aa-a968-3a9b64a6676f-istio-data\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-7qxgl\" (UID: \"13f8bac2-10c1-40aa-a968-3a9b64a6676f\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-7qxgl"
Nov 02 12:52:52.275526 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:52.275353    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13f8bac2-10c1-40aa-a968-3a9b64a6676f-workload-socket\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-7qxgl\" (UID: \"13f8bac2-10c1-40aa-a968-3a9b64a6676f\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-7qxgl"
Nov 02 12:52:52.275526 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:52.275369    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13f8bac2-10c1-40aa-a968-3a9b64a6676f-credential-socket\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-7qxgl\" (UID: \"13f8bac2-10c1-40aa-a968-3a9b64a6676f\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-7qxgl"
Nov 02 12:52:52.275526 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:52.275405    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53afdb1f-5ab4-4106-bccd-9ce04d9ced37-istio-data\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-znwwt\" (UID: \"53afdb1f-5ab4-4106-bccd-9ce04d9ced37\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-znwwt"
Nov 02 12:52:52.275526 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:52.275449    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53afdb1f-5ab4-4106-bccd-9ce04d9ced37-istio-podinfo\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-znwwt\" (UID: \"53afdb1f-5ab4-4106-bccd-9ce04d9ced37\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-znwwt"
Nov 02 12:52:52.275526 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:52.275494    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13f8bac2-10c1-40aa-a968-3a9b64a6676f-istio-envoy\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-7qxgl\" (UID: \"13f8bac2-10c1-40aa-a968-3a9b64a6676f\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-7qxgl"
Nov 02 12:52:52.275526 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:52.275523    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53afdb1f-5ab4-4106-bccd-9ce04d9ced37-istio-token\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-znwwt\" (UID: \"53afdb1f-5ab4-4106-bccd-9ce04d9ced37\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-znwwt"
Nov 02 12:52:52.276009 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:52.275576    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13f8bac2-10c1-40aa-a968-3a9b64a6676f-workload-certs\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-7qxgl\" (UID: \"13f8bac2-10c1-40aa-a968-3a9b64a6676f\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-7qxgl"
Nov 02 12:52:52.276009 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:52.275601    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13f8bac2-10c1-40aa-a968-3a9b64a6676f-istio-token\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-7qxgl\" (UID: \"13f8bac2-10c1-40aa-a968-3a9b64a6676f\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-7qxgl"
Nov 02 12:52:52.276009 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:52.275629    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53afdb1f-5ab4-4106-bccd-9ce04d9ced37-credential-socket\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-znwwt\" (UID: \"53afdb1f-5ab4-4106-bccd-9ce04d9ced37\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-znwwt"
Nov 02 12:52:52.276009 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:52.275723    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53afdb1f-5ab4-4106-bccd-9ce04d9ced37-workload-certs\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-znwwt\" (UID: \"53afdb1f-5ab4-4106-bccd-9ce04d9ced37\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-znwwt"
Nov 02 12:52:52.276009 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:52.275755    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53afdb1f-5ab4-4106-bccd-9ce04d9ced37-workload-socket\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-znwwt\" (UID: \"53afdb1f-5ab4-4106-bccd-9ce04d9ced37\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-znwwt"
Nov 02 12:52:52.276009 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:52.275794    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13f8bac2-10c1-40aa-a968-3a9b64a6676f-istiod-ca-cert\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-7qxgl\" (UID: \"13f8bac2-10c1-40aa-a968-3a9b64a6676f\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-7qxgl"
Nov 02 12:52:52.276009 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:52.275796    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13f8bac2-10c1-40aa-a968-3a9b64a6676f-istio-data\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-7qxgl\" (UID: \"13f8bac2-10c1-40aa-a968-3a9b64a6676f\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-7qxgl"
Nov 02 12:52:52.276009 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:52.275812    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13f8bac2-10c1-40aa-a968-3a9b64a6676f-credential-socket\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-7qxgl\" (UID: \"13f8bac2-10c1-40aa-a968-3a9b64a6676f\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-7qxgl"
Nov 02 12:52:52.276009 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:52.275825    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkbjj\" (UniqueName: \"kubernetes.io/projected/53afdb1f-5ab4-4106-bccd-9ce04d9ced37-kube-api-access-fkbjj\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-znwwt\" (UID: \"53afdb1f-5ab4-4106-bccd-9ce04d9ced37\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-znwwt"
Nov 02 12:52:52.276009 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:52.275869    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13f8bac2-10c1-40aa-a968-3a9b64a6676f-workload-socket\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-7qxgl\" (UID: \"13f8bac2-10c1-40aa-a968-3a9b64a6676f\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-7qxgl"
Nov 02 12:52:52.276009 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:52.275902    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkfb6\" (UniqueName: \"kubernetes.io/projected/13f8bac2-10c1-40aa-a968-3a9b64a6676f-kube-api-access-bkfb6\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-7qxgl\" (UID: \"13f8bac2-10c1-40aa-a968-3a9b64a6676f\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-7qxgl"
Nov 02 12:52:52.276606 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:52.276061    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13f8bac2-10c1-40aa-a968-3a9b64a6676f-workload-certs\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-7qxgl\" (UID: \"13f8bac2-10c1-40aa-a968-3a9b64a6676f\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-7qxgl"
Nov 02 12:52:52.276606 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:52.276144    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53afdb1f-5ab4-4106-bccd-9ce04d9ced37-workload-certs\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-znwwt\" (UID: \"53afdb1f-5ab4-4106-bccd-9ce04d9ced37\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-znwwt"
Nov 02 12:52:52.276606 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:52.276159    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53afdb1f-5ab4-4106-bccd-9ce04d9ced37-istiod-ca-cert\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-znwwt\" (UID: \"53afdb1f-5ab4-4106-bccd-9ce04d9ced37\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-znwwt"
Nov 02 12:52:52.276606 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:52.276446    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53afdb1f-5ab4-4106-bccd-9ce04d9ced37-istio-data\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-znwwt\" (UID: \"53afdb1f-5ab4-4106-bccd-9ce04d9ced37\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-znwwt"
Nov 02 12:52:52.276606 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:52.276542    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53afdb1f-5ab4-4106-bccd-9ce04d9ced37-workload-socket\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-znwwt\" (UID: \"53afdb1f-5ab4-4106-bccd-9ce04d9ced37\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-znwwt"
Nov 02 12:52:52.276905 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:52.276842    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53afdb1f-5ab4-4106-bccd-9ce04d9ced37-credential-socket\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-znwwt\" (UID: \"53afdb1f-5ab4-4106-bccd-9ce04d9ced37\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-znwwt"
Nov 02 12:52:52.277158 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:52.277131    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13f8bac2-10c1-40aa-a968-3a9b64a6676f-istiod-ca-cert\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-7qxgl\" (UID: \"13f8bac2-10c1-40aa-a968-3a9b64a6676f\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-7qxgl"
Nov 02 12:52:52.277948 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:52.277924    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53afdb1f-5ab4-4106-bccd-9ce04d9ced37-istio-envoy\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-znwwt\" (UID: \"53afdb1f-5ab4-4106-bccd-9ce04d9ced37\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-znwwt"
Nov 02 12:52:52.278170 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:52.278146    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13f8bac2-10c1-40aa-a968-3a9b64a6676f-istio-podinfo\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-7qxgl\" (UID: \"13f8bac2-10c1-40aa-a968-3a9b64a6676f\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-7qxgl"
Nov 02 12:52:52.278274 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:52.278226    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13f8bac2-10c1-40aa-a968-3a9b64a6676f-istio-envoy\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-7qxgl\" (UID: \"13f8bac2-10c1-40aa-a968-3a9b64a6676f\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-7qxgl"
Nov 02 12:52:52.278511 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:52.278494    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53afdb1f-5ab4-4106-bccd-9ce04d9ced37-istio-podinfo\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-znwwt\" (UID: \"53afdb1f-5ab4-4106-bccd-9ce04d9ced37\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-znwwt"
Nov 02 12:52:52.300409 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:52.300374    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53afdb1f-5ab4-4106-bccd-9ce04d9ced37-istio-token\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-znwwt\" (UID: \"53afdb1f-5ab4-4106-bccd-9ce04d9ced37\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-znwwt"
Nov 02 12:52:52.300409 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:52.300399    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkfb6\" (UniqueName: \"kubernetes.io/projected/13f8bac2-10c1-40aa-a968-3a9b64a6676f-kube-api-access-bkfb6\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-7qxgl\" (UID: \"13f8bac2-10c1-40aa-a968-3a9b64a6676f\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-7qxgl"
Nov 02 12:52:52.300725 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:52.300702    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13f8bac2-10c1-40aa-a968-3a9b64a6676f-istio-token\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-7qxgl\" (UID: \"13f8bac2-10c1-40aa-a968-3a9b64a6676f\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-7qxgl"
Nov 02 12:52:52.300775 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:52.300760    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkbjj\" (UniqueName: \"kubernetes.io/projected/53afdb1f-5ab4-4106-bccd-9ce04d9ced37-kube-api-access-fkbjj\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-znwwt\" (UID: \"53afdb1f-5ab4-4106-bccd-9ce04d9ced37\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-znwwt"
Nov 02 12:52:52.363998 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:52.363963    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-znwwt"
Nov 02 12:52:52.375558 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:52.375534    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-7qxgl"
Nov 02 12:52:52.697156 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:52:52.697122    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53afdb1f_5ab4_4106_bccd_9ce04d9ced37.slice/crio-f8c1ca967a05566d23a2d20a929cd2d90b5c2cf613cb9ba529972bfddeaa74cc WatchSource:0}: Error finding container f8c1ca967a05566d23a2d20a929cd2d90b5c2cf613cb9ba529972bfddeaa74cc: Status 404 returned error can't find the container with id f8c1ca967a05566d23a2d20a929cd2d90b5c2cf613cb9ba529972bfddeaa74cc
Nov 02 12:52:52.699168 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:52.699142    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-znwwt"]
Nov 02 12:52:52.700427 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:52.700383    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:52:52.700536 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:52.700481    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:52:52.700536 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:52.700525    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:52:52.702026 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:52.702006    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-7qxgl"]
Nov 02 12:52:52.710481 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:52:52.710454    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13f8bac2_10c1_40aa_a968_3a9b64a6676f.slice/crio-e2551fb75e560198534bc03c43845072c09f4b3c37dd35f68e551e82a4383f63 WatchSource:0}: Error finding container e2551fb75e560198534bc03c43845072c09f4b3c37dd35f68e551e82a4383f63: Status 404 returned error can't find the container with id e2551fb75e560198534bc03c43845072c09f4b3c37dd35f68e551e82a4383f63
Nov 02 12:52:52.713023 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:52.712992    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:52:52.713105 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:52.713066    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:52:52.713158 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:52.713113    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:52:53.676828 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:53.676787    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-7qxgl" event={"ID":"13f8bac2-10c1-40aa-a968-3a9b64a6676f","Type":"ContainerStarted","Data":"5a797e1223770184135d924d845ace05020d8abd016fbd85d030252a6e4bd07e"}
Nov 02 12:52:53.677449 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:53.677416    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-7qxgl" event={"ID":"13f8bac2-10c1-40aa-a968-3a9b64a6676f","Type":"ContainerStarted","Data":"e2551fb75e560198534bc03c43845072c09f4b3c37dd35f68e551e82a4383f63"}
Nov 02 12:52:53.680890 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:53.680864    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-znwwt" event={"ID":"53afdb1f-5ab4-4106-bccd-9ce04d9ced37","Type":"ContainerStarted","Data":"df2df1e59e30a10900104ef6b717fdfe588a3983fc690b685b499b712a7a0bee"}
Nov 02 12:52:53.681069 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:53.681056    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-znwwt" event={"ID":"53afdb1f-5ab4-4106-bccd-9ce04d9ced37","Type":"ContainerStarted","Data":"f8c1ca967a05566d23a2d20a929cd2d90b5c2cf613cb9ba529972bfddeaa74cc"}
Nov 02 12:52:53.764245 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:53.764195    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-7qxgl" podStartSLOduration=1.76417946 podStartE2EDuration="1.76417946s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:52:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:52:53.762964168 +0000 UTC m=+1570.018363265" watchObservedRunningTime="2026-11-02 12:52:53.76417946 +0000 UTC m=+1570.019578619"
Nov 02 12:52:53.832668 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:53.826180    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-znwwt" podStartSLOduration=1.8261546389999999 podStartE2EDuration="1.826154639s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:52:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:52:53.82415961 +0000 UTC m=+1570.079558705" watchObservedRunningTime="2026-11-02 12:52:53.826154639 +0000 UTC m=+1570.081553735"
Nov 02 12:52:54.367319 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:54.367239    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-znwwt"
Nov 02 12:52:54.379991 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:54.379954    4559 patch_prober.go:29] interesting pod/e2e-worker-w1-61e341-openshift-default-59d97476dc-znwwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" start-of-body=
Nov 02 12:52:54.382587 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:54.382548    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-znwwt" podUID="53afdb1f-5ab4-4106-bccd-9ce04d9ced37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused"
Nov 02 12:52:54.399221 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:54.381768    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-7qxgl"
Nov 02 12:52:54.399499 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:54.381829    4559 patch_prober.go:29] interesting pod/e2e-worker-w1-61e341-openshift-default-59d97476dc-7qxgl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" start-of-body=
Nov 02 12:52:54.399667 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:54.399623    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-7qxgl" podUID="13f8bac2-10c1-40aa-a968-3a9b64a6676f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused"
Nov 02 12:52:55.375720 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:55.373196    4559 patch_prober.go:29] interesting pod/e2e-worker-w1-61e341-openshift-default-59d97476dc-znwwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" start-of-body=
Nov 02 12:52:55.375720 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:55.373279    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-znwwt" podUID="53afdb1f-5ab4-4106-bccd-9ce04d9ced37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused"
Nov 02 12:52:55.378101 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:55.377992    4559 patch_prober.go:29] interesting pod/e2e-worker-w1-61e341-openshift-default-59d97476dc-7qxgl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" start-of-body=
Nov 02 12:52:55.378101 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:55.378046    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-7qxgl" podUID="13f8bac2-10c1-40aa-a968-3a9b64a6676f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused"
Nov 02 12:52:55.536665 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:55.527818    4559 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-843978/distinct-w4-843978-kserve-6fcf588858-4nrwx" podUID="82e5d483-e95b-43e6-9135-af960823427a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:52:55.638724 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:55.637653    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-61e341/distinct-2-w1-61e341-kserve-954cf956f-n7jmg" podUID="c5e1a5f8-97b4-469d-be9a-eae7ca995283" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded"
Nov 02 12:52:56.013155 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:56.012080    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843978/distinct-w4-843978-kserve-6fcf588858-4nrwx" podUID="82e5d483-e95b-43e6-9135-af960823427a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:52:56.063811 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:56.055277    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-61e341/premium-w1-61e341-kserve-746477b99-645g6" podUID="02ff21d3-5a37-493a-ab95-8e4784c29123" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded"
Nov 02 12:52:56.507352 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:56.507307    4559 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-61e341/premium-w1-61e341-kserve-746477b99-645g6" podUID="02ff21d3-5a37-493a-ab95-8e4784c29123" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/health\": context deadline exceeded"
Nov 02 12:52:56.541774 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:56.512321    4559 patch_prober.go:29] interesting pod/e2e-worker-w1-61e341-openshift-default-59d97476dc-7qxgl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" start-of-body=
Nov 02 12:52:56.542080 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:56.542047    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-7qxgl" podUID="13f8bac2-10c1-40aa-a968-3a9b64a6676f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused"
Nov 02 12:52:56.542242 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:56.541736    4559 patch_prober.go:29] interesting pod/e2e-worker-w1-61e341-openshift-default-59d97476dc-znwwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" start-of-body=
Nov 02 12:52:56.542380 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:56.542360    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-znwwt" podUID="53afdb1f-5ab4-4106-bccd-9ce04d9ced37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused"
Nov 02 12:52:56.607813 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:56.607766    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-61e341/distinct-w1-61e341-kserve-787cd566f4-cjj8s" podUID="6886ce28-5800-402d-b893-f92109e289a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Nov 02 12:52:56.649662 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:56.645803    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-61e341/distinct-2-w1-61e341-kserve-954cf956f-n7jmg" podUID="c5e1a5f8-97b4-469d-be9a-eae7ca995283" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded"
Nov 02 12:52:56.649662 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:56.647565    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-61e341/distinct-2-w1-61e341-kserve-954cf956f-n7jmg"
Nov 02 12:52:57.018470 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:57.015792    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843978/distinct-w4-843978-kserve-6fcf588858-4nrwx" podUID="82e5d483-e95b-43e6-9135-af960823427a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:52:57.018470 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:57.016298    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-843978/distinct-w4-843978-kserve-6fcf588858-4nrwx"
Nov 02 12:52:57.067081 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:57.067035    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-61e341/premium-w1-61e341-kserve-746477b99-645g6" podUID="02ff21d3-5a37-493a-ab95-8e4784c29123" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:52:57.069589 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:57.069549    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-61e341/premium-w1-61e341-kserve-746477b99-645g6"
Nov 02 12:52:57.375820 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:57.375701    4559 patch_prober.go:29] interesting pod/e2e-worker-w1-61e341-openshift-default-59d97476dc-znwwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" start-of-body=
Nov 02 12:52:57.401073 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:57.401031    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-znwwt" podUID="53afdb1f-5ab4-4106-bccd-9ce04d9ced37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused"
Nov 02 12:52:57.410941 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:57.410813    4559 patch_prober.go:29] interesting pod/e2e-worker-w1-61e341-openshift-default-59d97476dc-7qxgl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" start-of-body=
Nov 02 12:52:57.410941 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:57.410877    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-7qxgl" podUID="13f8bac2-10c1-40aa-a968-3a9b64a6676f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused"
Nov 02 12:52:57.614855 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:57.613947    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-61e341/distinct-w1-61e341-kserve-787cd566f4-cjj8s" podUID="6886ce28-5800-402d-b893-f92109e289a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:52:57.614855 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:57.614303    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-61e341/distinct-w1-61e341-kserve-787cd566f4-cjj8s"
Nov 02 12:52:57.670323 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:57.648566    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-61e341/distinct-2-w1-61e341-kserve-954cf956f-n7jmg" podUID="c5e1a5f8-97b4-469d-be9a-eae7ca995283" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:52:58.024742 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:58.017741    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843978/distinct-w4-843978-kserve-6fcf588858-4nrwx" podUID="82e5d483-e95b-43e6-9135-af960823427a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:52:58.075825 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:58.075757    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-61e341/premium-w1-61e341-kserve-746477b99-645g6" podUID="02ff21d3-5a37-493a-ab95-8e4784c29123" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:52:58.388722 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:58.367021    4559 patch_prober.go:29] interesting pod/e2e-worker-w1-61e341-openshift-default-59d97476dc-znwwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" start-of-body=
Nov 02 12:52:58.413770 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:58.413718    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-znwwt" podUID="53afdb1f-5ab4-4106-bccd-9ce04d9ced37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused"
Nov 02 12:52:58.457665 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:58.377066    4559 patch_prober.go:29] interesting pod/e2e-worker-w1-61e341-openshift-default-59d97476dc-7qxgl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" start-of-body=
Nov 02 12:52:58.469665 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:58.465718    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-7qxgl" podUID="13f8bac2-10c1-40aa-a968-3a9b64a6676f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused"
Nov 02 12:52:58.617666 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:58.616805    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-61e341/distinct-w1-61e341-kserve-787cd566f4-cjj8s" podUID="6886ce28-5800-402d-b893-f92109e289a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:52:58.662760 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:58.648991    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-61e341/distinct-2-w1-61e341-kserve-954cf956f-n7jmg" podUID="c5e1a5f8-97b4-469d-be9a-eae7ca995283" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:52:59.032663 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:59.032059    4559 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-61e341/distinct-w1-61e341-kserve-787cd566f4-cjj8s" podUID="6886ce28-5800-402d-b893-f92109e289a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:52:59.032663 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:59.032328    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843978/distinct-w4-843978-kserve-6fcf588858-4nrwx" podUID="82e5d483-e95b-43e6-9135-af960823427a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:52:59.068663 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:59.068608    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-61e341/distinct-w1-61e341-kserve-787cd566f4-cjj8s"
Nov 02 12:52:59.083666 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:59.083314    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-61e341/premium-w1-61e341-kserve-746477b99-645g6" podUID="02ff21d3-5a37-493a-ab95-8e4784c29123" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:52:59.088162 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:59.088137    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-61e341/distinct-2-w1-61e341-kserve-954cf956f-n7jmg"
Nov 02 12:52:59.267999 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:59.267966    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-843978/distinct-w4-843978-kserve-6fcf588858-4nrwx"
Nov 02 12:52:59.271924 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:59.271897    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-61e341/premium-w1-61e341-kserve-746477b99-645g6"
Nov 02 12:52:59.364426 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:59.364336    4559 patch_prober.go:29] interesting pod/e2e-worker-w1-61e341-openshift-default-59d97476dc-znwwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" start-of-body=
Nov 02 12:52:59.364426 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:59.364397    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-znwwt" podUID="53afdb1f-5ab4-4106-bccd-9ce04d9ced37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused"
Nov 02 12:52:59.376004 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:59.375969    4559 patch_prober.go:29] interesting pod/e2e-worker-w1-61e341-openshift-default-59d97476dc-7qxgl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" start-of-body=
Nov 02 12:52:59.376178 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:52:59.376028    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-7qxgl" podUID="13f8bac2-10c1-40aa-a968-3a9b64a6676f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused"
Nov 02 12:53:00.364682 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:00.364618    4559 patch_prober.go:29] interesting pod/e2e-worker-w1-61e341-openshift-default-59d97476dc-znwwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" start-of-body=
Nov 02 12:53:00.365160 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:00.364694    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-znwwt" podUID="53afdb1f-5ab4-4106-bccd-9ce04d9ced37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused"
Nov 02 12:53:00.376239 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:00.376204    4559 patch_prober.go:29] interesting pod/e2e-worker-w1-61e341-openshift-default-59d97476dc-7qxgl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" start-of-body=
Nov 02 12:53:00.376363 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:00.376270    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-7qxgl" podUID="13f8bac2-10c1-40aa-a968-3a9b64a6676f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused"
Nov 02 12:53:01.365373 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:01.365335    4559 patch_prober.go:29] interesting pod/e2e-worker-w1-61e341-openshift-default-59d97476dc-znwwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" start-of-body=
Nov 02 12:53:01.365798 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:01.365391    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-znwwt" podUID="53afdb1f-5ab4-4106-bccd-9ce04d9ced37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused"
Nov 02 12:53:01.375844 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:01.375814    4559 patch_prober.go:29] interesting pod/e2e-worker-w1-61e341-openshift-default-59d97476dc-7qxgl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" start-of-body=
Nov 02 12:53:01.375937 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:01.375866    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-7qxgl" podUID="13f8bac2-10c1-40aa-a968-3a9b64a6676f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused"
Nov 02 12:53:01.743666 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:01.743235    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-d7801d-6ffffb74dc-prdc6"]
Nov 02 12:53:01.743666 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:01.743488    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d7801d-6ffffb74dc-prdc6" podUID="c9479aa3-35be-4c23-95e6-efc7d46583cd" containerName="maas-api" containerID="cri-o://b835a2c0f7b3c90e3593fab5b8274c6c5f0cced9ae7ffa0c162a52eb26d57eab" gracePeriod=30
Nov 02 12:53:01.919150 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:01.919113    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d7801d-678658877b-bwjnd"]
Nov 02 12:53:01.919386 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:01.919360    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-d7801d-678658877b-bwjnd" podUID="997a3029-5f17-4a07-be79-0d8f7aa2e4b2" containerName="payload-processing" containerID="cri-o://5c30e4e48efdac5cb874cea03d931c7dae2f79f13dda0c7f96228b86830431e1" gracePeriod=30
Nov 02 12:53:01.953550 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:01.953509    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d7801d-5f9f5c649f-9frfl"]
Nov 02 12:53:01.953822 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:01.953790    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d7801d-5f9f5c649f-9frfl" podUID="d3e81e99-3871-429a-b725-37cf024fbce1" containerName="payload-pre-processing" containerID="cri-o://98637dd2dea79b341d14b5c84b86de69922c1148d7757c5834a8cae749092fa3" gracePeriod=30
Nov 02 12:53:02.129681 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:02.127776    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d7801d-6ffffb74dc-prdc6"
Nov 02 12:53:02.198696 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:02.198484    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9479aa3-35be-4c23-95e6-efc7d46583cd-kube-api-access-b8xtp\" (UniqueName: \"kubernetes.io/projected/c9479aa3-35be-4c23-95e6-efc7d46583cd-kube-api-access-b8xtp\") pod \"c9479aa3-35be-4c23-95e6-efc7d46583cd\" (UID: \"c9479aa3-35be-4c23-95e6-efc7d46583cd\") "
Nov 02 12:53:02.200876 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:02.199057    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9479aa3-35be-4c23-95e6-efc7d46583cd-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9479aa3-35be-4c23-95e6-efc7d46583cd-openshift-service-ca\") pod \"c9479aa3-35be-4c23-95e6-efc7d46583cd\" (UID: \"c9479aa3-35be-4c23-95e6-efc7d46583cd\") "
Nov 02 12:53:02.200876 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:02.199384    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c9479aa3-35be-4c23-95e6-efc7d46583cd-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9479aa3-35be-4c23-95e6-efc7d46583cd-metrics-certs\") pod \"c9479aa3-35be-4c23-95e6-efc7d46583cd\" (UID: \"c9479aa3-35be-4c23-95e6-efc7d46583cd\") "
Nov 02 12:53:02.200876 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:02.199501    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c9479aa3-35be-4c23-95e6-efc7d46583cd-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9479aa3-35be-4c23-95e6-efc7d46583cd-maas-api-tls\") pod \"c9479aa3-35be-4c23-95e6-efc7d46583cd\" (UID: \"c9479aa3-35be-4c23-95e6-efc7d46583cd\") "
Nov 02 12:53:02.204123 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:02.203917    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9479aa3-35be-4c23-95e6-efc7d46583cd-maas-api-tls" pod "c9479aa3-35be-4c23-95e6-efc7d46583cd" (UID: "c9479aa3-35be-4c23-95e6-efc7d46583cd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:53:02.204123 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:02.204081    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9479aa3-35be-4c23-95e6-efc7d46583cd-openshift-service-ca" pod "c9479aa3-35be-4c23-95e6-efc7d46583cd" (UID: "c9479aa3-35be-4c23-95e6-efc7d46583cd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:53:02.206077 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:02.206031    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9479aa3-35be-4c23-95e6-efc7d46583cd-kube-api-access-b8xtp" pod "c9479aa3-35be-4c23-95e6-efc7d46583cd" (UID: "c9479aa3-35be-4c23-95e6-efc7d46583cd"). InnerVolumeSpecName "kube-api-access-b8xtp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:53:02.206207 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:02.206096    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9479aa3-35be-4c23-95e6-efc7d46583cd-metrics-certs" pod "c9479aa3-35be-4c23-95e6-efc7d46583cd" (UID: "c9479aa3-35be-4c23-95e6-efc7d46583cd"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:53:02.277057 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:02.276995    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-d7801d-678658877b-bwjnd"
Nov 02 12:53:02.433735 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:02.413236    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/997a3029-5f17-4a07-be79-0d8f7aa2e4b2-kube-api-access-mvtlf\" (UniqueName: \"kubernetes.io/projected/997a3029-5f17-4a07-be79-0d8f7aa2e4b2-kube-api-access-mvtlf\") pod \"997a3029-5f17-4a07-be79-0d8f7aa2e4b2\" (UID: \"997a3029-5f17-4a07-be79-0d8f7aa2e4b2\") "
Nov 02 12:53:02.433735 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:02.413380    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/997a3029-5f17-4a07-be79-0d8f7aa2e4b2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/997a3029-5f17-4a07-be79-0d8f7aa2e4b2-plugins-config-volume\") pod \"997a3029-5f17-4a07-be79-0d8f7aa2e4b2\" (UID: \"997a3029-5f17-4a07-be79-0d8f7aa2e4b2\") "
Nov 02 12:53:02.433735 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:02.413883    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b8xtp\" (UniqueName: \"kubernetes.io/projected/c9479aa3-35be-4c23-95e6-efc7d46583cd-kube-api-access-b8xtp\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:53:02.433735 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:02.413906    4559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9479aa3-35be-4c23-95e6-efc7d46583cd-openshift-service-ca\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:53:02.433735 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:02.413920    4559 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9479aa3-35be-4c23-95e6-efc7d46583cd-metrics-certs\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:53:02.433735 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:02.413934    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9479aa3-35be-4c23-95e6-efc7d46583cd-maas-api-tls\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:53:02.462666 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:02.452480    4559 patch_prober.go:29] interesting pod/e2e-worker-w1-61e341-openshift-default-59d97476dc-znwwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" start-of-body=
Nov 02 12:53:02.462666 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:02.452546    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-znwwt" podUID="53afdb1f-5ab4-4106-bccd-9ce04d9ced37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused"
Nov 02 12:53:02.462666 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:02.453370    4559 patch_prober.go:29] interesting pod/e2e-worker-w1-61e341-openshift-default-59d97476dc-7qxgl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" start-of-body=
Nov 02 12:53:02.462666 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:02.453401    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-7qxgl" podUID="13f8bac2-10c1-40aa-a968-3a9b64a6676f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused"
Nov 02 12:53:02.462666 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:02.462159    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/997a3029-5f17-4a07-be79-0d8f7aa2e4b2-kube-api-access-mvtlf" pod "997a3029-5f17-4a07-be79-0d8f7aa2e4b2" (UID: "997a3029-5f17-4a07-be79-0d8f7aa2e4b2"). InnerVolumeSpecName "kube-api-access-mvtlf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:53:02.509867 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:02.509829    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/997a3029-5f17-4a07-be79-0d8f7aa2e4b2-plugins-config-volume" pod "997a3029-5f17-4a07-be79-0d8f7aa2e4b2" (UID: "997a3029-5f17-4a07-be79-0d8f7aa2e4b2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:53:02.522755 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:02.515413    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mvtlf\" (UniqueName: \"kubernetes.io/projected/997a3029-5f17-4a07-be79-0d8f7aa2e4b2-kube-api-access-mvtlf\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:53:02.522755 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:02.515452    4559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/997a3029-5f17-4a07-be79-0d8f7aa2e4b2-plugins-config-volume\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:53:02.533432 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:02.533400    4559 generic.go:356] "Generic (PLEG): container finished" podID="c9479aa3-35be-4c23-95e6-efc7d46583cd" containerID="b835a2c0f7b3c90e3593fab5b8274c6c5f0cced9ae7ffa0c162a52eb26d57eab" exitCode=0
Nov 02 12:53:02.533584 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:02.533566    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d7801d-6ffffb74dc-prdc6" event={"ID":"c9479aa3-35be-4c23-95e6-efc7d46583cd","Type":"ContainerDied","Data":"b835a2c0f7b3c90e3593fab5b8274c6c5f0cced9ae7ffa0c162a52eb26d57eab"}
Nov 02 12:53:02.533700 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:02.533630    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d7801d-6ffffb74dc-prdc6" event={"ID":"c9479aa3-35be-4c23-95e6-efc7d46583cd","Type":"ContainerDied","Data":"a5363013607a4b7d89876ff7b1329a2d80a182ff6693c8ebf17d44615cc1b998"}
Nov 02 12:53:02.533700 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:02.533694    4559 scope.go:122] "RemoveContainer" containerID="b835a2c0f7b3c90e3593fab5b8274c6c5f0cced9ae7ffa0c162a52eb26d57eab"
Nov 02 12:53:02.534871 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:02.534718    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d7801d-6ffffb74dc-prdc6"
Nov 02 12:53:02.542252 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:02.542170    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d7801d-5f9f5c649f-9frfl"
Nov 02 12:53:02.546748 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:02.545278    4559 generic.go:356] "Generic (PLEG): container finished" podID="997a3029-5f17-4a07-be79-0d8f7aa2e4b2" containerID="5c30e4e48efdac5cb874cea03d931c7dae2f79f13dda0c7f96228b86830431e1" exitCode=0
Nov 02 12:53:02.546748 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:02.545375    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-d7801d-678658877b-bwjnd" event={"ID":"997a3029-5f17-4a07-be79-0d8f7aa2e4b2","Type":"ContainerDied","Data":"5c30e4e48efdac5cb874cea03d931c7dae2f79f13dda0c7f96228b86830431e1"}
Nov 02 12:53:02.546748 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:02.545419    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-d7801d-678658877b-bwjnd" event={"ID":"997a3029-5f17-4a07-be79-0d8f7aa2e4b2","Type":"ContainerDied","Data":"f136fbf0202915e31efb8a07957fee1c4a08517909f41eb41e82e8955e891131"}
Nov 02 12:53:02.546748 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:02.545497    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-d7801d-678658877b-bwjnd"
Nov 02 12:53:02.559016 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:02.558920    4559 scope.go:122] "RemoveContainer" containerID="b835a2c0f7b3c90e3593fab5b8274c6c5f0cced9ae7ffa0c162a52eb26d57eab"
Nov 02 12:53:02.565364 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:53:02.559687    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b835a2c0f7b3c90e3593fab5b8274c6c5f0cced9ae7ffa0c162a52eb26d57eab\": container with ID starting with b835a2c0f7b3c90e3593fab5b8274c6c5f0cced9ae7ffa0c162a52eb26d57eab not found: ID does not exist" containerID="b835a2c0f7b3c90e3593fab5b8274c6c5f0cced9ae7ffa0c162a52eb26d57eab"
Nov 02 12:53:02.565364 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:02.559728    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b835a2c0f7b3c90e3593fab5b8274c6c5f0cced9ae7ffa0c162a52eb26d57eab"} err="failed to get container status \"b835a2c0f7b3c90e3593fab5b8274c6c5f0cced9ae7ffa0c162a52eb26d57eab\": rpc error: code = NotFound desc = could not find container \"b835a2c0f7b3c90e3593fab5b8274c6c5f0cced9ae7ffa0c162a52eb26d57eab\": container with ID starting with b835a2c0f7b3c90e3593fab5b8274c6c5f0cced9ae7ffa0c162a52eb26d57eab not found: ID does not exist"
Nov 02 12:53:02.565364 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:02.559755    4559 scope.go:122] "RemoveContainer" containerID="5c30e4e48efdac5cb874cea03d931c7dae2f79f13dda0c7f96228b86830431e1"
Nov 02 12:53:02.574698 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:02.574415    4559 generic.go:356] "Generic (PLEG): container finished" podID="d3e81e99-3871-429a-b725-37cf024fbce1" containerID="98637dd2dea79b341d14b5c84b86de69922c1148d7757c5834a8cae749092fa3" exitCode=0
Nov 02 12:53:02.574698 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:02.574490    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d7801d-5f9f5c649f-9frfl" event={"ID":"d3e81e99-3871-429a-b725-37cf024fbce1","Type":"ContainerDied","Data":"98637dd2dea79b341d14b5c84b86de69922c1148d7757c5834a8cae749092fa3"}
Nov 02 12:53:02.574698 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:02.574599    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d7801d-5f9f5c649f-9frfl"
Nov 02 12:53:02.589592 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:02.589548    4559 scope.go:122] "RemoveContainer" containerID="5c30e4e48efdac5cb874cea03d931c7dae2f79f13dda0c7f96228b86830431e1"
Nov 02 12:53:02.590252 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:53:02.590089    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c30e4e48efdac5cb874cea03d931c7dae2f79f13dda0c7f96228b86830431e1\": container with ID starting with 5c30e4e48efdac5cb874cea03d931c7dae2f79f13dda0c7f96228b86830431e1 not found: ID does not exist" containerID="5c30e4e48efdac5cb874cea03d931c7dae2f79f13dda0c7f96228b86830431e1"
Nov 02 12:53:02.590252 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:02.590130    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c30e4e48efdac5cb874cea03d931c7dae2f79f13dda0c7f96228b86830431e1"} err="failed to get container status \"5c30e4e48efdac5cb874cea03d931c7dae2f79f13dda0c7f96228b86830431e1\": rpc error: code = NotFound desc = could not find container \"5c30e4e48efdac5cb874cea03d931c7dae2f79f13dda0c7f96228b86830431e1\": container with ID starting with 5c30e4e48efdac5cb874cea03d931c7dae2f79f13dda0c7f96228b86830431e1 not found: ID does not exist"
Nov 02 12:53:02.590252 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:02.590155    4559 scope.go:122] "RemoveContainer" containerID="98637dd2dea79b341d14b5c84b86de69922c1148d7757c5834a8cae749092fa3"
Nov 02 12:53:02.616379 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:02.616350    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3e81e99-3871-429a-b725-37cf024fbce1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3e81e99-3871-429a-b725-37cf024fbce1-plugins-config-volume\") pod \"d3e81e99-3871-429a-b725-37cf024fbce1\" (UID: \"d3e81e99-3871-429a-b725-37cf024fbce1\") "
Nov 02 12:53:02.616564 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:02.616447    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3e81e99-3871-429a-b725-37cf024fbce1-kube-api-access-89hgr\" (UniqueName: \"kubernetes.io/projected/d3e81e99-3871-429a-b725-37cf024fbce1-kube-api-access-89hgr\") pod \"d3e81e99-3871-429a-b725-37cf024fbce1\" (UID: \"d3e81e99-3871-429a-b725-37cf024fbce1\") "
Nov 02 12:53:02.616732 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:02.616709    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3e81e99-3871-429a-b725-37cf024fbce1-plugins-config-volume" pod "d3e81e99-3871-429a-b725-37cf024fbce1" (UID: "d3e81e99-3871-429a-b725-37cf024fbce1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:53:02.617052 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:02.617029    4559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3e81e99-3871-429a-b725-37cf024fbce1-plugins-config-volume\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:53:02.618710 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:02.618687    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3e81e99-3871-429a-b725-37cf024fbce1-kube-api-access-89hgr" pod "d3e81e99-3871-429a-b725-37cf024fbce1" (UID: "d3e81e99-3871-429a-b725-37cf024fbce1"). InnerVolumeSpecName "kube-api-access-89hgr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:53:02.725173 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:02.721920    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-89hgr\" (UniqueName: \"kubernetes.io/projected/d3e81e99-3871-429a-b725-37cf024fbce1-kube-api-access-89hgr\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:53:02.751452 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:02.751389    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-d7801d-6ffffb74dc-prdc6"]
Nov 02 12:53:02.757662 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:02.757592    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-d7801d-6ffffb74dc-prdc6"]
Nov 02 12:53:02.802112 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:02.802026    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d7801d-678658877b-bwjnd"]
Nov 02 12:53:02.806892 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:02.806841    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d7801d-678658877b-bwjnd"]
Nov 02 12:53:02.926210 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:02.926170    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d7801d-5f9f5c649f-9frfl"]
Nov 02 12:53:02.932480 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:02.932449    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d7801d-5f9f5c649f-9frfl"]
Nov 02 12:53:03.119708 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:03.115494    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-fbc46"]
Nov 02 12:53:03.119708 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:03.116320    4559 cpu_manager.go:414] "containerMap: removing container" podUID="d3e81e99-3871-429a-b725-37cf024fbce1" containerName="payload-pre-processing"
Nov 02 12:53:03.119708 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:03.116342    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3e81e99-3871-429a-b725-37cf024fbce1" containerName="payload-pre-processing"
Nov 02 12:53:03.119708 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:03.116409    4559 cpu_manager.go:414] "containerMap: removing container" podUID="997a3029-5f17-4a07-be79-0d8f7aa2e4b2" containerName="payload-processing"
Nov 02 12:53:03.119708 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:03.116418    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="997a3029-5f17-4a07-be79-0d8f7aa2e4b2" containerName="payload-processing"
Nov 02 12:53:03.119708 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:03.116433    4559 cpu_manager.go:414] "containerMap: removing container" podUID="c9479aa3-35be-4c23-95e6-efc7d46583cd" containerName="maas-api"
Nov 02 12:53:03.119708 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:03.116442    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9479aa3-35be-4c23-95e6-efc7d46583cd" containerName="maas-api"
Nov 02 12:53:03.119708 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:03.116588    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="997a3029-5f17-4a07-be79-0d8f7aa2e4b2" containerName="payload-processing"
Nov 02 12:53:03.119708 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:03.116600    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9479aa3-35be-4c23-95e6-efc7d46583cd" containerName="maas-api"
Nov 02 12:53:03.119708 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:03.116614    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3e81e99-3871-429a-b725-37cf024fbce1" containerName="payload-pre-processing"
Nov 02 12:53:03.123368 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:03.123330    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-fbc46"
Nov 02 12:53:03.145580 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:03.145298    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-fbc46"]
Nov 02 12:53:03.229797 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:03.229758    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/988e5461-e95e-408c-b145-7253a1a21056-istio-envoy\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-fbc46\" (UID: \"988e5461-e95e-408c-b145-7253a1a21056\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-fbc46"
Nov 02 12:53:03.230141 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:03.230116    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/988e5461-e95e-408c-b145-7253a1a21056-istio-data\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-fbc46\" (UID: \"988e5461-e95e-408c-b145-7253a1a21056\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-fbc46"
Nov 02 12:53:03.230367 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:03.230348    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mccs7\" (UniqueName: \"kubernetes.io/projected/988e5461-e95e-408c-b145-7253a1a21056-kube-api-access-mccs7\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-fbc46\" (UID: \"988e5461-e95e-408c-b145-7253a1a21056\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-fbc46"
Nov 02 12:53:03.230488 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:03.230472    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/988e5461-e95e-408c-b145-7253a1a21056-workload-certs\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-fbc46\" (UID: \"988e5461-e95e-408c-b145-7253a1a21056\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-fbc46"
Nov 02 12:53:03.230613 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:03.230598    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/988e5461-e95e-408c-b145-7253a1a21056-istiod-ca-cert\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-fbc46\" (UID: \"988e5461-e95e-408c-b145-7253a1a21056\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-fbc46"
Nov 02 12:53:03.230800 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:03.230781    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/988e5461-e95e-408c-b145-7253a1a21056-istio-token\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-fbc46\" (UID: \"988e5461-e95e-408c-b145-7253a1a21056\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-fbc46"
Nov 02 12:53:03.230930 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:03.230915    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/988e5461-e95e-408c-b145-7253a1a21056-credential-socket\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-fbc46\" (UID: \"988e5461-e95e-408c-b145-7253a1a21056\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-fbc46"
Nov 02 12:53:03.231072 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:03.231057    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/988e5461-e95e-408c-b145-7253a1a21056-istio-podinfo\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-fbc46\" (UID: \"988e5461-e95e-408c-b145-7253a1a21056\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-fbc46"
Nov 02 12:53:03.231197 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:03.231182    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/988e5461-e95e-408c-b145-7253a1a21056-workload-socket\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-fbc46\" (UID: \"988e5461-e95e-408c-b145-7253a1a21056\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-fbc46"
Nov 02 12:53:03.332534 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:03.332476    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/988e5461-e95e-408c-b145-7253a1a21056-istio-podinfo\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-fbc46\" (UID: \"988e5461-e95e-408c-b145-7253a1a21056\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-fbc46"
Nov 02 12:53:03.333218 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:03.332625    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/988e5461-e95e-408c-b145-7253a1a21056-workload-socket\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-fbc46\" (UID: \"988e5461-e95e-408c-b145-7253a1a21056\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-fbc46"
Nov 02 12:53:03.333359 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:03.333273    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/988e5461-e95e-408c-b145-7253a1a21056-istio-envoy\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-fbc46\" (UID: \"988e5461-e95e-408c-b145-7253a1a21056\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-fbc46"
Nov 02 12:53:03.333442 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:03.333380    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/988e5461-e95e-408c-b145-7253a1a21056-istio-data\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-fbc46\" (UID: \"988e5461-e95e-408c-b145-7253a1a21056\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-fbc46"
Nov 02 12:53:03.333514 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:03.333502    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mccs7\" (UniqueName: \"kubernetes.io/projected/988e5461-e95e-408c-b145-7253a1a21056-kube-api-access-mccs7\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-fbc46\" (UID: \"988e5461-e95e-408c-b145-7253a1a21056\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-fbc46"
Nov 02 12:53:03.333584 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:03.333539    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/988e5461-e95e-408c-b145-7253a1a21056-workload-certs\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-fbc46\" (UID: \"988e5461-e95e-408c-b145-7253a1a21056\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-fbc46"
Nov 02 12:53:03.333584 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:03.333574    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/988e5461-e95e-408c-b145-7253a1a21056-istiod-ca-cert\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-fbc46\" (UID: \"988e5461-e95e-408c-b145-7253a1a21056\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-fbc46"
Nov 02 12:53:03.333747 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:03.333676    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/988e5461-e95e-408c-b145-7253a1a21056-istio-token\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-fbc46\" (UID: \"988e5461-e95e-408c-b145-7253a1a21056\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-fbc46"
Nov 02 12:53:03.333747 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:03.333717    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/988e5461-e95e-408c-b145-7253a1a21056-credential-socket\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-fbc46\" (UID: \"988e5461-e95e-408c-b145-7253a1a21056\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-fbc46"
Nov 02 12:53:03.333877 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:03.333848    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/988e5461-e95e-408c-b145-7253a1a21056-workload-socket\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-fbc46\" (UID: \"988e5461-e95e-408c-b145-7253a1a21056\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-fbc46"
Nov 02 12:53:03.334521 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:03.334129    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/988e5461-e95e-408c-b145-7253a1a21056-credential-socket\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-fbc46\" (UID: \"988e5461-e95e-408c-b145-7253a1a21056\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-fbc46"
Nov 02 12:53:03.334625 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:03.334543    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/988e5461-e95e-408c-b145-7253a1a21056-istio-data\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-fbc46\" (UID: \"988e5461-e95e-408c-b145-7253a1a21056\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-fbc46"
Nov 02 12:53:03.334729 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:03.334687    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/988e5461-e95e-408c-b145-7253a1a21056-workload-certs\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-fbc46\" (UID: \"988e5461-e95e-408c-b145-7253a1a21056\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-fbc46"
Nov 02 12:53:03.334729 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:03.334712    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/988e5461-e95e-408c-b145-7253a1a21056-istiod-ca-cert\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-fbc46\" (UID: \"988e5461-e95e-408c-b145-7253a1a21056\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-fbc46"
Nov 02 12:53:03.336062 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:03.336021    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/988e5461-e95e-408c-b145-7253a1a21056-istio-podinfo\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-fbc46\" (UID: \"988e5461-e95e-408c-b145-7253a1a21056\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-fbc46"
Nov 02 12:53:03.337763 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:03.337736    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/988e5461-e95e-408c-b145-7253a1a21056-istio-envoy\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-fbc46\" (UID: \"988e5461-e95e-408c-b145-7253a1a21056\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-fbc46"
Nov 02 12:53:03.359607 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:03.359570    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mccs7\" (UniqueName: \"kubernetes.io/projected/988e5461-e95e-408c-b145-7253a1a21056-kube-api-access-mccs7\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-fbc46\" (UID: \"988e5461-e95e-408c-b145-7253a1a21056\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-fbc46"
Nov 02 12:53:03.360108 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:03.360079    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/988e5461-e95e-408c-b145-7253a1a21056-istio-token\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-fbc46\" (UID: \"988e5461-e95e-408c-b145-7253a1a21056\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-fbc46"
Nov 02 12:53:03.459255 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:03.459205    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-fbc46"
Nov 02 12:53:03.575961 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:03.574624    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-7qxgl"
Nov 02 12:53:03.575961 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:03.575705    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-7qxgl"
Nov 02 12:53:03.577144 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:03.577082    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-7qxgl"
Nov 02 12:53:03.603991 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:03.603958    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-znwwt"
Nov 02 12:53:03.604609 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:03.604557    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-znwwt"
Nov 02 12:53:03.605752 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:03.605731    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-znwwt"
Nov 02 12:53:03.817523 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:53:03.817490    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod988e5461_e95e_408c_b145_7253a1a21056.slice/crio-ccc27a630aea470d8a537fb23e7a2ae1c995c1300fdda53f2a1fe8fbc3e2a866 WatchSource:0}: Error finding container ccc27a630aea470d8a537fb23e7a2ae1c995c1300fdda53f2a1fe8fbc3e2a866: Status 404 returned error can't find the container with id ccc27a630aea470d8a537fb23e7a2ae1c995c1300fdda53f2a1fe8fbc3e2a866
Nov 02 12:53:03.818499 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:03.818475    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-fbc46"]
Nov 02 12:53:03.820214 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:03.820171    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:53:03.820314 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:03.820268    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:53:03.820359 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:03.820310    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:53:03.898768 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:03.898735    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="997a3029-5f17-4a07-be79-0d8f7aa2e4b2" path="/var/lib/kubelet/pods/997a3029-5f17-4a07-be79-0d8f7aa2e4b2/volumes"
Nov 02 12:53:03.899421 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:03.899398    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9479aa3-35be-4c23-95e6-efc7d46583cd" path="/var/lib/kubelet/pods/c9479aa3-35be-4c23-95e6-efc7d46583cd/volumes"
Nov 02 12:53:03.899990 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:03.899972    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3e81e99-3871-429a-b725-37cf024fbce1" path="/var/lib/kubelet/pods/d3e81e99-3871-429a-b725-37cf024fbce1/volumes"
Nov 02 12:53:04.638787 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:04.638726    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-fbc46" event={"ID":"988e5461-e95e-408c-b145-7253a1a21056","Type":"ContainerStarted","Data":"8d70670962ad5c661ab1ecfd443fb8be9b3543ba0a63d44d92a0cac91b173323"}
Nov 02 12:53:04.639473 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:04.639447    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-fbc46" event={"ID":"988e5461-e95e-408c-b145-7253a1a21056","Type":"ContainerStarted","Data":"ccc27a630aea470d8a537fb23e7a2ae1c995c1300fdda53f2a1fe8fbc3e2a866"}
Nov 02 12:53:05.459727 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:05.459694    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-fbc46"
Nov 02 12:53:05.461288 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:05.461259    4559 patch_prober.go:29] interesting pod/e2e-worker-w2-95ee26-openshift-default-66679d8f57-fbc46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body=
Nov 02 12:53:05.461433 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:05.461318    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-fbc46" podUID="988e5461-e95e-408c-b145-7253a1a21056" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused"
Nov 02 12:53:06.460398 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:06.460359    4559 patch_prober.go:29] interesting pod/e2e-worker-w2-95ee26-openshift-default-66679d8f57-fbc46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body=
Nov 02 12:53:06.460857 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:06.460428    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-fbc46" podUID="988e5461-e95e-408c-b145-7253a1a21056" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused"
Nov 02 12:53:07.464409 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:07.464361    4559 patch_prober.go:29] interesting pod/e2e-worker-w2-95ee26-openshift-default-66679d8f57-fbc46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body=
Nov 02 12:53:07.464938 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:07.464424    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-fbc46" podUID="988e5461-e95e-408c-b145-7253a1a21056" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused"
Nov 02 12:53:08.459501 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:08.459462    4559 patch_prober.go:29] interesting pod/e2e-worker-w2-95ee26-openshift-default-66679d8f57-fbc46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body=
Nov 02 12:53:08.459756 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:08.459523    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-fbc46" podUID="988e5461-e95e-408c-b145-7253a1a21056" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused"
Nov 02 12:53:09.460370 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:09.460327    4559 patch_prober.go:29] interesting pod/e2e-worker-w2-95ee26-openshift-default-66679d8f57-fbc46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body=
Nov 02 12:53:09.460866 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:09.460395    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-fbc46" podUID="988e5461-e95e-408c-b145-7253a1a21056" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused"
Nov 02 12:53:10.460407 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:10.460371    4559 patch_prober.go:29] interesting pod/e2e-worker-w2-95ee26-openshift-default-66679d8f57-fbc46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body=
Nov 02 12:53:10.460917 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:10.460429    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-fbc46" podUID="988e5461-e95e-408c-b145-7253a1a21056" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused"
Nov 02 12:53:10.951680 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:10.951409    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-fbc46" podStartSLOduration=7.951388669 podStartE2EDuration="7.951388669s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:53:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:53:04.715986127 +0000 UTC m=+1580.971385247" watchObservedRunningTime="2026-11-02 12:53:10.951388669 +0000 UTC m=+1587.206787766"
Nov 02 12:53:10.958322 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:10.956931    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-d7801d-openshift-default-674cd67b5-xgb9d"]
Nov 02 12:53:10.958322 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:10.957601    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-d7801d-openshift-default-674cd67b5-xgb9d" podUID="03b6b83c-10cd-4876-b17a-96120c2317b7" containerName="istio-proxy" containerID="cri-o://ee7f13ca3c8c1367922192eb56747411b40607e8ed457ce506472100e803ad05" gracePeriod=30
Nov 02 12:53:11.489897 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:11.489862    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-fbc46"
Nov 02 12:53:11.490345 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:11.490226    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-fbc46"
Nov 02 12:53:11.491652 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:11.491612    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-fbc46"
Nov 02 12:53:12.999253 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:12.999211    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-c0d4cb-openshift-default-6798dbb74f-rslvw"]
Nov 02 12:53:13.008703 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:13.008623    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-c0d4cb-openshift-default-6798dbb74f-rslvw"
Nov 02 12:53:13.054916 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:13.054883    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-c0d4cb-openshift-default-dockercfg-qcdpw\""
Nov 02 12:53:13.061693 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:13.061661    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-c0d4cb-openshift-default-6798dbb74f-rslvw"]
Nov 02 12:53:13.175000 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:13.174952    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24bcb862-30cd-44b9-920e-a919d80678a9-istio-data\") pod \"e2e-praxis-a-c0d4cb-openshift-default-6798dbb74f-rslvw\" (UID: \"24bcb862-30cd-44b9-920e-a919d80678a9\") " pod="openshift-ingress/e2e-praxis-a-c0d4cb-openshift-default-6798dbb74f-rslvw"
Nov 02 12:53:13.175336 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:13.175307    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24bcb862-30cd-44b9-920e-a919d80678a9-istiod-ca-cert\") pod \"e2e-praxis-a-c0d4cb-openshift-default-6798dbb74f-rslvw\" (UID: \"24bcb862-30cd-44b9-920e-a919d80678a9\") " pod="openshift-ingress/e2e-praxis-a-c0d4cb-openshift-default-6798dbb74f-rslvw"
Nov 02 12:53:13.175550 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:13.175532    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24bcb862-30cd-44b9-920e-a919d80678a9-istio-envoy\") pod \"e2e-praxis-a-c0d4cb-openshift-default-6798dbb74f-rslvw\" (UID: \"24bcb862-30cd-44b9-920e-a919d80678a9\") " pod="openshift-ingress/e2e-praxis-a-c0d4cb-openshift-default-6798dbb74f-rslvw"
Nov 02 12:53:13.175710 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:13.175693    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24bcb862-30cd-44b9-920e-a919d80678a9-istio-podinfo\") pod \"e2e-praxis-a-c0d4cb-openshift-default-6798dbb74f-rslvw\" (UID: \"24bcb862-30cd-44b9-920e-a919d80678a9\") " pod="openshift-ingress/e2e-praxis-a-c0d4cb-openshift-default-6798dbb74f-rslvw"
Nov 02 12:53:13.175882 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:13.175857    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24bcb862-30cd-44b9-920e-a919d80678a9-workload-socket\") pod \"e2e-praxis-a-c0d4cb-openshift-default-6798dbb74f-rslvw\" (UID: \"24bcb862-30cd-44b9-920e-a919d80678a9\") " pod="openshift-ingress/e2e-praxis-a-c0d4cb-openshift-default-6798dbb74f-rslvw"
Nov 02 12:53:13.176085 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:13.176058    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2sxww\" (UniqueName: \"kubernetes.io/projected/24bcb862-30cd-44b9-920e-a919d80678a9-kube-api-access-2sxww\") pod \"e2e-praxis-a-c0d4cb-openshift-default-6798dbb74f-rslvw\" (UID: \"24bcb862-30cd-44b9-920e-a919d80678a9\") " pod="openshift-ingress/e2e-praxis-a-c0d4cb-openshift-default-6798dbb74f-rslvw"
Nov 02 12:53:13.176205 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:13.176190    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24bcb862-30cd-44b9-920e-a919d80678a9-credential-socket\") pod \"e2e-praxis-a-c0d4cb-openshift-default-6798dbb74f-rslvw\" (UID: \"24bcb862-30cd-44b9-920e-a919d80678a9\") " pod="openshift-ingress/e2e-praxis-a-c0d4cb-openshift-default-6798dbb74f-rslvw"
Nov 02 12:53:13.176335 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:13.176320    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24bcb862-30cd-44b9-920e-a919d80678a9-workload-certs\") pod \"e2e-praxis-a-c0d4cb-openshift-default-6798dbb74f-rslvw\" (UID: \"24bcb862-30cd-44b9-920e-a919d80678a9\") " pod="openshift-ingress/e2e-praxis-a-c0d4cb-openshift-default-6798dbb74f-rslvw"
Nov 02 12:53:13.176468 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:13.176446    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24bcb862-30cd-44b9-920e-a919d80678a9-istio-token\") pod \"e2e-praxis-a-c0d4cb-openshift-default-6798dbb74f-rslvw\" (UID: \"24bcb862-30cd-44b9-920e-a919d80678a9\") " pod="openshift-ingress/e2e-praxis-a-c0d4cb-openshift-default-6798dbb74f-rslvw"
Nov 02 12:53:13.282778 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:13.280177    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24bcb862-30cd-44b9-920e-a919d80678a9-istiod-ca-cert\") pod \"e2e-praxis-a-c0d4cb-openshift-default-6798dbb74f-rslvw\" (UID: \"24bcb862-30cd-44b9-920e-a919d80678a9\") " pod="openshift-ingress/e2e-praxis-a-c0d4cb-openshift-default-6798dbb74f-rslvw"
Nov 02 12:53:13.282778 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:13.280306    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24bcb862-30cd-44b9-920e-a919d80678a9-istio-envoy\") pod \"e2e-praxis-a-c0d4cb-openshift-default-6798dbb74f-rslvw\" (UID: \"24bcb862-30cd-44b9-920e-a919d80678a9\") " pod="openshift-ingress/e2e-praxis-a-c0d4cb-openshift-default-6798dbb74f-rslvw"
Nov 02 12:53:13.282778 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:13.280346    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24bcb862-30cd-44b9-920e-a919d80678a9-istio-podinfo\") pod \"e2e-praxis-a-c0d4cb-openshift-default-6798dbb74f-rslvw\" (UID: \"24bcb862-30cd-44b9-920e-a919d80678a9\") " pod="openshift-ingress/e2e-praxis-a-c0d4cb-openshift-default-6798dbb74f-rslvw"
Nov 02 12:53:13.282778 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:13.280399    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24bcb862-30cd-44b9-920e-a919d80678a9-workload-socket\") pod \"e2e-praxis-a-c0d4cb-openshift-default-6798dbb74f-rslvw\" (UID: \"24bcb862-30cd-44b9-920e-a919d80678a9\") " pod="openshift-ingress/e2e-praxis-a-c0d4cb-openshift-default-6798dbb74f-rslvw"
Nov 02 12:53:13.282778 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:13.280504    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2sxww\" (UniqueName: \"kubernetes.io/projected/24bcb862-30cd-44b9-920e-a919d80678a9-kube-api-access-2sxww\") pod \"e2e-praxis-a-c0d4cb-openshift-default-6798dbb74f-rslvw\" (UID: \"24bcb862-30cd-44b9-920e-a919d80678a9\") " pod="openshift-ingress/e2e-praxis-a-c0d4cb-openshift-default-6798dbb74f-rslvw"
Nov 02 12:53:13.282778 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:13.280564    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24bcb862-30cd-44b9-920e-a919d80678a9-credential-socket\") pod \"e2e-praxis-a-c0d4cb-openshift-default-6798dbb74f-rslvw\" (UID: \"24bcb862-30cd-44b9-920e-a919d80678a9\") " pod="openshift-ingress/e2e-praxis-a-c0d4cb-openshift-default-6798dbb74f-rslvw"
Nov 02 12:53:13.282778 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:13.280618    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24bcb862-30cd-44b9-920e-a919d80678a9-workload-certs\") pod \"e2e-praxis-a-c0d4cb-openshift-default-6798dbb74f-rslvw\" (UID: \"24bcb862-30cd-44b9-920e-a919d80678a9\") " pod="openshift-ingress/e2e-praxis-a-c0d4cb-openshift-default-6798dbb74f-rslvw"
Nov 02 12:53:13.282778 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:13.280668    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24bcb862-30cd-44b9-920e-a919d80678a9-istio-token\") pod \"e2e-praxis-a-c0d4cb-openshift-default-6798dbb74f-rslvw\" (UID: \"24bcb862-30cd-44b9-920e-a919d80678a9\") " pod="openshift-ingress/e2e-praxis-a-c0d4cb-openshift-default-6798dbb74f-rslvw"
Nov 02 12:53:13.282778 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:13.280894    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24bcb862-30cd-44b9-920e-a919d80678a9-istio-data\") pod \"e2e-praxis-a-c0d4cb-openshift-default-6798dbb74f-rslvw\" (UID: \"24bcb862-30cd-44b9-920e-a919d80678a9\") " pod="openshift-ingress/e2e-praxis-a-c0d4cb-openshift-default-6798dbb74f-rslvw"
Nov 02 12:53:13.282778 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:13.281588    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24bcb862-30cd-44b9-920e-a919d80678a9-istio-data\") pod \"e2e-praxis-a-c0d4cb-openshift-default-6798dbb74f-rslvw\" (UID: \"24bcb862-30cd-44b9-920e-a919d80678a9\") " pod="openshift-ingress/e2e-praxis-a-c0d4cb-openshift-default-6798dbb74f-rslvw"
Nov 02 12:53:13.282778 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:13.282010    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24bcb862-30cd-44b9-920e-a919d80678a9-workload-socket\") pod \"e2e-praxis-a-c0d4cb-openshift-default-6798dbb74f-rslvw\" (UID: \"24bcb862-30cd-44b9-920e-a919d80678a9\") " pod="openshift-ingress/e2e-praxis-a-c0d4cb-openshift-default-6798dbb74f-rslvw"
Nov 02 12:53:13.282778 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:13.282506    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24bcb862-30cd-44b9-920e-a919d80678a9-credential-socket\") pod \"e2e-praxis-a-c0d4cb-openshift-default-6798dbb74f-rslvw\" (UID: \"24bcb862-30cd-44b9-920e-a919d80678a9\") " pod="openshift-ingress/e2e-praxis-a-c0d4cb-openshift-default-6798dbb74f-rslvw"
Nov 02 12:53:13.282778 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:13.282719    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24bcb862-30cd-44b9-920e-a919d80678a9-workload-certs\") pod \"e2e-praxis-a-c0d4cb-openshift-default-6798dbb74f-rslvw\" (UID: \"24bcb862-30cd-44b9-920e-a919d80678a9\") " pod="openshift-ingress/e2e-praxis-a-c0d4cb-openshift-default-6798dbb74f-rslvw"
Nov 02 12:53:13.285481 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:13.283589    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24bcb862-30cd-44b9-920e-a919d80678a9-istiod-ca-cert\") pod \"e2e-praxis-a-c0d4cb-openshift-default-6798dbb74f-rslvw\" (UID: \"24bcb862-30cd-44b9-920e-a919d80678a9\") " pod="openshift-ingress/e2e-praxis-a-c0d4cb-openshift-default-6798dbb74f-rslvw"
Nov 02 12:53:13.299690 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:13.296461    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24bcb862-30cd-44b9-920e-a919d80678a9-istio-podinfo\") pod \"e2e-praxis-a-c0d4cb-openshift-default-6798dbb74f-rslvw\" (UID: \"24bcb862-30cd-44b9-920e-a919d80678a9\") " pod="openshift-ingress/e2e-praxis-a-c0d4cb-openshift-default-6798dbb74f-rslvw"
Nov 02 12:53:13.316675 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:13.312319    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2sxww\" (UniqueName: \"kubernetes.io/projected/24bcb862-30cd-44b9-920e-a919d80678a9-kube-api-access-2sxww\") pod \"e2e-praxis-a-c0d4cb-openshift-default-6798dbb74f-rslvw\" (UID: \"24bcb862-30cd-44b9-920e-a919d80678a9\") " pod="openshift-ingress/e2e-praxis-a-c0d4cb-openshift-default-6798dbb74f-rslvw"
Nov 02 12:53:13.326250 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:13.320727    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24bcb862-30cd-44b9-920e-a919d80678a9-istio-envoy\") pod \"e2e-praxis-a-c0d4cb-openshift-default-6798dbb74f-rslvw\" (UID: \"24bcb862-30cd-44b9-920e-a919d80678a9\") " pod="openshift-ingress/e2e-praxis-a-c0d4cb-openshift-default-6798dbb74f-rslvw"
Nov 02 12:53:13.327150 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:13.327109    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24bcb862-30cd-44b9-920e-a919d80678a9-istio-token\") pod \"e2e-praxis-a-c0d4cb-openshift-default-6798dbb74f-rslvw\" (UID: \"24bcb862-30cd-44b9-920e-a919d80678a9\") " pod="openshift-ingress/e2e-praxis-a-c0d4cb-openshift-default-6798dbb74f-rslvw"
Nov 02 12:53:13.343087 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:13.343053    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-c0d4cb-openshift-default-6798dbb74f-rslvw"
Nov 02 12:53:13.673495 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:53:13.673469    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24bcb862_30cd_44b9_920e_a919d80678a9.slice/crio-0c87267e2e1782df03326cd495e0dc9251a6247360e71e0359604da21728b88b WatchSource:0}: Error finding container 0c87267e2e1782df03326cd495e0dc9251a6247360e71e0359604da21728b88b: Status 404 returned error can't find the container with id 0c87267e2e1782df03326cd495e0dc9251a6247360e71e0359604da21728b88b
Nov 02 12:53:13.674744 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:13.674716    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-c0d4cb-openshift-default-6798dbb74f-rslvw"]
Nov 02 12:53:13.676431 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:13.676395    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:53:13.676504 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:13.676466    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:53:13.676504 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:13.676496    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:53:13.728705 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:13.728677    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-c0d4cb-openshift-default-6798dbb74f-rslvw" event={"ID":"24bcb862-30cd-44b9-920e-a919d80678a9","Type":"ContainerStarted","Data":"0c87267e2e1782df03326cd495e0dc9251a6247360e71e0359604da21728b88b"}
Nov 02 12:53:14.737089 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:14.737050    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-c0d4cb-openshift-default-6798dbb74f-rslvw" event={"ID":"24bcb862-30cd-44b9-920e-a919d80678a9","Type":"ContainerStarted","Data":"693fedc45dfde9ab92b41dc6b96b36bffe723cac46d4a469e5d74ee9285ce816"}
Nov 02 12:53:14.818464 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:14.818414    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-c0d4cb-openshift-default-6798dbb74f-rslvw" podStartSLOduration=2.818399436 podStartE2EDuration="2.818399436s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:53:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:53:14.817215952 +0000 UTC m=+1591.072615047" watchObservedRunningTime="2026-11-02 12:53:14.818399436 +0000 UTC m=+1591.073798530"
Nov 02 12:53:14.819940 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:14.819912    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"]
Nov 02 12:53:15.087746 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:53:15.087660    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqcmj" containerName="extractor"
Nov 02 12:53:15.343672 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:15.343573    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-c0d4cb-openshift-default-6798dbb74f-rslvw"
Nov 02 12:53:15.348941 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:15.348912    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-c0d4cb-openshift-default-6798dbb74f-rslvw"
Nov 02 12:53:15.758658 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:15.758068    4559 patch_prober.go:29] interesting pod/e2e-ipp-b-d7801d-openshift-default-674cd67b5-xgb9d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 12:53:15.758658 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:15.758187    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-d7801d-openshift-default-674cd67b5-xgb9d" podUID="03b6b83c-10cd-4876-b17a-96120c2317b7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 12:53:15.795669 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:15.795101    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-c0d4cb-openshift-default-6798dbb74f-rslvw"
Nov 02 12:53:15.849692 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:15.838130    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-c0d4cb-openshift-default-6798dbb74f-rslvw"
Nov 02 12:53:16.838000 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:16.836664    4559 generic.go:356] "Generic (PLEG): container finished" podID="03b6b83c-10cd-4876-b17a-96120c2317b7" containerID="ee7f13ca3c8c1367922192eb56747411b40607e8ed457ce506472100e803ad05" exitCode=0
Nov 02 12:53:16.838000 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:16.837723    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d7801d-openshift-default-674cd67b5-xgb9d" event={"ID":"03b6b83c-10cd-4876-b17a-96120c2317b7","Type":"ContainerDied","Data":"ee7f13ca3c8c1367922192eb56747411b40607e8ed457ce506472100e803ad05"}
Nov 02 12:53:16.838000 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:16.837786    4559 scope.go:122] "RemoveContainer" containerID="6224548b7e582bcc650188ee8bfdd325f69efc976323a76a62849fbafd0ecc29"
Nov 02 12:53:17.067672 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:17.064804    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843978/distinct-w4-843978-kserve-6fcf588858-4nrwx" podUID="82e5d483-e95b-43e6-9135-af960823427a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:53:17.127846 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:17.127737    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-61e341/distinct-2-w1-61e341-kserve-954cf956f-n7jmg" podUID="c5e1a5f8-97b4-469d-be9a-eae7ca995283" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:53:17.369665 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:17.362799    4559 patch_prober.go:29] interesting pod/e2e-worker-w2-95ee26-openshift-default-66679d8f57-bh68g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 12:53:17.369665 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:17.362882    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-bh68g" podUID="76dc13d1-a3a7-4f47-85a2-f596cafbcb94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:53:17.686661 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:17.677769    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-61e341/distinct-w1-61e341-kserve-787cd566f4-cjj8s" podUID="6886ce28-5800-402d-b893-f92109e289a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:53:17.971716 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:17.968367    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-d7801d-openshift-default-674cd67b5-xgb9d"
Nov 02 12:53:18.021948 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:18.021909    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d7801d-openshift-default-674cd67b5-xgb9d" event={"ID":"03b6b83c-10cd-4876-b17a-96120c2317b7","Type":"ContainerDied","Data":"2c512caa65bb6bb6e2021a95f4eaa40d0128dd350385ffa63dea79221720c621"}
Nov 02 12:53:18.044787 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:18.044748    4559 scope.go:122] "RemoveContainer" containerID="ee7f13ca3c8c1367922192eb56747411b40607e8ed457ce506472100e803ad05"
Nov 02 12:53:18.047117 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:18.046922    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03b6b83c-10cd-4876-b17a-96120c2317b7-credential-socket" pod "03b6b83c-10cd-4876-b17a-96120c2317b7" (UID: "03b6b83c-10cd-4876-b17a-96120c2317b7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:53:18.047751 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:18.046965    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03b6b83c-10cd-4876-b17a-96120c2317b7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03b6b83c-10cd-4876-b17a-96120c2317b7-credential-socket\") pod \"03b6b83c-10cd-4876-b17a-96120c2317b7\" (UID: \"03b6b83c-10cd-4876-b17a-96120c2317b7\") "
Nov 02 12:53:18.048013 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:18.047996    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/03b6b83c-10cd-4876-b17a-96120c2317b7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03b6b83c-10cd-4876-b17a-96120c2317b7-istio-podinfo\") pod \"03b6b83c-10cd-4876-b17a-96120c2317b7\" (UID: \"03b6b83c-10cd-4876-b17a-96120c2317b7\") "
Nov 02 12:53:18.048768 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:18.048748    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03b6b83c-10cd-4876-b17a-96120c2317b7-kube-api-access-rjs8r\" (UniqueName: \"kubernetes.io/projected/03b6b83c-10cd-4876-b17a-96120c2317b7-kube-api-access-rjs8r\") pod \"03b6b83c-10cd-4876-b17a-96120c2317b7\" (UID: \"03b6b83c-10cd-4876-b17a-96120c2317b7\") "
Nov 02 12:53:18.055908 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:18.055884    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03b6b83c-10cd-4876-b17a-96120c2317b7-istio-token\" (UniqueName: \"kubernetes.io/projected/03b6b83c-10cd-4876-b17a-96120c2317b7-istio-token\") pod \"03b6b83c-10cd-4876-b17a-96120c2317b7\" (UID: \"03b6b83c-10cd-4876-b17a-96120c2317b7\") "
Nov 02 12:53:18.085080 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:18.066873    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843978/distinct-w4-843978-kserve-6fcf588858-4nrwx" podUID="82e5d483-e95b-43e6-9135-af960823427a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:53:18.085080 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:18.068551    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-843978/distinct-w4-843978-kserve-6fcf588858-4nrwx"
Nov 02 12:53:18.085080 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:18.084777    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03b6b83c-10cd-4876-b17a-96120c2317b7-workload-certs" pod "03b6b83c-10cd-4876-b17a-96120c2317b7" (UID: "03b6b83c-10cd-4876-b17a-96120c2317b7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:53:18.148339 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:18.097057    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-61e341/premium-w1-61e341-kserve-746477b99-645g6" podUID="02ff21d3-5a37-493a-ab95-8e4784c29123" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:53:18.148339 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:18.101254    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03b6b83c-10cd-4876-b17a-96120c2317b7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03b6b83c-10cd-4876-b17a-96120c2317b7-workload-certs\") pod \"03b6b83c-10cd-4876-b17a-96120c2317b7\" (UID: \"03b6b83c-10cd-4876-b17a-96120c2317b7\") "
Nov 02 12:53:18.148339 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:18.101354    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03b6b83c-10cd-4876-b17a-96120c2317b7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03b6b83c-10cd-4876-b17a-96120c2317b7-workload-socket\") pod \"03b6b83c-10cd-4876-b17a-96120c2317b7\" (UID: \"03b6b83c-10cd-4876-b17a-96120c2317b7\") "
Nov 02 12:53:18.148339 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:18.101412    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03b6b83c-10cd-4876-b17a-96120c2317b7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03b6b83c-10cd-4876-b17a-96120c2317b7-istio-envoy\") pod \"03b6b83c-10cd-4876-b17a-96120c2317b7\" (UID: \"03b6b83c-10cd-4876-b17a-96120c2317b7\") "
Nov 02 12:53:18.148339 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:18.101445    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03b6b83c-10cd-4876-b17a-96120c2317b7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03b6b83c-10cd-4876-b17a-96120c2317b7-istiod-ca-cert\") pod \"03b6b83c-10cd-4876-b17a-96120c2317b7\" (UID: \"03b6b83c-10cd-4876-b17a-96120c2317b7\") "
Nov 02 12:53:18.148339 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:18.101499    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03b6b83c-10cd-4876-b17a-96120c2317b7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03b6b83c-10cd-4876-b17a-96120c2317b7-istio-data\") pod \"03b6b83c-10cd-4876-b17a-96120c2317b7\" (UID: \"03b6b83c-10cd-4876-b17a-96120c2317b7\") "
Nov 02 12:53:18.148339 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:18.101900    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03b6b83c-10cd-4876-b17a-96120c2317b7-workload-socket" pod "03b6b83c-10cd-4876-b17a-96120c2317b7" (UID: "03b6b83c-10cd-4876-b17a-96120c2317b7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:53:18.333253 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:18.333162    4559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03b6b83c-10cd-4876-b17a-96120c2317b7-credential-socket\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:53:18.333515 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:18.333492    4559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03b6b83c-10cd-4876-b17a-96120c2317b7-workload-certs\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:53:18.333670 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:18.333655    4559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03b6b83c-10cd-4876-b17a-96120c2317b7-workload-socket\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:53:18.338764 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:18.336679    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-61e341/distinct-2-w1-61e341-kserve-954cf956f-n7jmg" podUID="c5e1a5f8-97b4-469d-be9a-eae7ca995283" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded"
Nov 02 12:53:18.338764 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:18.337111    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-61e341/distinct-2-w1-61e341-kserve-954cf956f-n7jmg"
Nov 02 12:53:18.341706 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:18.341582    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03b6b83c-10cd-4876-b17a-96120c2317b7-istio-data" pod "03b6b83c-10cd-4876-b17a-96120c2317b7" (UID: "03b6b83c-10cd-4876-b17a-96120c2317b7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:53:18.350722 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:18.344865    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/03b6b83c-10cd-4876-b17a-96120c2317b7-istio-podinfo" pod "03b6b83c-10cd-4876-b17a-96120c2317b7" (UID: "03b6b83c-10cd-4876-b17a-96120c2317b7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 12:53:18.350722 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:18.348030    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03b6b83c-10cd-4876-b17a-96120c2317b7-istiod-ca-cert" pod "03b6b83c-10cd-4876-b17a-96120c2317b7" (UID: "03b6b83c-10cd-4876-b17a-96120c2317b7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:53:18.450674 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:18.438340    4559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03b6b83c-10cd-4876-b17a-96120c2317b7-istio-podinfo\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:53:18.671374 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:18.671346    4559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03b6b83c-10cd-4876-b17a-96120c2317b7-istiod-ca-cert\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:53:18.699805 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:18.699775    4559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03b6b83c-10cd-4876-b17a-96120c2317b7-istio-data\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:53:18.727911 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:18.693847    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-61e341/distinct-w1-61e341-kserve-787cd566f4-cjj8s" podUID="6886ce28-5800-402d-b893-f92109e289a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:53:18.728403 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:18.511288    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03b6b83c-10cd-4876-b17a-96120c2317b7-istio-token" pod "03b6b83c-10cd-4876-b17a-96120c2317b7" (UID: "03b6b83c-10cd-4876-b17a-96120c2317b7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:53:18.969851 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:18.881656    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03b6b83c-10cd-4876-b17a-96120c2317b7-istio-envoy" pod "03b6b83c-10cd-4876-b17a-96120c2317b7" (UID: "03b6b83c-10cd-4876-b17a-96120c2317b7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:53:18.991904 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:18.952779    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03b6b83c-10cd-4876-b17a-96120c2317b7-kube-api-access-rjs8r" pod "03b6b83c-10cd-4876-b17a-96120c2317b7" (UID: "03b6b83c-10cd-4876-b17a-96120c2317b7"). InnerVolumeSpecName "kube-api-access-rjs8r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:53:18.991904 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:18.983864    4559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03b6b83c-10cd-4876-b17a-96120c2317b7-istio-token\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:53:18.991904 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:18.734837    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-61e341/distinct-w1-61e341-kserve-787cd566f4-cjj8s"
Nov 02 12:53:19.017913 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:19.017870    4559 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-61e341/distinct-w1-61e341-kserve-787cd566f4-cjj8s" podUID="6886ce28-5800-402d-b893-f92109e289a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:53:19.217248 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:19.217205    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843978/distinct-w4-843978-kserve-6fcf588858-4nrwx" podUID="82e5d483-e95b-43e6-9135-af960823427a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Nov 02 12:53:19.237105 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:19.237009    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-d7801d-openshift-default-674cd67b5-xgb9d"
Nov 02 12:53:19.534876 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:19.534149    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rjs8r\" (UniqueName: \"kubernetes.io/projected/03b6b83c-10cd-4876-b17a-96120c2317b7-kube-api-access-rjs8r\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:53:19.534876 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:19.534182    4559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03b6b83c-10cd-4876-b17a-96120c2317b7-istio-envoy\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:53:19.540782 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:19.540741    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-843978/distinct-w4-843978-kserve-6fcf588858-4nrwx"
Nov 02 12:53:19.540967 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:19.540824    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-61e341/distinct-2-w1-61e341-kserve-954cf956f-n7jmg"
Nov 02 12:53:19.540967 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:19.540862    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-61e341/distinct-w1-61e341-kserve-787cd566f4-cjj8s"
Nov 02 12:53:19.641065 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:19.641025    4559 patch_prober.go:29] interesting pod/e2e-worker-w1-61e341-openshift-default-59d97476dc-7qxgl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.173:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 12:53:19.641355 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:19.641322    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-7qxgl" podUID="13f8bac2-10c1-40aa-a968-3a9b64a6676f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.173:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:53:20.424496 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:53:20.416436    4559 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03b6b83c_10cd_4876_b17a_96120c2317b7.slice/crio-2c512caa65bb6bb6e2021a95f4eaa40d0128dd350385ffa63dea79221720c621\": RecentStats: unable to find data in memory cache]"
Nov 02 12:53:20.570139 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:20.570091    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-61e341/distinct-2-w1-61e341-kserve-954cf956f-n7jmg" podUID="c5e1a5f8-97b4-469d-be9a-eae7ca995283" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:53:20.599734 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:20.599696    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-d7801d-openshift-default-674cd67b5-xgb9d"]
Nov 02 12:53:20.645872 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:20.644745    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-d7801d-openshift-default-674cd67b5-xgb9d"]
Nov 02 12:53:21.012035 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:21.012001    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-c0d4cb-867cbccf7f-pmph4"]
Nov 02 12:53:21.012560 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:21.012547    4559 cpu_manager.go:414] "containerMap: removing container" podUID="03b6b83c-10cd-4876-b17a-96120c2317b7" containerName="istio-proxy"
Nov 02 12:53:21.012607 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:21.012562    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03b6b83c-10cd-4876-b17a-96120c2317b7" containerName="istio-proxy"
Nov 02 12:53:21.012607 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:21.012580    4559 cpu_manager.go:414] "containerMap: removing container" podUID="03b6b83c-10cd-4876-b17a-96120c2317b7" containerName="istio-proxy"
Nov 02 12:53:21.012607 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:21.012585    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03b6b83c-10cd-4876-b17a-96120c2317b7" containerName="istio-proxy"
Nov 02 12:53:21.012724 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:21.012675    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="03b6b83c-10cd-4876-b17a-96120c2317b7" containerName="istio-proxy"
Nov 02 12:53:21.012724 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:21.012685    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="03b6b83c-10cd-4876-b17a-96120c2317b7" containerName="istio-proxy"
Nov 02 12:53:21.017439 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:21.017416    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c0d4cb-867cbccf7f-pmph4"
Nov 02 12:53:21.033803 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:21.033782    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-c0d4cb-dockercfg-zjhfz\""
Nov 02 12:53:21.033803 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:21.033783    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-c0d4cb\""
Nov 02 12:53:21.037766 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:21.037740    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-c0d4cb-867cbccf7f-pmph4"]
Nov 02 12:53:21.063903 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:21.063867    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8hrp\" (UniqueName: \"kubernetes.io/projected/97e9abd0-ac59-4b5a-88cf-174dbf24c305-kube-api-access-h8hrp\") pod \"payload-pre-processing-e2e-praxis-a-c0d4cb-867cbccf7f-pmph4\" (UID: \"97e9abd0-ac59-4b5a-88cf-174dbf24c305\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c0d4cb-867cbccf7f-pmph4"
Nov 02 12:53:21.064032 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:21.064010    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97e9abd0-ac59-4b5a-88cf-174dbf24c305-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-c0d4cb-867cbccf7f-pmph4\" (UID: \"97e9abd0-ac59-4b5a-88cf-174dbf24c305\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c0d4cb-867cbccf7f-pmph4"
Nov 02 12:53:21.097401 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:21.097372    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-c0d4cb-98c64fd4c-qs6bn"]
Nov 02 12:53:21.101578 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:21.101562    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-c0d4cb-98c64fd4c-qs6bn"
Nov 02 12:53:21.166685 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:21.166620    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8hrp\" (UniqueName: \"kubernetes.io/projected/97e9abd0-ac59-4b5a-88cf-174dbf24c305-kube-api-access-h8hrp\") pod \"payload-pre-processing-e2e-praxis-a-c0d4cb-867cbccf7f-pmph4\" (UID: \"97e9abd0-ac59-4b5a-88cf-174dbf24c305\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c0d4cb-867cbccf7f-pmph4"
Nov 02 12:53:21.167394 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:21.167363    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd77be34-4903-4c63-82d2-1b8951d975de-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-c0d4cb-98c64fd4c-qs6bn\" (UID: \"bd77be34-4903-4c63-82d2-1b8951d975de\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-c0d4cb-98c64fd4c-qs6bn"
Nov 02 12:53:21.169003 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:21.167698    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97e9abd0-ac59-4b5a-88cf-174dbf24c305-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-c0d4cb-867cbccf7f-pmph4\" (UID: \"97e9abd0-ac59-4b5a-88cf-174dbf24c305\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c0d4cb-867cbccf7f-pmph4"
Nov 02 12:53:21.181797 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:21.170148    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqhx9\" (UniqueName: \"kubernetes.io/projected/bd77be34-4903-4c63-82d2-1b8951d975de-kube-api-access-qqhx9\") pod \"payload-processing-e2e-praxis-a-c0d4cb-98c64fd4c-qs6bn\" (UID: \"bd77be34-4903-4c63-82d2-1b8951d975de\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-c0d4cb-98c64fd4c-qs6bn"
Nov 02 12:53:21.196333 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:21.170022    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97e9abd0-ac59-4b5a-88cf-174dbf24c305-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-c0d4cb-867cbccf7f-pmph4\" (UID: \"97e9abd0-ac59-4b5a-88cf-174dbf24c305\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c0d4cb-867cbccf7f-pmph4"
Nov 02 12:53:21.196565 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:21.184608    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-c0d4cb-98c64fd4c-qs6bn"]
Nov 02 12:53:21.292981 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:21.292865    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8hrp\" (UniqueName: \"kubernetes.io/projected/97e9abd0-ac59-4b5a-88cf-174dbf24c305-kube-api-access-h8hrp\") pod \"payload-pre-processing-e2e-praxis-a-c0d4cb-867cbccf7f-pmph4\" (UID: \"97e9abd0-ac59-4b5a-88cf-174dbf24c305\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c0d4cb-867cbccf7f-pmph4"
Nov 02 12:53:21.310394 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:21.307861    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd77be34-4903-4c63-82d2-1b8951d975de-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-c0d4cb-98c64fd4c-qs6bn\" (UID: \"bd77be34-4903-4c63-82d2-1b8951d975de\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-c0d4cb-98c64fd4c-qs6bn"
Nov 02 12:53:21.310824 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:21.310795    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqhx9\" (UniqueName: \"kubernetes.io/projected/bd77be34-4903-4c63-82d2-1b8951d975de-kube-api-access-qqhx9\") pod \"payload-processing-e2e-praxis-a-c0d4cb-98c64fd4c-qs6bn\" (UID: \"bd77be34-4903-4c63-82d2-1b8951d975de\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-c0d4cb-98c64fd4c-qs6bn"
Nov 02 12:53:21.312460 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:21.312420    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd77be34-4903-4c63-82d2-1b8951d975de-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-c0d4cb-98c64fd4c-qs6bn\" (UID: \"bd77be34-4903-4c63-82d2-1b8951d975de\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-c0d4cb-98c64fd4c-qs6bn"
Nov 02 12:53:21.331526 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:21.331492    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c0d4cb-867cbccf7f-pmph4"
Nov 02 12:53:21.343784 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:21.343583    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqhx9\" (UniqueName: \"kubernetes.io/projected/bd77be34-4903-4c63-82d2-1b8951d975de-kube-api-access-qqhx9\") pod \"payload-processing-e2e-praxis-a-c0d4cb-98c64fd4c-qs6bn\" (UID: \"bd77be34-4903-4c63-82d2-1b8951d975de\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-c0d4cb-98c64fd4c-qs6bn"
Nov 02 12:53:21.417138 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:21.416829    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-c0d4cb-98c64fd4c-qs6bn"
Nov 02 12:53:21.643700 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:53:21.643667    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97e9abd0_ac59_4b5a_88cf_174dbf24c305.slice/crio-7cedc7e2c23066a56491c6aaf9023984fa22a602e7516ec9a1c2fa1d56e9da3f WatchSource:0}: Error finding container 7cedc7e2c23066a56491c6aaf9023984fa22a602e7516ec9a1c2fa1d56e9da3f: Status 404 returned error can't find the container with id 7cedc7e2c23066a56491c6aaf9023984fa22a602e7516ec9a1c2fa1d56e9da3f
Nov 02 12:53:21.644519 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:21.644493    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-c0d4cb-867cbccf7f-pmph4"]
Nov 02 12:53:21.737522 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:53:21.737490    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd77be34_4903_4c63_82d2_1b8951d975de.slice/crio-1627f0fe3169f288164526923535aced0bf463523961729bd95886de47154a53 WatchSource:0}: Error finding container 1627f0fe3169f288164526923535aced0bf463523961729bd95886de47154a53: Status 404 returned error can't find the container with id 1627f0fe3169f288164526923535aced0bf463523961729bd95886de47154a53
Nov 02 12:53:21.739185 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:21.739154    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-c0d4cb-98c64fd4c-qs6bn"]
Nov 02 12:53:21.896082 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:21.895986    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03b6b83c-10cd-4876-b17a-96120c2317b7" path="/var/lib/kubelet/pods/03b6b83c-10cd-4876-b17a-96120c2317b7/volumes"
Nov 02 12:53:22.430101 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:22.430069    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-c0d4cb-98c64fd4c-qs6bn" event={"ID":"bd77be34-4903-4c63-82d2-1b8951d975de","Type":"ContainerStarted","Data":"2f125c060abb91d6014b62c80b724ff0148325bc98e2f042e0dfd25a95f74524"}
Nov 02 12:53:22.430294 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:22.430121    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-c0d4cb-98c64fd4c-qs6bn" event={"ID":"bd77be34-4903-4c63-82d2-1b8951d975de","Type":"ContainerStarted","Data":"1627f0fe3169f288164526923535aced0bf463523961729bd95886de47154a53"}
Nov 02 12:53:22.430294 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:22.430284    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-c0d4cb-98c64fd4c-qs6bn"
Nov 02 12:53:22.431867 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:22.431843    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c0d4cb-867cbccf7f-pmph4" event={"ID":"97e9abd0-ac59-4b5a-88cf-174dbf24c305","Type":"ContainerStarted","Data":"f755ee04d01c7f3d58b850f2df7e42492d23991c912151676e95a1c41a82da52"}
Nov 02 12:53:22.431981 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:22.431879    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c0d4cb-867cbccf7f-pmph4" event={"ID":"97e9abd0-ac59-4b5a-88cf-174dbf24c305","Type":"ContainerStarted","Data":"7cedc7e2c23066a56491c6aaf9023984fa22a602e7516ec9a1c2fa1d56e9da3f"}
Nov 02 12:53:22.431981 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:22.431914    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c0d4cb-867cbccf7f-pmph4"
Nov 02 12:53:22.472498 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:22.472441    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-c0d4cb-98c64fd4c-qs6bn" podStartSLOduration=1.472427158 podStartE2EDuration="1.472427158s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:53:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:53:22.47230501 +0000 UTC m=+1598.727704105" watchObservedRunningTime="2026-11-02 12:53:22.472427158 +0000 UTC m=+1598.727826253"
Nov 02 12:53:22.516036 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:22.515986    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c0d4cb-867cbccf7f-pmph4" podStartSLOduration=2.5159719750000002 podStartE2EDuration="2.515971975s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:53:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:53:22.514206762 +0000 UTC m=+1598.769605858" watchObservedRunningTime="2026-11-02 12:53:22.515971975 +0000 UTC m=+1598.771371069"
Nov 02 12:53:23.400060 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:23.400026    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-a6f2c0-openshift-default-7f47858fbf-x45sj"]
Nov 02 12:53:23.405480 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:23.405456    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-a6f2c0-openshift-default-7f47858fbf-x45sj"
Nov 02 12:53:23.421958 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:23.421934    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-a6f2c0-openshift-default-dockercfg-7djh5\""
Nov 02 12:53:23.443579 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:23.443547    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-a6f2c0-openshift-default-7f47858fbf-x45sj"]
Nov 02 12:53:23.454814 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:23.454785    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27dd21d8-102d-45c3-8e54-3c1e846829de-workload-certs\") pod \"e2e-shared-a-w1-a6f2c0-openshift-default-7f47858fbf-x45sj\" (UID: \"27dd21d8-102d-45c3-8e54-3c1e846829de\") " pod="openshift-ingress/e2e-shared-a-w1-a6f2c0-openshift-default-7f47858fbf-x45sj"
Nov 02 12:53:23.455003 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:23.454834    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27dd21d8-102d-45c3-8e54-3c1e846829de-istiod-ca-cert\") pod \"e2e-shared-a-w1-a6f2c0-openshift-default-7f47858fbf-x45sj\" (UID: \"27dd21d8-102d-45c3-8e54-3c1e846829de\") " pod="openshift-ingress/e2e-shared-a-w1-a6f2c0-openshift-default-7f47858fbf-x45sj"
Nov 02 12:53:23.455003 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:23.454972    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27dd21d8-102d-45c3-8e54-3c1e846829de-istio-data\") pod \"e2e-shared-a-w1-a6f2c0-openshift-default-7f47858fbf-x45sj\" (UID: \"27dd21d8-102d-45c3-8e54-3c1e846829de\") " pod="openshift-ingress/e2e-shared-a-w1-a6f2c0-openshift-default-7f47858fbf-x45sj"
Nov 02 12:53:23.455227 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:23.455136    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27dd21d8-102d-45c3-8e54-3c1e846829de-credential-socket\") pod \"e2e-shared-a-w1-a6f2c0-openshift-default-7f47858fbf-x45sj\" (UID: \"27dd21d8-102d-45c3-8e54-3c1e846829de\") " pod="openshift-ingress/e2e-shared-a-w1-a6f2c0-openshift-default-7f47858fbf-x45sj"
Nov 02 12:53:23.455227 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:23.455164    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffhrw\" (UniqueName: \"kubernetes.io/projected/27dd21d8-102d-45c3-8e54-3c1e846829de-kube-api-access-ffhrw\") pod \"e2e-shared-a-w1-a6f2c0-openshift-default-7f47858fbf-x45sj\" (UID: \"27dd21d8-102d-45c3-8e54-3c1e846829de\") " pod="openshift-ingress/e2e-shared-a-w1-a6f2c0-openshift-default-7f47858fbf-x45sj"
Nov 02 12:53:23.455355 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:23.455274    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27dd21d8-102d-45c3-8e54-3c1e846829de-istio-podinfo\") pod \"e2e-shared-a-w1-a6f2c0-openshift-default-7f47858fbf-x45sj\" (UID: \"27dd21d8-102d-45c3-8e54-3c1e846829de\") " pod="openshift-ingress/e2e-shared-a-w1-a6f2c0-openshift-default-7f47858fbf-x45sj"
Nov 02 12:53:23.455355 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:23.455326    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27dd21d8-102d-45c3-8e54-3c1e846829de-workload-socket\") pod \"e2e-shared-a-w1-a6f2c0-openshift-default-7f47858fbf-x45sj\" (UID: \"27dd21d8-102d-45c3-8e54-3c1e846829de\") " pod="openshift-ingress/e2e-shared-a-w1-a6f2c0-openshift-default-7f47858fbf-x45sj"
Nov 02 12:53:23.455481 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:23.455354    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27dd21d8-102d-45c3-8e54-3c1e846829de-istio-envoy\") pod \"e2e-shared-a-w1-a6f2c0-openshift-default-7f47858fbf-x45sj\" (UID: \"27dd21d8-102d-45c3-8e54-3c1e846829de\") " pod="openshift-ingress/e2e-shared-a-w1-a6f2c0-openshift-default-7f47858fbf-x45sj"
Nov 02 12:53:23.455481 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:23.455389    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27dd21d8-102d-45c3-8e54-3c1e846829de-istio-token\") pod \"e2e-shared-a-w1-a6f2c0-openshift-default-7f47858fbf-x45sj\" (UID: \"27dd21d8-102d-45c3-8e54-3c1e846829de\") " pod="openshift-ingress/e2e-shared-a-w1-a6f2c0-openshift-default-7f47858fbf-x45sj"
Nov 02 12:53:23.561608 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:23.561558    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27dd21d8-102d-45c3-8e54-3c1e846829de-istio-podinfo\") pod \"e2e-shared-a-w1-a6f2c0-openshift-default-7f47858fbf-x45sj\" (UID: \"27dd21d8-102d-45c3-8e54-3c1e846829de\") " pod="openshift-ingress/e2e-shared-a-w1-a6f2c0-openshift-default-7f47858fbf-x45sj"
Nov 02 12:53:23.561933 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:23.561899    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27dd21d8-102d-45c3-8e54-3c1e846829de-workload-socket\") pod \"e2e-shared-a-w1-a6f2c0-openshift-default-7f47858fbf-x45sj\" (UID: \"27dd21d8-102d-45c3-8e54-3c1e846829de\") " pod="openshift-ingress/e2e-shared-a-w1-a6f2c0-openshift-default-7f47858fbf-x45sj"
Nov 02 12:53:23.562093 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:23.562075    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27dd21d8-102d-45c3-8e54-3c1e846829de-istio-envoy\") pod \"e2e-shared-a-w1-a6f2c0-openshift-default-7f47858fbf-x45sj\" (UID: \"27dd21d8-102d-45c3-8e54-3c1e846829de\") " pod="openshift-ingress/e2e-shared-a-w1-a6f2c0-openshift-default-7f47858fbf-x45sj"
Nov 02 12:53:23.562276 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:23.562248    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27dd21d8-102d-45c3-8e54-3c1e846829de-istio-token\") pod \"e2e-shared-a-w1-a6f2c0-openshift-default-7f47858fbf-x45sj\" (UID: \"27dd21d8-102d-45c3-8e54-3c1e846829de\") " pod="openshift-ingress/e2e-shared-a-w1-a6f2c0-openshift-default-7f47858fbf-x45sj"
Nov 02 12:53:23.562443 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:23.562409    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27dd21d8-102d-45c3-8e54-3c1e846829de-workload-certs\") pod \"e2e-shared-a-w1-a6f2c0-openshift-default-7f47858fbf-x45sj\" (UID: \"27dd21d8-102d-45c3-8e54-3c1e846829de\") " pod="openshift-ingress/e2e-shared-a-w1-a6f2c0-openshift-default-7f47858fbf-x45sj"
Nov 02 12:53:23.562694 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:23.562629    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27dd21d8-102d-45c3-8e54-3c1e846829de-istiod-ca-cert\") pod \"e2e-shared-a-w1-a6f2c0-openshift-default-7f47858fbf-x45sj\" (UID: \"27dd21d8-102d-45c3-8e54-3c1e846829de\") " pod="openshift-ingress/e2e-shared-a-w1-a6f2c0-openshift-default-7f47858fbf-x45sj"
Nov 02 12:53:23.562902 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:23.562884    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27dd21d8-102d-45c3-8e54-3c1e846829de-istio-data\") pod \"e2e-shared-a-w1-a6f2c0-openshift-default-7f47858fbf-x45sj\" (UID: \"27dd21d8-102d-45c3-8e54-3c1e846829de\") " pod="openshift-ingress/e2e-shared-a-w1-a6f2c0-openshift-default-7f47858fbf-x45sj"
Nov 02 12:53:23.563258 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:23.563236    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27dd21d8-102d-45c3-8e54-3c1e846829de-credential-socket\") pod \"e2e-shared-a-w1-a6f2c0-openshift-default-7f47858fbf-x45sj\" (UID: \"27dd21d8-102d-45c3-8e54-3c1e846829de\") " pod="openshift-ingress/e2e-shared-a-w1-a6f2c0-openshift-default-7f47858fbf-x45sj"
Nov 02 12:53:23.563411 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:23.563387    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffhrw\" (UniqueName: \"kubernetes.io/projected/27dd21d8-102d-45c3-8e54-3c1e846829de-kube-api-access-ffhrw\") pod \"e2e-shared-a-w1-a6f2c0-openshift-default-7f47858fbf-x45sj\" (UID: \"27dd21d8-102d-45c3-8e54-3c1e846829de\") " pod="openshift-ingress/e2e-shared-a-w1-a6f2c0-openshift-default-7f47858fbf-x45sj"
Nov 02 12:53:23.577413 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:23.577381    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27dd21d8-102d-45c3-8e54-3c1e846829de-istiod-ca-cert\") pod \"e2e-shared-a-w1-a6f2c0-openshift-default-7f47858fbf-x45sj\" (UID: \"27dd21d8-102d-45c3-8e54-3c1e846829de\") " pod="openshift-ingress/e2e-shared-a-w1-a6f2c0-openshift-default-7f47858fbf-x45sj"
Nov 02 12:53:23.581662 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:23.580513    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27dd21d8-102d-45c3-8e54-3c1e846829de-credential-socket\") pod \"e2e-shared-a-w1-a6f2c0-openshift-default-7f47858fbf-x45sj\" (UID: \"27dd21d8-102d-45c3-8e54-3c1e846829de\") " pod="openshift-ingress/e2e-shared-a-w1-a6f2c0-openshift-default-7f47858fbf-x45sj"
Nov 02 12:53:23.581662 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:23.580812    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27dd21d8-102d-45c3-8e54-3c1e846829de-workload-certs\") pod \"e2e-shared-a-w1-a6f2c0-openshift-default-7f47858fbf-x45sj\" (UID: \"27dd21d8-102d-45c3-8e54-3c1e846829de\") " pod="openshift-ingress/e2e-shared-a-w1-a6f2c0-openshift-default-7f47858fbf-x45sj"
Nov 02 12:53:23.581662 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:23.581068    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27dd21d8-102d-45c3-8e54-3c1e846829de-workload-socket\") pod \"e2e-shared-a-w1-a6f2c0-openshift-default-7f47858fbf-x45sj\" (UID: \"27dd21d8-102d-45c3-8e54-3c1e846829de\") " pod="openshift-ingress/e2e-shared-a-w1-a6f2c0-openshift-default-7f47858fbf-x45sj"
Nov 02 12:53:23.586764 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:23.586738    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27dd21d8-102d-45c3-8e54-3c1e846829de-istio-data\") pod \"e2e-shared-a-w1-a6f2c0-openshift-default-7f47858fbf-x45sj\" (UID: \"27dd21d8-102d-45c3-8e54-3c1e846829de\") " pod="openshift-ingress/e2e-shared-a-w1-a6f2c0-openshift-default-7f47858fbf-x45sj"
Nov 02 12:53:23.591720 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:23.587941    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27dd21d8-102d-45c3-8e54-3c1e846829de-istio-podinfo\") pod \"e2e-shared-a-w1-a6f2c0-openshift-default-7f47858fbf-x45sj\" (UID: \"27dd21d8-102d-45c3-8e54-3c1e846829de\") " pod="openshift-ingress/e2e-shared-a-w1-a6f2c0-openshift-default-7f47858fbf-x45sj"
Nov 02 12:53:23.609255 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:23.609228    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27dd21d8-102d-45c3-8e54-3c1e846829de-istio-envoy\") pod \"e2e-shared-a-w1-a6f2c0-openshift-default-7f47858fbf-x45sj\" (UID: \"27dd21d8-102d-45c3-8e54-3c1e846829de\") " pod="openshift-ingress/e2e-shared-a-w1-a6f2c0-openshift-default-7f47858fbf-x45sj"
Nov 02 12:53:23.621518 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:23.621455    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27dd21d8-102d-45c3-8e54-3c1e846829de-istio-token\") pod \"e2e-shared-a-w1-a6f2c0-openshift-default-7f47858fbf-x45sj\" (UID: \"27dd21d8-102d-45c3-8e54-3c1e846829de\") " pod="openshift-ingress/e2e-shared-a-w1-a6f2c0-openshift-default-7f47858fbf-x45sj"
Nov 02 12:53:23.642455 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:23.642421    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffhrw\" (UniqueName: \"kubernetes.io/projected/27dd21d8-102d-45c3-8e54-3c1e846829de-kube-api-access-ffhrw\") pod \"e2e-shared-a-w1-a6f2c0-openshift-default-7f47858fbf-x45sj\" (UID: \"27dd21d8-102d-45c3-8e54-3c1e846829de\") " pod="openshift-ingress/e2e-shared-a-w1-a6f2c0-openshift-default-7f47858fbf-x45sj"
Nov 02 12:53:23.723426 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:23.723396    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-a6f2c0-openshift-default-7f47858fbf-x45sj"
Nov 02 12:53:23.959737 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:23.958159    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-ed56b9-openshift-default-789859d697-5f5lv"]
Nov 02 12:53:23.989262 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:23.988086    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-ed56b9-openshift-default-789859d697-5f5lv"]
Nov 02 12:53:23.989262 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:23.988560    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-ed56b9-openshift-default-789859d697-5f5lv"
Nov 02 12:53:24.007821 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:24.005244    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-ed56b9-openshift-default-dockercfg-ndgfq\""
Nov 02 12:53:24.137868 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:24.136423    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abd3e462-db3a-4c9a-9fbc-1b5cb3550ef9-workload-socket\") pod \"e2e-shared-a-w2-ed56b9-openshift-default-789859d697-5f5lv\" (UID: \"abd3e462-db3a-4c9a-9fbc-1b5cb3550ef9\") " pod="openshift-ingress/e2e-shared-a-w2-ed56b9-openshift-default-789859d697-5f5lv"
Nov 02 12:53:24.137868 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:24.136467    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abd3e462-db3a-4c9a-9fbc-1b5cb3550ef9-istio-token\") pod \"e2e-shared-a-w2-ed56b9-openshift-default-789859d697-5f5lv\" (UID: \"abd3e462-db3a-4c9a-9fbc-1b5cb3550ef9\") " pod="openshift-ingress/e2e-shared-a-w2-ed56b9-openshift-default-789859d697-5f5lv"
Nov 02 12:53:24.137868 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:24.136676    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abd3e462-db3a-4c9a-9fbc-1b5cb3550ef9-istio-data\") pod \"e2e-shared-a-w2-ed56b9-openshift-default-789859d697-5f5lv\" (UID: \"abd3e462-db3a-4c9a-9fbc-1b5cb3550ef9\") " pod="openshift-ingress/e2e-shared-a-w2-ed56b9-openshift-default-789859d697-5f5lv"
Nov 02 12:53:24.137868 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:24.136718    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hzsq\" (UniqueName: \"kubernetes.io/projected/abd3e462-db3a-4c9a-9fbc-1b5cb3550ef9-kube-api-access-9hzsq\") pod \"e2e-shared-a-w2-ed56b9-openshift-default-789859d697-5f5lv\" (UID: \"abd3e462-db3a-4c9a-9fbc-1b5cb3550ef9\") " pod="openshift-ingress/e2e-shared-a-w2-ed56b9-openshift-default-789859d697-5f5lv"
Nov 02 12:53:24.137868 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:24.136764    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abd3e462-db3a-4c9a-9fbc-1b5cb3550ef9-credential-socket\") pod \"e2e-shared-a-w2-ed56b9-openshift-default-789859d697-5f5lv\" (UID: \"abd3e462-db3a-4c9a-9fbc-1b5cb3550ef9\") " pod="openshift-ingress/e2e-shared-a-w2-ed56b9-openshift-default-789859d697-5f5lv"
Nov 02 12:53:24.137868 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:24.136819    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abd3e462-db3a-4c9a-9fbc-1b5cb3550ef9-workload-certs\") pod \"e2e-shared-a-w2-ed56b9-openshift-default-789859d697-5f5lv\" (UID: \"abd3e462-db3a-4c9a-9fbc-1b5cb3550ef9\") " pod="openshift-ingress/e2e-shared-a-w2-ed56b9-openshift-default-789859d697-5f5lv"
Nov 02 12:53:24.137868 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:24.136863    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abd3e462-db3a-4c9a-9fbc-1b5cb3550ef9-istiod-ca-cert\") pod \"e2e-shared-a-w2-ed56b9-openshift-default-789859d697-5f5lv\" (UID: \"abd3e462-db3a-4c9a-9fbc-1b5cb3550ef9\") " pod="openshift-ingress/e2e-shared-a-w2-ed56b9-openshift-default-789859d697-5f5lv"
Nov 02 12:53:24.137868 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:24.136945    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abd3e462-db3a-4c9a-9fbc-1b5cb3550ef9-istio-envoy\") pod \"e2e-shared-a-w2-ed56b9-openshift-default-789859d697-5f5lv\" (UID: \"abd3e462-db3a-4c9a-9fbc-1b5cb3550ef9\") " pod="openshift-ingress/e2e-shared-a-w2-ed56b9-openshift-default-789859d697-5f5lv"
Nov 02 12:53:24.137868 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:24.137105    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abd3e462-db3a-4c9a-9fbc-1b5cb3550ef9-istio-podinfo\") pod \"e2e-shared-a-w2-ed56b9-openshift-default-789859d697-5f5lv\" (UID: \"abd3e462-db3a-4c9a-9fbc-1b5cb3550ef9\") " pod="openshift-ingress/e2e-shared-a-w2-ed56b9-openshift-default-789859d697-5f5lv"
Nov 02 12:53:24.193420 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:24.193389    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-a6f2c0-openshift-default-7f47858fbf-x45sj"]
Nov 02 12:53:24.193666 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:53:24.193596    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27dd21d8_102d_45c3_8e54_3c1e846829de.slice/crio-c1b558c51d1777b1ca557894c40688c6d9ec421be9f21dfb05111c5794390baf WatchSource:0}: Error finding container c1b558c51d1777b1ca557894c40688c6d9ec421be9f21dfb05111c5794390baf: Status 404 returned error can't find the container with id c1b558c51d1777b1ca557894c40688c6d9ec421be9f21dfb05111c5794390baf
Nov 02 12:53:24.196494 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:24.196457    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:53:24.196611 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:24.196538    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:53:24.196611 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:24.196583    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:53:24.238210 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:24.238149    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hzsq\" (UniqueName: \"kubernetes.io/projected/abd3e462-db3a-4c9a-9fbc-1b5cb3550ef9-kube-api-access-9hzsq\") pod \"e2e-shared-a-w2-ed56b9-openshift-default-789859d697-5f5lv\" (UID: \"abd3e462-db3a-4c9a-9fbc-1b5cb3550ef9\") " pod="openshift-ingress/e2e-shared-a-w2-ed56b9-openshift-default-789859d697-5f5lv"
Nov 02 12:53:24.238333 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:24.238207    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abd3e462-db3a-4c9a-9fbc-1b5cb3550ef9-credential-socket\") pod \"e2e-shared-a-w2-ed56b9-openshift-default-789859d697-5f5lv\" (UID: \"abd3e462-db3a-4c9a-9fbc-1b5cb3550ef9\") " pod="openshift-ingress/e2e-shared-a-w2-ed56b9-openshift-default-789859d697-5f5lv"
Nov 02 12:53:24.238333 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:24.238245    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abd3e462-db3a-4c9a-9fbc-1b5cb3550ef9-workload-certs\") pod \"e2e-shared-a-w2-ed56b9-openshift-default-789859d697-5f5lv\" (UID: \"abd3e462-db3a-4c9a-9fbc-1b5cb3550ef9\") " pod="openshift-ingress/e2e-shared-a-w2-ed56b9-openshift-default-789859d697-5f5lv"
Nov 02 12:53:24.238333 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:24.238283    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abd3e462-db3a-4c9a-9fbc-1b5cb3550ef9-istiod-ca-cert\") pod \"e2e-shared-a-w2-ed56b9-openshift-default-789859d697-5f5lv\" (UID: \"abd3e462-db3a-4c9a-9fbc-1b5cb3550ef9\") " pod="openshift-ingress/e2e-shared-a-w2-ed56b9-openshift-default-789859d697-5f5lv"
Nov 02 12:53:24.238507 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:24.238348    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abd3e462-db3a-4c9a-9fbc-1b5cb3550ef9-istio-envoy\") pod \"e2e-shared-a-w2-ed56b9-openshift-default-789859d697-5f5lv\" (UID: \"abd3e462-db3a-4c9a-9fbc-1b5cb3550ef9\") " pod="openshift-ingress/e2e-shared-a-w2-ed56b9-openshift-default-789859d697-5f5lv"
Nov 02 12:53:24.238507 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:24.238476    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abd3e462-db3a-4c9a-9fbc-1b5cb3550ef9-istio-podinfo\") pod \"e2e-shared-a-w2-ed56b9-openshift-default-789859d697-5f5lv\" (UID: \"abd3e462-db3a-4c9a-9fbc-1b5cb3550ef9\") " pod="openshift-ingress/e2e-shared-a-w2-ed56b9-openshift-default-789859d697-5f5lv"
Nov 02 12:53:24.238606 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:24.238546    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abd3e462-db3a-4c9a-9fbc-1b5cb3550ef9-workload-socket\") pod \"e2e-shared-a-w2-ed56b9-openshift-default-789859d697-5f5lv\" (UID: \"abd3e462-db3a-4c9a-9fbc-1b5cb3550ef9\") " pod="openshift-ingress/e2e-shared-a-w2-ed56b9-openshift-default-789859d697-5f5lv"
Nov 02 12:53:24.238606 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:24.238573    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abd3e462-db3a-4c9a-9fbc-1b5cb3550ef9-istio-token\") pod \"e2e-shared-a-w2-ed56b9-openshift-default-789859d697-5f5lv\" (UID: \"abd3e462-db3a-4c9a-9fbc-1b5cb3550ef9\") " pod="openshift-ingress/e2e-shared-a-w2-ed56b9-openshift-default-789859d697-5f5lv"
Nov 02 12:53:24.238800 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:24.238726    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abd3e462-db3a-4c9a-9fbc-1b5cb3550ef9-istio-data\") pod \"e2e-shared-a-w2-ed56b9-openshift-default-789859d697-5f5lv\" (UID: \"abd3e462-db3a-4c9a-9fbc-1b5cb3550ef9\") " pod="openshift-ingress/e2e-shared-a-w2-ed56b9-openshift-default-789859d697-5f5lv"
Nov 02 12:53:24.239155 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:24.239131    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abd3e462-db3a-4c9a-9fbc-1b5cb3550ef9-istiod-ca-cert\") pod \"e2e-shared-a-w2-ed56b9-openshift-default-789859d697-5f5lv\" (UID: \"abd3e462-db3a-4c9a-9fbc-1b5cb3550ef9\") " pod="openshift-ingress/e2e-shared-a-w2-ed56b9-openshift-default-789859d697-5f5lv"
Nov 02 12:53:24.239400 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:24.239378    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abd3e462-db3a-4c9a-9fbc-1b5cb3550ef9-workload-certs\") pod \"e2e-shared-a-w2-ed56b9-openshift-default-789859d697-5f5lv\" (UID: \"abd3e462-db3a-4c9a-9fbc-1b5cb3550ef9\") " pod="openshift-ingress/e2e-shared-a-w2-ed56b9-openshift-default-789859d697-5f5lv"
Nov 02 12:53:24.240199 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:24.240171    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abd3e462-db3a-4c9a-9fbc-1b5cb3550ef9-credential-socket\") pod \"e2e-shared-a-w2-ed56b9-openshift-default-789859d697-5f5lv\" (UID: \"abd3e462-db3a-4c9a-9fbc-1b5cb3550ef9\") " pod="openshift-ingress/e2e-shared-a-w2-ed56b9-openshift-default-789859d697-5f5lv"
Nov 02 12:53:24.240301 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:24.240196    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abd3e462-db3a-4c9a-9fbc-1b5cb3550ef9-workload-socket\") pod \"e2e-shared-a-w2-ed56b9-openshift-default-789859d697-5f5lv\" (UID: \"abd3e462-db3a-4c9a-9fbc-1b5cb3550ef9\") " pod="openshift-ingress/e2e-shared-a-w2-ed56b9-openshift-default-789859d697-5f5lv"
Nov 02 12:53:24.240301 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:24.240285    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abd3e462-db3a-4c9a-9fbc-1b5cb3550ef9-istio-data\") pod \"e2e-shared-a-w2-ed56b9-openshift-default-789859d697-5f5lv\" (UID: \"abd3e462-db3a-4c9a-9fbc-1b5cb3550ef9\") " pod="openshift-ingress/e2e-shared-a-w2-ed56b9-openshift-default-789859d697-5f5lv"
Nov 02 12:53:24.241279 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:24.241258    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abd3e462-db3a-4c9a-9fbc-1b5cb3550ef9-istio-podinfo\") pod \"e2e-shared-a-w2-ed56b9-openshift-default-789859d697-5f5lv\" (UID: \"abd3e462-db3a-4c9a-9fbc-1b5cb3550ef9\") " pod="openshift-ingress/e2e-shared-a-w2-ed56b9-openshift-default-789859d697-5f5lv"
Nov 02 12:53:24.241914 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:24.241899    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abd3e462-db3a-4c9a-9fbc-1b5cb3550ef9-istio-envoy\") pod \"e2e-shared-a-w2-ed56b9-openshift-default-789859d697-5f5lv\" (UID: \"abd3e462-db3a-4c9a-9fbc-1b5cb3550ef9\") " pod="openshift-ingress/e2e-shared-a-w2-ed56b9-openshift-default-789859d697-5f5lv"
Nov 02 12:53:24.292508 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:24.292465    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abd3e462-db3a-4c9a-9fbc-1b5cb3550ef9-istio-token\") pod \"e2e-shared-a-w2-ed56b9-openshift-default-789859d697-5f5lv\" (UID: \"abd3e462-db3a-4c9a-9fbc-1b5cb3550ef9\") " pod="openshift-ingress/e2e-shared-a-w2-ed56b9-openshift-default-789859d697-5f5lv"
Nov 02 12:53:24.307031 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:24.306994    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hzsq\" (UniqueName: \"kubernetes.io/projected/abd3e462-db3a-4c9a-9fbc-1b5cb3550ef9-kube-api-access-9hzsq\") pod \"e2e-shared-a-w2-ed56b9-openshift-default-789859d697-5f5lv\" (UID: \"abd3e462-db3a-4c9a-9fbc-1b5cb3550ef9\") " pod="openshift-ingress/e2e-shared-a-w2-ed56b9-openshift-default-789859d697-5f5lv"
Nov 02 12:53:24.314512 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:24.314279    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-ed56b9-openshift-default-789859d697-5f5lv"
Nov 02 12:53:24.463963 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:24.463925    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-a6f2c0-openshift-default-7f47858fbf-x45sj" event={"ID":"27dd21d8-102d-45c3-8e54-3c1e846829de","Type":"ContainerStarted","Data":"214bc758f94617b4bb60976c80c20ad95dc5e3249685f127b36f6acc044ddbbf"}
Nov 02 12:53:24.464338 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:24.463988    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-a6f2c0-openshift-default-7f47858fbf-x45sj" event={"ID":"27dd21d8-102d-45c3-8e54-3c1e846829de","Type":"ContainerStarted","Data":"c1b558c51d1777b1ca557894c40688c6d9ec421be9f21dfb05111c5794390baf"}
Nov 02 12:53:24.575308 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:24.571424    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-a6f2c0-openshift-default-7f47858fbf-x45sj" podStartSLOduration=1.5714031830000001 podStartE2EDuration="1.571403183s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:53:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:53:24.569379954 +0000 UTC m=+1600.824779050" watchObservedRunningTime="2026-11-02 12:53:24.571403183 +0000 UTC m=+1600.826802279"
Nov 02 12:53:24.687867 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:24.687834    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-ed56b9-openshift-default-789859d697-5f5lv"]
Nov 02 12:53:24.688297 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:53:24.688266    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabd3e462_db3a_4c9a_9fbc_1b5cb3550ef9.slice/crio-34dc62a4a4e26e0cbfe078f3f914f46263cd7d1c5a7ae7a9d0ff30779c47666d WatchSource:0}: Error finding container 34dc62a4a4e26e0cbfe078f3f914f46263cd7d1c5a7ae7a9d0ff30779c47666d: Status 404 returned error can't find the container with id 34dc62a4a4e26e0cbfe078f3f914f46263cd7d1c5a7ae7a9d0ff30779c47666d
Nov 02 12:53:24.691052 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:24.691012    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:53:24.691155 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:24.691101    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:53:24.691155 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:24.691147    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:53:24.745070 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:24.744961    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-a6f2c0-openshift-default-7f47858fbf-x45sj"
Nov 02 12:53:25.474165 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:25.474126    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-ed56b9-openshift-default-789859d697-5f5lv" event={"ID":"abd3e462-db3a-4c9a-9fbc-1b5cb3550ef9","Type":"ContainerStarted","Data":"087d74e1c8b7ed7ed4a40876cfb5f43cc55df856350c1acc8542fd8efadee767"}
Nov 02 12:53:25.474562 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:25.474188    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-ed56b9-openshift-default-789859d697-5f5lv" event={"ID":"abd3e462-db3a-4c9a-9fbc-1b5cb3550ef9","Type":"ContainerStarted","Data":"34dc62a4a4e26e0cbfe078f3f914f46263cd7d1c5a7ae7a9d0ff30779c47666d"}
Nov 02 12:53:25.566698 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:25.566615    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-ed56b9-openshift-default-789859d697-5f5lv" podStartSLOduration=2.566602128 podStartE2EDuration="2.566602128s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:53:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:53:25.563478376 +0000 UTC m=+1601.818877471" watchObservedRunningTime="2026-11-02 12:53:25.566602128 +0000 UTC m=+1601.822001222"
Nov 02 12:53:25.742434 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:25.742349    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-a6f2c0-openshift-default-7f47858fbf-x45sj"
Nov 02 12:53:26.049844 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:26.049761    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-e7561a-openshift-default-7598f89b79-m5rrm"]
Nov 02 12:53:26.055576 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:26.054911    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-e7561a-openshift-default-7598f89b79-m5rrm"
Nov 02 12:53:26.074800 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:26.074773    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-e7561a-openshift-default-dockercfg-87qsn\""
Nov 02 12:53:26.088062 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:26.088030    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-e7561a-openshift-default-7598f89b79-m5rrm"]
Nov 02 12:53:26.137594 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:26.137560    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ts9j\" (UniqueName: \"kubernetes.io/projected/4293ef7c-ce26-475f-98e1-ebf007c35f00-kube-api-access-5ts9j\") pod \"e2e-shared-a-w3-e7561a-openshift-default-7598f89b79-m5rrm\" (UID: \"4293ef7c-ce26-475f-98e1-ebf007c35f00\") " pod="openshift-ingress/e2e-shared-a-w3-e7561a-openshift-default-7598f89b79-m5rrm"
Nov 02 12:53:26.137820 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:26.137718    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4293ef7c-ce26-475f-98e1-ebf007c35f00-workload-socket\") pod \"e2e-shared-a-w3-e7561a-openshift-default-7598f89b79-m5rrm\" (UID: \"4293ef7c-ce26-475f-98e1-ebf007c35f00\") " pod="openshift-ingress/e2e-shared-a-w3-e7561a-openshift-default-7598f89b79-m5rrm"
Nov 02 12:53:26.137820 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:26.137780    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4293ef7c-ce26-475f-98e1-ebf007c35f00-istio-token\") pod \"e2e-shared-a-w3-e7561a-openshift-default-7598f89b79-m5rrm\" (UID: \"4293ef7c-ce26-475f-98e1-ebf007c35f00\") " pod="openshift-ingress/e2e-shared-a-w3-e7561a-openshift-default-7598f89b79-m5rrm"
Nov 02 12:53:26.137820 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:26.137802    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4293ef7c-ce26-475f-98e1-ebf007c35f00-istio-podinfo\") pod \"e2e-shared-a-w3-e7561a-openshift-default-7598f89b79-m5rrm\" (UID: \"4293ef7c-ce26-475f-98e1-ebf007c35f00\") " pod="openshift-ingress/e2e-shared-a-w3-e7561a-openshift-default-7598f89b79-m5rrm"
Nov 02 12:53:26.137963 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:26.137906    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4293ef7c-ce26-475f-98e1-ebf007c35f00-workload-certs\") pod \"e2e-shared-a-w3-e7561a-openshift-default-7598f89b79-m5rrm\" (UID: \"4293ef7c-ce26-475f-98e1-ebf007c35f00\") " pod="openshift-ingress/e2e-shared-a-w3-e7561a-openshift-default-7598f89b79-m5rrm"
Nov 02 12:53:26.138011 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:26.137971    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4293ef7c-ce26-475f-98e1-ebf007c35f00-istio-data\") pod \"e2e-shared-a-w3-e7561a-openshift-default-7598f89b79-m5rrm\" (UID: \"4293ef7c-ce26-475f-98e1-ebf007c35f00\") " pod="openshift-ingress/e2e-shared-a-w3-e7561a-openshift-default-7598f89b79-m5rrm"
Nov 02 12:53:26.138059 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:26.138045    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4293ef7c-ce26-475f-98e1-ebf007c35f00-credential-socket\") pod \"e2e-shared-a-w3-e7561a-openshift-default-7598f89b79-m5rrm\" (UID: \"4293ef7c-ce26-475f-98e1-ebf007c35f00\") " pod="openshift-ingress/e2e-shared-a-w3-e7561a-openshift-default-7598f89b79-m5rrm"
Nov 02 12:53:26.138096 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:26.138076    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4293ef7c-ce26-475f-98e1-ebf007c35f00-istio-envoy\") pod \"e2e-shared-a-w3-e7561a-openshift-default-7598f89b79-m5rrm\" (UID: \"4293ef7c-ce26-475f-98e1-ebf007c35f00\") " pod="openshift-ingress/e2e-shared-a-w3-e7561a-openshift-default-7598f89b79-m5rrm"
Nov 02 12:53:26.138150 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:26.138106    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4293ef7c-ce26-475f-98e1-ebf007c35f00-istiod-ca-cert\") pod \"e2e-shared-a-w3-e7561a-openshift-default-7598f89b79-m5rrm\" (UID: \"4293ef7c-ce26-475f-98e1-ebf007c35f00\") " pod="openshift-ingress/e2e-shared-a-w3-e7561a-openshift-default-7598f89b79-m5rrm"
Nov 02 12:53:26.242665 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:26.241663    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4293ef7c-ce26-475f-98e1-ebf007c35f00-credential-socket\") pod \"e2e-shared-a-w3-e7561a-openshift-default-7598f89b79-m5rrm\" (UID: \"4293ef7c-ce26-475f-98e1-ebf007c35f00\") " pod="openshift-ingress/e2e-shared-a-w3-e7561a-openshift-default-7598f89b79-m5rrm"
Nov 02 12:53:26.242665 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:26.241741    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4293ef7c-ce26-475f-98e1-ebf007c35f00-istio-envoy\") pod \"e2e-shared-a-w3-e7561a-openshift-default-7598f89b79-m5rrm\" (UID: \"4293ef7c-ce26-475f-98e1-ebf007c35f00\") " pod="openshift-ingress/e2e-shared-a-w3-e7561a-openshift-default-7598f89b79-m5rrm"
Nov 02 12:53:26.242665 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:26.241809    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4293ef7c-ce26-475f-98e1-ebf007c35f00-istiod-ca-cert\") pod \"e2e-shared-a-w3-e7561a-openshift-default-7598f89b79-m5rrm\" (UID: \"4293ef7c-ce26-475f-98e1-ebf007c35f00\") " pod="openshift-ingress/e2e-shared-a-w3-e7561a-openshift-default-7598f89b79-m5rrm"
Nov 02 12:53:26.242665 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:26.241908    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ts9j\" (UniqueName: \"kubernetes.io/projected/4293ef7c-ce26-475f-98e1-ebf007c35f00-kube-api-access-5ts9j\") pod \"e2e-shared-a-w3-e7561a-openshift-default-7598f89b79-m5rrm\" (UID: \"4293ef7c-ce26-475f-98e1-ebf007c35f00\") " pod="openshift-ingress/e2e-shared-a-w3-e7561a-openshift-default-7598f89b79-m5rrm"
Nov 02 12:53:26.242665 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:26.242012    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4293ef7c-ce26-475f-98e1-ebf007c35f00-workload-socket\") pod \"e2e-shared-a-w3-e7561a-openshift-default-7598f89b79-m5rrm\" (UID: \"4293ef7c-ce26-475f-98e1-ebf007c35f00\") " pod="openshift-ingress/e2e-shared-a-w3-e7561a-openshift-default-7598f89b79-m5rrm"
Nov 02 12:53:26.242665 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:26.242051    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4293ef7c-ce26-475f-98e1-ebf007c35f00-istio-token\") pod \"e2e-shared-a-w3-e7561a-openshift-default-7598f89b79-m5rrm\" (UID: \"4293ef7c-ce26-475f-98e1-ebf007c35f00\") " pod="openshift-ingress/e2e-shared-a-w3-e7561a-openshift-default-7598f89b79-m5rrm"
Nov 02 12:53:26.242665 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:26.242077    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4293ef7c-ce26-475f-98e1-ebf007c35f00-istio-podinfo\") pod \"e2e-shared-a-w3-e7561a-openshift-default-7598f89b79-m5rrm\" (UID: \"4293ef7c-ce26-475f-98e1-ebf007c35f00\") " pod="openshift-ingress/e2e-shared-a-w3-e7561a-openshift-default-7598f89b79-m5rrm"
Nov 02 12:53:26.242665 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:26.242135    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4293ef7c-ce26-475f-98e1-ebf007c35f00-workload-certs\") pod \"e2e-shared-a-w3-e7561a-openshift-default-7598f89b79-m5rrm\" (UID: \"4293ef7c-ce26-475f-98e1-ebf007c35f00\") " pod="openshift-ingress/e2e-shared-a-w3-e7561a-openshift-default-7598f89b79-m5rrm"
Nov 02 12:53:26.242665 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:26.242245    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4293ef7c-ce26-475f-98e1-ebf007c35f00-istio-data\") pod \"e2e-shared-a-w3-e7561a-openshift-default-7598f89b79-m5rrm\" (UID: \"4293ef7c-ce26-475f-98e1-ebf007c35f00\") " pod="openshift-ingress/e2e-shared-a-w3-e7561a-openshift-default-7598f89b79-m5rrm"
Nov 02 12:53:26.243319 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:26.242725    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4293ef7c-ce26-475f-98e1-ebf007c35f00-istio-data\") pod \"e2e-shared-a-w3-e7561a-openshift-default-7598f89b79-m5rrm\" (UID: \"4293ef7c-ce26-475f-98e1-ebf007c35f00\") " pod="openshift-ingress/e2e-shared-a-w3-e7561a-openshift-default-7598f89b79-m5rrm"
Nov 02 12:53:26.243319 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:26.242976    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4293ef7c-ce26-475f-98e1-ebf007c35f00-credential-socket\") pod \"e2e-shared-a-w3-e7561a-openshift-default-7598f89b79-m5rrm\" (UID: \"4293ef7c-ce26-475f-98e1-ebf007c35f00\") " pod="openshift-ingress/e2e-shared-a-w3-e7561a-openshift-default-7598f89b79-m5rrm"
Nov 02 12:53:26.254671 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:26.243527    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4293ef7c-ce26-475f-98e1-ebf007c35f00-workload-socket\") pod \"e2e-shared-a-w3-e7561a-openshift-default-7598f89b79-m5rrm\" (UID: \"4293ef7c-ce26-475f-98e1-ebf007c35f00\") " pod="openshift-ingress/e2e-shared-a-w3-e7561a-openshift-default-7598f89b79-m5rrm"
Nov 02 12:53:26.254671 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:26.244187    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4293ef7c-ce26-475f-98e1-ebf007c35f00-istiod-ca-cert\") pod \"e2e-shared-a-w3-e7561a-openshift-default-7598f89b79-m5rrm\" (UID: \"4293ef7c-ce26-475f-98e1-ebf007c35f00\") " pod="openshift-ingress/e2e-shared-a-w3-e7561a-openshift-default-7598f89b79-m5rrm"
Nov 02 12:53:26.254671 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:26.246678    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4293ef7c-ce26-475f-98e1-ebf007c35f00-workload-certs\") pod \"e2e-shared-a-w3-e7561a-openshift-default-7598f89b79-m5rrm\" (UID: \"4293ef7c-ce26-475f-98e1-ebf007c35f00\") " pod="openshift-ingress/e2e-shared-a-w3-e7561a-openshift-default-7598f89b79-m5rrm"
Nov 02 12:53:26.254671 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:26.253166    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4293ef7c-ce26-475f-98e1-ebf007c35f00-istio-podinfo\") pod \"e2e-shared-a-w3-e7561a-openshift-default-7598f89b79-m5rrm\" (UID: \"4293ef7c-ce26-475f-98e1-ebf007c35f00\") " pod="openshift-ingress/e2e-shared-a-w3-e7561a-openshift-default-7598f89b79-m5rrm"
Nov 02 12:53:26.279661 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:26.278877    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4293ef7c-ce26-475f-98e1-ebf007c35f00-istio-envoy\") pod \"e2e-shared-a-w3-e7561a-openshift-default-7598f89b79-m5rrm\" (UID: \"4293ef7c-ce26-475f-98e1-ebf007c35f00\") " pod="openshift-ingress/e2e-shared-a-w3-e7561a-openshift-default-7598f89b79-m5rrm"
Nov 02 12:53:26.305359 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:26.299485    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ts9j\" (UniqueName: \"kubernetes.io/projected/4293ef7c-ce26-475f-98e1-ebf007c35f00-kube-api-access-5ts9j\") pod \"e2e-shared-a-w3-e7561a-openshift-default-7598f89b79-m5rrm\" (UID: \"4293ef7c-ce26-475f-98e1-ebf007c35f00\") " pod="openshift-ingress/e2e-shared-a-w3-e7561a-openshift-default-7598f89b79-m5rrm"
Nov 02 12:53:26.305359 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:26.300533    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4293ef7c-ce26-475f-98e1-ebf007c35f00-istio-token\") pod \"e2e-shared-a-w3-e7561a-openshift-default-7598f89b79-m5rrm\" (UID: \"4293ef7c-ce26-475f-98e1-ebf007c35f00\") " pod="openshift-ingress/e2e-shared-a-w3-e7561a-openshift-default-7598f89b79-m5rrm"
Nov 02 12:53:26.324839 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:26.319013    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-ed56b9-openshift-default-789859d697-5f5lv"
Nov 02 12:53:26.379610 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:26.377812    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-e7561a-openshift-default-7598f89b79-m5rrm"
Nov 02 12:53:26.392260 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:26.391962    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-ed56b9-openshift-default-789859d697-5f5lv"
Nov 02 12:53:26.485304 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:26.485276    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-ed56b9-openshift-default-789859d697-5f5lv"
Nov 02 12:53:26.485746 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:26.485692    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-a6f2c0-openshift-default-7f47858fbf-x45sj"
Nov 02 12:53:26.486933 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:26.486858    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-ed56b9-openshift-default-789859d697-5f5lv"
Nov 02 12:53:26.486933 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:26.486908    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-a6f2c0-openshift-default-7f47858fbf-x45sj"
Nov 02 12:53:26.783162 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:26.783124    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-e7561a-openshift-default-7598f89b79-m5rrm"]
Nov 02 12:53:26.783725 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:26.783689    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:53:26.783804 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:26.783757    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:53:26.783804 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:26.783787    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:53:27.492348 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:27.492313    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-e7561a-openshift-default-7598f89b79-m5rrm" event={"ID":"4293ef7c-ce26-475f-98e1-ebf007c35f00","Type":"ContainerStarted","Data":"ccbfe98cbcb6cee4011b7e65b5332a6b8f935dc779efd6365f629dd39731a796"}
Nov 02 12:53:27.492816 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:27.492386    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-e7561a-openshift-default-7598f89b79-m5rrm" event={"ID":"4293ef7c-ce26-475f-98e1-ebf007c35f00","Type":"ContainerStarted","Data":"c10bb62b8e4efbbe9bffb82478bf968f2dfc4253acd122ba82acc1215d0a93ea"}
Nov 02 12:53:27.580070 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:27.580019    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-e7561a-openshift-default-7598f89b79-m5rrm" podStartSLOduration=1.5800055689999999 podStartE2EDuration="1.580005569s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:53:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:53:27.565303522 +0000 UTC m=+1603.820702614" watchObservedRunningTime="2026-11-02 12:53:27.580005569 +0000 UTC m=+1603.835404664"
Nov 02 12:53:28.378815 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:28.378782    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-e7561a-openshift-default-7598f89b79-m5rrm"
Nov 02 12:53:28.384211 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:28.384185    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-e7561a-openshift-default-7598f89b79-m5rrm"
Nov 02 12:53:28.500710 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:28.500679    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-e7561a-openshift-default-7598f89b79-m5rrm"
Nov 02 12:53:28.501355 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:28.501335    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-e7561a-openshift-default-7598f89b79-m5rrm"
Nov 02 12:53:29.749364 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:29.749327    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a6f2c0-d46bdcf55-7nw8f"]
Nov 02 12:53:29.754251 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:29.754232    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a6f2c0-d46bdcf55-7nw8f"
Nov 02 12:53:29.779342 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:29.779308    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a6f2c0-d46bdcf55-7nw8f"]
Nov 02 12:53:29.782546 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:29.782518    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w1-a6f2c0-dockercfg-gfgmg\""
Nov 02 12:53:29.792389 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:29.792364    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbsx9\" (UniqueName: \"kubernetes.io/projected/da10c6a3-b295-4997-84fc-ffe8ebb90061-kube-api-access-xbsx9\") pod \"payload-pre-processing-e2e-shared-a-w1-a6f2c0-d46bdcf55-7nw8f\" (UID: \"da10c6a3-b295-4997-84fc-ffe8ebb90061\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a6f2c0-d46bdcf55-7nw8f"
Nov 02 12:53:29.792517 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:29.792411    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da10c6a3-b295-4997-84fc-ffe8ebb90061-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-a6f2c0-d46bdcf55-7nw8f\" (UID: \"da10c6a3-b295-4997-84fc-ffe8ebb90061\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a6f2c0-d46bdcf55-7nw8f"
Nov 02 12:53:29.872700 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:29.870991    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w1-a6f2c0\""
Nov 02 12:53:29.904662 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:29.893070    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da10c6a3-b295-4997-84fc-ffe8ebb90061-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-a6f2c0-d46bdcf55-7nw8f\" (UID: \"da10c6a3-b295-4997-84fc-ffe8ebb90061\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a6f2c0-d46bdcf55-7nw8f"
Nov 02 12:53:29.914218 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:29.914170    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbsx9\" (UniqueName: \"kubernetes.io/projected/da10c6a3-b295-4997-84fc-ffe8ebb90061-kube-api-access-xbsx9\") pod \"payload-pre-processing-e2e-shared-a-w1-a6f2c0-d46bdcf55-7nw8f\" (UID: \"da10c6a3-b295-4997-84fc-ffe8ebb90061\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a6f2c0-d46bdcf55-7nw8f"
Nov 02 12:53:29.915589 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:29.905834    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da10c6a3-b295-4997-84fc-ffe8ebb90061-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-a6f2c0-d46bdcf55-7nw8f\" (UID: \"da10c6a3-b295-4997-84fc-ffe8ebb90061\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a6f2c0-d46bdcf55-7nw8f"
Nov 02 12:53:29.964680 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:29.959378    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbsx9\" (UniqueName: \"kubernetes.io/projected/da10c6a3-b295-4997-84fc-ffe8ebb90061-kube-api-access-xbsx9\") pod \"payload-pre-processing-e2e-shared-a-w1-a6f2c0-d46bdcf55-7nw8f\" (UID: \"da10c6a3-b295-4997-84fc-ffe8ebb90061\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a6f2c0-d46bdcf55-7nw8f"
Nov 02 12:53:30.071714 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:30.071272    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a6f2c0-d46bdcf55-7nw8f"
Nov 02 12:53:30.426554 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:30.426524    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a6f2c0-d46bdcf55-7nw8f"]
Nov 02 12:53:30.426702 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:53:30.426547    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda10c6a3_b295_4997_84fc_ffe8ebb90061.slice/crio-91d1e848e333f8376301d59e7a8df48327583ab99364547a3da981760d95c957 WatchSource:0}: Error finding container 91d1e848e333f8376301d59e7a8df48327583ab99364547a3da981760d95c957: Status 404 returned error can't find the container with id 91d1e848e333f8376301d59e7a8df48327583ab99364547a3da981760d95c957
Nov 02 12:53:30.599568 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:30.599532    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a6f2c0-d46bdcf55-7nw8f" event={"ID":"da10c6a3-b295-4997-84fc-ffe8ebb90061","Type":"ContainerStarted","Data":"cc72ae59767debf7cdb18e6bb565deeba6674c7c578e00332ae45d2325d9480c"}
Nov 02 12:53:30.599781 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:30.599591    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a6f2c0-d46bdcf55-7nw8f" event={"ID":"da10c6a3-b295-4997-84fc-ffe8ebb90061","Type":"ContainerStarted","Data":"91d1e848e333f8376301d59e7a8df48327583ab99364547a3da981760d95c957"}
Nov 02 12:53:30.599781 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:30.599622    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a6f2c0-d46bdcf55-7nw8f"
Nov 02 12:53:30.652828 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:30.652772    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a6f2c0-d46bdcf55-7nw8f" podStartSLOduration=1.65275726 podStartE2EDuration="1.65275726s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:53:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:53:30.651231761 +0000 UTC m=+1606.906630855" watchObservedRunningTime="2026-11-02 12:53:30.65275726 +0000 UTC m=+1606.908156358"
Nov 02 12:53:32.470630 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:32.470594    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-ed56b9-7bc9f55c5-qjvr7"]
Nov 02 12:53:32.474987 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:32.474967    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-ed56b9-7bc9f55c5-qjvr7"
Nov 02 12:53:32.507461 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:32.507430    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-ed56b9\""
Nov 02 12:53:32.507696 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:32.507504    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-ed56b9-dockercfg-vj2x9\""
Nov 02 12:53:32.509129 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:32.509107    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-ed56b9-7bc9f55c5-qjvr7"]
Nov 02 12:53:32.572454 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:32.572322    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4m6q\" (UniqueName: \"kubernetes.io/projected/7c89b44f-fa2c-4379-a933-9a8e7d010bfd-kube-api-access-t4m6q\") pod \"payload-processing-e2e-shared-a-w2-ed56b9-7bc9f55c5-qjvr7\" (UID: \"7c89b44f-fa2c-4379-a933-9a8e7d010bfd\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-ed56b9-7bc9f55c5-qjvr7"
Nov 02 12:53:32.572896 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:32.572868    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c89b44f-fa2c-4379-a933-9a8e7d010bfd-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-ed56b9-7bc9f55c5-qjvr7\" (UID: \"7c89b44f-fa2c-4379-a933-9a8e7d010bfd\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-ed56b9-7bc9f55c5-qjvr7"
Nov 02 12:53:32.692557 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:32.692508    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c89b44f-fa2c-4379-a933-9a8e7d010bfd-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-ed56b9-7bc9f55c5-qjvr7\" (UID: \"7c89b44f-fa2c-4379-a933-9a8e7d010bfd\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-ed56b9-7bc9f55c5-qjvr7"
Nov 02 12:53:32.693040 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:32.693014    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4m6q\" (UniqueName: \"kubernetes.io/projected/7c89b44f-fa2c-4379-a933-9a8e7d010bfd-kube-api-access-t4m6q\") pod \"payload-processing-e2e-shared-a-w2-ed56b9-7bc9f55c5-qjvr7\" (UID: \"7c89b44f-fa2c-4379-a933-9a8e7d010bfd\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-ed56b9-7bc9f55c5-qjvr7"
Nov 02 12:53:32.718667 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:32.711468    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c89b44f-fa2c-4379-a933-9a8e7d010bfd-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-ed56b9-7bc9f55c5-qjvr7\" (UID: \"7c89b44f-fa2c-4379-a933-9a8e7d010bfd\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-ed56b9-7bc9f55c5-qjvr7"
Nov 02 12:53:32.783707 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:32.777172    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4m6q\" (UniqueName: \"kubernetes.io/projected/7c89b44f-fa2c-4379-a933-9a8e7d010bfd-kube-api-access-t4m6q\") pod \"payload-processing-e2e-shared-a-w2-ed56b9-7bc9f55c5-qjvr7\" (UID: \"7c89b44f-fa2c-4379-a933-9a8e7d010bfd\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-ed56b9-7bc9f55c5-qjvr7"
Nov 02 12:53:32.802670 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:32.790204    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-ed56b9-7bc9f55c5-qjvr7"
Nov 02 12:53:33.188160 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:53:33.188126    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c89b44f_fa2c_4379_a933_9a8e7d010bfd.slice/crio-ca81d9dd4f68fc85f585cf9c1cd2a9d5174fdb36a592fa196b08919fe3216f33 WatchSource:0}: Error finding container ca81d9dd4f68fc85f585cf9c1cd2a9d5174fdb36a592fa196b08919fe3216f33: Status 404 returned error can't find the container with id ca81d9dd4f68fc85f585cf9c1cd2a9d5174fdb36a592fa196b08919fe3216f33
Nov 02 12:53:33.190624 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:33.190593    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-ed56b9-7bc9f55c5-qjvr7"]
Nov 02 12:53:33.458317 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:33.458224    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c0d4cb-867cbccf7f-pmph4"
Nov 02 12:53:33.458317 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:33.458296    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-c0d4cb-98c64fd4c-qs6bn"
Nov 02 12:53:33.732050 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:33.731963    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-ed56b9-7bc9f55c5-qjvr7" event={"ID":"7c89b44f-fa2c-4379-a933-9a8e7d010bfd","Type":"ContainerStarted","Data":"839ab0ddb5712c3a205d8812562e16fb9303e8177aa61abb94a511641adb89e5"}
Nov 02 12:53:33.732050 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:33.732016    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-ed56b9-7bc9f55c5-qjvr7" event={"ID":"7c89b44f-fa2c-4379-a933-9a8e7d010bfd","Type":"ContainerStarted","Data":"ca81d9dd4f68fc85f585cf9c1cd2a9d5174fdb36a592fa196b08919fe3216f33"}
Nov 02 12:53:33.732050 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:33.732039    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-ed56b9-7bc9f55c5-qjvr7"
Nov 02 12:53:33.797099 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:33.797037    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-ed56b9-7bc9f55c5-qjvr7" podStartSLOduration=1.797019493 podStartE2EDuration="1.797019493s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:53:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:53:33.795767867 +0000 UTC m=+1610.051166961" watchObservedRunningTime="2026-11-02 12:53:33.797019493 +0000 UTC m=+1610.052418589"
Nov 02 12:53:35.391009 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:35.390970    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-c0d4cb-7465cdd564-w4nwd"]
Nov 02 12:53:35.395263 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:35.395240    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c0d4cb-7465cdd564-w4nwd"
Nov 02 12:53:35.417475 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:35.417451    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-c0d4cb-7465cdd564-w4nwd"]
Nov 02 12:53:35.424776 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:35.424735    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-c0d4cb-serving-cert\""
Nov 02 12:53:35.485402 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:35.485357    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5baf3254-0f52-4b0a-8b31-9ae96e0aa816-maas-api-tls\") pod \"maas-api-e2e-praxis-a-c0d4cb-7465cdd564-w4nwd\" (UID: \"5baf3254-0f52-4b0a-8b31-9ae96e0aa816\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c0d4cb-7465cdd564-w4nwd"
Nov 02 12:53:35.485602 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:35.485585    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5baf3254-0f52-4b0a-8b31-9ae96e0aa816-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-c0d4cb-7465cdd564-w4nwd\" (UID: \"5baf3254-0f52-4b0a-8b31-9ae96e0aa816\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c0d4cb-7465cdd564-w4nwd"
Nov 02 12:53:35.485795 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:35.485780    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gc9rp\" (UniqueName: \"kubernetes.io/projected/5baf3254-0f52-4b0a-8b31-9ae96e0aa816-kube-api-access-gc9rp\") pod \"maas-api-e2e-praxis-a-c0d4cb-7465cdd564-w4nwd\" (UID: \"5baf3254-0f52-4b0a-8b31-9ae96e0aa816\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c0d4cb-7465cdd564-w4nwd"
Nov 02 12:53:35.485846 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:35.485820    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5baf3254-0f52-4b0a-8b31-9ae96e0aa816-metrics-certs\") pod \"maas-api-e2e-praxis-a-c0d4cb-7465cdd564-w4nwd\" (UID: \"5baf3254-0f52-4b0a-8b31-9ae96e0aa816\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c0d4cb-7465cdd564-w4nwd"
Nov 02 12:53:35.586654 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:35.586607    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gc9rp\" (UniqueName: \"kubernetes.io/projected/5baf3254-0f52-4b0a-8b31-9ae96e0aa816-kube-api-access-gc9rp\") pod \"maas-api-e2e-praxis-a-c0d4cb-7465cdd564-w4nwd\" (UID: \"5baf3254-0f52-4b0a-8b31-9ae96e0aa816\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c0d4cb-7465cdd564-w4nwd"
Nov 02 12:53:35.586829 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:35.586702    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5baf3254-0f52-4b0a-8b31-9ae96e0aa816-metrics-certs\") pod \"maas-api-e2e-praxis-a-c0d4cb-7465cdd564-w4nwd\" (UID: \"5baf3254-0f52-4b0a-8b31-9ae96e0aa816\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c0d4cb-7465cdd564-w4nwd"
Nov 02 12:53:35.586829 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:35.586769    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5baf3254-0f52-4b0a-8b31-9ae96e0aa816-maas-api-tls\") pod \"maas-api-e2e-praxis-a-c0d4cb-7465cdd564-w4nwd\" (UID: \"5baf3254-0f52-4b0a-8b31-9ae96e0aa816\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c0d4cb-7465cdd564-w4nwd"
Nov 02 12:53:35.586952 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:35.586880    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5baf3254-0f52-4b0a-8b31-9ae96e0aa816-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-c0d4cb-7465cdd564-w4nwd\" (UID: \"5baf3254-0f52-4b0a-8b31-9ae96e0aa816\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c0d4cb-7465cdd564-w4nwd"
Nov 02 12:53:35.587549 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:35.587519    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5baf3254-0f52-4b0a-8b31-9ae96e0aa816-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-c0d4cb-7465cdd564-w4nwd\" (UID: \"5baf3254-0f52-4b0a-8b31-9ae96e0aa816\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c0d4cb-7465cdd564-w4nwd"
Nov 02 12:53:35.589223 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:35.589187    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5baf3254-0f52-4b0a-8b31-9ae96e0aa816-metrics-certs\") pod \"maas-api-e2e-praxis-a-c0d4cb-7465cdd564-w4nwd\" (UID: \"5baf3254-0f52-4b0a-8b31-9ae96e0aa816\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c0d4cb-7465cdd564-w4nwd"
Nov 02 12:53:35.589353 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:35.589310    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5baf3254-0f52-4b0a-8b31-9ae96e0aa816-maas-api-tls\") pod \"maas-api-e2e-praxis-a-c0d4cb-7465cdd564-w4nwd\" (UID: \"5baf3254-0f52-4b0a-8b31-9ae96e0aa816\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c0d4cb-7465cdd564-w4nwd"
Nov 02 12:53:35.621111 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:35.621082    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gc9rp\" (UniqueName: \"kubernetes.io/projected/5baf3254-0f52-4b0a-8b31-9ae96e0aa816-kube-api-access-gc9rp\") pod \"maas-api-e2e-praxis-a-c0d4cb-7465cdd564-w4nwd\" (UID: \"5baf3254-0f52-4b0a-8b31-9ae96e0aa816\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c0d4cb-7465cdd564-w4nwd"
Nov 02 12:53:35.719662 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:35.716101    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c0d4cb-7465cdd564-w4nwd"
Nov 02 12:53:36.126768 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:36.126659    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-c0d4cb-7465cdd564-w4nwd"]
Nov 02 12:53:36.785872 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:36.785779    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c0d4cb-7465cdd564-w4nwd" event={"ID":"5baf3254-0f52-4b0a-8b31-9ae96e0aa816","Type":"ContainerStarted","Data":"e70dc7b23a567a95b8fb6b3c0d4c98b6d278d7796d99057b7b2f5bd6d96b2bb5"}
Nov 02 12:53:36.785872 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:36.785851    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c0d4cb-7465cdd564-w4nwd" event={"ID":"5baf3254-0f52-4b0a-8b31-9ae96e0aa816","Type":"ContainerStarted","Data":"392efa535702c442012810a6dccb7aa209134eefd144046d19cac0375246ede9"}
Nov 02 12:53:36.786372 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:36.785975    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c0d4cb-7465cdd564-w4nwd"
Nov 02 12:53:36.831291 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:36.831235    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c0d4cb-7465cdd564-w4nwd" podStartSLOduration=1.439136418 podStartE2EDuration="1.831221253s" totalImagesPullingTime="392.084835ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:53:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:53:36.829344746 +0000 UTC m=+1613.084743853" watchObservedRunningTime="2026-11-02 12:53:36.831221253 +0000 UTC m=+1613.086620348"
Nov 02 12:53:41.608473 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:41.608446    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a6f2c0-d46bdcf55-7nw8f"
Nov 02 12:53:42.822569 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:42.822539    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c0d4cb-7465cdd564-w4nwd"
Nov 02 12:53:44.679650 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:44.679614    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-v9zqm"]
Nov 02 12:53:44.685169 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:44.685148    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-v9zqm"
Nov 02 12:53:44.718675 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:44.718616    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-v9zqm"]
Nov 02 12:53:44.730390 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:44.730356    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-qdrs6"]
Nov 02 12:53:44.734950 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:44.734924    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-qdrs6"
Nov 02 12:53:44.742415 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:44.742391    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-ed56b9-7bc9f55c5-qjvr7"
Nov 02 12:53:44.766848 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:44.766817    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-qdrs6"]
Nov 02 12:53:44.780177 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:44.780147    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlszr\" (UniqueName: \"kubernetes.io/projected/ef76ad04-45eb-4b3b-b314-e93122bb18f5-kube-api-access-mlszr\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-v9zqm\" (UID: \"ef76ad04-45eb-4b3b-b314-e93122bb18f5\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-v9zqm"
Nov 02 12:53:44.780361 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:44.780230    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef76ad04-45eb-4b3b-b314-e93122bb18f5-istio-envoy\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-v9zqm\" (UID: \"ef76ad04-45eb-4b3b-b314-e93122bb18f5\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-v9zqm"
Nov 02 12:53:44.780361 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:44.780260    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef76ad04-45eb-4b3b-b314-e93122bb18f5-istio-podinfo\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-v9zqm\" (UID: \"ef76ad04-45eb-4b3b-b314-e93122bb18f5\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-v9zqm"
Nov 02 12:53:44.780361 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:44.780295    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef76ad04-45eb-4b3b-b314-e93122bb18f5-istio-token\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-v9zqm\" (UID: \"ef76ad04-45eb-4b3b-b314-e93122bb18f5\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-v9zqm"
Nov 02 12:53:44.780595 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:44.780449    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef76ad04-45eb-4b3b-b314-e93122bb18f5-workload-certs\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-v9zqm\" (UID: \"ef76ad04-45eb-4b3b-b314-e93122bb18f5\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-v9zqm"
Nov 02 12:53:44.780595 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:44.780520    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef76ad04-45eb-4b3b-b314-e93122bb18f5-istio-data\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-v9zqm\" (UID: \"ef76ad04-45eb-4b3b-b314-e93122bb18f5\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-v9zqm"
Nov 02 12:53:44.780595 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:44.780572    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef76ad04-45eb-4b3b-b314-e93122bb18f5-workload-socket\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-v9zqm\" (UID: \"ef76ad04-45eb-4b3b-b314-e93122bb18f5\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-v9zqm"
Nov 02 12:53:44.780803 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:44.780677    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef76ad04-45eb-4b3b-b314-e93122bb18f5-istiod-ca-cert\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-v9zqm\" (UID: \"ef76ad04-45eb-4b3b-b314-e93122bb18f5\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-v9zqm"
Nov 02 12:53:44.780803 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:44.780756    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef76ad04-45eb-4b3b-b314-e93122bb18f5-credential-socket\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-v9zqm\" (UID: \"ef76ad04-45eb-4b3b-b314-e93122bb18f5\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-v9zqm"
Nov 02 12:53:44.882197 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:44.882156    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef76ad04-45eb-4b3b-b314-e93122bb18f5-credential-socket\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-v9zqm\" (UID: \"ef76ad04-45eb-4b3b-b314-e93122bb18f5\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-v9zqm"
Nov 02 12:53:44.882403 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:44.882214    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3aabb9b6-9382-42d9-8c52-4d89e361c25b-istio-token\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-qdrs6\" (UID: \"3aabb9b6-9382-42d9-8c52-4d89e361c25b\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-qdrs6"
Nov 02 12:53:44.882403 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:44.882254    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlszr\" (UniqueName: \"kubernetes.io/projected/ef76ad04-45eb-4b3b-b314-e93122bb18f5-kube-api-access-mlszr\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-v9zqm\" (UID: \"ef76ad04-45eb-4b3b-b314-e93122bb18f5\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-v9zqm"
Nov 02 12:53:44.882403 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:44.882379    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3aabb9b6-9382-42d9-8c52-4d89e361c25b-istio-envoy\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-qdrs6\" (UID: \"3aabb9b6-9382-42d9-8c52-4d89e361c25b\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-qdrs6"
Nov 02 12:53:44.882580 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:44.882503    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef76ad04-45eb-4b3b-b314-e93122bb18f5-istio-envoy\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-v9zqm\" (UID: \"ef76ad04-45eb-4b3b-b314-e93122bb18f5\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-v9zqm"
Nov 02 12:53:44.882580 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:44.882509    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef76ad04-45eb-4b3b-b314-e93122bb18f5-credential-socket\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-v9zqm\" (UID: \"ef76ad04-45eb-4b3b-b314-e93122bb18f5\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-v9zqm"
Nov 02 12:53:44.882580 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:44.882522    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3aabb9b6-9382-42d9-8c52-4d89e361c25b-istio-data\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-qdrs6\" (UID: \"3aabb9b6-9382-42d9-8c52-4d89e361c25b\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-qdrs6"
Nov 02 12:53:44.882580 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:44.882549    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef76ad04-45eb-4b3b-b314-e93122bb18f5-istio-podinfo\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-v9zqm\" (UID: \"ef76ad04-45eb-4b3b-b314-e93122bb18f5\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-v9zqm"
Nov 02 12:53:44.882841 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:44.882581    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef76ad04-45eb-4b3b-b314-e93122bb18f5-istio-token\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-v9zqm\" (UID: \"ef76ad04-45eb-4b3b-b314-e93122bb18f5\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-v9zqm"
Nov 02 12:53:44.882841 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:44.882606    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtsvd\" (UniqueName: \"kubernetes.io/projected/3aabb9b6-9382-42d9-8c52-4d89e361c25b-kube-api-access-rtsvd\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-qdrs6\" (UID: \"3aabb9b6-9382-42d9-8c52-4d89e361c25b\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-qdrs6"
Nov 02 12:53:44.882841 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:44.882705    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3aabb9b6-9382-42d9-8c52-4d89e361c25b-istiod-ca-cert\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-qdrs6\" (UID: \"3aabb9b6-9382-42d9-8c52-4d89e361c25b\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-qdrs6"
Nov 02 12:53:44.882841 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:44.882775    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3aabb9b6-9382-42d9-8c52-4d89e361c25b-credential-socket\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-qdrs6\" (UID: \"3aabb9b6-9382-42d9-8c52-4d89e361c25b\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-qdrs6"
Nov 02 12:53:44.883015 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:44.882920    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3aabb9b6-9382-42d9-8c52-4d89e361c25b-workload-certs\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-qdrs6\" (UID: \"3aabb9b6-9382-42d9-8c52-4d89e361c25b\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-qdrs6"
Nov 02 12:53:44.883015 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:44.882977    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3aabb9b6-9382-42d9-8c52-4d89e361c25b-istio-podinfo\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-qdrs6\" (UID: \"3aabb9b6-9382-42d9-8c52-4d89e361c25b\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-qdrs6"
Nov 02 12:53:44.883133 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:44.883045    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3aabb9b6-9382-42d9-8c52-4d89e361c25b-workload-socket\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-qdrs6\" (UID: \"3aabb9b6-9382-42d9-8c52-4d89e361c25b\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-qdrs6"
Nov 02 12:53:44.883133 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:44.883120    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef76ad04-45eb-4b3b-b314-e93122bb18f5-workload-certs\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-v9zqm\" (UID: \"ef76ad04-45eb-4b3b-b314-e93122bb18f5\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-v9zqm"
Nov 02 12:53:44.883241 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:44.883202    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef76ad04-45eb-4b3b-b314-e93122bb18f5-istio-data\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-v9zqm\" (UID: \"ef76ad04-45eb-4b3b-b314-e93122bb18f5\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-v9zqm"
Nov 02 12:53:44.883295 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:44.883274    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef76ad04-45eb-4b3b-b314-e93122bb18f5-workload-socket\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-v9zqm\" (UID: \"ef76ad04-45eb-4b3b-b314-e93122bb18f5\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-v9zqm"
Nov 02 12:53:44.883351 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:44.883299    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef76ad04-45eb-4b3b-b314-e93122bb18f5-istiod-ca-cert\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-v9zqm\" (UID: \"ef76ad04-45eb-4b3b-b314-e93122bb18f5\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-v9zqm"
Nov 02 12:53:44.883503 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:44.883471    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef76ad04-45eb-4b3b-b314-e93122bb18f5-workload-certs\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-v9zqm\" (UID: \"ef76ad04-45eb-4b3b-b314-e93122bb18f5\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-v9zqm"
Nov 02 12:53:44.883699 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:44.883550    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef76ad04-45eb-4b3b-b314-e93122bb18f5-istio-data\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-v9zqm\" (UID: \"ef76ad04-45eb-4b3b-b314-e93122bb18f5\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-v9zqm"
Nov 02 12:53:44.883699 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:44.883686    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef76ad04-45eb-4b3b-b314-e93122bb18f5-workload-socket\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-v9zqm\" (UID: \"ef76ad04-45eb-4b3b-b314-e93122bb18f5\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-v9zqm"
Nov 02 12:53:44.883991 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:44.883969    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef76ad04-45eb-4b3b-b314-e93122bb18f5-istiod-ca-cert\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-v9zqm\" (UID: \"ef76ad04-45eb-4b3b-b314-e93122bb18f5\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-v9zqm"
Nov 02 12:53:44.884839 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:44.884818    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef76ad04-45eb-4b3b-b314-e93122bb18f5-istio-envoy\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-v9zqm\" (UID: \"ef76ad04-45eb-4b3b-b314-e93122bb18f5\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-v9zqm"
Nov 02 12:53:44.885232 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:44.885210    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef76ad04-45eb-4b3b-b314-e93122bb18f5-istio-podinfo\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-v9zqm\" (UID: \"ef76ad04-45eb-4b3b-b314-e93122bb18f5\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-v9zqm"
Nov 02 12:53:44.906866 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:44.906842    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef76ad04-45eb-4b3b-b314-e93122bb18f5-istio-token\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-v9zqm\" (UID: \"ef76ad04-45eb-4b3b-b314-e93122bb18f5\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-v9zqm"
Nov 02 12:53:44.908057 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:44.908032    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlszr\" (UniqueName: \"kubernetes.io/projected/ef76ad04-45eb-4b3b-b314-e93122bb18f5-kube-api-access-mlszr\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-v9zqm\" (UID: \"ef76ad04-45eb-4b3b-b314-e93122bb18f5\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-v9zqm"
Nov 02 12:53:44.987250 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:44.987155    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3aabb9b6-9382-42d9-8c52-4d89e361c25b-istio-token\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-qdrs6\" (UID: \"3aabb9b6-9382-42d9-8c52-4d89e361c25b\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-qdrs6"
Nov 02 12:53:44.987250 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:44.987222    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3aabb9b6-9382-42d9-8c52-4d89e361c25b-istio-envoy\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-qdrs6\" (UID: \"3aabb9b6-9382-42d9-8c52-4d89e361c25b\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-qdrs6"
Nov 02 12:53:44.987485 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:44.987263    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3aabb9b6-9382-42d9-8c52-4d89e361c25b-istio-data\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-qdrs6\" (UID: \"3aabb9b6-9382-42d9-8c52-4d89e361c25b\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-qdrs6"
Nov 02 12:53:44.987485 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:44.987296    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtsvd\" (UniqueName: \"kubernetes.io/projected/3aabb9b6-9382-42d9-8c52-4d89e361c25b-kube-api-access-rtsvd\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-qdrs6\" (UID: \"3aabb9b6-9382-42d9-8c52-4d89e361c25b\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-qdrs6"
Nov 02 12:53:44.987485 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:44.987331    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3aabb9b6-9382-42d9-8c52-4d89e361c25b-istiod-ca-cert\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-qdrs6\" (UID: \"3aabb9b6-9382-42d9-8c52-4d89e361c25b\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-qdrs6"
Nov 02 12:53:44.987485 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:44.987396    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3aabb9b6-9382-42d9-8c52-4d89e361c25b-credential-socket\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-qdrs6\" (UID: \"3aabb9b6-9382-42d9-8c52-4d89e361c25b\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-qdrs6"
Nov 02 12:53:44.987485 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:44.987468    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3aabb9b6-9382-42d9-8c52-4d89e361c25b-workload-certs\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-qdrs6\" (UID: \"3aabb9b6-9382-42d9-8c52-4d89e361c25b\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-qdrs6"
Nov 02 12:53:44.987777 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:44.987493    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3aabb9b6-9382-42d9-8c52-4d89e361c25b-istio-podinfo\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-qdrs6\" (UID: \"3aabb9b6-9382-42d9-8c52-4d89e361c25b\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-qdrs6"
Nov 02 12:53:44.987777 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:44.987514    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3aabb9b6-9382-42d9-8c52-4d89e361c25b-workload-socket\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-qdrs6\" (UID: \"3aabb9b6-9382-42d9-8c52-4d89e361c25b\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-qdrs6"
Nov 02 12:53:44.988630 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:44.988222    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3aabb9b6-9382-42d9-8c52-4d89e361c25b-workload-socket\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-qdrs6\" (UID: \"3aabb9b6-9382-42d9-8c52-4d89e361c25b\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-qdrs6"
Nov 02 12:53:44.988630 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:44.988297    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3aabb9b6-9382-42d9-8c52-4d89e361c25b-istio-data\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-qdrs6\" (UID: \"3aabb9b6-9382-42d9-8c52-4d89e361c25b\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-qdrs6"
Nov 02 12:53:44.988630 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:44.988502    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3aabb9b6-9382-42d9-8c52-4d89e361c25b-credential-socket\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-qdrs6\" (UID: \"3aabb9b6-9382-42d9-8c52-4d89e361c25b\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-qdrs6"
Nov 02 12:53:44.988630 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:44.988584    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3aabb9b6-9382-42d9-8c52-4d89e361c25b-istiod-ca-cert\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-qdrs6\" (UID: \"3aabb9b6-9382-42d9-8c52-4d89e361c25b\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-qdrs6"
Nov 02 12:53:44.988976 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:44.988674    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3aabb9b6-9382-42d9-8c52-4d89e361c25b-workload-certs\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-qdrs6\" (UID: \"3aabb9b6-9382-42d9-8c52-4d89e361c25b\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-qdrs6"
Nov 02 12:53:44.995345 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:44.992112    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3aabb9b6-9382-42d9-8c52-4d89e361c25b-istio-envoy\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-qdrs6\" (UID: \"3aabb9b6-9382-42d9-8c52-4d89e361c25b\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-qdrs6"
Nov 02 12:53:45.013684 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:45.012801    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-v9zqm"
Nov 02 12:53:45.016690 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:45.014280    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3aabb9b6-9382-42d9-8c52-4d89e361c25b-istio-podinfo\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-qdrs6\" (UID: \"3aabb9b6-9382-42d9-8c52-4d89e361c25b\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-qdrs6"
Nov 02 12:53:45.025770 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:45.024961    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3aabb9b6-9382-42d9-8c52-4d89e361c25b-istio-token\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-qdrs6\" (UID: \"3aabb9b6-9382-42d9-8c52-4d89e361c25b\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-qdrs6"
Nov 02 12:53:45.026503 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:45.026475    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtsvd\" (UniqueName: \"kubernetes.io/projected/3aabb9b6-9382-42d9-8c52-4d89e361c25b-kube-api-access-rtsvd\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-qdrs6\" (UID: \"3aabb9b6-9382-42d9-8c52-4d89e361c25b\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-qdrs6"
Nov 02 12:53:45.052358 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:45.052329    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-qdrs6"
Nov 02 12:53:45.096016 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:53:45.095934    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqcmj" containerName="extractor"
Nov 02 12:53:45.300030 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:45.299525    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ed56b9-5f957cb677-8j2rp"]
Nov 02 12:53:45.309701 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:45.307677    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ed56b9-5f957cb677-8j2rp"]
Nov 02 12:53:45.309701 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:45.307790    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ed56b9-5f957cb677-8j2rp"
Nov 02 12:53:45.340676 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:45.340588    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-ed56b9-serving-cert\""
Nov 02 12:53:45.372714 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:45.372683    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-v9zqm"]
Nov 02 12:53:45.382697 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:53:45.382665    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef76ad04_45eb_4b3b_b314_e93122bb18f5.slice/crio-ade342c7d53f0fa54032a2fc71a2d3f0234e84568b94fe8f5177bacd0dee65eb WatchSource:0}: Error finding container ade342c7d53f0fa54032a2fc71a2d3f0234e84568b94fe8f5177bacd0dee65eb: Status 404 returned error can't find the container with id ade342c7d53f0fa54032a2fc71a2d3f0234e84568b94fe8f5177bacd0dee65eb
Nov 02 12:53:45.385562 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:45.385528    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:53:45.385678 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:45.385598    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:53:45.385678 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:45.385654    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:53:45.411282 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:45.411257    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/95acd739-ef82-4350-a68c-7a40ae0acbe5-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-ed56b9-5f957cb677-8j2rp\" (UID: \"95acd739-ef82-4350-a68c-7a40ae0acbe5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ed56b9-5f957cb677-8j2rp"
Nov 02 12:53:45.411387 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:45.411371    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/95acd739-ef82-4350-a68c-7a40ae0acbe5-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-ed56b9-5f957cb677-8j2rp\" (UID: \"95acd739-ef82-4350-a68c-7a40ae0acbe5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ed56b9-5f957cb677-8j2rp"
Nov 02 12:53:45.411552 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:45.411540    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/95acd739-ef82-4350-a68c-7a40ae0acbe5-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-ed56b9-5f957cb677-8j2rp\" (UID: \"95acd739-ef82-4350-a68c-7a40ae0acbe5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ed56b9-5f957cb677-8j2rp"
Nov 02 12:53:45.411613 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:45.411598    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjqgm\" (UniqueName: \"kubernetes.io/projected/95acd739-ef82-4350-a68c-7a40ae0acbe5-kube-api-access-wjqgm\") pod \"maas-api-e2e-shared-a-w2-ed56b9-5f957cb677-8j2rp\" (UID: \"95acd739-ef82-4350-a68c-7a40ae0acbe5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ed56b9-5f957cb677-8j2rp"
Nov 02 12:53:45.412108 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:45.412089    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-qdrs6"]
Nov 02 12:53:45.417926 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:53:45.417900    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3aabb9b6_9382_42d9_8c52_4d89e361c25b.slice/crio-5eb4183d3e04fda9ccba08abfff666b95b3b17bbae5094e2f706bcd2f17b46c4 WatchSource:0}: Error finding container 5eb4183d3e04fda9ccba08abfff666b95b3b17bbae5094e2f706bcd2f17b46c4: Status 404 returned error can't find the container with id 5eb4183d3e04fda9ccba08abfff666b95b3b17bbae5094e2f706bcd2f17b46c4
Nov 02 12:53:45.420360 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:45.420326    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:53:45.420437 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:45.420394    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:53:45.420437 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:45.420425    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:53:45.514478 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:45.513532    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/95acd739-ef82-4350-a68c-7a40ae0acbe5-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-ed56b9-5f957cb677-8j2rp\" (UID: \"95acd739-ef82-4350-a68c-7a40ae0acbe5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ed56b9-5f957cb677-8j2rp"
Nov 02 12:53:45.514478 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:45.513663    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjqgm\" (UniqueName: \"kubernetes.io/projected/95acd739-ef82-4350-a68c-7a40ae0acbe5-kube-api-access-wjqgm\") pod \"maas-api-e2e-shared-a-w2-ed56b9-5f957cb677-8j2rp\" (UID: \"95acd739-ef82-4350-a68c-7a40ae0acbe5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ed56b9-5f957cb677-8j2rp"
Nov 02 12:53:45.514478 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:45.513734    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/95acd739-ef82-4350-a68c-7a40ae0acbe5-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-ed56b9-5f957cb677-8j2rp\" (UID: \"95acd739-ef82-4350-a68c-7a40ae0acbe5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ed56b9-5f957cb677-8j2rp"
Nov 02 12:53:45.514478 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:45.513829    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/95acd739-ef82-4350-a68c-7a40ae0acbe5-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-ed56b9-5f957cb677-8j2rp\" (UID: \"95acd739-ef82-4350-a68c-7a40ae0acbe5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ed56b9-5f957cb677-8j2rp"
Nov 02 12:53:45.540152 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:45.527043    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/95acd739-ef82-4350-a68c-7a40ae0acbe5-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-ed56b9-5f957cb677-8j2rp\" (UID: \"95acd739-ef82-4350-a68c-7a40ae0acbe5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ed56b9-5f957cb677-8j2rp"
Nov 02 12:53:45.540152 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:45.537411    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/95acd739-ef82-4350-a68c-7a40ae0acbe5-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-ed56b9-5f957cb677-8j2rp\" (UID: \"95acd739-ef82-4350-a68c-7a40ae0acbe5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ed56b9-5f957cb677-8j2rp"
Nov 02 12:53:45.555755 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:45.542860    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/95acd739-ef82-4350-a68c-7a40ae0acbe5-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-ed56b9-5f957cb677-8j2rp\" (UID: \"95acd739-ef82-4350-a68c-7a40ae0acbe5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ed56b9-5f957cb677-8j2rp"
Nov 02 12:53:45.572663 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:45.569780    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjqgm\" (UniqueName: \"kubernetes.io/projected/95acd739-ef82-4350-a68c-7a40ae0acbe5-kube-api-access-wjqgm\") pod \"maas-api-e2e-shared-a-w2-ed56b9-5f957cb677-8j2rp\" (UID: \"95acd739-ef82-4350-a68c-7a40ae0acbe5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ed56b9-5f957cb677-8j2rp"
Nov 02 12:53:45.628086 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:45.627850    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ed56b9-5f957cb677-8j2rp"
Nov 02 12:53:45.872743 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:45.872689    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-qdrs6" event={"ID":"3aabb9b6-9382-42d9-8c52-4d89e361c25b","Type":"ContainerStarted","Data":"45af072c4ede5a3117cf04d4c7314ff8c012723495d29dd227e1391ed307b0a9"}
Nov 02 12:53:45.873340 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:45.872768    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-qdrs6" event={"ID":"3aabb9b6-9382-42d9-8c52-4d89e361c25b","Type":"ContainerStarted","Data":"5eb4183d3e04fda9ccba08abfff666b95b3b17bbae5094e2f706bcd2f17b46c4"}
Nov 02 12:53:45.874735 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:45.874707    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-v9zqm" event={"ID":"ef76ad04-45eb-4b3b-b314-e93122bb18f5","Type":"ContainerStarted","Data":"70e5ba11c272127890ba4f3d4cba2d3574441162cb27abe1498ce4870f5b0a4c"}
Nov 02 12:53:45.874829 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:45.874762    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-v9zqm" event={"ID":"ef76ad04-45eb-4b3b-b314-e93122bb18f5","Type":"ContainerStarted","Data":"ade342c7d53f0fa54032a2fc71a2d3f0234e84568b94fe8f5177bacd0dee65eb"}
Nov 02 12:53:45.930731 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:45.930629    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-qdrs6" podStartSLOduration=1.930609922 podStartE2EDuration="1.930609922s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:53:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:53:45.929269892 +0000 UTC m=+1622.184669000" watchObservedRunningTime="2026-11-02 12:53:45.930609922 +0000 UTC m=+1622.186009018"
Nov 02 12:53:45.971877 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:53:45.971844    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95acd739_ef82_4350_a68c_7a40ae0acbe5.slice/crio-1491d35cf638ec42190d9ccffff7f789fe938263d6862596adf1fbf5f9e52ac3 WatchSource:0}: Error finding container 1491d35cf638ec42190d9ccffff7f789fe938263d6862596adf1fbf5f9e52ac3: Status 404 returned error can't find the container with id 1491d35cf638ec42190d9ccffff7f789fe938263d6862596adf1fbf5f9e52ac3
Nov 02 12:53:45.972173 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:45.972149    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ed56b9-5f957cb677-8j2rp"]
Nov 02 12:53:46.014050 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:46.014013    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-v9zqm"
Nov 02 12:53:46.015506 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:46.015468    4559 patch_prober.go:29] interesting pod/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-v9zqm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" start-of-body=
Nov 02 12:53:46.015689 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:46.015539    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-v9zqm" podUID="ef76ad04-45eb-4b3b-b314-e93122bb18f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused"
Nov 02 12:53:46.059797 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:46.059761    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-qdrs6"
Nov 02 12:53:46.060302 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:46.060173    4559 patch_prober.go:29] interesting pod/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-qdrs6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" start-of-body=
Nov 02 12:53:46.060302 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:46.060237    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-qdrs6" podUID="3aabb9b6-9382-42d9-8c52-4d89e361c25b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused"
Nov 02 12:53:46.937718 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:46.937608    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ed56b9-5f957cb677-8j2rp" event={"ID":"95acd739-ef82-4350-a68c-7a40ae0acbe5","Type":"ContainerStarted","Data":"b2367a193246f6c5ac4b1037fe0eea981729682519ea7bc275635d2729097323"}
Nov 02 12:53:46.939227 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:46.939200    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ed56b9-5f957cb677-8j2rp" event={"ID":"95acd739-ef82-4350-a68c-7a40ae0acbe5","Type":"ContainerStarted","Data":"1491d35cf638ec42190d9ccffff7f789fe938263d6862596adf1fbf5f9e52ac3"}
Nov 02 12:53:46.939438 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:46.939425    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ed56b9-5f957cb677-8j2rp"
Nov 02 12:53:47.039252 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:47.039215    4559 patch_prober.go:29] interesting pod/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-v9zqm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" start-of-body=
Nov 02 12:53:47.039580 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:47.039533    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-v9zqm" podUID="ef76ad04-45eb-4b3b-b314-e93122bb18f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused"
Nov 02 12:53:47.040125 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:47.040068    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-v9zqm" podStartSLOduration=3.040047835 podStartE2EDuration="3.040047835s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:53:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:53:46.002766987 +0000 UTC m=+1622.258166084" watchObservedRunningTime="2026-11-02 12:53:47.040047835 +0000 UTC m=+1623.295446931"
Nov 02 12:53:47.053393 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:47.053350    4559 patch_prober.go:29] interesting pod/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-qdrs6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" start-of-body=
Nov 02 12:53:47.053578 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:47.053405    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-qdrs6" podUID="3aabb9b6-9382-42d9-8c52-4d89e361c25b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused"
Nov 02 12:53:48.016665 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:48.016044    4559 patch_prober.go:29] interesting pod/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-v9zqm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" start-of-body=
Nov 02 12:53:48.016665 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:48.016108    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-v9zqm" podUID="ef76ad04-45eb-4b3b-b314-e93122bb18f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused"
Nov 02 12:53:48.052789 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:48.052758    4559 patch_prober.go:29] interesting pod/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-qdrs6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" start-of-body=
Nov 02 12:53:48.053021 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:48.052999    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-qdrs6" podUID="3aabb9b6-9382-42d9-8c52-4d89e361c25b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused"
Nov 02 12:53:49.017111 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:49.017068    4559 patch_prober.go:29] interesting pod/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-v9zqm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" start-of-body=
Nov 02 12:53:49.017793 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:49.017762    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-v9zqm" podUID="ef76ad04-45eb-4b3b-b314-e93122bb18f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused"
Nov 02 12:53:49.053429 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:49.053395    4559 patch_prober.go:29] interesting pod/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-qdrs6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" start-of-body=
Nov 02 12:53:49.053600 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:49.053447    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-qdrs6" podUID="3aabb9b6-9382-42d9-8c52-4d89e361c25b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused"
Nov 02 12:53:50.020663 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:50.019775    4559 patch_prober.go:29] interesting pod/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-v9zqm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" start-of-body=
Nov 02 12:53:50.020663 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:50.019837    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-v9zqm" podUID="ef76ad04-45eb-4b3b-b314-e93122bb18f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused"
Nov 02 12:53:50.065665 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:50.037339    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ed56b9-5f957cb677-8j2rp" podStartSLOduration=4.645662061 podStartE2EDuration="5.037313972s" totalImagesPullingTime="391.651911ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:53:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:53:47.009909095 +0000 UTC m=+1623.265308192" watchObservedRunningTime="2026-11-02 12:53:50.037313972 +0000 UTC m=+1626.292713068"
Nov 02 12:53:50.068363 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:50.068328    4559 patch_prober.go:29] interesting pod/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-qdrs6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" start-of-body=
Nov 02 12:53:50.068622 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:50.068591    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-qdrs6" podUID="3aabb9b6-9382-42d9-8c52-4d89e361c25b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused"
Nov 02 12:53:50.090626 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:50.090588    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-8032f3-openshift-default-765d774fdc-mhf7j"]
Nov 02 12:53:50.152671 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:50.151930    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-8032f3-openshift-default-765d774fdc-mhf7j"
Nov 02 12:53:50.213663 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:50.186578    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-8032f3-openshift-default-765d774fdc-mhf7j"]
Nov 02 12:53:50.383014 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:50.223434    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d73ae6a-aaa7-4946-baa5-03ab6d91d403-credential-socket\") pod \"e2e-shared-a-w0-8032f3-openshift-default-765d774fdc-mhf7j\" (UID: \"5d73ae6a-aaa7-4946-baa5-03ab6d91d403\") " pod="openshift-ingress/e2e-shared-a-w0-8032f3-openshift-default-765d774fdc-mhf7j"
Nov 02 12:53:50.383489 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:50.383456    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5z47\" (UniqueName: \"kubernetes.io/projected/5d73ae6a-aaa7-4946-baa5-03ab6d91d403-kube-api-access-b5z47\") pod \"e2e-shared-a-w0-8032f3-openshift-default-765d774fdc-mhf7j\" (UID: \"5d73ae6a-aaa7-4946-baa5-03ab6d91d403\") " pod="openshift-ingress/e2e-shared-a-w0-8032f3-openshift-default-765d774fdc-mhf7j"
Nov 02 12:53:50.391701 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:50.225501    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-8032f3-openshift-default-dockercfg-9qpzf\""
Nov 02 12:53:50.391953 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:50.383743    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d73ae6a-aaa7-4946-baa5-03ab6d91d403-istio-token\") pod \"e2e-shared-a-w0-8032f3-openshift-default-765d774fdc-mhf7j\" (UID: \"5d73ae6a-aaa7-4946-baa5-03ab6d91d403\") " pod="openshift-ingress/e2e-shared-a-w0-8032f3-openshift-default-765d774fdc-mhf7j"
Nov 02 12:53:50.400851 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:50.400815    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d73ae6a-aaa7-4946-baa5-03ab6d91d403-istio-podinfo\") pod \"e2e-shared-a-w0-8032f3-openshift-default-765d774fdc-mhf7j\" (UID: \"5d73ae6a-aaa7-4946-baa5-03ab6d91d403\") " pod="openshift-ingress/e2e-shared-a-w0-8032f3-openshift-default-765d774fdc-mhf7j"
Nov 02 12:53:50.401230 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:50.401206    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d73ae6a-aaa7-4946-baa5-03ab6d91d403-istio-data\") pod \"e2e-shared-a-w0-8032f3-openshift-default-765d774fdc-mhf7j\" (UID: \"5d73ae6a-aaa7-4946-baa5-03ab6d91d403\") " pod="openshift-ingress/e2e-shared-a-w0-8032f3-openshift-default-765d774fdc-mhf7j"
Nov 02 12:53:50.401442 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:50.401421    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d73ae6a-aaa7-4946-baa5-03ab6d91d403-workload-certs\") pod \"e2e-shared-a-w0-8032f3-openshift-default-765d774fdc-mhf7j\" (UID: \"5d73ae6a-aaa7-4946-baa5-03ab6d91d403\") " pod="openshift-ingress/e2e-shared-a-w0-8032f3-openshift-default-765d774fdc-mhf7j"
Nov 02 12:53:50.405662 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:50.401714    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d73ae6a-aaa7-4946-baa5-03ab6d91d403-workload-socket\") pod \"e2e-shared-a-w0-8032f3-openshift-default-765d774fdc-mhf7j\" (UID: \"5d73ae6a-aaa7-4946-baa5-03ab6d91d403\") " pod="openshift-ingress/e2e-shared-a-w0-8032f3-openshift-default-765d774fdc-mhf7j"
Nov 02 12:53:50.406000 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:50.405978    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d73ae6a-aaa7-4946-baa5-03ab6d91d403-istiod-ca-cert\") pod \"e2e-shared-a-w0-8032f3-openshift-default-765d774fdc-mhf7j\" (UID: \"5d73ae6a-aaa7-4946-baa5-03ab6d91d403\") " pod="openshift-ingress/e2e-shared-a-w0-8032f3-openshift-default-765d774fdc-mhf7j"
Nov 02 12:53:50.413077 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:50.413045    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d73ae6a-aaa7-4946-baa5-03ab6d91d403-istio-envoy\") pod \"e2e-shared-a-w0-8032f3-openshift-default-765d774fdc-mhf7j\" (UID: \"5d73ae6a-aaa7-4946-baa5-03ab6d91d403\") " pod="openshift-ingress/e2e-shared-a-w0-8032f3-openshift-default-765d774fdc-mhf7j"
Nov 02 12:53:50.525668 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:50.517107    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d73ae6a-aaa7-4946-baa5-03ab6d91d403-credential-socket\") pod \"e2e-shared-a-w0-8032f3-openshift-default-765d774fdc-mhf7j\" (UID: \"5d73ae6a-aaa7-4946-baa5-03ab6d91d403\") " pod="openshift-ingress/e2e-shared-a-w0-8032f3-openshift-default-765d774fdc-mhf7j"
Nov 02 12:53:50.546823 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:50.526243    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d73ae6a-aaa7-4946-baa5-03ab6d91d403-credential-socket\") pod \"e2e-shared-a-w0-8032f3-openshift-default-765d774fdc-mhf7j\" (UID: \"5d73ae6a-aaa7-4946-baa5-03ab6d91d403\") " pod="openshift-ingress/e2e-shared-a-w0-8032f3-openshift-default-765d774fdc-mhf7j"
Nov 02 12:53:50.547056 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:50.546803    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5z47\" (UniqueName: \"kubernetes.io/projected/5d73ae6a-aaa7-4946-baa5-03ab6d91d403-kube-api-access-b5z47\") pod \"e2e-shared-a-w0-8032f3-openshift-default-765d774fdc-mhf7j\" (UID: \"5d73ae6a-aaa7-4946-baa5-03ab6d91d403\") " pod="openshift-ingress/e2e-shared-a-w0-8032f3-openshift-default-765d774fdc-mhf7j"
Nov 02 12:53:50.547056 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:50.546928    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d73ae6a-aaa7-4946-baa5-03ab6d91d403-istio-token\") pod \"e2e-shared-a-w0-8032f3-openshift-default-765d774fdc-mhf7j\" (UID: \"5d73ae6a-aaa7-4946-baa5-03ab6d91d403\") " pod="openshift-ingress/e2e-shared-a-w0-8032f3-openshift-default-765d774fdc-mhf7j"
Nov 02 12:53:50.547299 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:50.547276    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d73ae6a-aaa7-4946-baa5-03ab6d91d403-istio-podinfo\") pod \"e2e-shared-a-w0-8032f3-openshift-default-765d774fdc-mhf7j\" (UID: \"5d73ae6a-aaa7-4946-baa5-03ab6d91d403\") " pod="openshift-ingress/e2e-shared-a-w0-8032f3-openshift-default-765d774fdc-mhf7j"
Nov 02 12:53:50.547388 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:50.547371    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d73ae6a-aaa7-4946-baa5-03ab6d91d403-istio-data\") pod \"e2e-shared-a-w0-8032f3-openshift-default-765d774fdc-mhf7j\" (UID: \"5d73ae6a-aaa7-4946-baa5-03ab6d91d403\") " pod="openshift-ingress/e2e-shared-a-w0-8032f3-openshift-default-765d774fdc-mhf7j"
Nov 02 12:53:50.547452 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:50.547421    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d73ae6a-aaa7-4946-baa5-03ab6d91d403-workload-certs\") pod \"e2e-shared-a-w0-8032f3-openshift-default-765d774fdc-mhf7j\" (UID: \"5d73ae6a-aaa7-4946-baa5-03ab6d91d403\") " pod="openshift-ingress/e2e-shared-a-w0-8032f3-openshift-default-765d774fdc-mhf7j"
Nov 02 12:53:50.547510 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:50.547498    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d73ae6a-aaa7-4946-baa5-03ab6d91d403-workload-socket\") pod \"e2e-shared-a-w0-8032f3-openshift-default-765d774fdc-mhf7j\" (UID: \"5d73ae6a-aaa7-4946-baa5-03ab6d91d403\") " pod="openshift-ingress/e2e-shared-a-w0-8032f3-openshift-default-765d774fdc-mhf7j"
Nov 02 12:53:50.547772 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:50.547717    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d73ae6a-aaa7-4946-baa5-03ab6d91d403-istio-data\") pod \"e2e-shared-a-w0-8032f3-openshift-default-765d774fdc-mhf7j\" (UID: \"5d73ae6a-aaa7-4946-baa5-03ab6d91d403\") " pod="openshift-ingress/e2e-shared-a-w0-8032f3-openshift-default-765d774fdc-mhf7j"
Nov 02 12:53:50.547887 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:50.547814    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d73ae6a-aaa7-4946-baa5-03ab6d91d403-workload-certs\") pod \"e2e-shared-a-w0-8032f3-openshift-default-765d774fdc-mhf7j\" (UID: \"5d73ae6a-aaa7-4946-baa5-03ab6d91d403\") " pod="openshift-ingress/e2e-shared-a-w0-8032f3-openshift-default-765d774fdc-mhf7j"
Nov 02 12:53:50.548078 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:50.548037    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d73ae6a-aaa7-4946-baa5-03ab6d91d403-istiod-ca-cert\") pod \"e2e-shared-a-w0-8032f3-openshift-default-765d774fdc-mhf7j\" (UID: \"5d73ae6a-aaa7-4946-baa5-03ab6d91d403\") " pod="openshift-ingress/e2e-shared-a-w0-8032f3-openshift-default-765d774fdc-mhf7j"
Nov 02 12:53:50.548164 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:50.548084    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d73ae6a-aaa7-4946-baa5-03ab6d91d403-workload-socket\") pod \"e2e-shared-a-w0-8032f3-openshift-default-765d774fdc-mhf7j\" (UID: \"5d73ae6a-aaa7-4946-baa5-03ab6d91d403\") " pod="openshift-ingress/e2e-shared-a-w0-8032f3-openshift-default-765d774fdc-mhf7j"
Nov 02 12:53:50.548164 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:50.548096    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d73ae6a-aaa7-4946-baa5-03ab6d91d403-istio-envoy\") pod \"e2e-shared-a-w0-8032f3-openshift-default-765d774fdc-mhf7j\" (UID: \"5d73ae6a-aaa7-4946-baa5-03ab6d91d403\") " pod="openshift-ingress/e2e-shared-a-w0-8032f3-openshift-default-765d774fdc-mhf7j"
Nov 02 12:53:50.548835 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:50.548815    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d73ae6a-aaa7-4946-baa5-03ab6d91d403-istiod-ca-cert\") pod \"e2e-shared-a-w0-8032f3-openshift-default-765d774fdc-mhf7j\" (UID: \"5d73ae6a-aaa7-4946-baa5-03ab6d91d403\") " pod="openshift-ingress/e2e-shared-a-w0-8032f3-openshift-default-765d774fdc-mhf7j"
Nov 02 12:53:50.551181 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:50.551159    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d73ae6a-aaa7-4946-baa5-03ab6d91d403-istio-envoy\") pod \"e2e-shared-a-w0-8032f3-openshift-default-765d774fdc-mhf7j\" (UID: \"5d73ae6a-aaa7-4946-baa5-03ab6d91d403\") " pod="openshift-ingress/e2e-shared-a-w0-8032f3-openshift-default-765d774fdc-mhf7j"
Nov 02 12:53:50.551298 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:50.551282    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d73ae6a-aaa7-4946-baa5-03ab6d91d403-istio-podinfo\") pod \"e2e-shared-a-w0-8032f3-openshift-default-765d774fdc-mhf7j\" (UID: \"5d73ae6a-aaa7-4946-baa5-03ab6d91d403\") " pod="openshift-ingress/e2e-shared-a-w0-8032f3-openshift-default-765d774fdc-mhf7j"
Nov 02 12:53:50.604307 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:50.604224    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d73ae6a-aaa7-4946-baa5-03ab6d91d403-istio-token\") pod \"e2e-shared-a-w0-8032f3-openshift-default-765d774fdc-mhf7j\" (UID: \"5d73ae6a-aaa7-4946-baa5-03ab6d91d403\") " pod="openshift-ingress/e2e-shared-a-w0-8032f3-openshift-default-765d774fdc-mhf7j"
Nov 02 12:53:50.641390 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:50.641052    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5z47\" (UniqueName: \"kubernetes.io/projected/5d73ae6a-aaa7-4946-baa5-03ab6d91d403-kube-api-access-b5z47\") pod \"e2e-shared-a-w0-8032f3-openshift-default-765d774fdc-mhf7j\" (UID: \"5d73ae6a-aaa7-4946-baa5-03ab6d91d403\") " pod="openshift-ingress/e2e-shared-a-w0-8032f3-openshift-default-765d774fdc-mhf7j"
Nov 02 12:53:50.677663 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:50.670627    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-8032f3-openshift-default-765d774fdc-mhf7j"
Nov 02 12:53:51.024665 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:51.020168    4559 patch_prober.go:29] interesting pod/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-v9zqm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" start-of-body=
Nov 02 12:53:51.024665 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:51.020234    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-v9zqm" podUID="ef76ad04-45eb-4b3b-b314-e93122bb18f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused"
Nov 02 12:53:51.078665 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:51.073871    4559 patch_prober.go:29] interesting pod/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-qdrs6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" start-of-body=
Nov 02 12:53:51.078665 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:51.073982    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-qdrs6" podUID="3aabb9b6-9382-42d9-8c52-4d89e361c25b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused"
Nov 02 12:53:51.471671 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:51.460593    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-8032f3-openshift-default-765d774fdc-mhf7j"]
Nov 02 12:53:51.507569 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:51.507525    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:53:51.507862 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:51.507838    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:53:51.508004 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:51.507987    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:53:52.015136 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:52.015020    4559 patch_prober.go:29] interesting pod/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-v9zqm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" start-of-body=
Nov 02 12:53:52.015136 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:52.015085    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-v9zqm" podUID="ef76ad04-45eb-4b3b-b314-e93122bb18f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused"
Nov 02 12:53:52.053227 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:52.053179    4559 patch_prober.go:29] interesting pod/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-qdrs6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" start-of-body=
Nov 02 12:53:52.053767 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:52.053251    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-qdrs6" podUID="3aabb9b6-9382-42d9-8c52-4d89e361c25b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused"
Nov 02 12:53:52.156653 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:52.156614    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-8032f3-openshift-default-765d774fdc-mhf7j" event={"ID":"5d73ae6a-aaa7-4946-baa5-03ab6d91d403","Type":"ContainerStarted","Data":"6d770e61b37efd4981a0b48fd44b029c55d19eb0c7dd7b8fb5ce75a89a9b3034"}
Nov 02 12:53:52.156863 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:52.156680    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-8032f3-openshift-default-765d774fdc-mhf7j" event={"ID":"5d73ae6a-aaa7-4946-baa5-03ab6d91d403","Type":"ContainerStarted","Data":"31c052c90dfeeb7b01d5bc09963bd57e26812fd159d35390bf98573ddd489d77"}
Nov 02 12:53:52.219665 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:52.218680    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-8032f3-openshift-default-765d774fdc-mhf7j" podStartSLOduration=3.2186605090000002 podStartE2EDuration="3.218660509s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:53:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:53:52.215592723 +0000 UTC m=+1628.470991818" watchObservedRunningTime="2026-11-02 12:53:52.218660509 +0000 UTC m=+1628.474059603"
Nov 02 12:53:52.782164 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:52.782134    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-8032f3-openshift-default-765d774fdc-mhf7j"
Nov 02 12:53:52.800772 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:52.800736    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-8032f3-openshift-default-765d774fdc-mhf7j"
Nov 02 12:53:53.168553 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:53.168515    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-8032f3-openshift-default-765d774fdc-mhf7j"
Nov 02 12:53:53.169996 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:53.169971    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-8032f3-openshift-default-765d774fdc-mhf7j"
Nov 02 12:53:53.526116 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:53.525448    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-qdrs6"
Nov 02 12:53:53.526983 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:53.526961    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-qdrs6"
Nov 02 12:53:53.527865 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:53.527841    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-qdrs6"
Nov 02 12:53:53.549349 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:53.549008    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-v9zqm"
Nov 02 12:53:53.550041 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:53.550023    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-v9zqm"
Nov 02 12:53:53.550851 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:53.550829    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-v9zqm"
Nov 02 12:53:53.986660 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:53.982738    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ed56b9-5f957cb677-8j2rp"
Nov 02 12:53:57.377003 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:57.369448    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-8032f3-86595cb565-rrbnr"]
Nov 02 12:53:57.377003 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:57.376553    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-8032f3-86595cb565-rrbnr"
Nov 02 12:53:57.405344 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:57.405214    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w0-8032f3-dockercfg-lf775\""
Nov 02 12:53:57.405344 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:57.405317    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w0-8032f3\""
Nov 02 12:53:57.406778 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:57.406744    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-8032f3-86595cb565-rrbnr"]
Nov 02 12:53:57.472006 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:57.471975    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4cfq\" (UniqueName: \"kubernetes.io/projected/45b5f260-c257-485e-88bd-c87dbe6cb4d9-kube-api-access-x4cfq\") pod \"payload-processing-e2e-shared-a-w0-8032f3-86595cb565-rrbnr\" (UID: \"45b5f260-c257-485e-88bd-c87dbe6cb4d9\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-8032f3-86595cb565-rrbnr"
Nov 02 12:53:57.472006 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:57.472010    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45b5f260-c257-485e-88bd-c87dbe6cb4d9-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-8032f3-86595cb565-rrbnr\" (UID: \"45b5f260-c257-485e-88bd-c87dbe6cb4d9\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-8032f3-86595cb565-rrbnr"
Nov 02 12:53:57.573436 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:57.573396    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4cfq\" (UniqueName: \"kubernetes.io/projected/45b5f260-c257-485e-88bd-c87dbe6cb4d9-kube-api-access-x4cfq\") pod \"payload-processing-e2e-shared-a-w0-8032f3-86595cb565-rrbnr\" (UID: \"45b5f260-c257-485e-88bd-c87dbe6cb4d9\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-8032f3-86595cb565-rrbnr"
Nov 02 12:53:57.573661 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:57.573446    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45b5f260-c257-485e-88bd-c87dbe6cb4d9-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-8032f3-86595cb565-rrbnr\" (UID: \"45b5f260-c257-485e-88bd-c87dbe6cb4d9\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-8032f3-86595cb565-rrbnr"
Nov 02 12:53:57.574156 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:57.574132    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45b5f260-c257-485e-88bd-c87dbe6cb4d9-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-8032f3-86595cb565-rrbnr\" (UID: \"45b5f260-c257-485e-88bd-c87dbe6cb4d9\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-8032f3-86595cb565-rrbnr"
Nov 02 12:53:57.599153 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:57.599115    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4cfq\" (UniqueName: \"kubernetes.io/projected/45b5f260-c257-485e-88bd-c87dbe6cb4d9-kube-api-access-x4cfq\") pod \"payload-processing-e2e-shared-a-w0-8032f3-86595cb565-rrbnr\" (UID: \"45b5f260-c257-485e-88bd-c87dbe6cb4d9\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-8032f3-86595cb565-rrbnr"
Nov 02 12:53:57.708490 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:57.708456    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-8032f3-86595cb565-rrbnr"
Nov 02 12:53:58.028795 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:53:58.028766    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45b5f260_c257_485e_88bd_c87dbe6cb4d9.slice/crio-2f3b184e31e5b04d9c1289e8e4aedb29ac54b910fd0e58f1d4c2ff35c805f2ca WatchSource:0}: Error finding container 2f3b184e31e5b04d9c1289e8e4aedb29ac54b910fd0e58f1d4c2ff35c805f2ca: Status 404 returned error can't find the container with id 2f3b184e31e5b04d9c1289e8e4aedb29ac54b910fd0e58f1d4c2ff35c805f2ca
Nov 02 12:53:58.029575 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:58.029551    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-8032f3-86595cb565-rrbnr"]
Nov 02 12:53:58.395580 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:58.395476    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-8032f3-86595cb565-rrbnr" event={"ID":"45b5f260-c257-485e-88bd-c87dbe6cb4d9","Type":"ContainerStarted","Data":"f465c93f0524f69e3979c69ec7e5841f1b0d61460f3150d55188ebcb71a6f072"}
Nov 02 12:53:58.395580 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:58.395569    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-8032f3-86595cb565-rrbnr" event={"ID":"45b5f260-c257-485e-88bd-c87dbe6cb4d9","Type":"ContainerStarted","Data":"2f3b184e31e5b04d9c1289e8e4aedb29ac54b910fd0e58f1d4c2ff35c805f2ca"}
Nov 02 12:53:58.396106 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:58.395594    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-8032f3-86595cb565-rrbnr"
Nov 02 12:53:58.468084 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:58.468016    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-8032f3-86595cb565-rrbnr" podStartSLOduration=1.4679929139999999 podStartE2EDuration="1.467992914s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:53:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:53:58.444327881 +0000 UTC m=+1634.699726976" watchObservedRunningTime="2026-11-02 12:53:58.467992914 +0000 UTC m=+1634.723392010"
Nov 02 12:53:59.402625 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:53:59.402588    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"]
Nov 02 12:54:02.474533 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:02.473381    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843978/distinct-w4-843978-kserve-6fcf588858-4nrwx" podUID="82e5d483-e95b-43e6-9135-af960823427a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:54:02.474533 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:02.474022    4559 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-61e341/distinct-2-w1-61e341-kserve-954cf956f-n7jmg" podUID="c5e1a5f8-97b4-469d-be9a-eae7ca995283" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": context deadline exceeded"
Nov 02 12:54:02.509691 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:02.509629    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-61e341/distinct-w1-61e341-kserve-787cd566f4-cjj8s" podUID="6886ce28-5800-402d-b893-f92109e289a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:54:02.579198 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:02.579147    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-61e341/distinct-2-w1-61e341-kserve-954cf956f-n7jmg" podUID="c5e1a5f8-97b4-469d-be9a-eae7ca995283" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded"
Nov 02 12:54:03.091662 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:03.090795    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-61e341/premium-w1-61e341-kserve-746477b99-645g6" podUID="02ff21d3-5a37-493a-ab95-8e4784c29123" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:54:03.492665 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:03.490801    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843978/distinct-w4-843978-kserve-6fcf588858-4nrwx" podUID="82e5d483-e95b-43e6-9135-af960823427a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:54:03.512846 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:03.512801    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-61e341/distinct-w1-61e341-kserve-787cd566f4-cjj8s" podUID="6886ce28-5800-402d-b893-f92109e289a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Nov 02 12:54:03.514664 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:03.514623    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-843978/distinct-w4-843978-kserve-6fcf588858-4nrwx"
Nov 02 12:54:03.526127 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:03.526100    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-61e341/distinct-w1-61e341-kserve-787cd566f4-cjj8s"
Nov 02 12:54:04.082508 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:04.082473    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-61e341/distinct-w1-61e341-kserve-787cd566f4-cjj8s"
Nov 02 12:54:04.174544 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:04.172358    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-843978/distinct-w4-843978-kserve-6fcf588858-4nrwx"
Nov 02 12:54:08.536669 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:08.536625    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-1aa18c-openshift-default-7d7f44f669-xz5m4"]
Nov 02 12:54:08.542901 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:08.542876    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-1aa18c-openshift-default-7d7f44f669-xz5m4"
Nov 02 12:54:08.575036 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:08.574999    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-1aa18c-openshift-default-7d7f44f669-xz5m4"]
Nov 02 12:54:08.578982 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:08.575379    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-1aa18c-openshift-default-dockercfg-wk4h7\""
Nov 02 12:54:08.732663 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:08.725973    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8debb7bd-d249-47d9-8776-f2be8f0d4f27-istiod-ca-cert\") pod \"e2e-shared-b-w1-1aa18c-openshift-default-7d7f44f669-xz5m4\" (UID: \"8debb7bd-d249-47d9-8776-f2be8f0d4f27\") " pod="openshift-ingress/e2e-shared-b-w1-1aa18c-openshift-default-7d7f44f669-xz5m4"
Nov 02 12:54:08.732663 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:08.726052    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8debb7bd-d249-47d9-8776-f2be8f0d4f27-workload-certs\") pod \"e2e-shared-b-w1-1aa18c-openshift-default-7d7f44f669-xz5m4\" (UID: \"8debb7bd-d249-47d9-8776-f2be8f0d4f27\") " pod="openshift-ingress/e2e-shared-b-w1-1aa18c-openshift-default-7d7f44f669-xz5m4"
Nov 02 12:54:08.732663 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:08.726121    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8debb7bd-d249-47d9-8776-f2be8f0d4f27-istio-data\") pod \"e2e-shared-b-w1-1aa18c-openshift-default-7d7f44f669-xz5m4\" (UID: \"8debb7bd-d249-47d9-8776-f2be8f0d4f27\") " pod="openshift-ingress/e2e-shared-b-w1-1aa18c-openshift-default-7d7f44f669-xz5m4"
Nov 02 12:54:08.732663 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:08.726265    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8debb7bd-d249-47d9-8776-f2be8f0d4f27-istio-envoy\") pod \"e2e-shared-b-w1-1aa18c-openshift-default-7d7f44f669-xz5m4\" (UID: \"8debb7bd-d249-47d9-8776-f2be8f0d4f27\") " pod="openshift-ingress/e2e-shared-b-w1-1aa18c-openshift-default-7d7f44f669-xz5m4"
Nov 02 12:54:08.732663 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:08.726354    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8debb7bd-d249-47d9-8776-f2be8f0d4f27-credential-socket\") pod \"e2e-shared-b-w1-1aa18c-openshift-default-7d7f44f669-xz5m4\" (UID: \"8debb7bd-d249-47d9-8776-f2be8f0d4f27\") " pod="openshift-ingress/e2e-shared-b-w1-1aa18c-openshift-default-7d7f44f669-xz5m4"
Nov 02 12:54:08.732663 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:08.726406    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8debb7bd-d249-47d9-8776-f2be8f0d4f27-workload-socket\") pod \"e2e-shared-b-w1-1aa18c-openshift-default-7d7f44f669-xz5m4\" (UID: \"8debb7bd-d249-47d9-8776-f2be8f0d4f27\") " pod="openshift-ingress/e2e-shared-b-w1-1aa18c-openshift-default-7d7f44f669-xz5m4"
Nov 02 12:54:08.732663 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:08.726454    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lp29f\" (UniqueName: \"kubernetes.io/projected/8debb7bd-d249-47d9-8776-f2be8f0d4f27-kube-api-access-lp29f\") pod \"e2e-shared-b-w1-1aa18c-openshift-default-7d7f44f669-xz5m4\" (UID: \"8debb7bd-d249-47d9-8776-f2be8f0d4f27\") " pod="openshift-ingress/e2e-shared-b-w1-1aa18c-openshift-default-7d7f44f669-xz5m4"
Nov 02 12:54:08.732663 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:08.726497    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8debb7bd-d249-47d9-8776-f2be8f0d4f27-istio-podinfo\") pod \"e2e-shared-b-w1-1aa18c-openshift-default-7d7f44f669-xz5m4\" (UID: \"8debb7bd-d249-47d9-8776-f2be8f0d4f27\") " pod="openshift-ingress/e2e-shared-b-w1-1aa18c-openshift-default-7d7f44f669-xz5m4"
Nov 02 12:54:08.732663 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:08.726594    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8debb7bd-d249-47d9-8776-f2be8f0d4f27-istio-token\") pod \"e2e-shared-b-w1-1aa18c-openshift-default-7d7f44f669-xz5m4\" (UID: \"8debb7bd-d249-47d9-8776-f2be8f0d4f27\") " pod="openshift-ingress/e2e-shared-b-w1-1aa18c-openshift-default-7d7f44f669-xz5m4"
Nov 02 12:54:08.846659 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:08.828676    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8debb7bd-d249-47d9-8776-f2be8f0d4f27-istio-envoy\") pod \"e2e-shared-b-w1-1aa18c-openshift-default-7d7f44f669-xz5m4\" (UID: \"8debb7bd-d249-47d9-8776-f2be8f0d4f27\") " pod="openshift-ingress/e2e-shared-b-w1-1aa18c-openshift-default-7d7f44f669-xz5m4"
Nov 02 12:54:08.846659 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:08.828760    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8debb7bd-d249-47d9-8776-f2be8f0d4f27-credential-socket\") pod \"e2e-shared-b-w1-1aa18c-openshift-default-7d7f44f669-xz5m4\" (UID: \"8debb7bd-d249-47d9-8776-f2be8f0d4f27\") " pod="openshift-ingress/e2e-shared-b-w1-1aa18c-openshift-default-7d7f44f669-xz5m4"
Nov 02 12:54:08.846659 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:08.828799    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8debb7bd-d249-47d9-8776-f2be8f0d4f27-workload-socket\") pod \"e2e-shared-b-w1-1aa18c-openshift-default-7d7f44f669-xz5m4\" (UID: \"8debb7bd-d249-47d9-8776-f2be8f0d4f27\") " pod="openshift-ingress/e2e-shared-b-w1-1aa18c-openshift-default-7d7f44f669-xz5m4"
Nov 02 12:54:08.846659 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:08.828837    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lp29f\" (UniqueName: \"kubernetes.io/projected/8debb7bd-d249-47d9-8776-f2be8f0d4f27-kube-api-access-lp29f\") pod \"e2e-shared-b-w1-1aa18c-openshift-default-7d7f44f669-xz5m4\" (UID: \"8debb7bd-d249-47d9-8776-f2be8f0d4f27\") " pod="openshift-ingress/e2e-shared-b-w1-1aa18c-openshift-default-7d7f44f669-xz5m4"
Nov 02 12:54:08.846659 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:08.828871    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8debb7bd-d249-47d9-8776-f2be8f0d4f27-istio-podinfo\") pod \"e2e-shared-b-w1-1aa18c-openshift-default-7d7f44f669-xz5m4\" (UID: \"8debb7bd-d249-47d9-8776-f2be8f0d4f27\") " pod="openshift-ingress/e2e-shared-b-w1-1aa18c-openshift-default-7d7f44f669-xz5m4"
Nov 02 12:54:08.846659 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:08.828941    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8debb7bd-d249-47d9-8776-f2be8f0d4f27-istio-token\") pod \"e2e-shared-b-w1-1aa18c-openshift-default-7d7f44f669-xz5m4\" (UID: \"8debb7bd-d249-47d9-8776-f2be8f0d4f27\") " pod="openshift-ingress/e2e-shared-b-w1-1aa18c-openshift-default-7d7f44f669-xz5m4"
Nov 02 12:54:08.846659 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:08.829017    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8debb7bd-d249-47d9-8776-f2be8f0d4f27-istiod-ca-cert\") pod \"e2e-shared-b-w1-1aa18c-openshift-default-7d7f44f669-xz5m4\" (UID: \"8debb7bd-d249-47d9-8776-f2be8f0d4f27\") " pod="openshift-ingress/e2e-shared-b-w1-1aa18c-openshift-default-7d7f44f669-xz5m4"
Nov 02 12:54:08.846659 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:08.829052    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8debb7bd-d249-47d9-8776-f2be8f0d4f27-workload-certs\") pod \"e2e-shared-b-w1-1aa18c-openshift-default-7d7f44f669-xz5m4\" (UID: \"8debb7bd-d249-47d9-8776-f2be8f0d4f27\") " pod="openshift-ingress/e2e-shared-b-w1-1aa18c-openshift-default-7d7f44f669-xz5m4"
Nov 02 12:54:08.846659 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:08.829102    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8debb7bd-d249-47d9-8776-f2be8f0d4f27-istio-data\") pod \"e2e-shared-b-w1-1aa18c-openshift-default-7d7f44f669-xz5m4\" (UID: \"8debb7bd-d249-47d9-8776-f2be8f0d4f27\") " pod="openshift-ingress/e2e-shared-b-w1-1aa18c-openshift-default-7d7f44f669-xz5m4"
Nov 02 12:54:08.846659 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:08.829666    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8debb7bd-d249-47d9-8776-f2be8f0d4f27-istio-data\") pod \"e2e-shared-b-w1-1aa18c-openshift-default-7d7f44f669-xz5m4\" (UID: \"8debb7bd-d249-47d9-8776-f2be8f0d4f27\") " pod="openshift-ingress/e2e-shared-b-w1-1aa18c-openshift-default-7d7f44f669-xz5m4"
Nov 02 12:54:08.846659 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:08.840133    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8debb7bd-d249-47d9-8776-f2be8f0d4f27-istio-envoy\") pod \"e2e-shared-b-w1-1aa18c-openshift-default-7d7f44f669-xz5m4\" (UID: \"8debb7bd-d249-47d9-8776-f2be8f0d4f27\") " pod="openshift-ingress/e2e-shared-b-w1-1aa18c-openshift-default-7d7f44f669-xz5m4"
Nov 02 12:54:08.846659 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:08.840387    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8debb7bd-d249-47d9-8776-f2be8f0d4f27-credential-socket\") pod \"e2e-shared-b-w1-1aa18c-openshift-default-7d7f44f669-xz5m4\" (UID: \"8debb7bd-d249-47d9-8776-f2be8f0d4f27\") " pod="openshift-ingress/e2e-shared-b-w1-1aa18c-openshift-default-7d7f44f669-xz5m4"
Nov 02 12:54:08.846659 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:08.840584    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8debb7bd-d249-47d9-8776-f2be8f0d4f27-workload-socket\") pod \"e2e-shared-b-w1-1aa18c-openshift-default-7d7f44f669-xz5m4\" (UID: \"8debb7bd-d249-47d9-8776-f2be8f0d4f27\") " pod="openshift-ingress/e2e-shared-b-w1-1aa18c-openshift-default-7d7f44f669-xz5m4"
Nov 02 12:54:08.846659 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:08.843939    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8debb7bd-d249-47d9-8776-f2be8f0d4f27-istiod-ca-cert\") pod \"e2e-shared-b-w1-1aa18c-openshift-default-7d7f44f669-xz5m4\" (UID: \"8debb7bd-d249-47d9-8776-f2be8f0d4f27\") " pod="openshift-ingress/e2e-shared-b-w1-1aa18c-openshift-default-7d7f44f669-xz5m4"
Nov 02 12:54:08.846659 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:08.844175    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8debb7bd-d249-47d9-8776-f2be8f0d4f27-workload-certs\") pod \"e2e-shared-b-w1-1aa18c-openshift-default-7d7f44f669-xz5m4\" (UID: \"8debb7bd-d249-47d9-8776-f2be8f0d4f27\") " pod="openshift-ingress/e2e-shared-b-w1-1aa18c-openshift-default-7d7f44f669-xz5m4"
Nov 02 12:54:08.891287 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:08.891209    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8debb7bd-d249-47d9-8776-f2be8f0d4f27-istio-podinfo\") pod \"e2e-shared-b-w1-1aa18c-openshift-default-7d7f44f669-xz5m4\" (UID: \"8debb7bd-d249-47d9-8776-f2be8f0d4f27\") " pod="openshift-ingress/e2e-shared-b-w1-1aa18c-openshift-default-7d7f44f669-xz5m4"
Nov 02 12:54:08.894563 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:08.894462    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lp29f\" (UniqueName: \"kubernetes.io/projected/8debb7bd-d249-47d9-8776-f2be8f0d4f27-kube-api-access-lp29f\") pod \"e2e-shared-b-w1-1aa18c-openshift-default-7d7f44f669-xz5m4\" (UID: \"8debb7bd-d249-47d9-8776-f2be8f0d4f27\") " pod="openshift-ingress/e2e-shared-b-w1-1aa18c-openshift-default-7d7f44f669-xz5m4"
Nov 02 12:54:08.911854 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:08.904925    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8debb7bd-d249-47d9-8776-f2be8f0d4f27-istio-token\") pod \"e2e-shared-b-w1-1aa18c-openshift-default-7d7f44f669-xz5m4\" (UID: \"8debb7bd-d249-47d9-8776-f2be8f0d4f27\") " pod="openshift-ingress/e2e-shared-b-w1-1aa18c-openshift-default-7d7f44f669-xz5m4"
Nov 02 12:54:09.171908 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:09.171875    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-1aa18c-openshift-default-7d7f44f669-xz5m4"
Nov 02 12:54:09.482193 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:09.482102    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-8032f3-86595cb565-rrbnr"
Nov 02 12:54:09.539083 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:09.539041    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:54:09.539509 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:09.539136    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:54:09.539509 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:09.539181    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:54:09.542129 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:09.542108    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-1aa18c-openshift-default-7d7f44f669-xz5m4"]
Nov 02 12:54:09.901372 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:09.901281    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-1aa18c-openshift-default-7d7f44f669-xz5m4" event={"ID":"8debb7bd-d249-47d9-8776-f2be8f0d4f27","Type":"ContainerStarted","Data":"66a2714cf5cd54c62bed7a6eb34297b3139d6f16a0ad9d88a2404ca6ef03071e"}
Nov 02 12:54:09.901372 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:09.901343    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-1aa18c-openshift-default-7d7f44f669-xz5m4" event={"ID":"8debb7bd-d249-47d9-8776-f2be8f0d4f27","Type":"ContainerStarted","Data":"ba55c7015f5e24ddbd03269a9ed034111edb4193bb60d2051785143a169e76e5"}
Nov 02 12:54:10.178974 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:10.178174    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-1aa18c-openshift-default-7d7f44f669-xz5m4"
Nov 02 12:54:10.179287 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:10.179262    4559 patch_prober.go:29] interesting pod/e2e-shared-b-w1-1aa18c-openshift-default-7d7f44f669-xz5m4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" start-of-body=
Nov 02 12:54:10.179452 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:10.179428    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-1aa18c-openshift-default-7d7f44f669-xz5m4" podUID="8debb7bd-d249-47d9-8776-f2be8f0d4f27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused"
Nov 02 12:54:11.178009 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:11.177974    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-1aa18c-openshift-default-7d7f44f669-xz5m4"
Nov 02 12:54:11.249733 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:11.249675    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-1aa18c-openshift-default-7d7f44f669-xz5m4" podStartSLOduration=3.249656243 podStartE2EDuration="3.249656243s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:54:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:54:09.96807045 +0000 UTC m=+1646.223469544" watchObservedRunningTime="2026-11-02 12:54:11.249656243 +0000 UTC m=+1647.505055340"
Nov 02 12:54:11.916929 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:11.916894    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-1aa18c-openshift-default-7d7f44f669-xz5m4"
Nov 02 12:54:11.917861 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:11.917839    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-1aa18c-openshift-default-7d7f44f669-xz5m4"
Nov 02 12:54:12.527818 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:12.527777    4559 patch_prober.go:29] interesting pod/e2e-shared-a-w1-a6f2c0-openshift-default-7f47858fbf-x45sj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.178:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 12:54:12.528325 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:12.527856    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w1-a6f2c0-openshift-default-7f47858fbf-x45sj" podUID="27dd21d8-102d-45c3-8e54-3c1e846829de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.178:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:54:12.554830 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:12.554783    4559 patch_prober.go:29] interesting pod/e2e-shared-a-w2-ed56b9-openshift-default-789859d697-5f5lv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 12:54:12.554981 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:12.554874    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-ed56b9-openshift-default-789859d697-5f5lv" podUID="abd3e462-db3a-4c9a-9fbc-1b5cb3550ef9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:54:14.118022 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:14.028602    4559 generic.go:356] "Generic (PLEG): container finished" podID="abd3e462-db3a-4c9a-9fbc-1b5cb3550ef9" containerID="087d74e1c8b7ed7ed4a40876cfb5f43cc55df856350c1acc8542fd8efadee767" exitCode=0
Nov 02 12:54:14.118022 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:14.028677    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-ed56b9-openshift-default-789859d697-5f5lv" event={"ID":"abd3e462-db3a-4c9a-9fbc-1b5cb3550ef9","Type":"ContainerDied","Data":"087d74e1c8b7ed7ed4a40876cfb5f43cc55df856350c1acc8542fd8efadee767"}
Nov 02 12:54:14.118022 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:14.029403    4559 scope.go:122] "RemoveContainer" containerID="087d74e1c8b7ed7ed4a40876cfb5f43cc55df856350c1acc8542fd8efadee767"
Nov 02 12:54:14.118022 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:14.031107    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:54:14.118022 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:14.031172    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:54:14.118022 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:14.031206    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:54:14.541787 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:14.541753    4559 patch_prober.go:29] interesting pod/e2e-shared-a-w3-e7561a-openshift-default-7598f89b79-m5rrm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 12:54:14.542050 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:14.541806    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-e7561a-openshift-default-7598f89b79-m5rrm" podUID="4293ef7c-ce26-475f-98e1-ebf007c35f00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:54:15.114667 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:15.093998    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-ed56b9-openshift-default-789859d697-5f5lv" event={"ID":"abd3e462-db3a-4c9a-9fbc-1b5cb3550ef9","Type":"ContainerStarted","Data":"fc7c8f5f7a49d000b0f3bfcd56620a16dac5d7e80d0f3a450144f5c9a4196cdc"}
Nov 02 12:54:15.306622 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:15.264912    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-8032f3-74cb7cfb7d-56656"]
Nov 02 12:54:15.306622 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:54:15.280989    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqcmj" containerName="extractor"
Nov 02 12:54:15.306622 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:15.306465    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-8032f3-74cb7cfb7d-56656"]
Nov 02 12:54:15.513369 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:15.306628    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-8032f3-74cb7cfb7d-56656"
Nov 02 12:54:15.513369 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:15.320101    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-ed56b9-openshift-default-789859d697-5f5lv"
Nov 02 12:54:15.513369 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:15.320877    4559 patch_prober.go:29] interesting pod/e2e-shared-a-w2-ed56b9-openshift-default-789859d697-5f5lv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" start-of-body=
Nov 02 12:54:15.513369 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:15.320964    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-ed56b9-openshift-default-789859d697-5f5lv" podUID="abd3e462-db3a-4c9a-9fbc-1b5cb3550ef9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused"
Nov 02 12:54:15.513369 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:15.323727    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w0-8032f3-serving-cert\""
Nov 02 12:54:15.513369 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:15.375393    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b66c2e1-91f4-4204-be78-73c05fc42258-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-8032f3-74cb7cfb7d-56656\" (UID: \"8b66c2e1-91f4-4204-be78-73c05fc42258\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-8032f3-74cb7cfb7d-56656"
Nov 02 12:54:15.513369 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:15.375439    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b66c2e1-91f4-4204-be78-73c05fc42258-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-8032f3-74cb7cfb7d-56656\" (UID: \"8b66c2e1-91f4-4204-be78-73c05fc42258\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-8032f3-74cb7cfb7d-56656"
Nov 02 12:54:15.513369 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:15.375559    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q65nn\" (UniqueName: \"kubernetes.io/projected/8b66c2e1-91f4-4204-be78-73c05fc42258-kube-api-access-q65nn\") pod \"maas-api-e2e-shared-a-w0-8032f3-74cb7cfb7d-56656\" (UID: \"8b66c2e1-91f4-4204-be78-73c05fc42258\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-8032f3-74cb7cfb7d-56656"
Nov 02 12:54:15.513369 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:15.375720    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b66c2e1-91f4-4204-be78-73c05fc42258-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-8032f3-74cb7cfb7d-56656\" (UID: \"8b66c2e1-91f4-4204-be78-73c05fc42258\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-8032f3-74cb7cfb7d-56656"
Nov 02 12:54:15.513369 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:15.477802    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b66c2e1-91f4-4204-be78-73c05fc42258-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-8032f3-74cb7cfb7d-56656\" (UID: \"8b66c2e1-91f4-4204-be78-73c05fc42258\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-8032f3-74cb7cfb7d-56656"
Nov 02 12:54:15.513369 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:15.477848    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b66c2e1-91f4-4204-be78-73c05fc42258-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-8032f3-74cb7cfb7d-56656\" (UID: \"8b66c2e1-91f4-4204-be78-73c05fc42258\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-8032f3-74cb7cfb7d-56656"
Nov 02 12:54:15.513369 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:15.477889    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q65nn\" (UniqueName: \"kubernetes.io/projected/8b66c2e1-91f4-4204-be78-73c05fc42258-kube-api-access-q65nn\") pod \"maas-api-e2e-shared-a-w0-8032f3-74cb7cfb7d-56656\" (UID: \"8b66c2e1-91f4-4204-be78-73c05fc42258\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-8032f3-74cb7cfb7d-56656"
Nov 02 12:54:15.513369 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:15.477961    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b66c2e1-91f4-4204-be78-73c05fc42258-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-8032f3-74cb7cfb7d-56656\" (UID: \"8b66c2e1-91f4-4204-be78-73c05fc42258\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-8032f3-74cb7cfb7d-56656"
Nov 02 12:54:15.513369 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:15.490607    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b66c2e1-91f4-4204-be78-73c05fc42258-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-8032f3-74cb7cfb7d-56656\" (UID: \"8b66c2e1-91f4-4204-be78-73c05fc42258\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-8032f3-74cb7cfb7d-56656"
Nov 02 12:54:15.513369 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:15.501678    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b66c2e1-91f4-4204-be78-73c05fc42258-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-8032f3-74cb7cfb7d-56656\" (UID: \"8b66c2e1-91f4-4204-be78-73c05fc42258\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-8032f3-74cb7cfb7d-56656"
Nov 02 12:54:15.513369 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:15.511841    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b66c2e1-91f4-4204-be78-73c05fc42258-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-8032f3-74cb7cfb7d-56656\" (UID: \"8b66c2e1-91f4-4204-be78-73c05fc42258\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-8032f3-74cb7cfb7d-56656"
Nov 02 12:54:15.530721 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:15.530574    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q65nn\" (UniqueName: \"kubernetes.io/projected/8b66c2e1-91f4-4204-be78-73c05fc42258-kube-api-access-q65nn\") pod \"maas-api-e2e-shared-a-w0-8032f3-74cb7cfb7d-56656\" (UID: \"8b66c2e1-91f4-4204-be78-73c05fc42258\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-8032f3-74cb7cfb7d-56656"
Nov 02 12:54:15.649330 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:15.649295    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-8032f3-74cb7cfb7d-56656"
Nov 02 12:54:16.361375 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:54:16.042372    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b66c2e1_91f4_4204_be78_73c05fc42258.slice/crio-84ddd7e55f6200462d42b8dfbc26e3ebd549695d5e1d301462b8008b7492a51d WatchSource:0}: Error finding container 84ddd7e55f6200462d42b8dfbc26e3ebd549695d5e1d301462b8008b7492a51d: Status 404 returned error can't find the container with id 84ddd7e55f6200462d42b8dfbc26e3ebd549695d5e1d301462b8008b7492a51d
Nov 02 12:54:16.361375 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:16.042519    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-8032f3-74cb7cfb7d-56656"]
Nov 02 12:54:16.361375 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:16.043932    4559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 12:54:16.361375 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:16.205827    4559 generic.go:356] "Generic (PLEG): container finished" podID="4293ef7c-ce26-475f-98e1-ebf007c35f00" containerID="ccbfe98cbcb6cee4011b7e65b5332a6b8f935dc779efd6365f629dd39731a796" exitCode=0
Nov 02 12:54:16.361375 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:16.206030    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-e7561a-openshift-default-7598f89b79-m5rrm" event={"ID":"4293ef7c-ce26-475f-98e1-ebf007c35f00","Type":"ContainerDied","Data":"ccbfe98cbcb6cee4011b7e65b5332a6b8f935dc779efd6365f629dd39731a796"}
Nov 02 12:54:16.361375 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:16.207025    4559 scope.go:122] "RemoveContainer" containerID="ccbfe98cbcb6cee4011b7e65b5332a6b8f935dc779efd6365f629dd39731a796"
Nov 02 12:54:16.361375 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:16.210082    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:54:16.361375 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:16.210180    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:54:16.361375 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:16.210271    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:54:16.361375 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:16.230512    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-8032f3-74cb7cfb7d-56656" event={"ID":"8b66c2e1-91f4-4204-be78-73c05fc42258","Type":"ContainerStarted","Data":"84ddd7e55f6200462d42b8dfbc26e3ebd549695d5e1d301462b8008b7492a51d"}
Nov 02 12:54:16.361375 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:16.314619    4559 patch_prober.go:29] interesting pod/e2e-shared-a-w2-ed56b9-openshift-default-789859d697-5f5lv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" start-of-body=
Nov 02 12:54:16.361375 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:16.314694    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-ed56b9-openshift-default-789859d697-5f5lv" podUID="abd3e462-db3a-4c9a-9fbc-1b5cb3550ef9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused"
Nov 02 12:54:17.353875 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:17.353832    4559 patch_prober.go:29] interesting pod/e2e-shared-a-w2-ed56b9-openshift-default-789859d697-5f5lv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" start-of-body=
Nov 02 12:54:17.354163 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:17.354132    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-ed56b9-openshift-default-789859d697-5f5lv" podUID="abd3e462-db3a-4c9a-9fbc-1b5cb3550ef9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused"
Nov 02 12:54:17.410664 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:17.407419    4559 generic.go:356] "Generic (PLEG): container finished" podID="27dd21d8-102d-45c3-8e54-3c1e846829de" containerID="214bc758f94617b4bb60976c80c20ad95dc5e3249685f127b36f6acc044ddbbf" exitCode=0
Nov 02 12:54:17.410664 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:17.407555    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-a6f2c0-openshift-default-7f47858fbf-x45sj" event={"ID":"27dd21d8-102d-45c3-8e54-3c1e846829de","Type":"ContainerDied","Data":"214bc758f94617b4bb60976c80c20ad95dc5e3249685f127b36f6acc044ddbbf"}
Nov 02 12:54:17.410664 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:17.408146    4559 scope.go:122] "RemoveContainer" containerID="214bc758f94617b4bb60976c80c20ad95dc5e3249685f127b36f6acc044ddbbf"
Nov 02 12:54:17.463727 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:17.447829    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:54:17.463727 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:17.447947    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:54:17.463727 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:17.447998    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:54:17.558752 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:17.558632    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-e7561a-openshift-default-7598f89b79-m5rrm" event={"ID":"4293ef7c-ce26-475f-98e1-ebf007c35f00","Type":"ContainerStarted","Data":"19b2b5345b9ea13a7075f8ba886ab0f6af8f2e3a48dc9e42611fd54bd9467bf2"}
Nov 02 12:54:17.562699 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:17.562611    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-8032f3-74cb7cfb7d-56656" event={"ID":"8b66c2e1-91f4-4204-be78-73c05fc42258","Type":"ContainerStarted","Data":"609e0c4adfd669ba77b10ea7556750ca4e78e3e8a8b59fbf40fd746e4c3489f8"}
Nov 02 12:54:17.563566 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:17.563543    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-8032f3-74cb7cfb7d-56656"
Nov 02 12:54:17.917668 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:17.914124    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-8032f3-74cb7cfb7d-56656" podStartSLOduration=2.509211317 podStartE2EDuration="2.914102594s" totalImagesPullingTime="404.891277ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:54:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:54:17.744822394 +0000 UTC m=+1654.000221491" watchObservedRunningTime="2026-11-02 12:54:17.914102594 +0000 UTC m=+1654.169501690"
Nov 02 12:54:18.327666 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:18.323397    4559 patch_prober.go:29] interesting pod/e2e-shared-a-w2-ed56b9-openshift-default-789859d697-5f5lv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" start-of-body=
Nov 02 12:54:18.327666 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:18.323465    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-ed56b9-openshift-default-789859d697-5f5lv" podUID="abd3e462-db3a-4c9a-9fbc-1b5cb3550ef9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused"
Nov 02 12:54:18.378665 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:18.378615    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-e7561a-openshift-default-7598f89b79-m5rrm"
Nov 02 12:54:18.379251 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:18.379123    4559 patch_prober.go:29] interesting pod/e2e-shared-a-w3-e7561a-openshift-default-7598f89b79-m5rrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body=
Nov 02 12:54:18.379376 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:18.379281    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-e7561a-openshift-default-7598f89b79-m5rrm" podUID="4293ef7c-ce26-475f-98e1-ebf007c35f00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused"
Nov 02 12:54:18.571995 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:18.571951    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-a6f2c0-openshift-default-7f47858fbf-x45sj" event={"ID":"27dd21d8-102d-45c3-8e54-3c1e846829de","Type":"ContainerStarted","Data":"d8d8080416b5f46c65cb189efafd288d2033b0c353dbc7c7e54ba0319aaba737"}
Nov 02 12:54:18.729108 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:18.728967    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-a6f2c0-openshift-default-7f47858fbf-x45sj"
Nov 02 12:54:19.323347 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:19.323309    4559 patch_prober.go:29] interesting pod/e2e-shared-a-w2-ed56b9-openshift-default-789859d697-5f5lv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" start-of-body=
Nov 02 12:54:19.324061 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:19.323613    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-ed56b9-openshift-default-789859d697-5f5lv" podUID="abd3e462-db3a-4c9a-9fbc-1b5cb3550ef9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused"
Nov 02 12:54:19.379289 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:19.379257    4559 patch_prober.go:29] interesting pod/e2e-shared-a-w3-e7561a-openshift-default-7598f89b79-m5rrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body=
Nov 02 12:54:19.379495 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:19.379312    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-e7561a-openshift-default-7598f89b79-m5rrm" podUID="4293ef7c-ce26-475f-98e1-ebf007c35f00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused"
Nov 02 12:54:19.725016 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:19.724978    4559 patch_prober.go:29] interesting pod/e2e-shared-a-w1-a6f2c0-openshift-default-7f47858fbf-x45sj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" start-of-body=
Nov 02 12:54:19.725661 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:19.725603    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-a6f2c0-openshift-default-7f47858fbf-x45sj" podUID="27dd21d8-102d-45c3-8e54-3c1e846829de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused"
Nov 02 12:54:20.315099 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:20.315060    4559 patch_prober.go:29] interesting pod/e2e-shared-a-w2-ed56b9-openshift-default-789859d697-5f5lv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" start-of-body=
Nov 02 12:54:20.315398 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:20.315365    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-ed56b9-openshift-default-789859d697-5f5lv" podUID="abd3e462-db3a-4c9a-9fbc-1b5cb3550ef9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused"
Nov 02 12:54:20.378713 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:20.378674    4559 patch_prober.go:29] interesting pod/e2e-shared-a-w3-e7561a-openshift-default-7598f89b79-m5rrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body=
Nov 02 12:54:20.378891 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:20.378755    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-e7561a-openshift-default-7598f89b79-m5rrm" podUID="4293ef7c-ce26-475f-98e1-ebf007c35f00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused"
Nov 02 12:54:20.724663 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:20.723830    4559 patch_prober.go:29] interesting pod/e2e-shared-a-w1-a6f2c0-openshift-default-7f47858fbf-x45sj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" start-of-body=
Nov 02 12:54:20.724663 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:20.723899    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-a6f2c0-openshift-default-7f47858fbf-x45sj" podUID="27dd21d8-102d-45c3-8e54-3c1e846829de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused"
Nov 02 12:54:21.378763 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:21.378723    4559 patch_prober.go:29] interesting pod/e2e-shared-a-w3-e7561a-openshift-default-7598f89b79-m5rrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body=
Nov 02 12:54:21.379256 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:21.378778    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-e7561a-openshift-default-7598f89b79-m5rrm" podUID="4293ef7c-ce26-475f-98e1-ebf007c35f00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused"
Nov 02 12:54:21.472269 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:21.472183    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-ed56b9-openshift-default-789859d697-5f5lv"
Nov 02 12:54:21.472782 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:21.472756    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-ed56b9-openshift-default-789859d697-5f5lv"
Nov 02 12:54:21.473659 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:21.473624    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-ed56b9-openshift-default-789859d697-5f5lv"
Nov 02 12:54:21.772543 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:21.755601    4559 patch_prober.go:29] interesting pod/e2e-shared-a-w1-a6f2c0-openshift-default-7f47858fbf-x45sj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" start-of-body=
Nov 02 12:54:21.772543 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:21.755694    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-a6f2c0-openshift-default-7f47858fbf-x45sj" podUID="27dd21d8-102d-45c3-8e54-3c1e846829de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused"
Nov 02 12:54:22.380357 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:22.380320    4559 patch_prober.go:29] interesting pod/e2e-shared-a-w3-e7561a-openshift-default-7598f89b79-m5rrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body=
Nov 02 12:54:22.380788 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:22.380377    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-e7561a-openshift-default-7598f89b79-m5rrm" podUID="4293ef7c-ce26-475f-98e1-ebf007c35f00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused"
Nov 02 12:54:22.743516 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:22.743465    4559 patch_prober.go:29] interesting pod/e2e-shared-a-w1-a6f2c0-openshift-default-7f47858fbf-x45sj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" start-of-body=
Nov 02 12:54:22.743844 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:22.743799    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-a6f2c0-openshift-default-7f47858fbf-x45sj" podUID="27dd21d8-102d-45c3-8e54-3c1e846829de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused"
Nov 02 12:54:23.378502 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:23.378467    4559 patch_prober.go:29] interesting pod/e2e-shared-a-w3-e7561a-openshift-default-7598f89b79-m5rrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body=
Nov 02 12:54:23.378709 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:23.378520    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-e7561a-openshift-default-7598f89b79-m5rrm" podUID="4293ef7c-ce26-475f-98e1-ebf007c35f00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused"
Nov 02 12:54:23.739665 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:23.737020    4559 patch_prober.go:29] interesting pod/e2e-shared-a-w1-a6f2c0-openshift-default-7f47858fbf-x45sj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" start-of-body=
Nov 02 12:54:23.739665 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:23.737106    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-a6f2c0-openshift-default-7f47858fbf-x45sj" podUID="27dd21d8-102d-45c3-8e54-3c1e846829de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused"
Nov 02 12:54:24.124235 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:24.124148    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-6416f0-7f56ccb547-q7k5v"]
Nov 02 12:54:24.128758 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:24.128730    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-6416f0-7f56ccb547-q7k5v"
Nov 02 12:54:24.134596 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:24.134550    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/459db7e8-1294-4758-93a4-1fc9b9320595-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-6416f0-7f56ccb547-q7k5v\" (UID: \"459db7e8-1294-4758-93a4-1fc9b9320595\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-6416f0-7f56ccb547-q7k5v"
Nov 02 12:54:24.134759 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:24.134617    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bh4k\" (UniqueName: \"kubernetes.io/projected/459db7e8-1294-4758-93a4-1fc9b9320595-kube-api-access-5bh4k\") pod \"payload-pre-processing-e2e-shared-b-w3-6416f0-7f56ccb547-q7k5v\" (UID: \"459db7e8-1294-4758-93a4-1fc9b9320595\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-6416f0-7f56ccb547-q7k5v"
Nov 02 12:54:24.151960 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:24.151931    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-6416f0-7f56ccb547-q7k5v"]
Nov 02 12:54:24.159086 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:24.159041    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-6416f0-dockercfg-mmgb2\""
Nov 02 12:54:24.159086 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:24.159041    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-6416f0\""
Nov 02 12:54:24.236367 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:24.236304    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/459db7e8-1294-4758-93a4-1fc9b9320595-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-6416f0-7f56ccb547-q7k5v\" (UID: \"459db7e8-1294-4758-93a4-1fc9b9320595\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-6416f0-7f56ccb547-q7k5v"
Nov 02 12:54:24.236681 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:24.236653    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bh4k\" (UniqueName: \"kubernetes.io/projected/459db7e8-1294-4758-93a4-1fc9b9320595-kube-api-access-5bh4k\") pod \"payload-pre-processing-e2e-shared-b-w3-6416f0-7f56ccb547-q7k5v\" (UID: \"459db7e8-1294-4758-93a4-1fc9b9320595\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-6416f0-7f56ccb547-q7k5v"
Nov 02 12:54:24.238164 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:24.238118    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/459db7e8-1294-4758-93a4-1fc9b9320595-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-6416f0-7f56ccb547-q7k5v\" (UID: \"459db7e8-1294-4758-93a4-1fc9b9320595\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-6416f0-7f56ccb547-q7k5v"
Nov 02 12:54:24.325659 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:24.324774    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bh4k\" (UniqueName: \"kubernetes.io/projected/459db7e8-1294-4758-93a4-1fc9b9320595-kube-api-access-5bh4k\") pod \"payload-pre-processing-e2e-shared-b-w3-6416f0-7f56ccb547-q7k5v\" (UID: \"459db7e8-1294-4758-93a4-1fc9b9320595\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-6416f0-7f56ccb547-q7k5v"
Nov 02 12:54:24.429660 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:24.426209    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-e7561a-openshift-default-7598f89b79-m5rrm"
Nov 02 12:54:24.429660 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:24.428563    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-e7561a-openshift-default-7598f89b79-m5rrm"
Nov 02 12:54:24.567711 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:24.558908    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-e7561a-openshift-default-7598f89b79-m5rrm"
Nov 02 12:54:24.567711 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:24.559512    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-6416f0-7f56ccb547-q7k5v"
Nov 02 12:54:24.623117 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:24.623069    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-8032f3-74cb7cfb7d-56656"
Nov 02 12:54:24.728209 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:24.728135    4559 patch_prober.go:29] interesting pod/e2e-shared-a-w1-a6f2c0-openshift-default-7f47858fbf-x45sj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" start-of-body=
Nov 02 12:54:24.728991 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:24.728942    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-a6f2c0-openshift-default-7f47858fbf-x45sj" podUID="27dd21d8-102d-45c3-8e54-3c1e846829de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused"
Nov 02 12:54:25.348744 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:25.341865    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-6416f0-7f56ccb547-q7k5v"]
Nov 02 12:54:25.724455 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:25.724415    4559 patch_prober.go:29] interesting pod/e2e-shared-a-w1-a6f2c0-openshift-default-7f47858fbf-x45sj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" start-of-body=
Nov 02 12:54:25.724654 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:25.724477    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-a6f2c0-openshift-default-7f47858fbf-x45sj" podUID="27dd21d8-102d-45c3-8e54-3c1e846829de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused"
Nov 02 12:54:26.107597 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:26.107506    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-6416f0-7f56ccb547-q7k5v" event={"ID":"459db7e8-1294-4758-93a4-1fc9b9320595","Type":"ContainerStarted","Data":"8aee7ce49b53f0aed49cce17e60deb50fa3b7a9c09916c2e6a856ce8a5078d61"}
Nov 02 12:54:26.107798 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:26.107666    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-6416f0-7f56ccb547-q7k5v"
Nov 02 12:54:26.107798 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:26.107703    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-6416f0-7f56ccb547-q7k5v" event={"ID":"459db7e8-1294-4758-93a4-1fc9b9320595","Type":"ContainerStarted","Data":"6a095b9ea54263fe206d5a135bee8f6c9afcf27d12f6d6ae10f501e117c210b1"}
Nov 02 12:54:26.167338 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:26.167289    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-6416f0-7f56ccb547-q7k5v" podStartSLOduration=2.167276979 podStartE2EDuration="2.167276979s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:54:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:54:26.1659091 +0000 UTC m=+1662.421308194" watchObservedRunningTime="2026-11-02 12:54:26.167276979 +0000 UTC m=+1662.422676091"
Nov 02 12:54:26.486051 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:26.486021    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-a6f2c0-openshift-default-7f47858fbf-x45sj"
Nov 02 12:54:26.728569 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:26.728535    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-a6f2c0-openshift-default-7f47858fbf-x45sj"
Nov 02 12:54:27.115740 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:27.115714    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-a6f2c0-openshift-default-7f47858fbf-x45sj"
Nov 02 12:54:31.997449 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:31.997410    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-74ba71-fdcf98f5f-dp92m"]
Nov 02 12:54:32.001899 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:32.001882    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-74ba71-fdcf98f5f-dp92m"
Nov 02 12:54:32.025053 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:32.025030    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-74ba71-serving-cert\""
Nov 02 12:54:32.037055 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:32.037028    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-74ba71-fdcf98f5f-dp92m"]
Nov 02 12:54:32.082258 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:32.082226    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7c4bc95-0804-468f-9395-d87d4cbdf8cf-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-74ba71-fdcf98f5f-dp92m\" (UID: \"a7c4bc95-0804-468f-9395-d87d4cbdf8cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-74ba71-fdcf98f5f-dp92m"
Nov 02 12:54:32.082446 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:32.082287    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a7c4bc95-0804-468f-9395-d87d4cbdf8cf-metrics-certs\") pod \"maas-api-e2e-praxis-b-74ba71-fdcf98f5f-dp92m\" (UID: \"a7c4bc95-0804-468f-9395-d87d4cbdf8cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-74ba71-fdcf98f5f-dp92m"
Nov 02 12:54:32.082446 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:32.082432    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9pbw\" (UniqueName: \"kubernetes.io/projected/a7c4bc95-0804-468f-9395-d87d4cbdf8cf-kube-api-access-w9pbw\") pod \"maas-api-e2e-praxis-b-74ba71-fdcf98f5f-dp92m\" (UID: \"a7c4bc95-0804-468f-9395-d87d4cbdf8cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-74ba71-fdcf98f5f-dp92m"
Nov 02 12:54:32.082587 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:32.082531    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a7c4bc95-0804-468f-9395-d87d4cbdf8cf-maas-api-tls\") pod \"maas-api-e2e-praxis-b-74ba71-fdcf98f5f-dp92m\" (UID: \"a7c4bc95-0804-468f-9395-d87d4cbdf8cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-74ba71-fdcf98f5f-dp92m"
Nov 02 12:54:32.188662 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:32.185574    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9pbw\" (UniqueName: \"kubernetes.io/projected/a7c4bc95-0804-468f-9395-d87d4cbdf8cf-kube-api-access-w9pbw\") pod \"maas-api-e2e-praxis-b-74ba71-fdcf98f5f-dp92m\" (UID: \"a7c4bc95-0804-468f-9395-d87d4cbdf8cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-74ba71-fdcf98f5f-dp92m"
Nov 02 12:54:32.188662 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:32.185741    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a7c4bc95-0804-468f-9395-d87d4cbdf8cf-maas-api-tls\") pod \"maas-api-e2e-praxis-b-74ba71-fdcf98f5f-dp92m\" (UID: \"a7c4bc95-0804-468f-9395-d87d4cbdf8cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-74ba71-fdcf98f5f-dp92m"
Nov 02 12:54:32.188662 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:32.185894    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7c4bc95-0804-468f-9395-d87d4cbdf8cf-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-74ba71-fdcf98f5f-dp92m\" (UID: \"a7c4bc95-0804-468f-9395-d87d4cbdf8cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-74ba71-fdcf98f5f-dp92m"
Nov 02 12:54:32.188662 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:32.185939    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a7c4bc95-0804-468f-9395-d87d4cbdf8cf-metrics-certs\") pod \"maas-api-e2e-praxis-b-74ba71-fdcf98f5f-dp92m\" (UID: \"a7c4bc95-0804-468f-9395-d87d4cbdf8cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-74ba71-fdcf98f5f-dp92m"
Nov 02 12:54:32.224667 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:54:32.218943    4559 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-b-74ba71-serving-cert: secret "maas-api-e2e-praxis-b-74ba71-serving-cert" not found
Nov 02 12:54:32.224667 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:54:32.219016    4559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a7c4bc95-0804-468f-9395-d87d4cbdf8cf-maas-api-tls podName:a7c4bc95-0804-468f-9395-d87d4cbdf8cf nodeName:}" failed. No retries permitted until 2026-11-02 12:54:32.718996388 +0000 UTC m=+1668.974395475 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a7c4bc95-0804-468f-9395-d87d4cbdf8cf-maas-api-tls") pod "maas-api-e2e-praxis-b-74ba71-fdcf98f5f-dp92m" (UID: "a7c4bc95-0804-468f-9395-d87d4cbdf8cf") : secret "maas-api-e2e-praxis-b-74ba71-serving-cert" not found
Nov 02 12:54:32.224667 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:32.219897    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7c4bc95-0804-468f-9395-d87d4cbdf8cf-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-74ba71-fdcf98f5f-dp92m\" (UID: \"a7c4bc95-0804-468f-9395-d87d4cbdf8cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-74ba71-fdcf98f5f-dp92m"
Nov 02 12:54:32.224667 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:32.221219    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a7c4bc95-0804-468f-9395-d87d4cbdf8cf-metrics-certs\") pod \"maas-api-e2e-praxis-b-74ba71-fdcf98f5f-dp92m\" (UID: \"a7c4bc95-0804-468f-9395-d87d4cbdf8cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-74ba71-fdcf98f5f-dp92m"
Nov 02 12:54:32.274727 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:32.274526    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9pbw\" (UniqueName: \"kubernetes.io/projected/a7c4bc95-0804-468f-9395-d87d4cbdf8cf-kube-api-access-w9pbw\") pod \"maas-api-e2e-praxis-b-74ba71-fdcf98f5f-dp92m\" (UID: \"a7c4bc95-0804-468f-9395-d87d4cbdf8cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-74ba71-fdcf98f5f-dp92m"
Nov 02 12:54:32.810660 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:32.806313    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a7c4bc95-0804-468f-9395-d87d4cbdf8cf-maas-api-tls\") pod \"maas-api-e2e-praxis-b-74ba71-fdcf98f5f-dp92m\" (UID: \"a7c4bc95-0804-468f-9395-d87d4cbdf8cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-74ba71-fdcf98f5f-dp92m"
Nov 02 12:54:32.826896 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:32.826250    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a7c4bc95-0804-468f-9395-d87d4cbdf8cf-maas-api-tls\") pod \"maas-api-e2e-praxis-b-74ba71-fdcf98f5f-dp92m\" (UID: \"a7c4bc95-0804-468f-9395-d87d4cbdf8cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-74ba71-fdcf98f5f-dp92m"
Nov 02 12:54:32.916335 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:32.916303    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-74ba71-fdcf98f5f-dp92m"
Nov 02 12:54:33.273917 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:33.261099    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-74ba71-fdcf98f5f-dp92m"]
Nov 02 12:54:34.285679 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:34.285612    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-74ba71-fdcf98f5f-dp92m" event={"ID":"a7c4bc95-0804-468f-9395-d87d4cbdf8cf","Type":"ContainerStarted","Data":"df533c79ac077625c53266a1ac84fcaead81fcb3bfa191d948ec92f54c6c9fbd"}
Nov 02 12:54:34.286071 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:34.285698    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-74ba71-fdcf98f5f-dp92m" event={"ID":"a7c4bc95-0804-468f-9395-d87d4cbdf8cf","Type":"ContainerStarted","Data":"f562829cd1281fd9594ec235dac5a6e508ae3a9b23edf43e0a039b994838205a"}
Nov 02 12:54:34.286071 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:34.285758    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-74ba71-fdcf98f5f-dp92m"
Nov 02 12:54:34.336220 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:34.336172    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-74ba71-fdcf98f5f-dp92m" podStartSLOduration=2.9598602080000003 podStartE2EDuration="3.336157699s" totalImagesPullingTime="376.297491ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:54:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:54:34.333836217 +0000 UTC m=+1670.589235312" watchObservedRunningTime="2026-11-02 12:54:34.336157699 +0000 UTC m=+1670.591556794"
Nov 02 12:54:37.117760 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:37.117731    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-6416f0-7f56ccb547-q7k5v"
Nov 02 12:54:40.297934 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:40.297903    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-74ba71-fdcf98f5f-dp92m"
Nov 02 12:54:43.053371 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:43.053327    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-1e14d4-7f57bfcbd8-9l28c"]
Nov 02 12:54:43.061331 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:43.061299    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-1e14d4-7f57bfcbd8-9l28c"
Nov 02 12:54:43.071480 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:43.071451    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-1e14d4-7f57bfcbd8-9l28c"]
Nov 02 12:54:43.077318 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:43.077281    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-1e14d4-serving-cert\""
Nov 02 12:54:43.230511 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:43.230479    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ec4ed15-a5c5-461f-b839-ef61b09f6309-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-1e14d4-7f57bfcbd8-9l28c\" (UID: \"6ec4ed15-a5c5-461f-b839-ef61b09f6309\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-1e14d4-7f57bfcbd8-9l28c"
Nov 02 12:54:43.230511 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:43.230516    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ec4ed15-a5c5-461f-b839-ef61b09f6309-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-1e14d4-7f57bfcbd8-9l28c\" (UID: \"6ec4ed15-a5c5-461f-b839-ef61b09f6309\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-1e14d4-7f57bfcbd8-9l28c"
Nov 02 12:54:43.230796 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:43.230684    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n22f4\" (UniqueName: \"kubernetes.io/projected/6ec4ed15-a5c5-461f-b839-ef61b09f6309-kube-api-access-n22f4\") pod \"maas-api-e2e-shared-b-w2-1e14d4-7f57bfcbd8-9l28c\" (UID: \"6ec4ed15-a5c5-461f-b839-ef61b09f6309\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-1e14d4-7f57bfcbd8-9l28c"
Nov 02 12:54:43.230852 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:43.230837    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ec4ed15-a5c5-461f-b839-ef61b09f6309-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-1e14d4-7f57bfcbd8-9l28c\" (UID: \"6ec4ed15-a5c5-461f-b839-ef61b09f6309\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-1e14d4-7f57bfcbd8-9l28c"
Nov 02 12:54:43.336548 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:43.335007    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ec4ed15-a5c5-461f-b839-ef61b09f6309-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-1e14d4-7f57bfcbd8-9l28c\" (UID: \"6ec4ed15-a5c5-461f-b839-ef61b09f6309\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-1e14d4-7f57bfcbd8-9l28c"
Nov 02 12:54:43.336548 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:43.335064    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ec4ed15-a5c5-461f-b839-ef61b09f6309-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-1e14d4-7f57bfcbd8-9l28c\" (UID: \"6ec4ed15-a5c5-461f-b839-ef61b09f6309\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-1e14d4-7f57bfcbd8-9l28c"
Nov 02 12:54:43.336548 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:43.335186    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n22f4\" (UniqueName: \"kubernetes.io/projected/6ec4ed15-a5c5-461f-b839-ef61b09f6309-kube-api-access-n22f4\") pod \"maas-api-e2e-shared-b-w2-1e14d4-7f57bfcbd8-9l28c\" (UID: \"6ec4ed15-a5c5-461f-b839-ef61b09f6309\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-1e14d4-7f57bfcbd8-9l28c"
Nov 02 12:54:43.336548 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:43.335391    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ec4ed15-a5c5-461f-b839-ef61b09f6309-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-1e14d4-7f57bfcbd8-9l28c\" (UID: \"6ec4ed15-a5c5-461f-b839-ef61b09f6309\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-1e14d4-7f57bfcbd8-9l28c"
Nov 02 12:54:43.336923 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:43.336720    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ec4ed15-a5c5-461f-b839-ef61b09f6309-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-1e14d4-7f57bfcbd8-9l28c\" (UID: \"6ec4ed15-a5c5-461f-b839-ef61b09f6309\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-1e14d4-7f57bfcbd8-9l28c"
Nov 02 12:54:43.342285 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:43.339407    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ec4ed15-a5c5-461f-b839-ef61b09f6309-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-1e14d4-7f57bfcbd8-9l28c\" (UID: \"6ec4ed15-a5c5-461f-b839-ef61b09f6309\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-1e14d4-7f57bfcbd8-9l28c"
Nov 02 12:54:43.342285 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:43.341254    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ec4ed15-a5c5-461f-b839-ef61b09f6309-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-1e14d4-7f57bfcbd8-9l28c\" (UID: \"6ec4ed15-a5c5-461f-b839-ef61b09f6309\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-1e14d4-7f57bfcbd8-9l28c"
Nov 02 12:54:43.387702 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:43.387630    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n22f4\" (UniqueName: \"kubernetes.io/projected/6ec4ed15-a5c5-461f-b839-ef61b09f6309-kube-api-access-n22f4\") pod \"maas-api-e2e-shared-b-w2-1e14d4-7f57bfcbd8-9l28c\" (UID: \"6ec4ed15-a5c5-461f-b839-ef61b09f6309\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-1e14d4-7f57bfcbd8-9l28c"
Nov 02 12:54:43.679882 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:43.679854    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-1e14d4-7f57bfcbd8-9l28c"
Nov 02 12:54:43.999725 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:43.996859    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-1e14d4-7f57bfcbd8-9l28c"]
Nov 02 12:54:44.326333 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:44.326255    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"]
Nov 02 12:54:44.498209 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:44.491988    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-1e14d4-7f57bfcbd8-9l28c" event={"ID":"6ec4ed15-a5c5-461f-b839-ef61b09f6309","Type":"ContainerStarted","Data":"7707c01c642dc767ebec3de6d0e8c90f2d0b2bd6aea25b9d270ab6a4e561055b"}
Nov 02 12:54:45.671662 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:54:45.670734    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqcmj" containerName="extractor"
Nov 02 12:54:45.709104 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:45.709026    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-1e14d4-7f57bfcbd8-9l28c" event={"ID":"6ec4ed15-a5c5-461f-b839-ef61b09f6309","Type":"ContainerStarted","Data":"348a76bf1dba8e6bd0880ef63793384582f20e48c82aa7156f577cacbecd85ec"}
Nov 02 12:54:45.709881 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:45.709801    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-1e14d4-7f57bfcbd8-9l28c"
Nov 02 12:54:45.778148 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:45.774772    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-1e14d4-7f57bfcbd8-9l28c" podStartSLOduration=3.1481927929999998 podStartE2EDuration="3.774749998s" totalImagesPullingTime="626.557205ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:54:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:54:45.772918379 +0000 UTC m=+1682.028317543" watchObservedRunningTime="2026-11-02 12:54:45.774749998 +0000 UTC m=+1682.030149097"
Nov 02 12:54:51.739700 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:51.739669    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-1e14d4-7f57bfcbd8-9l28c"
Nov 02 12:54:54.228792 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:54.228738    4559 patch_prober.go:29] interesting pod/e2e-shared-a-w0-8032f3-openshift-default-765d774fdc-mhf7j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.187:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 12:54:54.229304 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:54.228817    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w0-8032f3-openshift-default-765d774fdc-mhf7j" podUID="5d73ae6a-aaa7-4946-baa5-03ab6d91d403" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.187:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:54:56.854912 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:56.854878    4559 generic.go:356] "Generic (PLEG): container finished" podID="5d73ae6a-aaa7-4946-baa5-03ab6d91d403" containerID="6d770e61b37efd4981a0b48fd44b029c55d19eb0c7dd7b8fb5ce75a89a9b3034" exitCode=0
Nov 02 12:54:56.855342 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:56.854951    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-8032f3-openshift-default-765d774fdc-mhf7j" event={"ID":"5d73ae6a-aaa7-4946-baa5-03ab6d91d403","Type":"ContainerDied","Data":"6d770e61b37efd4981a0b48fd44b029c55d19eb0c7dd7b8fb5ce75a89a9b3034"}
Nov 02 12:54:56.855342 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:56.855335    4559 scope.go:122] "RemoveContainer" containerID="6d770e61b37efd4981a0b48fd44b029c55d19eb0c7dd7b8fb5ce75a89a9b3034"
Nov 02 12:54:56.856832 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:56.856802    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:54:56.856930 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:56.856869    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:54:56.856930 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:56.856899    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:54:57.864140 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:57.864107    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-8032f3-openshift-default-765d774fdc-mhf7j" event={"ID":"5d73ae6a-aaa7-4946-baa5-03ab6d91d403","Type":"ContainerStarted","Data":"3ab56461a8d9cd117c6b68400d9e00469020ae5db9744c2fdd3177ce5c40d2d6"}
Nov 02 12:54:58.782178 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:58.782148    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-8032f3-openshift-default-765d774fdc-mhf7j"
Nov 02 12:54:58.782397 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:58.782376    4559 patch_prober.go:29] interesting pod/e2e-shared-a-w0-8032f3-openshift-default-765d774fdc-mhf7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused" start-of-body=
Nov 02 12:54:58.782451 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:58.782420    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-8032f3-openshift-default-765d774fdc-mhf7j" podUID="5d73ae6a-aaa7-4946-baa5-03ab6d91d403" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused"
Nov 02 12:54:59.782230 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:59.782198    4559 patch_prober.go:29] interesting pod/e2e-shared-a-w0-8032f3-openshift-default-765d774fdc-mhf7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused" start-of-body=
Nov 02 12:54:59.782623 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:54:59.782255    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-8032f3-openshift-default-765d774fdc-mhf7j" podUID="5d73ae6a-aaa7-4946-baa5-03ab6d91d403" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused"
Nov 02 12:55:00.781984 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:55:00.781950    4559 patch_prober.go:29] interesting pod/e2e-shared-a-w0-8032f3-openshift-default-765d774fdc-mhf7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused" start-of-body=
Nov 02 12:55:00.782150 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:55:00.782004    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-8032f3-openshift-default-765d774fdc-mhf7j" podUID="5d73ae6a-aaa7-4946-baa5-03ab6d91d403" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused"
Nov 02 12:55:01.781871 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:55:01.781831    4559 patch_prober.go:29] interesting pod/e2e-shared-a-w0-8032f3-openshift-default-765d774fdc-mhf7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused" start-of-body=
Nov 02 12:55:01.782255 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:55:01.781911    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-8032f3-openshift-default-765d774fdc-mhf7j" podUID="5d73ae6a-aaa7-4946-baa5-03ab6d91d403" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused"
Nov 02 12:55:02.798662 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:55:02.791904    4559 patch_prober.go:29] interesting pod/e2e-shared-a-w0-8032f3-openshift-default-765d774fdc-mhf7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused" start-of-body=
Nov 02 12:55:02.798662 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:55:02.791977    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-8032f3-openshift-default-765d774fdc-mhf7j" podUID="5d73ae6a-aaa7-4946-baa5-03ab6d91d403" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused"
Nov 02 12:55:03.786065 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:55:03.786035    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-8032f3-openshift-default-765d774fdc-mhf7j"
Nov 02 12:55:03.786305 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:55:03.786284    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-8032f3-openshift-default-765d774fdc-mhf7j"
Nov 02 12:55:03.786875 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:55:03.786856    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-8032f3-openshift-default-765d774fdc-mhf7j"
Nov 02 12:55:09.565800 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:55:09.565766    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"]
Nov 02 12:55:15.113175 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:55:15.113134    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqcmj" containerName="extractor"
Nov 02 12:55:17.291142 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:55:17.291097    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-43be82-5d876bd44c-s6bm8"]
Nov 02 12:55:17.310885 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:55:17.310111    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-43be82-5d876bd44c-s6bm8"
Nov 02 12:55:17.311058 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:55:17.310010    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-43be82-5d876bd44c-s6bm8"]
Nov 02 12:55:17.331125 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:55:17.331099    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w0-43be82-serving-cert\""
Nov 02 12:55:17.455663 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:55:17.454827    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4030f96f-2e78-477c-8670-c47821b4a931-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-43be82-5d876bd44c-s6bm8\" (UID: \"4030f96f-2e78-477c-8670-c47821b4a931\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-43be82-5d876bd44c-s6bm8"
Nov 02 12:55:17.455663 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:55:17.454906    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4030f96f-2e78-477c-8670-c47821b4a931-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-43be82-5d876bd44c-s6bm8\" (UID: \"4030f96f-2e78-477c-8670-c47821b4a931\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-43be82-5d876bd44c-s6bm8"
Nov 02 12:55:17.455663 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:55:17.455021    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6h25\" (UniqueName: \"kubernetes.io/projected/4030f96f-2e78-477c-8670-c47821b4a931-kube-api-access-g6h25\") pod \"maas-api-e2e-shared-b-w0-43be82-5d876bd44c-s6bm8\" (UID: \"4030f96f-2e78-477c-8670-c47821b4a931\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-43be82-5d876bd44c-s6bm8"
Nov 02 12:55:17.455663 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:55:17.455194    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4030f96f-2e78-477c-8670-c47821b4a931-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-43be82-5d876bd44c-s6bm8\" (UID: \"4030f96f-2e78-477c-8670-c47821b4a931\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-43be82-5d876bd44c-s6bm8"
Nov 02 12:55:17.559798 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:55:17.559717    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4030f96f-2e78-477c-8670-c47821b4a931-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-43be82-5d876bd44c-s6bm8\" (UID: \"4030f96f-2e78-477c-8670-c47821b4a931\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-43be82-5d876bd44c-s6bm8"
Nov 02 12:55:17.560735 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:55:17.560614    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4030f96f-2e78-477c-8670-c47821b4a931-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-43be82-5d876bd44c-s6bm8\" (UID: \"4030f96f-2e78-477c-8670-c47821b4a931\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-43be82-5d876bd44c-s6bm8"
Nov 02 12:55:17.560969 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:55:17.560948    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4030f96f-2e78-477c-8670-c47821b4a931-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-43be82-5d876bd44c-s6bm8\" (UID: \"4030f96f-2e78-477c-8670-c47821b4a931\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-43be82-5d876bd44c-s6bm8"
Nov 02 12:55:17.561225 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:55:17.561204    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6h25\" (UniqueName: \"kubernetes.io/projected/4030f96f-2e78-477c-8670-c47821b4a931-kube-api-access-g6h25\") pod \"maas-api-e2e-shared-b-w0-43be82-5d876bd44c-s6bm8\" (UID: \"4030f96f-2e78-477c-8670-c47821b4a931\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-43be82-5d876bd44c-s6bm8"
Nov 02 12:55:17.561599 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:55:17.561575    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4030f96f-2e78-477c-8670-c47821b4a931-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-43be82-5d876bd44c-s6bm8\" (UID: \"4030f96f-2e78-477c-8670-c47821b4a931\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-43be82-5d876bd44c-s6bm8"
Nov 02 12:55:17.568790 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:55:17.568764    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4030f96f-2e78-477c-8670-c47821b4a931-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-43be82-5d876bd44c-s6bm8\" (UID: \"4030f96f-2e78-477c-8670-c47821b4a931\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-43be82-5d876bd44c-s6bm8"
Nov 02 12:55:17.569433 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:55:17.569377    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4030f96f-2e78-477c-8670-c47821b4a931-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-43be82-5d876bd44c-s6bm8\" (UID: \"4030f96f-2e78-477c-8670-c47821b4a931\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-43be82-5d876bd44c-s6bm8"
Nov 02 12:55:17.590177 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:55:17.590147    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6h25\" (UniqueName: \"kubernetes.io/projected/4030f96f-2e78-477c-8670-c47821b4a931-kube-api-access-g6h25\") pod \"maas-api-e2e-shared-b-w0-43be82-5d876bd44c-s6bm8\" (UID: \"4030f96f-2e78-477c-8670-c47821b4a931\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-43be82-5d876bd44c-s6bm8"
Nov 02 12:55:17.633394 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:55:17.633361    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-43be82-5d876bd44c-s6bm8"
Nov 02 12:55:17.945031 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:55:17.945001    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-43be82-5d876bd44c-s6bm8"]
Nov 02 12:55:17.945151 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:55:17.945052    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4030f96f_2e78_477c_8670_c47821b4a931.slice/crio-caac0a093d16286229028ada29053bd4a72f3f7953e0d0aa679c724be423f3b2 WatchSource:0}: Error finding container caac0a093d16286229028ada29053bd4a72f3f7953e0d0aa679c724be423f3b2: Status 404 returned error can't find the container with id caac0a093d16286229028ada29053bd4a72f3f7953e0d0aa679c724be423f3b2
Nov 02 12:55:18.164064 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:55:18.164031    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-43be82-5d876bd44c-s6bm8" event={"ID":"4030f96f-2e78-477c-8670-c47821b4a931","Type":"ContainerStarted","Data":"caac0a093d16286229028ada29053bd4a72f3f7953e0d0aa679c724be423f3b2"}
Nov 02 12:55:19.173871 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:55:19.173830    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-43be82-5d876bd44c-s6bm8" event={"ID":"4030f96f-2e78-477c-8670-c47821b4a931","Type":"ContainerStarted","Data":"ed26216e8698be654b7de5665a74fd6c8e80238ef41c88133ad4e5eaba918ec3"}
Nov 02 12:55:19.174955 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:55:19.174928    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-43be82-5d876bd44c-s6bm8"
Nov 02 12:55:19.225141 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:55:19.225075    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-43be82-5d876bd44c-s6bm8" podStartSLOduration=1.918956077 podStartE2EDuration="2.225059342s" totalImagesPullingTime="306.103265ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:55:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:55:19.221884812 +0000 UTC m=+1715.477283902" watchObservedRunningTime="2026-11-02 12:55:19.225059342 +0000 UTC m=+1715.480458465"
Nov 02 12:55:21.190184 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:55:21.190155    4559 generic.go:356] "Generic (PLEG): container finished" podID="8debb7bd-d249-47d9-8776-f2be8f0d4f27" containerID="66a2714cf5cd54c62bed7a6eb34297b3139d6f16a0ad9d88a2404ca6ef03071e" exitCode=0
Nov 02 12:55:21.190720 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:55:21.190222    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-1aa18c-openshift-default-7d7f44f669-xz5m4" event={"ID":"8debb7bd-d249-47d9-8776-f2be8f0d4f27","Type":"ContainerDied","Data":"66a2714cf5cd54c62bed7a6eb34297b3139d6f16a0ad9d88a2404ca6ef03071e"}
Nov 02 12:55:21.190720 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:55:21.190649    4559 scope.go:122] "RemoveContainer" containerID="66a2714cf5cd54c62bed7a6eb34297b3139d6f16a0ad9d88a2404ca6ef03071e"
Nov 02 12:55:21.192129 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:55:21.192099    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:55:21.192207 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:55:21.192180    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:55:21.192249 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:55:21.192232    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:55:22.217517 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:55:22.217481    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-1aa18c-openshift-default-7d7f44f669-xz5m4" event={"ID":"8debb7bd-d249-47d9-8776-f2be8f0d4f27","Type":"ContainerStarted","Data":"7e6c60c6cf892fd38b50a857246c2b4ee3fd1cf261261951a149d41e12dfb557"}
Nov 02 12:55:23.174852 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:55:23.174812    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-1aa18c-openshift-default-7d7f44f669-xz5m4"
Nov 02 12:55:23.175912 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:55:23.175884    4559 patch_prober.go:29] interesting pod/e2e-shared-b-w1-1aa18c-openshift-default-7d7f44f669-xz5m4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" start-of-body=
Nov 02 12:55:23.176060 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:55:23.175935    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-1aa18c-openshift-default-7d7f44f669-xz5m4" podUID="8debb7bd-d249-47d9-8776-f2be8f0d4f27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused"
Nov 02 12:55:24.172898 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:55:24.172857    4559 patch_prober.go:29] interesting pod/e2e-shared-b-w1-1aa18c-openshift-default-7d7f44f669-xz5m4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" start-of-body=
Nov 02 12:55:24.173287 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:55:24.172929    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-1aa18c-openshift-default-7d7f44f669-xz5m4" podUID="8debb7bd-d249-47d9-8776-f2be8f0d4f27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused"
Nov 02 12:55:25.172430 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:55:25.172394    4559 patch_prober.go:29] interesting pod/e2e-shared-b-w1-1aa18c-openshift-default-7d7f44f669-xz5m4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" start-of-body=
Nov 02 12:55:25.172620 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:55:25.172465    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-1aa18c-openshift-default-7d7f44f669-xz5m4" podUID="8debb7bd-d249-47d9-8776-f2be8f0d4f27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused"
Nov 02 12:55:26.172427 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:55:26.172386    4559 patch_prober.go:29] interesting pod/e2e-shared-b-w1-1aa18c-openshift-default-7d7f44f669-xz5m4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" start-of-body=
Nov 02 12:55:26.172900 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:55:26.172479    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-1aa18c-openshift-default-7d7f44f669-xz5m4" podUID="8debb7bd-d249-47d9-8776-f2be8f0d4f27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused"
Nov 02 12:55:26.211313 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:55:26.211286    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-43be82-5d876bd44c-s6bm8"
Nov 02 12:55:26.916966 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:55:26.916930    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-1aa18c-openshift-default-7d7f44f669-xz5m4"
Nov 02 12:55:27.172393 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:55:27.172359    4559 patch_prober.go:29] interesting pod/e2e-shared-b-w1-1aa18c-openshift-default-7d7f44f669-xz5m4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" start-of-body=
Nov 02 12:55:27.172587 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:55:27.172417    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-1aa18c-openshift-default-7d7f44f669-xz5m4" podUID="8debb7bd-d249-47d9-8776-f2be8f0d4f27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused"
Nov 02 12:55:28.177394 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:55:28.177363    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-1aa18c-openshift-default-7d7f44f669-xz5m4"
Nov 02 12:55:28.178715 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:55:28.178687    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-1aa18c-openshift-default-7d7f44f669-xz5m4"
Nov 02 12:55:38.450951 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:55:38.450880    4559 generic.go:356] "Generic (PLEG): container finished" podID="24bcb862-30cd-44b9-920e-a919d80678a9" containerID="693fedc45dfde9ab92b41dc6b96b36bffe723cac46d4a469e5d74ee9285ce816" exitCode=0
Nov 02 12:55:38.451487 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:55:38.450960    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-c0d4cb-openshift-default-6798dbb74f-rslvw" event={"ID":"24bcb862-30cd-44b9-920e-a919d80678a9","Type":"ContainerDied","Data":"693fedc45dfde9ab92b41dc6b96b36bffe723cac46d4a469e5d74ee9285ce816"}
Nov 02 12:55:38.451487 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:55:38.451415    4559 scope.go:122] "RemoveContainer" containerID="693fedc45dfde9ab92b41dc6b96b36bffe723cac46d4a469e5d74ee9285ce816"
Nov 02 12:55:38.453007 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:55:38.452976    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:55:38.453108 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:55:38.453045    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:55:38.453108 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:55:38.453078    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:55:39.459083 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:55:39.459010    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-c0d4cb-openshift-default-6798dbb74f-rslvw" event={"ID":"24bcb862-30cd-44b9-920e-a919d80678a9","Type":"ContainerStarted","Data":"a52dff0ab53917cde9b60354d9e55e43edabe19576d96c9498ede76ab44a91ca"}
Nov 02 12:55:40.343982 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:55:40.343952    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-c0d4cb-openshift-default-6798dbb74f-rslvw"
Nov 02 12:55:40.344167 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:55:40.344115    4559 patch_prober.go:29] interesting pod/e2e-praxis-a-c0d4cb-openshift-default-6798dbb74f-rslvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" start-of-body=
Nov 02 12:55:40.344220 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:55:40.344167    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-c0d4cb-openshift-default-6798dbb74f-rslvw" podUID="24bcb862-30cd-44b9-920e-a919d80678a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused"
Nov 02 12:55:41.345661 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:55:41.345493    4559 patch_prober.go:29] interesting pod/e2e-praxis-a-c0d4cb-openshift-default-6798dbb74f-rslvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" start-of-body=
Nov 02 12:55:41.345661 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:55:41.345590    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-c0d4cb-openshift-default-6798dbb74f-rslvw" podUID="24bcb862-30cd-44b9-920e-a919d80678a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused"
Nov 02 12:55:42.344385 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:55:42.344350    4559 patch_prober.go:29] interesting pod/e2e-praxis-a-c0d4cb-openshift-default-6798dbb74f-rslvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" start-of-body=
Nov 02 12:55:42.344599 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:55:42.344418    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-c0d4cb-openshift-default-6798dbb74f-rslvw" podUID="24bcb862-30cd-44b9-920e-a919d80678a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused"
Nov 02 12:55:43.344428 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:55:43.344397    4559 patch_prober.go:29] interesting pod/e2e-praxis-a-c0d4cb-openshift-default-6798dbb74f-rslvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" start-of-body=
Nov 02 12:55:43.344847 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:55:43.344459    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-c0d4cb-openshift-default-6798dbb74f-rslvw" podUID="24bcb862-30cd-44b9-920e-a919d80678a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused"
Nov 02 12:55:44.344083 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:55:44.344046    4559 patch_prober.go:29] interesting pod/e2e-praxis-a-c0d4cb-openshift-default-6798dbb74f-rslvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" start-of-body=
Nov 02 12:55:44.344271 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:55:44.344107    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-c0d4cb-openshift-default-6798dbb74f-rslvw" podUID="24bcb862-30cd-44b9-920e-a919d80678a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused"
Nov 02 12:55:45.085992 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:55:45.085956    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqcmj" containerName="extractor"
Nov 02 12:55:45.437722 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:55:45.437689    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-c0d4cb-openshift-default-6798dbb74f-rslvw"
Nov 02 12:55:45.440007 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:55:45.439987    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-c0d4cb-openshift-default-6798dbb74f-rslvw"
Nov 02 12:55:45.453269 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:55:45.453240    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-c0d4cb-openshift-default-6798dbb74f-rslvw"
Nov 02 12:55:49.499739 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:55:49.499699    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"]
Nov 02 12:55:56.706938 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:55:56.706858    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w0-8032f3/auth-test-model-8032f3-kserve-675786644f-2dmj4"]
Nov 02 12:55:56.712894 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:55:56.712869    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-8032f3/auth-test-model-8032f3-kserve-675786644f-2dmj4"
Nov 02 12:55:56.742664 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:55:56.742627    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-8032f3\"/\"openshift-service-ca.crt\""
Nov 02 12:55:56.742930 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:55:56.742912    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-8032f3\"/\"kube-root-ca.crt\""
Nov 02 12:55:56.743032 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:55:56.743000    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-8032f3\"/\"default-dockercfg-28fjx\""
Nov 02 12:55:56.743129 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:55:56.743113    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-8032f3\"/\"auth-test-model-8032f3-kserve-self-signed-certs\""
Nov 02 12:55:56.748387 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:55:56.748362    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-8032f3/auth-test-model-8032f3-kserve-675786644f-2dmj4"]
Nov 02 12:55:56.848497 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:55:56.848470    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8024fcc-5300-4b6c-84c0-7c2c30a0370a-home\") pod \"auth-test-model-8032f3-kserve-675786644f-2dmj4\" (UID: \"e8024fcc-5300-4b6c-84c0-7c2c30a0370a\") " pod="ai-tenant-e2e-shared-a-w0-8032f3/auth-test-model-8032f3-kserve-675786644f-2dmj4"
Nov 02 12:55:56.848668 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:55:56.848503    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8024fcc-5300-4b6c-84c0-7c2c30a0370a-tmp-dir\") pod \"auth-test-model-8032f3-kserve-675786644f-2dmj4\" (UID: \"e8024fcc-5300-4b6c-84c0-7c2c30a0370a\") " pod="ai-tenant-e2e-shared-a-w0-8032f3/auth-test-model-8032f3-kserve-675786644f-2dmj4"
Nov 02 12:55:56.848668 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:55:56.848556    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8024fcc-5300-4b6c-84c0-7c2c30a0370a-dshm\") pod \"auth-test-model-8032f3-kserve-675786644f-2dmj4\" (UID: \"e8024fcc-5300-4b6c-84c0-7c2c30a0370a\") " pod="ai-tenant-e2e-shared-a-w0-8032f3/auth-test-model-8032f3-kserve-675786644f-2dmj4"
Nov 02 12:55:56.848772 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:55:56.848682    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pcdsv\" (UniqueName: \"kubernetes.io/projected/e8024fcc-5300-4b6c-84c0-7c2c30a0370a-kube-api-access-pcdsv\") pod \"auth-test-model-8032f3-kserve-675786644f-2dmj4\" (UID: \"e8024fcc-5300-4b6c-84c0-7c2c30a0370a\") " pod="ai-tenant-e2e-shared-a-w0-8032f3/auth-test-model-8032f3-kserve-675786644f-2dmj4"
Nov 02 12:55:56.848772 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:55:56.848729    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8024fcc-5300-4b6c-84c0-7c2c30a0370a-tls-certs\") pod \"auth-test-model-8032f3-kserve-675786644f-2dmj4\" (UID: \"e8024fcc-5300-4b6c-84c0-7c2c30a0370a\") " pod="ai-tenant-e2e-shared-a-w0-8032f3/auth-test-model-8032f3-kserve-675786644f-2dmj4"
Nov 02 12:55:56.848772 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:55:56.848746    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8024fcc-5300-4b6c-84c0-7c2c30a0370a-model-cache\") pod \"auth-test-model-8032f3-kserve-675786644f-2dmj4\" (UID: \"e8024fcc-5300-4b6c-84c0-7c2c30a0370a\") " pod="ai-tenant-e2e-shared-a-w0-8032f3/auth-test-model-8032f3-kserve-675786644f-2dmj4"
Nov 02 12:55:56.951818 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:55:56.951776    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8024fcc-5300-4b6c-84c0-7c2c30a0370a-home\") pod \"auth-test-model-8032f3-kserve-675786644f-2dmj4\" (UID: \"e8024fcc-5300-4b6c-84c0-7c2c30a0370a\") " pod="ai-tenant-e2e-shared-a-w0-8032f3/auth-test-model-8032f3-kserve-675786644f-2dmj4"
Nov 02 12:55:56.952083 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:55:56.952061    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8024fcc-5300-4b6c-84c0-7c2c30a0370a-tmp-dir\") pod \"auth-test-model-8032f3-kserve-675786644f-2dmj4\" (UID: \"e8024fcc-5300-4b6c-84c0-7c2c30a0370a\") " pod="ai-tenant-e2e-shared-a-w0-8032f3/auth-test-model-8032f3-kserve-675786644f-2dmj4"
Nov 02 12:55:56.952323 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:55:56.952303    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8024fcc-5300-4b6c-84c0-7c2c30a0370a-dshm\") pod \"auth-test-model-8032f3-kserve-675786644f-2dmj4\" (UID: \"e8024fcc-5300-4b6c-84c0-7c2c30a0370a\") " pod="ai-tenant-e2e-shared-a-w0-8032f3/auth-test-model-8032f3-kserve-675786644f-2dmj4"
Nov 02 12:55:56.952604 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:55:56.952585    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pcdsv\" (UniqueName: \"kubernetes.io/projected/e8024fcc-5300-4b6c-84c0-7c2c30a0370a-kube-api-access-pcdsv\") pod \"auth-test-model-8032f3-kserve-675786644f-2dmj4\" (UID: \"e8024fcc-5300-4b6c-84c0-7c2c30a0370a\") " pod="ai-tenant-e2e-shared-a-w0-8032f3/auth-test-model-8032f3-kserve-675786644f-2dmj4"
Nov 02 12:55:56.952861 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:55:56.952842    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8024fcc-5300-4b6c-84c0-7c2c30a0370a-tls-certs\") pod \"auth-test-model-8032f3-kserve-675786644f-2dmj4\" (UID: \"e8024fcc-5300-4b6c-84c0-7c2c30a0370a\") " pod="ai-tenant-e2e-shared-a-w0-8032f3/auth-test-model-8032f3-kserve-675786644f-2dmj4"
Nov 02 12:55:56.953000 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:55:56.952980    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8024fcc-5300-4b6c-84c0-7c2c30a0370a-model-cache\") pod \"auth-test-model-8032f3-kserve-675786644f-2dmj4\" (UID: \"e8024fcc-5300-4b6c-84c0-7c2c30a0370a\") " pod="ai-tenant-e2e-shared-a-w0-8032f3/auth-test-model-8032f3-kserve-675786644f-2dmj4"
Nov 02 12:55:56.958693 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:55:56.953724    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8024fcc-5300-4b6c-84c0-7c2c30a0370a-model-cache\") pod \"auth-test-model-8032f3-kserve-675786644f-2dmj4\" (UID: \"e8024fcc-5300-4b6c-84c0-7c2c30a0370a\") " pod="ai-tenant-e2e-shared-a-w0-8032f3/auth-test-model-8032f3-kserve-675786644f-2dmj4"
Nov 02 12:55:56.964554 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:55:56.959694    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8024fcc-5300-4b6c-84c0-7c2c30a0370a-home\") pod \"auth-test-model-8032f3-kserve-675786644f-2dmj4\" (UID: \"e8024fcc-5300-4b6c-84c0-7c2c30a0370a\") " pod="ai-tenant-e2e-shared-a-w0-8032f3/auth-test-model-8032f3-kserve-675786644f-2dmj4"
Nov 02 12:55:56.965053 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:55:56.965028    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8024fcc-5300-4b6c-84c0-7c2c30a0370a-tmp-dir\") pod \"auth-test-model-8032f3-kserve-675786644f-2dmj4\" (UID: \"e8024fcc-5300-4b6c-84c0-7c2c30a0370a\") " pod="ai-tenant-e2e-shared-a-w0-8032f3/auth-test-model-8032f3-kserve-675786644f-2dmj4"
Nov 02 12:55:56.994659 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:55:56.973874    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8024fcc-5300-4b6c-84c0-7c2c30a0370a-dshm\") pod \"auth-test-model-8032f3-kserve-675786644f-2dmj4\" (UID: \"e8024fcc-5300-4b6c-84c0-7c2c30a0370a\") " pod="ai-tenant-e2e-shared-a-w0-8032f3/auth-test-model-8032f3-kserve-675786644f-2dmj4"
Nov 02 12:55:57.010098 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:55:57.010068    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pcdsv\" (UniqueName: \"kubernetes.io/projected/e8024fcc-5300-4b6c-84c0-7c2c30a0370a-kube-api-access-pcdsv\") pod \"auth-test-model-8032f3-kserve-675786644f-2dmj4\" (UID: \"e8024fcc-5300-4b6c-84c0-7c2c30a0370a\") " pod="ai-tenant-e2e-shared-a-w0-8032f3/auth-test-model-8032f3-kserve-675786644f-2dmj4"
Nov 02 12:55:57.024664 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:55:57.021349    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8024fcc-5300-4b6c-84c0-7c2c30a0370a-tls-certs\") pod \"auth-test-model-8032f3-kserve-675786644f-2dmj4\" (UID: \"e8024fcc-5300-4b6c-84c0-7c2c30a0370a\") " pod="ai-tenant-e2e-shared-a-w0-8032f3/auth-test-model-8032f3-kserve-675786644f-2dmj4"
Nov 02 12:55:57.034662 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:55:57.032013    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-8032f3/auth-test-model-8032f3-kserve-675786644f-2dmj4"
Nov 02 12:55:57.433250 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:55:57.430756    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-8032f3/auth-test-model-8032f3-kserve-675786644f-2dmj4"]
Nov 02 12:55:57.707810 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:55:57.707723    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-8032f3/auth-test-model-8032f3-kserve-675786644f-2dmj4" event={"ID":"e8024fcc-5300-4b6c-84c0-7c2c30a0370a","Type":"ContainerStarted","Data":"7783ba0607706736796a59dcfae865bf0505f812f9c5dd2e56d7f63d6c008209"}
Nov 02 12:55:57.707810 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:55:57.707772    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-8032f3/auth-test-model-8032f3-kserve-675786644f-2dmj4" event={"ID":"e8024fcc-5300-4b6c-84c0-7c2c30a0370a","Type":"ContainerStarted","Data":"ce7084a959e080e4a560c3456a2cd49f4c0dff45c5001d8222d0dbb425b5bfc2"}
Nov 02 12:55:57.756244 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:55:57.754214    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w0-8032f3/auth-test-model-8032f3-kserve-675786644f-2dmj4" podStartSLOduration=1.75419416 podStartE2EDuration="1.75419416s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:55:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:55:57.751222721 +0000 UTC m=+1754.006621815" watchObservedRunningTime="2026-11-02 12:55:57.75419416 +0000 UTC m=+1754.009593256"
Nov 02 12:55:58.033116 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:55:58.033045    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w0-8032f3/auth-test-model-8032f3-kserve-675786644f-2dmj4"
Nov 02 12:56:07.032753 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:56:07.032720    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w0-8032f3/auth-test-model-8032f3-kserve-675786644f-2dmj4"
Nov 02 12:56:07.045779 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:56:07.045753    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w0-8032f3/auth-test-model-8032f3-kserve-675786644f-2dmj4"
Nov 02 12:56:07.807209 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:56:07.807176    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w0-8032f3/auth-test-model-8032f3-kserve-675786644f-2dmj4"
Nov 02 12:56:15.087467 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:56:15.087427    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqcmj" containerName="extractor"
Nov 02 12:56:19.929127 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:56:19.929090    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-xs5zn"]
Nov 02 12:56:19.929664 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:56:19.929331    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-xs5zn" podUID="36990652-4474-4f19-94e5-8d540cdc06a1" containerName="istio-proxy" containerID="cri-o://c50dfe46f0cdb8ef0d49145840779c4a42d6ccd66cc4cdcd2707988cfc97a572" gracePeriod=30
Nov 02 12:56:22.508508 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:56:22.508471    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w2-1e14d4/auth-test-model-1e14d4-kserve-97cc85bb9-5jjh5"]
Nov 02 12:56:22.514429 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:56:22.514407    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-1e14d4/auth-test-model-1e14d4-kserve-97cc85bb9-5jjh5"
Nov 02 12:56:22.533249 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:56:22.533223    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-1e14d4/auth-test-model-1e14d4-kserve-97cc85bb9-5jjh5"]
Nov 02 12:56:22.542565 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:56:22.542534    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-1e14d4\"/\"default-dockercfg-6n4mn\""
Nov 02 12:56:22.542709 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:56:22.542559    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-1e14d4\"/\"auth-test-model-1e14d4-kserve-self-signed-certs\""
Nov 02 12:56:22.561482 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:56:22.561451    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9664d93b-e25a-430b-bef1-f4570d950694-model-cache\") pod \"auth-test-model-1e14d4-kserve-97cc85bb9-5jjh5\" (UID: \"9664d93b-e25a-430b-bef1-f4570d950694\") " pod="ai-tenant-e2e-shared-b-w2-1e14d4/auth-test-model-1e14d4-kserve-97cc85bb9-5jjh5"
Nov 02 12:56:22.561618 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:56:22.561545    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9664d93b-e25a-430b-bef1-f4570d950694-home\") pod \"auth-test-model-1e14d4-kserve-97cc85bb9-5jjh5\" (UID: \"9664d93b-e25a-430b-bef1-f4570d950694\") " pod="ai-tenant-e2e-shared-b-w2-1e14d4/auth-test-model-1e14d4-kserve-97cc85bb9-5jjh5"
Nov 02 12:56:22.561618 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:56:22.561573    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9664d93b-e25a-430b-bef1-f4570d950694-dshm\") pod \"auth-test-model-1e14d4-kserve-97cc85bb9-5jjh5\" (UID: \"9664d93b-e25a-430b-bef1-f4570d950694\") " pod="ai-tenant-e2e-shared-b-w2-1e14d4/auth-test-model-1e14d4-kserve-97cc85bb9-5jjh5"
Nov 02 12:56:22.561618 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:56:22.561604    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9664d93b-e25a-430b-bef1-f4570d950694-tls-certs\") pod \"auth-test-model-1e14d4-kserve-97cc85bb9-5jjh5\" (UID: \"9664d93b-e25a-430b-bef1-f4570d950694\") " pod="ai-tenant-e2e-shared-b-w2-1e14d4/auth-test-model-1e14d4-kserve-97cc85bb9-5jjh5"
Nov 02 12:56:22.561828 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:56:22.561812    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rp4vg\" (UniqueName: \"kubernetes.io/projected/9664d93b-e25a-430b-bef1-f4570d950694-kube-api-access-rp4vg\") pod \"auth-test-model-1e14d4-kserve-97cc85bb9-5jjh5\" (UID: \"9664d93b-e25a-430b-bef1-f4570d950694\") " pod="ai-tenant-e2e-shared-b-w2-1e14d4/auth-test-model-1e14d4-kserve-97cc85bb9-5jjh5"
Nov 02 12:56:22.562039 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:56:22.562023    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9664d93b-e25a-430b-bef1-f4570d950694-tmp-dir\") pod \"auth-test-model-1e14d4-kserve-97cc85bb9-5jjh5\" (UID: \"9664d93b-e25a-430b-bef1-f4570d950694\") " pod="ai-tenant-e2e-shared-b-w2-1e14d4/auth-test-model-1e14d4-kserve-97cc85bb9-5jjh5"
Nov 02 12:56:22.607015 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:56:22.606996    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-1e14d4\"/\"openshift-service-ca.crt\""
Nov 02 12:56:22.607130 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:56:22.607089    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-1e14d4\"/\"kube-root-ca.crt\""
Nov 02 12:56:22.663328 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:56:22.663301    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9664d93b-e25a-430b-bef1-f4570d950694-tmp-dir\") pod \"auth-test-model-1e14d4-kserve-97cc85bb9-5jjh5\" (UID: \"9664d93b-e25a-430b-bef1-f4570d950694\") " pod="ai-tenant-e2e-shared-b-w2-1e14d4/auth-test-model-1e14d4-kserve-97cc85bb9-5jjh5"
Nov 02 12:56:22.663445 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:56:22.663344    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9664d93b-e25a-430b-bef1-f4570d950694-model-cache\") pod \"auth-test-model-1e14d4-kserve-97cc85bb9-5jjh5\" (UID: \"9664d93b-e25a-430b-bef1-f4570d950694\") " pod="ai-tenant-e2e-shared-b-w2-1e14d4/auth-test-model-1e14d4-kserve-97cc85bb9-5jjh5"
Nov 02 12:56:22.663445 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:56:22.663382    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9664d93b-e25a-430b-bef1-f4570d950694-home\") pod \"auth-test-model-1e14d4-kserve-97cc85bb9-5jjh5\" (UID: \"9664d93b-e25a-430b-bef1-f4570d950694\") " pod="ai-tenant-e2e-shared-b-w2-1e14d4/auth-test-model-1e14d4-kserve-97cc85bb9-5jjh5"
Nov 02 12:56:22.663445 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:56:22.663398    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9664d93b-e25a-430b-bef1-f4570d950694-dshm\") pod \"auth-test-model-1e14d4-kserve-97cc85bb9-5jjh5\" (UID: \"9664d93b-e25a-430b-bef1-f4570d950694\") " pod="ai-tenant-e2e-shared-b-w2-1e14d4/auth-test-model-1e14d4-kserve-97cc85bb9-5jjh5"
Nov 02 12:56:22.663445 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:56:22.663418    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9664d93b-e25a-430b-bef1-f4570d950694-tls-certs\") pod \"auth-test-model-1e14d4-kserve-97cc85bb9-5jjh5\" (UID: \"9664d93b-e25a-430b-bef1-f4570d950694\") " pod="ai-tenant-e2e-shared-b-w2-1e14d4/auth-test-model-1e14d4-kserve-97cc85bb9-5jjh5"
Nov 02 12:56:22.663723 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:56:22.663521    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rp4vg\" (UniqueName: \"kubernetes.io/projected/9664d93b-e25a-430b-bef1-f4570d950694-kube-api-access-rp4vg\") pod \"auth-test-model-1e14d4-kserve-97cc85bb9-5jjh5\" (UID: \"9664d93b-e25a-430b-bef1-f4570d950694\") " pod="ai-tenant-e2e-shared-b-w2-1e14d4/auth-test-model-1e14d4-kserve-97cc85bb9-5jjh5"
Nov 02 12:56:22.663785 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:56:22.663732    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9664d93b-e25a-430b-bef1-f4570d950694-tmp-dir\") pod \"auth-test-model-1e14d4-kserve-97cc85bb9-5jjh5\" (UID: \"9664d93b-e25a-430b-bef1-f4570d950694\") " pod="ai-tenant-e2e-shared-b-w2-1e14d4/auth-test-model-1e14d4-kserve-97cc85bb9-5jjh5"
Nov 02 12:56:22.663846 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:56:22.663782    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9664d93b-e25a-430b-bef1-f4570d950694-home\") pod \"auth-test-model-1e14d4-kserve-97cc85bb9-5jjh5\" (UID: \"9664d93b-e25a-430b-bef1-f4570d950694\") " pod="ai-tenant-e2e-shared-b-w2-1e14d4/auth-test-model-1e14d4-kserve-97cc85bb9-5jjh5"
Nov 02 12:56:22.663906 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:56:22.663868    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9664d93b-e25a-430b-bef1-f4570d950694-model-cache\") pod \"auth-test-model-1e14d4-kserve-97cc85bb9-5jjh5\" (UID: \"9664d93b-e25a-430b-bef1-f4570d950694\") " pod="ai-tenant-e2e-shared-b-w2-1e14d4/auth-test-model-1e14d4-kserve-97cc85bb9-5jjh5"
Nov 02 12:56:22.665736 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:56:22.665715    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9664d93b-e25a-430b-bef1-f4570d950694-dshm\") pod \"auth-test-model-1e14d4-kserve-97cc85bb9-5jjh5\" (UID: \"9664d93b-e25a-430b-bef1-f4570d950694\") " pod="ai-tenant-e2e-shared-b-w2-1e14d4/auth-test-model-1e14d4-kserve-97cc85bb9-5jjh5"
Nov 02 12:56:22.666274 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:56:22.666251    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9664d93b-e25a-430b-bef1-f4570d950694-tls-certs\") pod \"auth-test-model-1e14d4-kserve-97cc85bb9-5jjh5\" (UID: \"9664d93b-e25a-430b-bef1-f4570d950694\") " pod="ai-tenant-e2e-shared-b-w2-1e14d4/auth-test-model-1e14d4-kserve-97cc85bb9-5jjh5"
Nov 02 12:56:22.687113 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:56:22.687087    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rp4vg\" (UniqueName: \"kubernetes.io/projected/9664d93b-e25a-430b-bef1-f4570d950694-kube-api-access-rp4vg\") pod \"auth-test-model-1e14d4-kserve-97cc85bb9-5jjh5\" (UID: \"9664d93b-e25a-430b-bef1-f4570d950694\") " pod="ai-tenant-e2e-shared-b-w2-1e14d4/auth-test-model-1e14d4-kserve-97cc85bb9-5jjh5"
Nov 02 12:56:22.842770 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:56:22.841343    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-1e14d4/auth-test-model-1e14d4-kserve-97cc85bb9-5jjh5"
Nov 02 12:56:23.221900 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:56:23.221860    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9664d93b_e25a_430b_bef1_f4570d950694.slice/crio-3f61eb7514d12af7dd4ef9eaea666c24d538e02ab2a8b0b5d35a127864d2e85f WatchSource:0}: Error finding container 3f61eb7514d12af7dd4ef9eaea666c24d538e02ab2a8b0b5d35a127864d2e85f: Status 404 returned error can't find the container with id 3f61eb7514d12af7dd4ef9eaea666c24d538e02ab2a8b0b5d35a127864d2e85f
Nov 02 12:56:23.224219 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:56:23.224188    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-1e14d4/auth-test-model-1e14d4-kserve-97cc85bb9-5jjh5"]
Nov 02 12:56:23.944100 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:56:23.944062    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-1e14d4/auth-test-model-1e14d4-kserve-97cc85bb9-5jjh5" event={"ID":"9664d93b-e25a-430b-bef1-f4570d950694","Type":"ContainerStarted","Data":"16fcf1f4c720cb22e6d39df2a40cfddb2dc550fbf9ea980193de9f432f67177d"}
Nov 02 12:56:23.944499 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:56:23.944112    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-1e14d4/auth-test-model-1e14d4-kserve-97cc85bb9-5jjh5" event={"ID":"9664d93b-e25a-430b-bef1-f4570d950694","Type":"ContainerStarted","Data":"3f61eb7514d12af7dd4ef9eaea666c24d538e02ab2a8b0b5d35a127864d2e85f"}
Nov 02 12:56:23.992133 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:56:23.992071    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w2-1e14d4/auth-test-model-1e14d4-kserve-97cc85bb9-5jjh5" podStartSLOduration=1.9920553760000002 podStartE2EDuration="1.992055376s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:56:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:56:23.986770879 +0000 UTC m=+1780.242169975" watchObservedRunningTime="2026-11-02 12:56:23.992055376 +0000 UTC m=+1780.247454473"
Nov 02 12:56:24.850578 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:56:24.846715    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w2-1e14d4/auth-test-model-1e14d4-kserve-97cc85bb9-5jjh5"
Nov 02 12:56:25.284461 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:56:25.284427    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-xs5zn"
Nov 02 12:56:25.445310 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:56:25.445277    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36990652-4474-4f19-94e5-8d540cdc06a1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36990652-4474-4f19-94e5-8d540cdc06a1-istio-envoy\") pod \"36990652-4474-4f19-94e5-8d540cdc06a1\" (UID: \"36990652-4474-4f19-94e5-8d540cdc06a1\") "
Nov 02 12:56:25.445498 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:56:25.445407    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36990652-4474-4f19-94e5-8d540cdc06a1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36990652-4474-4f19-94e5-8d540cdc06a1-istiod-ca-cert\") pod \"36990652-4474-4f19-94e5-8d540cdc06a1\" (UID: \"36990652-4474-4f19-94e5-8d540cdc06a1\") "
Nov 02 12:56:25.445498 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:56:25.445470    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36990652-4474-4f19-94e5-8d540cdc06a1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36990652-4474-4f19-94e5-8d540cdc06a1-credential-socket\") pod \"36990652-4474-4f19-94e5-8d540cdc06a1\" (UID: \"36990652-4474-4f19-94e5-8d540cdc06a1\") "
Nov 02 12:56:25.445605 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:56:25.445537    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36990652-4474-4f19-94e5-8d540cdc06a1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36990652-4474-4f19-94e5-8d540cdc06a1-workload-socket\") pod \"36990652-4474-4f19-94e5-8d540cdc06a1\" (UID: \"36990652-4474-4f19-94e5-8d540cdc06a1\") "
Nov 02 12:56:25.445708 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:56:25.445607    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/36990652-4474-4f19-94e5-8d540cdc06a1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36990652-4474-4f19-94e5-8d540cdc06a1-istio-podinfo\") pod \"36990652-4474-4f19-94e5-8d540cdc06a1\" (UID: \"36990652-4474-4f19-94e5-8d540cdc06a1\") "
Nov 02 12:56:25.445776 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:56:25.445736    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36990652-4474-4f19-94e5-8d540cdc06a1-kube-api-access-l8zrh\" (UniqueName: \"kubernetes.io/projected/36990652-4474-4f19-94e5-8d540cdc06a1-kube-api-access-l8zrh\") pod \"36990652-4474-4f19-94e5-8d540cdc06a1\" (UID: \"36990652-4474-4f19-94e5-8d540cdc06a1\") "
Nov 02 12:56:25.445836 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:56:25.445814    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36990652-4474-4f19-94e5-8d540cdc06a1-workload-socket" pod "36990652-4474-4f19-94e5-8d540cdc06a1" (UID: "36990652-4474-4f19-94e5-8d540cdc06a1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:56:25.445896 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:56:25.445819    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36990652-4474-4f19-94e5-8d540cdc06a1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36990652-4474-4f19-94e5-8d540cdc06a1-istio-data\") pod \"36990652-4474-4f19-94e5-8d540cdc06a1\" (UID: \"36990652-4474-4f19-94e5-8d540cdc06a1\") "
Nov 02 12:56:25.445896 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:56:25.445851    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36990652-4474-4f19-94e5-8d540cdc06a1-credential-socket" pod "36990652-4474-4f19-94e5-8d540cdc06a1" (UID: "36990652-4474-4f19-94e5-8d540cdc06a1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:56:25.445896 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:56:25.445875    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36990652-4474-4f19-94e5-8d540cdc06a1-istiod-ca-cert" pod "36990652-4474-4f19-94e5-8d540cdc06a1" (UID: "36990652-4474-4f19-94e5-8d540cdc06a1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:56:25.446060 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:56:25.445945    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36990652-4474-4f19-94e5-8d540cdc06a1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36990652-4474-4f19-94e5-8d540cdc06a1-workload-certs\") pod \"36990652-4474-4f19-94e5-8d540cdc06a1\" (UID: \"36990652-4474-4f19-94e5-8d540cdc06a1\") "
Nov 02 12:56:25.446060 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:56:25.446048    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36990652-4474-4f19-94e5-8d540cdc06a1-istio-token\" (UniqueName: \"kubernetes.io/projected/36990652-4474-4f19-94e5-8d540cdc06a1-istio-token\") pod \"36990652-4474-4f19-94e5-8d540cdc06a1\" (UID: \"36990652-4474-4f19-94e5-8d540cdc06a1\") "
Nov 02 12:56:25.446179 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:56:25.446079    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36990652-4474-4f19-94e5-8d540cdc06a1-istio-data" pod "36990652-4474-4f19-94e5-8d540cdc06a1" (UID: "36990652-4474-4f19-94e5-8d540cdc06a1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:56:25.446312 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:56:25.446291    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36990652-4474-4f19-94e5-8d540cdc06a1-workload-certs" pod "36990652-4474-4f19-94e5-8d540cdc06a1" (UID: "36990652-4474-4f19-94e5-8d540cdc06a1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:56:25.446831 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:56:25.446804    4559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36990652-4474-4f19-94e5-8d540cdc06a1-workload-socket\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:56:25.446831 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:56:25.446830    4559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36990652-4474-4f19-94e5-8d540cdc06a1-istio-data\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:56:25.446994 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:56:25.446845    4559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36990652-4474-4f19-94e5-8d540cdc06a1-workload-certs\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:56:25.446994 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:56:25.446858    4559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36990652-4474-4f19-94e5-8d540cdc06a1-istiod-ca-cert\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:56:25.446994 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:56:25.446872    4559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36990652-4474-4f19-94e5-8d540cdc06a1-credential-socket\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:56:25.447849 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:56:25.447828    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36990652-4474-4f19-94e5-8d540cdc06a1-istio-envoy" pod "36990652-4474-4f19-94e5-8d540cdc06a1" (UID: "36990652-4474-4f19-94e5-8d540cdc06a1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:56:25.448125 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:56:25.448075    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/36990652-4474-4f19-94e5-8d540cdc06a1-istio-podinfo" pod "36990652-4474-4f19-94e5-8d540cdc06a1" (UID: "36990652-4474-4f19-94e5-8d540cdc06a1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 12:56:25.448225 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:56:25.448209    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36990652-4474-4f19-94e5-8d540cdc06a1-kube-api-access-l8zrh" pod "36990652-4474-4f19-94e5-8d540cdc06a1" (UID: "36990652-4474-4f19-94e5-8d540cdc06a1"). InnerVolumeSpecName "kube-api-access-l8zrh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:56:25.448287 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:56:25.448273    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36990652-4474-4f19-94e5-8d540cdc06a1-istio-token" pod "36990652-4474-4f19-94e5-8d540cdc06a1" (UID: "36990652-4474-4f19-94e5-8d540cdc06a1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:56:25.548527 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:56:25.548497    4559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36990652-4474-4f19-94e5-8d540cdc06a1-istio-podinfo\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:56:25.548527 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:56:25.548523    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l8zrh\" (UniqueName: \"kubernetes.io/projected/36990652-4474-4f19-94e5-8d540cdc06a1-kube-api-access-l8zrh\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:56:25.548728 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:56:25.548534    4559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36990652-4474-4f19-94e5-8d540cdc06a1-istio-token\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:56:25.548728 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:56:25.548544    4559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36990652-4474-4f19-94e5-8d540cdc06a1-istio-envoy\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:56:25.960344 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:56:25.960315    4559 generic.go:356] "Generic (PLEG): container finished" podID="36990652-4474-4f19-94e5-8d540cdc06a1" containerID="c50dfe46f0cdb8ef0d49145840779c4a42d6ccd66cc4cdcd2707988cfc97a572" exitCode=0
Nov 02 12:56:25.960497 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:56:25.960358    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-xs5zn"
Nov 02 12:56:25.960497 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:56:25.960390    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-xs5zn" event={"ID":"36990652-4474-4f19-94e5-8d540cdc06a1","Type":"ContainerDied","Data":"c50dfe46f0cdb8ef0d49145840779c4a42d6ccd66cc4cdcd2707988cfc97a572"}
Nov 02 12:56:25.960497 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:56:25.960442    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-xs5zn" event={"ID":"36990652-4474-4f19-94e5-8d540cdc06a1","Type":"ContainerDied","Data":"11acc9bbb88b1b118fb12fd5b6519207f50e4b4996d0a1c887f07aa90c688592"}
Nov 02 12:56:25.960497 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:56:25.960464    4559 scope.go:122] "RemoveContainer" containerID="c50dfe46f0cdb8ef0d49145840779c4a42d6ccd66cc4cdcd2707988cfc97a572"
Nov 02 12:56:25.971322 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:56:25.971304    4559 scope.go:122] "RemoveContainer" containerID="c50dfe46f0cdb8ef0d49145840779c4a42d6ccd66cc4cdcd2707988cfc97a572"
Nov 02 12:56:25.971589 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:56:25.971567    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c50dfe46f0cdb8ef0d49145840779c4a42d6ccd66cc4cdcd2707988cfc97a572\": container with ID starting with c50dfe46f0cdb8ef0d49145840779c4a42d6ccd66cc4cdcd2707988cfc97a572 not found: ID does not exist" containerID="c50dfe46f0cdb8ef0d49145840779c4a42d6ccd66cc4cdcd2707988cfc97a572"
Nov 02 12:56:25.971834 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:56:25.971597    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c50dfe46f0cdb8ef0d49145840779c4a42d6ccd66cc4cdcd2707988cfc97a572"} err="failed to get container status \"c50dfe46f0cdb8ef0d49145840779c4a42d6ccd66cc4cdcd2707988cfc97a572\": rpc error: code = NotFound desc = could not find container \"c50dfe46f0cdb8ef0d49145840779c4a42d6ccd66cc4cdcd2707988cfc97a572\": container with ID starting with c50dfe46f0cdb8ef0d49145840779c4a42d6ccd66cc4cdcd2707988cfc97a572 not found: ID does not exist"
Nov 02 12:56:26.015146 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:56:26.015124    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-xs5zn"]
Nov 02 12:56:26.027889 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:56:26.027868    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-xs5zn"]
Nov 02 12:56:27.894943 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:56:27.894914    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36990652-4474-4f19-94e5-8d540cdc06a1" path="/var/lib/kubelet/pods/36990652-4474-4f19-94e5-8d540cdc06a1/volumes"
Nov 02 12:56:32.842633 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:56:32.842601    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w2-1e14d4/auth-test-model-1e14d4-kserve-97cc85bb9-5jjh5"
Nov 02 12:56:32.855568 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:56:32.855540    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w2-1e14d4/auth-test-model-1e14d4-kserve-97cc85bb9-5jjh5"
Nov 02 12:56:33.030537 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:56:33.030504    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w2-1e14d4/auth-test-model-1e14d4-kserve-97cc85bb9-5jjh5"
Nov 02 12:56:34.797673 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:56:34.797626    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w0-43be82/auth-test-model-43be82-kserve-5c6dfd4dfc-n4dl7"]
Nov 02 12:56:34.798585 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:56:34.798562    4559 cpu_manager.go:414] "containerMap: removing container" podUID="36990652-4474-4f19-94e5-8d540cdc06a1" containerName="istio-proxy"
Nov 02 12:56:34.798585 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:56:34.798586    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36990652-4474-4f19-94e5-8d540cdc06a1" containerName="istio-proxy"
Nov 02 12:56:34.798826 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:56:34.798759    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="36990652-4474-4f19-94e5-8d540cdc06a1" containerName="istio-proxy"
Nov 02 12:56:34.804204 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:56:34.804184    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-43be82/auth-test-model-43be82-kserve-5c6dfd4dfc-n4dl7"
Nov 02 12:56:34.822315 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:56:34.822290    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-43be82/auth-test-model-43be82-kserve-5c6dfd4dfc-n4dl7"]
Nov 02 12:56:34.829622 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:56:34.829587    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-43be82\"/\"kube-root-ca.crt\""
Nov 02 12:56:34.829773 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:56:34.829629    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-43be82\"/\"openshift-service-ca.crt\""
Nov 02 12:56:34.835480 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:56:34.835458    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-43be82\"/\"default-dockercfg-w5zw7\""
Nov 02 12:56:34.835480 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:56:34.835473    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-43be82\"/\"auth-test-model-43be82-kserve-self-signed-certs\""
Nov 02 12:56:34.936830 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:56:34.936803    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/750b979e-ef5c-4c06-b456-d1c6cdce27e5-dshm\") pod \"auth-test-model-43be82-kserve-5c6dfd4dfc-n4dl7\" (UID: \"750b979e-ef5c-4c06-b456-d1c6cdce27e5\") " pod="ai-tenant-e2e-shared-b-w0-43be82/auth-test-model-43be82-kserve-5c6dfd4dfc-n4dl7"
Nov 02 12:56:34.937065 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:56:34.936864    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbmf5\" (UniqueName: \"kubernetes.io/projected/750b979e-ef5c-4c06-b456-d1c6cdce27e5-kube-api-access-kbmf5\") pod \"auth-test-model-43be82-kserve-5c6dfd4dfc-n4dl7\" (UID: \"750b979e-ef5c-4c06-b456-d1c6cdce27e5\") " pod="ai-tenant-e2e-shared-b-w0-43be82/auth-test-model-43be82-kserve-5c6dfd4dfc-n4dl7"
Nov 02 12:56:34.937065 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:56:34.937042    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/750b979e-ef5c-4c06-b456-d1c6cdce27e5-tmp-dir\") pod \"auth-test-model-43be82-kserve-5c6dfd4dfc-n4dl7\" (UID: \"750b979e-ef5c-4c06-b456-d1c6cdce27e5\") " pod="ai-tenant-e2e-shared-b-w0-43be82/auth-test-model-43be82-kserve-5c6dfd4dfc-n4dl7"
Nov 02 12:56:34.937188 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:56:34.937116    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/750b979e-ef5c-4c06-b456-d1c6cdce27e5-tls-certs\") pod \"auth-test-model-43be82-kserve-5c6dfd4dfc-n4dl7\" (UID: \"750b979e-ef5c-4c06-b456-d1c6cdce27e5\") " pod="ai-tenant-e2e-shared-b-w0-43be82/auth-test-model-43be82-kserve-5c6dfd4dfc-n4dl7"
Nov 02 12:56:34.937468 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:56:34.937445    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/750b979e-ef5c-4c06-b456-d1c6cdce27e5-home\") pod \"auth-test-model-43be82-kserve-5c6dfd4dfc-n4dl7\" (UID: \"750b979e-ef5c-4c06-b456-d1c6cdce27e5\") " pod="ai-tenant-e2e-shared-b-w0-43be82/auth-test-model-43be82-kserve-5c6dfd4dfc-n4dl7"
Nov 02 12:56:34.937538 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:56:34.937518    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/750b979e-ef5c-4c06-b456-d1c6cdce27e5-model-cache\") pod \"auth-test-model-43be82-kserve-5c6dfd4dfc-n4dl7\" (UID: \"750b979e-ef5c-4c06-b456-d1c6cdce27e5\") " pod="ai-tenant-e2e-shared-b-w0-43be82/auth-test-model-43be82-kserve-5c6dfd4dfc-n4dl7"
Nov 02 12:56:35.059723 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:56:35.059553    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/750b979e-ef5c-4c06-b456-d1c6cdce27e5-home\") pod \"auth-test-model-43be82-kserve-5c6dfd4dfc-n4dl7\" (UID: \"750b979e-ef5c-4c06-b456-d1c6cdce27e5\") " pod="ai-tenant-e2e-shared-b-w0-43be82/auth-test-model-43be82-kserve-5c6dfd4dfc-n4dl7"
Nov 02 12:56:35.059723 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:56:35.059663    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/750b979e-ef5c-4c06-b456-d1c6cdce27e5-model-cache\") pod \"auth-test-model-43be82-kserve-5c6dfd4dfc-n4dl7\" (UID: \"750b979e-ef5c-4c06-b456-d1c6cdce27e5\") " pod="ai-tenant-e2e-shared-b-w0-43be82/auth-test-model-43be82-kserve-5c6dfd4dfc-n4dl7"
Nov 02 12:56:35.059969 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:56:35.059780    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/750b979e-ef5c-4c06-b456-d1c6cdce27e5-dshm\") pod \"auth-test-model-43be82-kserve-5c6dfd4dfc-n4dl7\" (UID: \"750b979e-ef5c-4c06-b456-d1c6cdce27e5\") " pod="ai-tenant-e2e-shared-b-w0-43be82/auth-test-model-43be82-kserve-5c6dfd4dfc-n4dl7"
Nov 02 12:56:35.059969 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:56:35.059813    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbmf5\" (UniqueName: \"kubernetes.io/projected/750b979e-ef5c-4c06-b456-d1c6cdce27e5-kube-api-access-kbmf5\") pod \"auth-test-model-43be82-kserve-5c6dfd4dfc-n4dl7\" (UID: \"750b979e-ef5c-4c06-b456-d1c6cdce27e5\") " pod="ai-tenant-e2e-shared-b-w0-43be82/auth-test-model-43be82-kserve-5c6dfd4dfc-n4dl7"
Nov 02 12:56:35.059969 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:56:35.059867    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/750b979e-ef5c-4c06-b456-d1c6cdce27e5-tmp-dir\") pod \"auth-test-model-43be82-kserve-5c6dfd4dfc-n4dl7\" (UID: \"750b979e-ef5c-4c06-b456-d1c6cdce27e5\") " pod="ai-tenant-e2e-shared-b-w0-43be82/auth-test-model-43be82-kserve-5c6dfd4dfc-n4dl7"
Nov 02 12:56:35.059969 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:56:35.059927    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/750b979e-ef5c-4c06-b456-d1c6cdce27e5-tls-certs\") pod \"auth-test-model-43be82-kserve-5c6dfd4dfc-n4dl7\" (UID: \"750b979e-ef5c-4c06-b456-d1c6cdce27e5\") " pod="ai-tenant-e2e-shared-b-w0-43be82/auth-test-model-43be82-kserve-5c6dfd4dfc-n4dl7"
Nov 02 12:56:35.071663 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:56:35.065335    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/750b979e-ef5c-4c06-b456-d1c6cdce27e5-home\") pod \"auth-test-model-43be82-kserve-5c6dfd4dfc-n4dl7\" (UID: \"750b979e-ef5c-4c06-b456-d1c6cdce27e5\") " pod="ai-tenant-e2e-shared-b-w0-43be82/auth-test-model-43be82-kserve-5c6dfd4dfc-n4dl7"
Nov 02 12:56:35.071663 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:56:35.065595    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/750b979e-ef5c-4c06-b456-d1c6cdce27e5-tmp-dir\") pod \"auth-test-model-43be82-kserve-5c6dfd4dfc-n4dl7\" (UID: \"750b979e-ef5c-4c06-b456-d1c6cdce27e5\") " pod="ai-tenant-e2e-shared-b-w0-43be82/auth-test-model-43be82-kserve-5c6dfd4dfc-n4dl7"
Nov 02 12:56:35.071663 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:56:35.065866    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/750b979e-ef5c-4c06-b456-d1c6cdce27e5-model-cache\") pod \"auth-test-model-43be82-kserve-5c6dfd4dfc-n4dl7\" (UID: \"750b979e-ef5c-4c06-b456-d1c6cdce27e5\") " pod="ai-tenant-e2e-shared-b-w0-43be82/auth-test-model-43be82-kserve-5c6dfd4dfc-n4dl7"
Nov 02 12:56:35.100697 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:56:35.098117    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/750b979e-ef5c-4c06-b456-d1c6cdce27e5-dshm\") pod \"auth-test-model-43be82-kserve-5c6dfd4dfc-n4dl7\" (UID: \"750b979e-ef5c-4c06-b456-d1c6cdce27e5\") " pod="ai-tenant-e2e-shared-b-w0-43be82/auth-test-model-43be82-kserve-5c6dfd4dfc-n4dl7"
Nov 02 12:56:35.141661 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:56:35.135067    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/750b979e-ef5c-4c06-b456-d1c6cdce27e5-tls-certs\") pod \"auth-test-model-43be82-kserve-5c6dfd4dfc-n4dl7\" (UID: \"750b979e-ef5c-4c06-b456-d1c6cdce27e5\") " pod="ai-tenant-e2e-shared-b-w0-43be82/auth-test-model-43be82-kserve-5c6dfd4dfc-n4dl7"
Nov 02 12:56:35.141661 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:56:35.135871    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbmf5\" (UniqueName: \"kubernetes.io/projected/750b979e-ef5c-4c06-b456-d1c6cdce27e5-kube-api-access-kbmf5\") pod \"auth-test-model-43be82-kserve-5c6dfd4dfc-n4dl7\" (UID: \"750b979e-ef5c-4c06-b456-d1c6cdce27e5\") " pod="ai-tenant-e2e-shared-b-w0-43be82/auth-test-model-43be82-kserve-5c6dfd4dfc-n4dl7"
Nov 02 12:56:35.420510 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:56:35.420476    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-43be82/auth-test-model-43be82-kserve-5c6dfd4dfc-n4dl7"
Nov 02 12:56:35.734488 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:56:35.734396    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod750b979e_ef5c_4c06_b456_d1c6cdce27e5.slice/crio-f4f229d9c68ad689d14a1e88d4520b1975f429e264a5855ee470d6ec5a0d79c5 WatchSource:0}: Error finding container f4f229d9c68ad689d14a1e88d4520b1975f429e264a5855ee470d6ec5a0d79c5: Status 404 returned error can't find the container with id f4f229d9c68ad689d14a1e88d4520b1975f429e264a5855ee470d6ec5a0d79c5
Nov 02 12:56:35.737886 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:56:35.737007    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-43be82/auth-test-model-43be82-kserve-5c6dfd4dfc-n4dl7"]
Nov 02 12:56:36.117668 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:56:36.117570    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-43be82/auth-test-model-43be82-kserve-5c6dfd4dfc-n4dl7" event={"ID":"750b979e-ef5c-4c06-b456-d1c6cdce27e5","Type":"ContainerStarted","Data":"7c04f3ca2a3fe91f92a0ed51e71dd8fca1238eb7ac2fbe84f9813f0388d570cb"}
Nov 02 12:56:36.117668 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:56:36.117608    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-43be82/auth-test-model-43be82-kserve-5c6dfd4dfc-n4dl7" event={"ID":"750b979e-ef5c-4c06-b456-d1c6cdce27e5","Type":"ContainerStarted","Data":"f4f229d9c68ad689d14a1e88d4520b1975f429e264a5855ee470d6ec5a0d79c5"}
Nov 02 12:56:36.162466 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:56:36.162416    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w0-43be82/auth-test-model-43be82-kserve-5c6dfd4dfc-n4dl7" podStartSLOduration=2.162403709 podStartE2EDuration="2.162403709s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:56:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:56:36.159624447 +0000 UTC m=+1792.415023542" watchObservedRunningTime="2026-11-02 12:56:36.162403709 +0000 UTC m=+1792.417802803"
Nov 02 12:56:36.423770 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:56:36.423733    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w0-43be82/auth-test-model-43be82-kserve-5c6dfd4dfc-n4dl7"
Nov 02 12:56:37.162774 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:56:37.162739    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"]
Nov 02 12:56:45.087001 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:56:45.086960    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqcmj" containerName="extractor"
Nov 02 12:56:45.420606 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:56:45.420571    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w0-43be82/auth-test-model-43be82-kserve-5c6dfd4dfc-n4dl7"
Nov 02 12:56:45.434025 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:56:45.433995    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w0-43be82/auth-test-model-43be82-kserve-5c6dfd4dfc-n4dl7"
Nov 02 12:56:45.891810 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:56:45.891721    4559 scope.go:122] "RemoveContainer" containerID="58c789e83735b6170c1b7ed603ece9861195ccf4eadf8b944732624640fadca3"
Nov 02 12:56:45.903343 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:56:45.903320    4559 scope.go:122] "RemoveContainer" containerID="81175c8b270bc8b000dfd7e74444dd2c44e7c29a7979bd7a11ee8c31591e9ac6"
Nov 02 12:56:46.265326 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:56:46.265295    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w0-43be82/auth-test-model-43be82-kserve-5c6dfd4dfc-n4dl7"
Nov 02 12:57:04.234060 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:57:04.234024    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-c0d4cb-98c64fd4c-qs6bn"]
Nov 02 12:57:04.272099 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:57:04.272055    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-c0d4cb-867cbccf7f-pmph4"]
Nov 02 12:57:09.234571 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:57:09.234494    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-c0d4cb-98c64fd4c-qs6bn" podUID="bd77be34-4903-4c63-82d2-1b8951d975de" containerName="payload-processing" containerID="cri-o://2f125c060abb91d6014b62c80b724ff0148325bc98e2f042e0dfd25a95f74524" gracePeriod=25
Nov 02 12:57:09.293023 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:57:09.292710    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c0d4cb-867cbccf7f-pmph4" podUID="97e9abd0-ac59-4b5a-88cf-174dbf24c305" containerName="payload-pre-processing" containerID="cri-o://f755ee04d01c7f3d58b850f2df7e42492d23991c912151676e95a1c41a82da52" gracePeriod=25
Nov 02 12:57:13.455892 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:57:13.455845    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-praxis-a-c0d4cb-98c64fd4c-qs6bn" podUID="bd77be34-4903-4c63-82d2-1b8951d975de" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 12:57:13.455892 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:57:13.455883    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c0d4cb-867cbccf7f-pmph4" podUID="97e9abd0-ac59-4b5a-88cf-174dbf24c305" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 12:57:15.089569 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:57:15.089533    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqcmj" containerName="extractor"
Nov 02 12:57:23.457495 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:57:23.457419    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-praxis-a-c0d4cb-98c64fd4c-qs6bn" podUID="bd77be34-4903-4c63-82d2-1b8951d975de" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 12:57:23.458149 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:57:23.458039    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c0d4cb-867cbccf7f-pmph4" podUID="97e9abd0-ac59-4b5a-88cf-174dbf24c305" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 12:57:25.846814 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:57:25.846730    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"]
Nov 02 12:57:29.597604 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:57:29.597559    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-c0d4cb-7465cdd564-w4nwd"]
Nov 02 12:57:29.598300 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:57:29.597855    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c0d4cb-7465cdd564-w4nwd" podUID="5baf3254-0f52-4b0a-8b31-9ae96e0aa816" containerName="maas-api" containerID="cri-o://e70dc7b23a567a95b8fb6b3c0d4c98b6d278d7796d99057b7b2f5bd6d96b2bb5" gracePeriod=30
Nov 02 12:57:29.616661 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:57:29.614279    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-c0d4cb-98c64fd4c-qs6bn"
Nov 02 12:57:29.680444 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:57:29.680408    4559 generic.go:356] "Generic (PLEG): container finished" podID="97e9abd0-ac59-4b5a-88cf-174dbf24c305" containerID="f755ee04d01c7f3d58b850f2df7e42492d23991c912151676e95a1c41a82da52" exitCode=0
Nov 02 12:57:29.680587 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:57:29.680479    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c0d4cb-867cbccf7f-pmph4" event={"ID":"97e9abd0-ac59-4b5a-88cf-174dbf24c305","Type":"ContainerDied","Data":"f755ee04d01c7f3d58b850f2df7e42492d23991c912151676e95a1c41a82da52"}
Nov 02 12:57:29.680587 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:57:29.680533    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c0d4cb-867cbccf7f-pmph4" event={"ID":"97e9abd0-ac59-4b5a-88cf-174dbf24c305","Type":"ContainerDied","Data":"7cedc7e2c23066a56491c6aaf9023984fa22a602e7516ec9a1c2fa1d56e9da3f"}
Nov 02 12:57:29.680587 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:57:29.680551    4559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7cedc7e2c23066a56491c6aaf9023984fa22a602e7516ec9a1c2fa1d56e9da3f"
Nov 02 12:57:29.682115 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:57:29.682090    4559 generic.go:356] "Generic (PLEG): container finished" podID="bd77be34-4903-4c63-82d2-1b8951d975de" containerID="2f125c060abb91d6014b62c80b724ff0148325bc98e2f042e0dfd25a95f74524" exitCode=0
Nov 02 12:57:29.682253 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:57:29.682150    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-c0d4cb-98c64fd4c-qs6bn"
Nov 02 12:57:29.682253 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:57:29.682172    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-c0d4cb-98c64fd4c-qs6bn" event={"ID":"bd77be34-4903-4c63-82d2-1b8951d975de","Type":"ContainerDied","Data":"2f125c060abb91d6014b62c80b724ff0148325bc98e2f042e0dfd25a95f74524"}
Nov 02 12:57:29.682253 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:57:29.682218    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-c0d4cb-98c64fd4c-qs6bn" event={"ID":"bd77be34-4903-4c63-82d2-1b8951d975de","Type":"ContainerDied","Data":"1627f0fe3169f288164526923535aced0bf463523961729bd95886de47154a53"}
Nov 02 12:57:29.682253 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:57:29.682244    4559 scope.go:122] "RemoveContainer" containerID="2f125c060abb91d6014b62c80b724ff0148325bc98e2f042e0dfd25a95f74524"
Nov 02 12:57:29.694918 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:57:29.694888    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c0d4cb-867cbccf7f-pmph4"
Nov 02 12:57:29.695249 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:57:29.695228    4559 scope.go:122] "RemoveContainer" containerID="2f125c060abb91d6014b62c80b724ff0148325bc98e2f042e0dfd25a95f74524"
Nov 02 12:57:29.695538 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:57:29.695515    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f125c060abb91d6014b62c80b724ff0148325bc98e2f042e0dfd25a95f74524\": container with ID starting with 2f125c060abb91d6014b62c80b724ff0148325bc98e2f042e0dfd25a95f74524 not found: ID does not exist" containerID="2f125c060abb91d6014b62c80b724ff0148325bc98e2f042e0dfd25a95f74524"
Nov 02 12:57:29.695667 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:57:29.695545    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f125c060abb91d6014b62c80b724ff0148325bc98e2f042e0dfd25a95f74524"} err="failed to get container status \"2f125c060abb91d6014b62c80b724ff0148325bc98e2f042e0dfd25a95f74524\": rpc error: code = NotFound desc = could not find container \"2f125c060abb91d6014b62c80b724ff0148325bc98e2f042e0dfd25a95f74524\": container with ID starting with 2f125c060abb91d6014b62c80b724ff0148325bc98e2f042e0dfd25a95f74524 not found: ID does not exist"
Nov 02 12:57:29.753694 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:57:29.753660    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd77be34-4903-4c63-82d2-1b8951d975de-kube-api-access-qqhx9\" (UniqueName: \"kubernetes.io/projected/bd77be34-4903-4c63-82d2-1b8951d975de-kube-api-access-qqhx9\") pod \"bd77be34-4903-4c63-82d2-1b8951d975de\" (UID: \"bd77be34-4903-4c63-82d2-1b8951d975de\") "
Nov 02 12:57:29.753887 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:57:29.753805    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd77be34-4903-4c63-82d2-1b8951d975de-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd77be34-4903-4c63-82d2-1b8951d975de-plugins-config-volume\") pod \"bd77be34-4903-4c63-82d2-1b8951d975de\" (UID: \"bd77be34-4903-4c63-82d2-1b8951d975de\") "
Nov 02 12:57:29.754261 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:57:29.754222    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd77be34-4903-4c63-82d2-1b8951d975de-plugins-config-volume" pod "bd77be34-4903-4c63-82d2-1b8951d975de" (UID: "bd77be34-4903-4c63-82d2-1b8951d975de"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:57:29.754631 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:57:29.754610    4559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd77be34-4903-4c63-82d2-1b8951d975de-plugins-config-volume\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:57:29.756094 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:57:29.756063    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd77be34-4903-4c63-82d2-1b8951d975de-kube-api-access-qqhx9" pod "bd77be34-4903-4c63-82d2-1b8951d975de" (UID: "bd77be34-4903-4c63-82d2-1b8951d975de"). InnerVolumeSpecName "kube-api-access-qqhx9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:57:29.862720 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:57:29.858254    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97e9abd0-ac59-4b5a-88cf-174dbf24c305-kube-api-access-h8hrp\" (UniqueName: \"kubernetes.io/projected/97e9abd0-ac59-4b5a-88cf-174dbf24c305-kube-api-access-h8hrp\") pod \"97e9abd0-ac59-4b5a-88cf-174dbf24c305\" (UID: \"97e9abd0-ac59-4b5a-88cf-174dbf24c305\") "
Nov 02 12:57:29.862720 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:57:29.858337    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97e9abd0-ac59-4b5a-88cf-174dbf24c305-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97e9abd0-ac59-4b5a-88cf-174dbf24c305-plugins-config-volume\") pod \"97e9abd0-ac59-4b5a-88cf-174dbf24c305\" (UID: \"97e9abd0-ac59-4b5a-88cf-174dbf24c305\") "
Nov 02 12:57:29.862720 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:57:29.859160    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqhx9\" (UniqueName: \"kubernetes.io/projected/bd77be34-4903-4c63-82d2-1b8951d975de-kube-api-access-qqhx9\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:57:29.890662 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:57:29.889996    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97e9abd0-ac59-4b5a-88cf-174dbf24c305-kube-api-access-h8hrp" pod "97e9abd0-ac59-4b5a-88cf-174dbf24c305" (UID: "97e9abd0-ac59-4b5a-88cf-174dbf24c305"). InnerVolumeSpecName "kube-api-access-h8hrp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:57:29.939876 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:57:29.939831    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97e9abd0-ac59-4b5a-88cf-174dbf24c305-plugins-config-volume" pod "97e9abd0-ac59-4b5a-88cf-174dbf24c305" (UID: "97e9abd0-ac59-4b5a-88cf-174dbf24c305"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:57:29.967666 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:57:29.961471    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h8hrp\" (UniqueName: \"kubernetes.io/projected/97e9abd0-ac59-4b5a-88cf-174dbf24c305-kube-api-access-h8hrp\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:57:29.967666 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:57:29.961512    4559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97e9abd0-ac59-4b5a-88cf-174dbf24c305-plugins-config-volume\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:57:30.181863 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:57:30.181821    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-c0d4cb-98c64fd4c-qs6bn"]
Nov 02 12:57:30.223975 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:57:30.223934    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-c0d4cb-98c64fd4c-qs6bn"]
Nov 02 12:57:30.508111 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:57:30.508085    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c0d4cb-7465cdd564-w4nwd"
Nov 02 12:57:30.513131 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:57:30.513105    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5baf3254-0f52-4b0a-8b31-9ae96e0aa816-kube-api-access-gc9rp\" (UniqueName: \"kubernetes.io/projected/5baf3254-0f52-4b0a-8b31-9ae96e0aa816-kube-api-access-gc9rp\") pod \"5baf3254-0f52-4b0a-8b31-9ae96e0aa816\" (UID: \"5baf3254-0f52-4b0a-8b31-9ae96e0aa816\") "
Nov 02 12:57:30.513253 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:57:30.513217    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5baf3254-0f52-4b0a-8b31-9ae96e0aa816-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5baf3254-0f52-4b0a-8b31-9ae96e0aa816-maas-api-tls\") pod \"5baf3254-0f52-4b0a-8b31-9ae96e0aa816\" (UID: \"5baf3254-0f52-4b0a-8b31-9ae96e0aa816\") "
Nov 02 12:57:30.513319 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:57:30.513283    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5baf3254-0f52-4b0a-8b31-9ae96e0aa816-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5baf3254-0f52-4b0a-8b31-9ae96e0aa816-openshift-service-ca\") pod \"5baf3254-0f52-4b0a-8b31-9ae96e0aa816\" (UID: \"5baf3254-0f52-4b0a-8b31-9ae96e0aa816\") "
Nov 02 12:57:30.513379 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:57:30.513318    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5baf3254-0f52-4b0a-8b31-9ae96e0aa816-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5baf3254-0f52-4b0a-8b31-9ae96e0aa816-metrics-certs\") pod \"5baf3254-0f52-4b0a-8b31-9ae96e0aa816\" (UID: \"5baf3254-0f52-4b0a-8b31-9ae96e0aa816\") "
Nov 02 12:57:30.513741 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:57:30.513707    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5baf3254-0f52-4b0a-8b31-9ae96e0aa816-openshift-service-ca" pod "5baf3254-0f52-4b0a-8b31-9ae96e0aa816" (UID: "5baf3254-0f52-4b0a-8b31-9ae96e0aa816"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:57:30.514074 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:57:30.514050    4559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5baf3254-0f52-4b0a-8b31-9ae96e0aa816-openshift-service-ca\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:57:30.515369 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:57:30.515338    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5baf3254-0f52-4b0a-8b31-9ae96e0aa816-maas-api-tls" pod "5baf3254-0f52-4b0a-8b31-9ae96e0aa816" (UID: "5baf3254-0f52-4b0a-8b31-9ae96e0aa816"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:57:30.515473 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:57:30.515402    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5baf3254-0f52-4b0a-8b31-9ae96e0aa816-kube-api-access-gc9rp" pod "5baf3254-0f52-4b0a-8b31-9ae96e0aa816" (UID: "5baf3254-0f52-4b0a-8b31-9ae96e0aa816"). InnerVolumeSpecName "kube-api-access-gc9rp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:57:30.515473 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:57:30.515440    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5baf3254-0f52-4b0a-8b31-9ae96e0aa816-metrics-certs" pod "5baf3254-0f52-4b0a-8b31-9ae96e0aa816" (UID: "5baf3254-0f52-4b0a-8b31-9ae96e0aa816"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:57:30.616035 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:57:30.615993    4559 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5baf3254-0f52-4b0a-8b31-9ae96e0aa816-metrics-certs\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:57:30.616035 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:57:30.616029    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gc9rp\" (UniqueName: \"kubernetes.io/projected/5baf3254-0f52-4b0a-8b31-9ae96e0aa816-kube-api-access-gc9rp\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:57:30.616035 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:57:30.616043    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5baf3254-0f52-4b0a-8b31-9ae96e0aa816-maas-api-tls\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:57:30.690357 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:57:30.690320    4559 generic.go:356] "Generic (PLEG): container finished" podID="5baf3254-0f52-4b0a-8b31-9ae96e0aa816" containerID="e70dc7b23a567a95b8fb6b3c0d4c98b6d278d7796d99057b7b2f5bd6d96b2bb5" exitCode=0
Nov 02 12:57:30.690568 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:57:30.690378    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c0d4cb-7465cdd564-w4nwd"
Nov 02 12:57:30.690568 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:57:30.690402    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c0d4cb-7465cdd564-w4nwd" event={"ID":"5baf3254-0f52-4b0a-8b31-9ae96e0aa816","Type":"ContainerDied","Data":"e70dc7b23a567a95b8fb6b3c0d4c98b6d278d7796d99057b7b2f5bd6d96b2bb5"}
Nov 02 12:57:30.690568 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:57:30.690458    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c0d4cb-7465cdd564-w4nwd" event={"ID":"5baf3254-0f52-4b0a-8b31-9ae96e0aa816","Type":"ContainerDied","Data":"392efa535702c442012810a6dccb7aa209134eefd144046d19cac0375246ede9"}
Nov 02 12:57:30.690568 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:57:30.690486    4559 scope.go:122] "RemoveContainer" containerID="e70dc7b23a567a95b8fb6b3c0d4c98b6d278d7796d99057b7b2f5bd6d96b2bb5"
Nov 02 12:57:30.691808 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:57:30.691790    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c0d4cb-867cbccf7f-pmph4"
Nov 02 12:57:30.705169 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:57:30.703784    4559 scope.go:122] "RemoveContainer" containerID="e70dc7b23a567a95b8fb6b3c0d4c98b6d278d7796d99057b7b2f5bd6d96b2bb5"
Nov 02 12:57:30.706515 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:57:30.706487    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e70dc7b23a567a95b8fb6b3c0d4c98b6d278d7796d99057b7b2f5bd6d96b2bb5\": container with ID starting with e70dc7b23a567a95b8fb6b3c0d4c98b6d278d7796d99057b7b2f5bd6d96b2bb5 not found: ID does not exist" containerID="e70dc7b23a567a95b8fb6b3c0d4c98b6d278d7796d99057b7b2f5bd6d96b2bb5"
Nov 02 12:57:30.706618 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:57:30.706522    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e70dc7b23a567a95b8fb6b3c0d4c98b6d278d7796d99057b7b2f5bd6d96b2bb5"} err="failed to get container status \"e70dc7b23a567a95b8fb6b3c0d4c98b6d278d7796d99057b7b2f5bd6d96b2bb5\": rpc error: code = NotFound desc = could not find container \"e70dc7b23a567a95b8fb6b3c0d4c98b6d278d7796d99057b7b2f5bd6d96b2bb5\": container with ID starting with e70dc7b23a567a95b8fb6b3c0d4c98b6d278d7796d99057b7b2f5bd6d96b2bb5 not found: ID does not exist"
Nov 02 12:57:30.745779 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:57:30.745701    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-c0d4cb-867cbccf7f-pmph4"]
Nov 02 12:57:30.751606 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:57:30.751575    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-c0d4cb-867cbccf7f-pmph4"]
Nov 02 12:57:30.785913 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:57:30.785873    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-c0d4cb-7465cdd564-w4nwd"]
Nov 02 12:57:30.791977 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:57:30.791945    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-c0d4cb-7465cdd564-w4nwd"]
Nov 02 12:57:31.919788 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:57:31.919747    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5baf3254-0f52-4b0a-8b31-9ae96e0aa816" path="/var/lib/kubelet/pods/5baf3254-0f52-4b0a-8b31-9ae96e0aa816/volumes"
Nov 02 12:57:31.921701 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:57:31.921306    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97e9abd0-ac59-4b5a-88cf-174dbf24c305" path="/var/lib/kubelet/pods/97e9abd0-ac59-4b5a-88cf-174dbf24c305/volumes"
Nov 02 12:57:31.939664 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:57:31.922810    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd77be34-4903-4c63-82d2-1b8951d975de" path="/var/lib/kubelet/pods/bd77be34-4903-4c63-82d2-1b8951d975de/volumes"
Nov 02 12:57:32.856404 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:57:32.856368    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-c0d4cb-openshift-default-6798dbb74f-rslvw"]
Nov 02 12:57:32.856695 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:57:32.856656    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-c0d4cb-openshift-default-6798dbb74f-rslvw" podUID="24bcb862-30cd-44b9-920e-a919d80678a9" containerName="istio-proxy" containerID="cri-o://a52dff0ab53917cde9b60354d9e55e43edabe19576d96c9498ede76ab44a91ca" gracePeriod=30
Nov 02 12:57:38.154386 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:57:38.154353    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-c0d4cb-openshift-default-6798dbb74f-rslvw"
Nov 02 12:57:38.190185 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:57:38.190149    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24bcb862-30cd-44b9-920e-a919d80678a9-kube-api-access-2sxww\" (UniqueName: \"kubernetes.io/projected/24bcb862-30cd-44b9-920e-a919d80678a9-kube-api-access-2sxww\") pod \"24bcb862-30cd-44b9-920e-a919d80678a9\" (UID: \"24bcb862-30cd-44b9-920e-a919d80678a9\") "
Nov 02 12:57:38.190383 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:57:38.190193    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24bcb862-30cd-44b9-920e-a919d80678a9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24bcb862-30cd-44b9-920e-a919d80678a9-credential-socket\") pod \"24bcb862-30cd-44b9-920e-a919d80678a9\" (UID: \"24bcb862-30cd-44b9-920e-a919d80678a9\") "
Nov 02 12:57:38.190383 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:57:38.190255    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24bcb862-30cd-44b9-920e-a919d80678a9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24bcb862-30cd-44b9-920e-a919d80678a9-workload-socket\") pod \"24bcb862-30cd-44b9-920e-a919d80678a9\" (UID: \"24bcb862-30cd-44b9-920e-a919d80678a9\") "
Nov 02 12:57:38.190383 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:57:38.190332    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24bcb862-30cd-44b9-920e-a919d80678a9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24bcb862-30cd-44b9-920e-a919d80678a9-workload-certs\") pod \"24bcb862-30cd-44b9-920e-a919d80678a9\" (UID: \"24bcb862-30cd-44b9-920e-a919d80678a9\") "
Nov 02 12:57:38.190573 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:57:38.190401    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/24bcb862-30cd-44b9-920e-a919d80678a9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24bcb862-30cd-44b9-920e-a919d80678a9-istio-podinfo\") pod \"24bcb862-30cd-44b9-920e-a919d80678a9\" (UID: \"24bcb862-30cd-44b9-920e-a919d80678a9\") "
Nov 02 12:57:38.190573 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:57:38.190489    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24bcb862-30cd-44b9-920e-a919d80678a9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24bcb862-30cd-44b9-920e-a919d80678a9-istio-data\") pod \"24bcb862-30cd-44b9-920e-a919d80678a9\" (UID: \"24bcb862-30cd-44b9-920e-a919d80678a9\") "
Nov 02 12:57:38.190573 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:57:38.190506    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24bcb862-30cd-44b9-920e-a919d80678a9-credential-socket" pod "24bcb862-30cd-44b9-920e-a919d80678a9" (UID: "24bcb862-30cd-44b9-920e-a919d80678a9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:57:38.190573 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:57:38.190553    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24bcb862-30cd-44b9-920e-a919d80678a9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24bcb862-30cd-44b9-920e-a919d80678a9-istio-envoy\") pod \"24bcb862-30cd-44b9-920e-a919d80678a9\" (UID: \"24bcb862-30cd-44b9-920e-a919d80678a9\") "
Nov 02 12:57:38.190573 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:57:38.190557    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24bcb862-30cd-44b9-920e-a919d80678a9-workload-socket" pod "24bcb862-30cd-44b9-920e-a919d80678a9" (UID: "24bcb862-30cd-44b9-920e-a919d80678a9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:57:38.190881 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:57:38.190589    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24bcb862-30cd-44b9-920e-a919d80678a9-workload-certs" pod "24bcb862-30cd-44b9-920e-a919d80678a9" (UID: "24bcb862-30cd-44b9-920e-a919d80678a9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:57:38.190881 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:57:38.190618    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24bcb862-30cd-44b9-920e-a919d80678a9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24bcb862-30cd-44b9-920e-a919d80678a9-istiod-ca-cert\") pod \"24bcb862-30cd-44b9-920e-a919d80678a9\" (UID: \"24bcb862-30cd-44b9-920e-a919d80678a9\") "
Nov 02 12:57:38.190881 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:57:38.190758    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24bcb862-30cd-44b9-920e-a919d80678a9-istio-token\" (UniqueName: \"kubernetes.io/projected/24bcb862-30cd-44b9-920e-a919d80678a9-istio-token\") pod \"24bcb862-30cd-44b9-920e-a919d80678a9\" (UID: \"24bcb862-30cd-44b9-920e-a919d80678a9\") "
Nov 02 12:57:38.190881 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:57:38.190768    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24bcb862-30cd-44b9-920e-a919d80678a9-istio-data" pod "24bcb862-30cd-44b9-920e-a919d80678a9" (UID: "24bcb862-30cd-44b9-920e-a919d80678a9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:57:38.191192 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:57:38.191004    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24bcb862-30cd-44b9-920e-a919d80678a9-istiod-ca-cert" pod "24bcb862-30cd-44b9-920e-a919d80678a9" (UID: "24bcb862-30cd-44b9-920e-a919d80678a9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:57:38.191440 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:57:38.191421    4559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24bcb862-30cd-44b9-920e-a919d80678a9-istio-data\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:57:38.191520 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:57:38.191443    4559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24bcb862-30cd-44b9-920e-a919d80678a9-istiod-ca-cert\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:57:38.191520 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:57:38.191455    4559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24bcb862-30cd-44b9-920e-a919d80678a9-credential-socket\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:57:38.191520 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:57:38.191464    4559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24bcb862-30cd-44b9-920e-a919d80678a9-workload-socket\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:57:38.191520 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:57:38.191473    4559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24bcb862-30cd-44b9-920e-a919d80678a9-workload-certs\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:57:38.193124 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:57:38.193099    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24bcb862-30cd-44b9-920e-a919d80678a9-kube-api-access-2sxww" pod "24bcb862-30cd-44b9-920e-a919d80678a9" (UID: "24bcb862-30cd-44b9-920e-a919d80678a9"). InnerVolumeSpecName "kube-api-access-2sxww". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:57:38.193248 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:57:38.193136    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24bcb862-30cd-44b9-920e-a919d80678a9-istio-token" pod "24bcb862-30cd-44b9-920e-a919d80678a9" (UID: "24bcb862-30cd-44b9-920e-a919d80678a9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:57:38.193248 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:57:38.193173    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24bcb862-30cd-44b9-920e-a919d80678a9-istio-envoy" pod "24bcb862-30cd-44b9-920e-a919d80678a9" (UID: "24bcb862-30cd-44b9-920e-a919d80678a9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:57:38.193248 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:57:38.193179    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/24bcb862-30cd-44b9-920e-a919d80678a9-istio-podinfo" pod "24bcb862-30cd-44b9-920e-a919d80678a9" (UID: "24bcb862-30cd-44b9-920e-a919d80678a9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 12:57:38.292795 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:57:38.292758    4559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24bcb862-30cd-44b9-920e-a919d80678a9-istio-token\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:57:38.292795 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:57:38.292790    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2sxww\" (UniqueName: \"kubernetes.io/projected/24bcb862-30cd-44b9-920e-a919d80678a9-kube-api-access-2sxww\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:57:38.292795 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:57:38.292801    4559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24bcb862-30cd-44b9-920e-a919d80678a9-istio-podinfo\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:57:38.293041 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:57:38.292810    4559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24bcb862-30cd-44b9-920e-a919d80678a9-istio-envoy\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:57:38.753850 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:57:38.753815    4559 generic.go:356] "Generic (PLEG): container finished" podID="24bcb862-30cd-44b9-920e-a919d80678a9" containerID="a52dff0ab53917cde9b60354d9e55e43edabe19576d96c9498ede76ab44a91ca" exitCode=0
Nov 02 12:57:38.754061 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:57:38.753873    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-c0d4cb-openshift-default-6798dbb74f-rslvw"
Nov 02 12:57:38.754061 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:57:38.753893    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-c0d4cb-openshift-default-6798dbb74f-rslvw" event={"ID":"24bcb862-30cd-44b9-920e-a919d80678a9","Type":"ContainerDied","Data":"a52dff0ab53917cde9b60354d9e55e43edabe19576d96c9498ede76ab44a91ca"}
Nov 02 12:57:38.754061 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:57:38.753963    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-c0d4cb-openshift-default-6798dbb74f-rslvw" event={"ID":"24bcb862-30cd-44b9-920e-a919d80678a9","Type":"ContainerDied","Data":"0c87267e2e1782df03326cd495e0dc9251a6247360e71e0359604da21728b88b"}
Nov 02 12:57:38.754061 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:57:38.753994    4559 scope.go:122] "RemoveContainer" containerID="a52dff0ab53917cde9b60354d9e55e43edabe19576d96c9498ede76ab44a91ca"
Nov 02 12:57:38.768046 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:57:38.768021    4559 scope.go:122] "RemoveContainer" containerID="693fedc45dfde9ab92b41dc6b96b36bffe723cac46d4a469e5d74ee9285ce816"
Nov 02 12:57:38.778101 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:57:38.778077    4559 scope.go:122] "RemoveContainer" containerID="a52dff0ab53917cde9b60354d9e55e43edabe19576d96c9498ede76ab44a91ca"
Nov 02 12:57:38.778412 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:57:38.778387    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a52dff0ab53917cde9b60354d9e55e43edabe19576d96c9498ede76ab44a91ca\": container with ID starting with a52dff0ab53917cde9b60354d9e55e43edabe19576d96c9498ede76ab44a91ca not found: ID does not exist" containerID="a52dff0ab53917cde9b60354d9e55e43edabe19576d96c9498ede76ab44a91ca"
Nov 02 12:57:38.778467 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:57:38.778422    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a52dff0ab53917cde9b60354d9e55e43edabe19576d96c9498ede76ab44a91ca"} err="failed to get container status \"a52dff0ab53917cde9b60354d9e55e43edabe19576d96c9498ede76ab44a91ca\": rpc error: code = NotFound desc = could not find container \"a52dff0ab53917cde9b60354d9e55e43edabe19576d96c9498ede76ab44a91ca\": container with ID starting with a52dff0ab53917cde9b60354d9e55e43edabe19576d96c9498ede76ab44a91ca not found: ID does not exist"
Nov 02 12:57:38.778467 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:57:38.778442    4559 scope.go:122] "RemoveContainer" containerID="693fedc45dfde9ab92b41dc6b96b36bffe723cac46d4a469e5d74ee9285ce816"
Nov 02 12:57:38.778720 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:57:38.778683    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"693fedc45dfde9ab92b41dc6b96b36bffe723cac46d4a469e5d74ee9285ce816\": container with ID starting with 693fedc45dfde9ab92b41dc6b96b36bffe723cac46d4a469e5d74ee9285ce816 not found: ID does not exist" containerID="693fedc45dfde9ab92b41dc6b96b36bffe723cac46d4a469e5d74ee9285ce816"
Nov 02 12:57:38.778791 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:57:38.778734    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"693fedc45dfde9ab92b41dc6b96b36bffe723cac46d4a469e5d74ee9285ce816"} err="failed to get container status \"693fedc45dfde9ab92b41dc6b96b36bffe723cac46d4a469e5d74ee9285ce816\": rpc error: code = NotFound desc = could not find container \"693fedc45dfde9ab92b41dc6b96b36bffe723cac46d4a469e5d74ee9285ce816\": container with ID starting with 693fedc45dfde9ab92b41dc6b96b36bffe723cac46d4a469e5d74ee9285ce816 not found: ID does not exist"
Nov 02 12:57:38.809473 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:57:38.809439    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-c0d4cb-openshift-default-6798dbb74f-rslvw"]
Nov 02 12:57:38.821819 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:57:38.821790    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-c0d4cb-openshift-default-6798dbb74f-rslvw"]
Nov 02 12:57:39.894228 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:57:39.894193    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24bcb862-30cd-44b9-920e-a919d80678a9" path="/var/lib/kubelet/pods/24bcb862-30cd-44b9-920e-a919d80678a9/volumes"
Nov 02 12:57:45.143836 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:57:45.143798    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqcmj" containerName="extractor"
Nov 02 12:58:01.964947 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:01.964916    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-74ba71-fdcf98f5f-dp92m"]
Nov 02 12:58:01.965396 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:01.965117    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-74ba71-fdcf98f5f-dp92m" podUID="a7c4bc95-0804-468f-9395-d87d4cbdf8cf" containerName="maas-api" containerID="cri-o://df533c79ac077625c53266a1ac84fcaead81fcb3bfa191d948ec92f54c6c9fbd" gracePeriod=30
Nov 02 12:58:02.465662 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:02.465311    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-74ba71-fdcf98f5f-dp92m"
Nov 02 12:58:02.594587 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:02.585273    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a7c4bc95-0804-468f-9395-d87d4cbdf8cf-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a7c4bc95-0804-468f-9395-d87d4cbdf8cf-metrics-certs\") pod \"a7c4bc95-0804-468f-9395-d87d4cbdf8cf\" (UID: \"a7c4bc95-0804-468f-9395-d87d4cbdf8cf\") "
Nov 02 12:58:02.594587 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:02.585490    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7c4bc95-0804-468f-9395-d87d4cbdf8cf-kube-api-access-w9pbw\" (UniqueName: \"kubernetes.io/projected/a7c4bc95-0804-468f-9395-d87d4cbdf8cf-kube-api-access-w9pbw\") pod \"a7c4bc95-0804-468f-9395-d87d4cbdf8cf\" (UID: \"a7c4bc95-0804-468f-9395-d87d4cbdf8cf\") "
Nov 02 12:58:02.594587 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:02.585705    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a7c4bc95-0804-468f-9395-d87d4cbdf8cf-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a7c4bc95-0804-468f-9395-d87d4cbdf8cf-maas-api-tls\") pod \"a7c4bc95-0804-468f-9395-d87d4cbdf8cf\" (UID: \"a7c4bc95-0804-468f-9395-d87d4cbdf8cf\") "
Nov 02 12:58:02.594587 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:02.585745    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7c4bc95-0804-468f-9395-d87d4cbdf8cf-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7c4bc95-0804-468f-9395-d87d4cbdf8cf-openshift-service-ca\") pod \"a7c4bc95-0804-468f-9395-d87d4cbdf8cf\" (UID: \"a7c4bc95-0804-468f-9395-d87d4cbdf8cf\") "
Nov 02 12:58:02.594587 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:02.592416    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7c4bc95-0804-468f-9395-d87d4cbdf8cf-kube-api-access-w9pbw" pod "a7c4bc95-0804-468f-9395-d87d4cbdf8cf" (UID: "a7c4bc95-0804-468f-9395-d87d4cbdf8cf"). InnerVolumeSpecName "kube-api-access-w9pbw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:58:02.602171 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:02.602119    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7c4bc95-0804-468f-9395-d87d4cbdf8cf-openshift-service-ca" pod "a7c4bc95-0804-468f-9395-d87d4cbdf8cf" (UID: "a7c4bc95-0804-468f-9395-d87d4cbdf8cf"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:58:02.605790 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:02.605758    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7c4bc95-0804-468f-9395-d87d4cbdf8cf-maas-api-tls" pod "a7c4bc95-0804-468f-9395-d87d4cbdf8cf" (UID: "a7c4bc95-0804-468f-9395-d87d4cbdf8cf"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:58:02.609961 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:02.606754    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7c4bc95-0804-468f-9395-d87d4cbdf8cf-metrics-certs" pod "a7c4bc95-0804-468f-9395-d87d4cbdf8cf" (UID: "a7c4bc95-0804-468f-9395-d87d4cbdf8cf"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:58:02.689941 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:02.689906    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a7c4bc95-0804-468f-9395-d87d4cbdf8cf-maas-api-tls\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:58:02.689941 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:02.689943    4559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7c4bc95-0804-468f-9395-d87d4cbdf8cf-openshift-service-ca\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:58:02.690164 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:02.689961    4559 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a7c4bc95-0804-468f-9395-d87d4cbdf8cf-metrics-certs\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:58:02.690164 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:02.689975    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w9pbw\" (UniqueName: \"kubernetes.io/projected/a7c4bc95-0804-468f-9395-d87d4cbdf8cf-kube-api-access-w9pbw\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:58:03.018837 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:03.018797    4559 generic.go:356] "Generic (PLEG): container finished" podID="a7c4bc95-0804-468f-9395-d87d4cbdf8cf" containerID="df533c79ac077625c53266a1ac84fcaead81fcb3bfa191d948ec92f54c6c9fbd" exitCode=0
Nov 02 12:58:03.019311 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:03.018834    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-74ba71-fdcf98f5f-dp92m" event={"ID":"a7c4bc95-0804-468f-9395-d87d4cbdf8cf","Type":"ContainerDied","Data":"df533c79ac077625c53266a1ac84fcaead81fcb3bfa191d948ec92f54c6c9fbd"}
Nov 02 12:58:03.019311 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:03.018876    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-74ba71-fdcf98f5f-dp92m"
Nov 02 12:58:03.019311 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:03.018901    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-74ba71-fdcf98f5f-dp92m" event={"ID":"a7c4bc95-0804-468f-9395-d87d4cbdf8cf","Type":"ContainerDied","Data":"f562829cd1281fd9594ec235dac5a6e508ae3a9b23edf43e0a039b994838205a"}
Nov 02 12:58:03.019311 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:03.018926    4559 scope.go:122] "RemoveContainer" containerID="df533c79ac077625c53266a1ac84fcaead81fcb3bfa191d948ec92f54c6c9fbd"
Nov 02 12:58:03.033286 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:03.033266    4559 scope.go:122] "RemoveContainer" containerID="df533c79ac077625c53266a1ac84fcaead81fcb3bfa191d948ec92f54c6c9fbd"
Nov 02 12:58:03.033584 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:58:03.033565    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df533c79ac077625c53266a1ac84fcaead81fcb3bfa191d948ec92f54c6c9fbd\": container with ID starting with df533c79ac077625c53266a1ac84fcaead81fcb3bfa191d948ec92f54c6c9fbd not found: ID does not exist" containerID="df533c79ac077625c53266a1ac84fcaead81fcb3bfa191d948ec92f54c6c9fbd"
Nov 02 12:58:03.033660 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:03.033592    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df533c79ac077625c53266a1ac84fcaead81fcb3bfa191d948ec92f54c6c9fbd"} err="failed to get container status \"df533c79ac077625c53266a1ac84fcaead81fcb3bfa191d948ec92f54c6c9fbd\": rpc error: code = NotFound desc = could not find container \"df533c79ac077625c53266a1ac84fcaead81fcb3bfa191d948ec92f54c6c9fbd\": container with ID starting with df533c79ac077625c53266a1ac84fcaead81fcb3bfa191d948ec92f54c6c9fbd not found: ID does not exist"
Nov 02 12:58:03.075714 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:03.075681    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-74ba71-fdcf98f5f-dp92m"]
Nov 02 12:58:03.078997 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:03.078974    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-74ba71-fdcf98f5f-dp92m"]
Nov 02 12:58:03.894623 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:03.894592    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7c4bc95-0804-468f-9395-d87d4cbdf8cf" path="/var/lib/kubelet/pods/a7c4bc95-0804-468f-9395-d87d4cbdf8cf/volumes"
Nov 02 12:58:13.462877 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:13.462844    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-explicit-e430c7-openshift-default-8649b6f6fc-vl6ql"]
Nov 02 12:58:13.464486 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:13.464453    4559 cpu_manager.go:414] "containerMap: removing container" podUID="bd77be34-4903-4c63-82d2-1b8951d975de" containerName="payload-processing"
Nov 02 12:58:13.464486 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:13.464482    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd77be34-4903-4c63-82d2-1b8951d975de" containerName="payload-processing"
Nov 02 12:58:13.464831 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:13.464512    4559 cpu_manager.go:414] "containerMap: removing container" podUID="24bcb862-30cd-44b9-920e-a919d80678a9" containerName="istio-proxy"
Nov 02 12:58:13.464831 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:13.464519    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24bcb862-30cd-44b9-920e-a919d80678a9" containerName="istio-proxy"
Nov 02 12:58:13.464831 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:13.464589    4559 cpu_manager.go:414] "containerMap: removing container" podUID="97e9abd0-ac59-4b5a-88cf-174dbf24c305" containerName="payload-pre-processing"
Nov 02 12:58:13.464831 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:13.464600    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97e9abd0-ac59-4b5a-88cf-174dbf24c305" containerName="payload-pre-processing"
Nov 02 12:58:13.464831 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:13.464612    4559 cpu_manager.go:414] "containerMap: removing container" podUID="a7c4bc95-0804-468f-9395-d87d4cbdf8cf" containerName="maas-api"
Nov 02 12:58:13.464831 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:13.464620    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7c4bc95-0804-468f-9395-d87d4cbdf8cf" containerName="maas-api"
Nov 02 12:58:13.464831 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:13.464664    4559 cpu_manager.go:414] "containerMap: removing container" podUID="5baf3254-0f52-4b0a-8b31-9ae96e0aa816" containerName="maas-api"
Nov 02 12:58:13.464831 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:13.464672    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5baf3254-0f52-4b0a-8b31-9ae96e0aa816" containerName="maas-api"
Nov 02 12:58:13.464831 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:13.464817    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="24bcb862-30cd-44b9-920e-a919d80678a9" containerName="istio-proxy"
Nov 02 12:58:13.464831 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:13.464834    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="97e9abd0-ac59-4b5a-88cf-174dbf24c305" containerName="payload-pre-processing"
Nov 02 12:58:13.465328 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:13.464842    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7c4bc95-0804-468f-9395-d87d4cbdf8cf" containerName="maas-api"
Nov 02 12:58:13.465328 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:13.464848    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="5baf3254-0f52-4b0a-8b31-9ae96e0aa816" containerName="maas-api"
Nov 02 12:58:13.465328 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:13.464858    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="24bcb862-30cd-44b9-920e-a919d80678a9" containerName="istio-proxy"
Nov 02 12:58:13.465328 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:13.464866    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd77be34-4903-4c63-82d2-1b8951d975de" containerName="payload-processing"
Nov 02 12:58:13.468731 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:13.468712    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-e430c7-openshift-default-8649b6f6fc-vl6ql"
Nov 02 12:58:13.480096 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:13.480068    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f8849e4-f808-45f0-b24a-e1e1c8b99974-workload-socket\") pod \"e2e-explicit-e430c7-openshift-default-8649b6f6fc-vl6ql\" (UID: \"6f8849e4-f808-45f0-b24a-e1e1c8b99974\") " pod="openshift-ingress/e2e-explicit-e430c7-openshift-default-8649b6f6fc-vl6ql"
Nov 02 12:58:13.480214 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:13.480122    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f8849e4-f808-45f0-b24a-e1e1c8b99974-workload-certs\") pod \"e2e-explicit-e430c7-openshift-default-8649b6f6fc-vl6ql\" (UID: \"6f8849e4-f808-45f0-b24a-e1e1c8b99974\") " pod="openshift-ingress/e2e-explicit-e430c7-openshift-default-8649b6f6fc-vl6ql"
Nov 02 12:58:13.480285 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:13.480265    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f8849e4-f808-45f0-b24a-e1e1c8b99974-istio-podinfo\") pod \"e2e-explicit-e430c7-openshift-default-8649b6f6fc-vl6ql\" (UID: \"6f8849e4-f808-45f0-b24a-e1e1c8b99974\") " pod="openshift-ingress/e2e-explicit-e430c7-openshift-default-8649b6f6fc-vl6ql"
Nov 02 12:58:13.480407 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:13.480386    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f8849e4-f808-45f0-b24a-e1e1c8b99974-credential-socket\") pod \"e2e-explicit-e430c7-openshift-default-8649b6f6fc-vl6ql\" (UID: \"6f8849e4-f808-45f0-b24a-e1e1c8b99974\") " pod="openshift-ingress/e2e-explicit-e430c7-openshift-default-8649b6f6fc-vl6ql"
Nov 02 12:58:13.480467 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:13.480445    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97dkk\" (UniqueName: \"kubernetes.io/projected/6f8849e4-f808-45f0-b24a-e1e1c8b99974-kube-api-access-97dkk\") pod \"e2e-explicit-e430c7-openshift-default-8649b6f6fc-vl6ql\" (UID: \"6f8849e4-f808-45f0-b24a-e1e1c8b99974\") " pod="openshift-ingress/e2e-explicit-e430c7-openshift-default-8649b6f6fc-vl6ql"
Nov 02 12:58:13.480525 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:13.480471    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f8849e4-f808-45f0-b24a-e1e1c8b99974-istio-envoy\") pod \"e2e-explicit-e430c7-openshift-default-8649b6f6fc-vl6ql\" (UID: \"6f8849e4-f808-45f0-b24a-e1e1c8b99974\") " pod="openshift-ingress/e2e-explicit-e430c7-openshift-default-8649b6f6fc-vl6ql"
Nov 02 12:58:13.480525 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:13.480515    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f8849e4-f808-45f0-b24a-e1e1c8b99974-istiod-ca-cert\") pod \"e2e-explicit-e430c7-openshift-default-8649b6f6fc-vl6ql\" (UID: \"6f8849e4-f808-45f0-b24a-e1e1c8b99974\") " pod="openshift-ingress/e2e-explicit-e430c7-openshift-default-8649b6f6fc-vl6ql"
Nov 02 12:58:13.480656 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:13.480612    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f8849e4-f808-45f0-b24a-e1e1c8b99974-istio-token\") pod \"e2e-explicit-e430c7-openshift-default-8649b6f6fc-vl6ql\" (UID: \"6f8849e4-f808-45f0-b24a-e1e1c8b99974\") " pod="openshift-ingress/e2e-explicit-e430c7-openshift-default-8649b6f6fc-vl6ql"
Nov 02 12:58:13.480730 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:13.480689    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f8849e4-f808-45f0-b24a-e1e1c8b99974-istio-data\") pod \"e2e-explicit-e430c7-openshift-default-8649b6f6fc-vl6ql\" (UID: \"6f8849e4-f808-45f0-b24a-e1e1c8b99974\") " pod="openshift-ingress/e2e-explicit-e430c7-openshift-default-8649b6f6fc-vl6ql"
Nov 02 12:58:13.507546 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:13.507513    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-e430c7-openshift-default-8649b6f6fc-vl6ql"]
Nov 02 12:58:13.509419 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:13.509397    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-explicit-e430c7-openshift-default-dockercfg-bzvqt\""
Nov 02 12:58:13.595955 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:13.584604    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f8849e4-f808-45f0-b24a-e1e1c8b99974-istio-envoy\") pod \"e2e-explicit-e430c7-openshift-default-8649b6f6fc-vl6ql\" (UID: \"6f8849e4-f808-45f0-b24a-e1e1c8b99974\") " pod="openshift-ingress/e2e-explicit-e430c7-openshift-default-8649b6f6fc-vl6ql"
Nov 02 12:58:13.595955 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:13.584677    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f8849e4-f808-45f0-b24a-e1e1c8b99974-istiod-ca-cert\") pod \"e2e-explicit-e430c7-openshift-default-8649b6f6fc-vl6ql\" (UID: \"6f8849e4-f808-45f0-b24a-e1e1c8b99974\") " pod="openshift-ingress/e2e-explicit-e430c7-openshift-default-8649b6f6fc-vl6ql"
Nov 02 12:58:13.595955 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:13.584760    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f8849e4-f808-45f0-b24a-e1e1c8b99974-istio-token\") pod \"e2e-explicit-e430c7-openshift-default-8649b6f6fc-vl6ql\" (UID: \"6f8849e4-f808-45f0-b24a-e1e1c8b99974\") " pod="openshift-ingress/e2e-explicit-e430c7-openshift-default-8649b6f6fc-vl6ql"
Nov 02 12:58:13.595955 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:13.584833    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f8849e4-f808-45f0-b24a-e1e1c8b99974-istio-data\") pod \"e2e-explicit-e430c7-openshift-default-8649b6f6fc-vl6ql\" (UID: \"6f8849e4-f808-45f0-b24a-e1e1c8b99974\") " pod="openshift-ingress/e2e-explicit-e430c7-openshift-default-8649b6f6fc-vl6ql"
Nov 02 12:58:13.595955 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:13.584917    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f8849e4-f808-45f0-b24a-e1e1c8b99974-workload-socket\") pod \"e2e-explicit-e430c7-openshift-default-8649b6f6fc-vl6ql\" (UID: \"6f8849e4-f808-45f0-b24a-e1e1c8b99974\") " pod="openshift-ingress/e2e-explicit-e430c7-openshift-default-8649b6f6fc-vl6ql"
Nov 02 12:58:13.595955 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:13.584987    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f8849e4-f808-45f0-b24a-e1e1c8b99974-workload-certs\") pod \"e2e-explicit-e430c7-openshift-default-8649b6f6fc-vl6ql\" (UID: \"6f8849e4-f808-45f0-b24a-e1e1c8b99974\") " pod="openshift-ingress/e2e-explicit-e430c7-openshift-default-8649b6f6fc-vl6ql"
Nov 02 12:58:13.595955 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:13.585074    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f8849e4-f808-45f0-b24a-e1e1c8b99974-istio-podinfo\") pod \"e2e-explicit-e430c7-openshift-default-8649b6f6fc-vl6ql\" (UID: \"6f8849e4-f808-45f0-b24a-e1e1c8b99974\") " pod="openshift-ingress/e2e-explicit-e430c7-openshift-default-8649b6f6fc-vl6ql"
Nov 02 12:58:13.595955 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:13.585174    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f8849e4-f808-45f0-b24a-e1e1c8b99974-credential-socket\") pod \"e2e-explicit-e430c7-openshift-default-8649b6f6fc-vl6ql\" (UID: \"6f8849e4-f808-45f0-b24a-e1e1c8b99974\") " pod="openshift-ingress/e2e-explicit-e430c7-openshift-default-8649b6f6fc-vl6ql"
Nov 02 12:58:13.595955 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:13.585251    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97dkk\" (UniqueName: \"kubernetes.io/projected/6f8849e4-f808-45f0-b24a-e1e1c8b99974-kube-api-access-97dkk\") pod \"e2e-explicit-e430c7-openshift-default-8649b6f6fc-vl6ql\" (UID: \"6f8849e4-f808-45f0-b24a-e1e1c8b99974\") " pod="openshift-ingress/e2e-explicit-e430c7-openshift-default-8649b6f6fc-vl6ql"
Nov 02 12:58:13.595955 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:13.591461    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f8849e4-f808-45f0-b24a-e1e1c8b99974-workload-socket\") pod \"e2e-explicit-e430c7-openshift-default-8649b6f6fc-vl6ql\" (UID: \"6f8849e4-f808-45f0-b24a-e1e1c8b99974\") " pod="openshift-ingress/e2e-explicit-e430c7-openshift-default-8649b6f6fc-vl6ql"
Nov 02 12:58:13.595955 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:13.592144    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f8849e4-f808-45f0-b24a-e1e1c8b99974-istiod-ca-cert\") pod \"e2e-explicit-e430c7-openshift-default-8649b6f6fc-vl6ql\" (UID: \"6f8849e4-f808-45f0-b24a-e1e1c8b99974\") " pod="openshift-ingress/e2e-explicit-e430c7-openshift-default-8649b6f6fc-vl6ql"
Nov 02 12:58:13.595955 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:13.593490    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f8849e4-f808-45f0-b24a-e1e1c8b99974-workload-certs\") pod \"e2e-explicit-e430c7-openshift-default-8649b6f6fc-vl6ql\" (UID: \"6f8849e4-f808-45f0-b24a-e1e1c8b99974\") " pod="openshift-ingress/e2e-explicit-e430c7-openshift-default-8649b6f6fc-vl6ql"
Nov 02 12:58:13.595955 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:13.593811    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f8849e4-f808-45f0-b24a-e1e1c8b99974-credential-socket\") pod \"e2e-explicit-e430c7-openshift-default-8649b6f6fc-vl6ql\" (UID: \"6f8849e4-f808-45f0-b24a-e1e1c8b99974\") " pod="openshift-ingress/e2e-explicit-e430c7-openshift-default-8649b6f6fc-vl6ql"
Nov 02 12:58:13.622675 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:13.603449    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f8849e4-f808-45f0-b24a-e1e1c8b99974-istio-data\") pod \"e2e-explicit-e430c7-openshift-default-8649b6f6fc-vl6ql\" (UID: \"6f8849e4-f808-45f0-b24a-e1e1c8b99974\") " pod="openshift-ingress/e2e-explicit-e430c7-openshift-default-8649b6f6fc-vl6ql"
Nov 02 12:58:13.648673 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:13.638677    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f8849e4-f808-45f0-b24a-e1e1c8b99974-istio-podinfo\") pod \"e2e-explicit-e430c7-openshift-default-8649b6f6fc-vl6ql\" (UID: \"6f8849e4-f808-45f0-b24a-e1e1c8b99974\") " pod="openshift-ingress/e2e-explicit-e430c7-openshift-default-8649b6f6fc-vl6ql"
Nov 02 12:58:13.669661 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:13.651978    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f8849e4-f808-45f0-b24a-e1e1c8b99974-istio-envoy\") pod \"e2e-explicit-e430c7-openshift-default-8649b6f6fc-vl6ql\" (UID: \"6f8849e4-f808-45f0-b24a-e1e1c8b99974\") " pod="openshift-ingress/e2e-explicit-e430c7-openshift-default-8649b6f6fc-vl6ql"
Nov 02 12:58:13.669661 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:13.659658    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f8849e4-f808-45f0-b24a-e1e1c8b99974-istio-token\") pod \"e2e-explicit-e430c7-openshift-default-8649b6f6fc-vl6ql\" (UID: \"6f8849e4-f808-45f0-b24a-e1e1c8b99974\") " pod="openshift-ingress/e2e-explicit-e430c7-openshift-default-8649b6f6fc-vl6ql"
Nov 02 12:58:13.669661 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:13.660551    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-97dkk\" (UniqueName: \"kubernetes.io/projected/6f8849e4-f808-45f0-b24a-e1e1c8b99974-kube-api-access-97dkk\") pod \"e2e-explicit-e430c7-openshift-default-8649b6f6fc-vl6ql\" (UID: \"6f8849e4-f808-45f0-b24a-e1e1c8b99974\") " pod="openshift-ingress/e2e-explicit-e430c7-openshift-default-8649b6f6fc-vl6ql"
Nov 02 12:58:13.788762 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:13.784384    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-e430c7-openshift-default-8649b6f6fc-vl6ql"
Nov 02 12:58:14.228470 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:58:14.228434    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f8849e4_f808_45f0_b24a_e1e1c8b99974.slice/crio-5abbbdcf6276879d5b8a643acac07f0b397f93eec4d335020ab869837795884a WatchSource:0}: Error finding container 5abbbdcf6276879d5b8a643acac07f0b397f93eec4d335020ab869837795884a: Status 404 returned error can't find the container with id 5abbbdcf6276879d5b8a643acac07f0b397f93eec4d335020ab869837795884a
Nov 02 12:58:14.228821 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:14.228798    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-e430c7-openshift-default-8649b6f6fc-vl6ql"]
Nov 02 12:58:14.231142 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:14.231110    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:58:14.231231 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:14.231177    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:58:14.231231 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:14.231208    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:58:15.087159 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:58:15.087122    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqcmj" containerName="extractor"
Nov 02 12:58:15.120897 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:15.120857    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-e430c7-openshift-default-8649b6f6fc-vl6ql" event={"ID":"6f8849e4-f808-45f0-b24a-e1e1c8b99974","Type":"ContainerStarted","Data":"01e7c9f818f941d68525708b9d7b5448400bc31b9777538360f08d5a61d40e6d"}
Nov 02 12:58:15.121102 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:15.120926    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-e430c7-openshift-default-8649b6f6fc-vl6ql" event={"ID":"6f8849e4-f808-45f0-b24a-e1e1c8b99974","Type":"ContainerStarted","Data":"5abbbdcf6276879d5b8a643acac07f0b397f93eec4d335020ab869837795884a"}
Nov 02 12:58:15.184764 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:15.184717    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-explicit-e430c7-openshift-default-8649b6f6fc-vl6ql" podStartSLOduration=2.184702824 podStartE2EDuration="2.184702824s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:58:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:58:15.183477178 +0000 UTC m=+1891.438876270" watchObservedRunningTime="2026-11-02 12:58:15.184702824 +0000 UTC m=+1891.440101918"
Nov 02 12:58:15.227336 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:15.227302    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"]
Nov 02 12:58:15.784886 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:15.784852    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-e430c7-openshift-default-8649b6f6fc-vl6ql"
Nov 02 12:58:15.791685 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:15.791659    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-e430c7-openshift-default-8649b6f6fc-vl6ql"
Nov 02 12:58:16.158724 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:16.157839    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-e430c7-openshift-default-8649b6f6fc-vl6ql"
Nov 02 12:58:16.162658 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:16.159581    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-e430c7-openshift-default-8649b6f6fc-vl6ql"
Nov 02 12:58:19.403393 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:19.403360    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-e430c7-85548db555-r7xkb"]
Nov 02 12:58:19.404114 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:19.404096    4559 cpu_manager.go:414] "containerMap: removing container" podUID="24bcb862-30cd-44b9-920e-a919d80678a9" containerName="istio-proxy"
Nov 02 12:58:19.404162 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:19.404115    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24bcb862-30cd-44b9-920e-a919d80678a9" containerName="istio-proxy"
Nov 02 12:58:19.408074 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:19.408057    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-e430c7-85548db555-r7xkb"
Nov 02 12:58:19.424587 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:19.424564    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-e430c7\""
Nov 02 12:58:19.432285 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:19.432250    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4j7h\" (UniqueName: \"kubernetes.io/projected/d9e9675e-a96b-4f43-a253-608bcd468854-kube-api-access-c4j7h\") pod \"payload-pre-processing-e2e-explicit-e430c7-85548db555-r7xkb\" (UID: \"d9e9675e-a96b-4f43-a253-608bcd468854\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-e430c7-85548db555-r7xkb"
Nov 02 12:58:19.432448 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:19.432398    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9e9675e-a96b-4f43-a253-608bcd468854-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-e430c7-85548db555-r7xkb\" (UID: \"d9e9675e-a96b-4f43-a253-608bcd468854\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-e430c7-85548db555-r7xkb"
Nov 02 12:58:19.437620 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:19.437587    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-e430c7-dockercfg-4gmfs\""
Nov 02 12:58:19.444996 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:19.444962    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-e430c7-85548db555-r7xkb"]
Nov 02 12:58:19.562677 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:19.550494    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4j7h\" (UniqueName: \"kubernetes.io/projected/d9e9675e-a96b-4f43-a253-608bcd468854-kube-api-access-c4j7h\") pod \"payload-pre-processing-e2e-explicit-e430c7-85548db555-r7xkb\" (UID: \"d9e9675e-a96b-4f43-a253-608bcd468854\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-e430c7-85548db555-r7xkb"
Nov 02 12:58:19.563126 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:19.563089    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9e9675e-a96b-4f43-a253-608bcd468854-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-e430c7-85548db555-r7xkb\" (UID: \"d9e9675e-a96b-4f43-a253-608bcd468854\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-e430c7-85548db555-r7xkb"
Nov 02 12:58:19.579996 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:19.579961    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9e9675e-a96b-4f43-a253-608bcd468854-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-e430c7-85548db555-r7xkb\" (UID: \"d9e9675e-a96b-4f43-a253-608bcd468854\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-e430c7-85548db555-r7xkb"
Nov 02 12:58:19.595681 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:19.595622    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-e430c7-75f68d8dd5-68xr4"]
Nov 02 12:58:19.653712 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:19.626928    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-e430c7-75f68d8dd5-68xr4"
Nov 02 12:58:19.704998 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:19.702435    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0cf4374b-846c-4a69-872f-6c03ed423b41-plugins-config-volume\") pod \"payload-processing-e2e-explicit-e430c7-75f68d8dd5-68xr4\" (UID: \"0cf4374b-846c-4a69-872f-6c03ed423b41\") " pod="openshift-ingress/payload-processing-e2e-explicit-e430c7-75f68d8dd5-68xr4"
Nov 02 12:58:19.704998 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:19.702500    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbdv6\" (UniqueName: \"kubernetes.io/projected/0cf4374b-846c-4a69-872f-6c03ed423b41-kube-api-access-cbdv6\") pod \"payload-processing-e2e-explicit-e430c7-75f68d8dd5-68xr4\" (UID: \"0cf4374b-846c-4a69-872f-6c03ed423b41\") " pod="openshift-ingress/payload-processing-e2e-explicit-e430c7-75f68d8dd5-68xr4"
Nov 02 12:58:19.756733 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:19.730726    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-e430c7-75f68d8dd5-68xr4"]
Nov 02 12:58:19.781852 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:19.781795    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4j7h\" (UniqueName: \"kubernetes.io/projected/d9e9675e-a96b-4f43-a253-608bcd468854-kube-api-access-c4j7h\") pod \"payload-pre-processing-e2e-explicit-e430c7-85548db555-r7xkb\" (UID: \"d9e9675e-a96b-4f43-a253-608bcd468854\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-e430c7-85548db555-r7xkb"
Nov 02 12:58:19.805833 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:19.803744    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0cf4374b-846c-4a69-872f-6c03ed423b41-plugins-config-volume\") pod \"payload-processing-e2e-explicit-e430c7-75f68d8dd5-68xr4\" (UID: \"0cf4374b-846c-4a69-872f-6c03ed423b41\") " pod="openshift-ingress/payload-processing-e2e-explicit-e430c7-75f68d8dd5-68xr4"
Nov 02 12:58:19.805833 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:19.803835    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbdv6\" (UniqueName: \"kubernetes.io/projected/0cf4374b-846c-4a69-872f-6c03ed423b41-kube-api-access-cbdv6\") pod \"payload-processing-e2e-explicit-e430c7-75f68d8dd5-68xr4\" (UID: \"0cf4374b-846c-4a69-872f-6c03ed423b41\") " pod="openshift-ingress/payload-processing-e2e-explicit-e430c7-75f68d8dd5-68xr4"
Nov 02 12:58:19.805833 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:19.804961    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0cf4374b-846c-4a69-872f-6c03ed423b41-plugins-config-volume\") pod \"payload-processing-e2e-explicit-e430c7-75f68d8dd5-68xr4\" (UID: \"0cf4374b-846c-4a69-872f-6c03ed423b41\") " pod="openshift-ingress/payload-processing-e2e-explicit-e430c7-75f68d8dd5-68xr4"
Nov 02 12:58:19.827833 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:19.827793    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbdv6\" (UniqueName: \"kubernetes.io/projected/0cf4374b-846c-4a69-872f-6c03ed423b41-kube-api-access-cbdv6\") pod \"payload-processing-e2e-explicit-e430c7-75f68d8dd5-68xr4\" (UID: \"0cf4374b-846c-4a69-872f-6c03ed423b41\") " pod="openshift-ingress/payload-processing-e2e-explicit-e430c7-75f68d8dd5-68xr4"
Nov 02 12:58:20.038660 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:20.038129    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-e430c7-85548db555-r7xkb"
Nov 02 12:58:20.120264 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:20.119942    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-e430c7-75f68d8dd5-68xr4"
Nov 02 12:58:20.358911 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:58:20.358883    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9e9675e_a96b_4f43_a253_608bcd468854.slice/crio-410caee7de487c59578d762cb4384d45def6a64b3ef92b71bf44fffcebb25178 WatchSource:0}: Error finding container 410caee7de487c59578d762cb4384d45def6a64b3ef92b71bf44fffcebb25178: Status 404 returned error can't find the container with id 410caee7de487c59578d762cb4384d45def6a64b3ef92b71bf44fffcebb25178
Nov 02 12:58:20.359459 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:20.359345    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-e430c7-85548db555-r7xkb"]
Nov 02 12:58:20.449484 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:58:20.449457    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cf4374b_846c_4a69_872f_6c03ed423b41.slice/crio-0f25d9420c18418724580c85e491d1831255c142430efe2acb2442a6ea120d40 WatchSource:0}: Error finding container 0f25d9420c18418724580c85e491d1831255c142430efe2acb2442a6ea120d40: Status 404 returned error can't find the container with id 0f25d9420c18418724580c85e491d1831255c142430efe2acb2442a6ea120d40
Nov 02 12:58:20.451723 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:20.451701    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-e430c7-75f68d8dd5-68xr4"]
Nov 02 12:58:21.199109 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:21.199071    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-e430c7-85548db555-r7xkb" event={"ID":"d9e9675e-a96b-4f43-a253-608bcd468854","Type":"ContainerStarted","Data":"dc249e4d6aa08c3984d95bc80dc2d9aff90d393b2425501d611b652a5ac230e8"}
Nov 02 12:58:21.199312 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:21.199146    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-e430c7-85548db555-r7xkb"
Nov 02 12:58:21.199312 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:21.199170    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-e430c7-85548db555-r7xkb" event={"ID":"d9e9675e-a96b-4f43-a253-608bcd468854","Type":"ContainerStarted","Data":"410caee7de487c59578d762cb4384d45def6a64b3ef92b71bf44fffcebb25178"}
Nov 02 12:58:21.200864 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:21.200836    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-e430c7-75f68d8dd5-68xr4" event={"ID":"0cf4374b-846c-4a69-872f-6c03ed423b41","Type":"ContainerStarted","Data":"d7178ca335fa285f4abfbc31c99174acc5d52716d84961cf94c43e6febf849bd"}
Nov 02 12:58:21.201008 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:21.200885    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-e430c7-75f68d8dd5-68xr4"
Nov 02 12:58:21.201008 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:21.200906    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-e430c7-75f68d8dd5-68xr4" event={"ID":"0cf4374b-846c-4a69-872f-6c03ed423b41","Type":"ContainerStarted","Data":"0f25d9420c18418724580c85e491d1831255c142430efe2acb2442a6ea120d40"}
Nov 02 12:58:21.242486 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:21.242438    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-e430c7-85548db555-r7xkb" podStartSLOduration=2.242424239 podStartE2EDuration="2.242424239s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:58:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:58:21.240184383 +0000 UTC m=+1897.495583479" watchObservedRunningTime="2026-11-02 12:58:21.242424239 +0000 UTC m=+1897.497823371"
Nov 02 12:58:21.283766 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:21.283713    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-e430c7-75f68d8dd5-68xr4" podStartSLOduration=2.283698951 podStartE2EDuration="2.283698951s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:58:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:58:21.282524249 +0000 UTC m=+1897.537923371" watchObservedRunningTime="2026-11-02 12:58:21.283698951 +0000 UTC m=+1897.539098046"
Nov 02 12:58:32.210219 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:32.210189    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-e430c7-75f68d8dd5-68xr4"
Nov 02 12:58:32.210738 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:32.210550    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-e430c7-85548db555-r7xkb"
Nov 02 12:58:40.124456 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:40.124418    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-e430c7-6f7d8d6ccc-9lkds"]
Nov 02 12:58:40.128957 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:40.128939    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-e430c7-6f7d8d6ccc-9lkds"
Nov 02 12:58:40.148169 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:40.148134    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-e430c7-serving-cert\""
Nov 02 12:58:40.150551 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:40.150526    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-e430c7-6f7d8d6ccc-9lkds"]
Nov 02 12:58:40.256697 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:40.256658    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b93c19bf-9bc0-4c51-a945-5b196d5dd80e-metrics-certs\") pod \"maas-api-e2e-explicit-e430c7-6f7d8d6ccc-9lkds\" (UID: \"b93c19bf-9bc0-4c51-a945-5b196d5dd80e\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-e430c7-6f7d8d6ccc-9lkds"
Nov 02 12:58:40.256918 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:40.256741    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zddtb\" (UniqueName: \"kubernetes.io/projected/b93c19bf-9bc0-4c51-a945-5b196d5dd80e-kube-api-access-zddtb\") pod \"maas-api-e2e-explicit-e430c7-6f7d8d6ccc-9lkds\" (UID: \"b93c19bf-9bc0-4c51-a945-5b196d5dd80e\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-e430c7-6f7d8d6ccc-9lkds"
Nov 02 12:58:40.256918 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:40.256821    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b93c19bf-9bc0-4c51-a945-5b196d5dd80e-openshift-service-ca\") pod \"maas-api-e2e-explicit-e430c7-6f7d8d6ccc-9lkds\" (UID: \"b93c19bf-9bc0-4c51-a945-5b196d5dd80e\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-e430c7-6f7d8d6ccc-9lkds"
Nov 02 12:58:40.257031 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:40.256940    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b93c19bf-9bc0-4c51-a945-5b196d5dd80e-maas-api-tls\") pod \"maas-api-e2e-explicit-e430c7-6f7d8d6ccc-9lkds\" (UID: \"b93c19bf-9bc0-4c51-a945-5b196d5dd80e\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-e430c7-6f7d8d6ccc-9lkds"
Nov 02 12:58:40.367367 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:40.367151    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b93c19bf-9bc0-4c51-a945-5b196d5dd80e-metrics-certs\") pod \"maas-api-e2e-explicit-e430c7-6f7d8d6ccc-9lkds\" (UID: \"b93c19bf-9bc0-4c51-a945-5b196d5dd80e\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-e430c7-6f7d8d6ccc-9lkds"
Nov 02 12:58:40.385711 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:40.368281    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zddtb\" (UniqueName: \"kubernetes.io/projected/b93c19bf-9bc0-4c51-a945-5b196d5dd80e-kube-api-access-zddtb\") pod \"maas-api-e2e-explicit-e430c7-6f7d8d6ccc-9lkds\" (UID: \"b93c19bf-9bc0-4c51-a945-5b196d5dd80e\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-e430c7-6f7d8d6ccc-9lkds"
Nov 02 12:58:40.408696 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:40.404031    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b93c19bf-9bc0-4c51-a945-5b196d5dd80e-openshift-service-ca\") pod \"maas-api-e2e-explicit-e430c7-6f7d8d6ccc-9lkds\" (UID: \"b93c19bf-9bc0-4c51-a945-5b196d5dd80e\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-e430c7-6f7d8d6ccc-9lkds"
Nov 02 12:58:40.408696 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:40.406837    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b93c19bf-9bc0-4c51-a945-5b196d5dd80e-openshift-service-ca\") pod \"maas-api-e2e-explicit-e430c7-6f7d8d6ccc-9lkds\" (UID: \"b93c19bf-9bc0-4c51-a945-5b196d5dd80e\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-e430c7-6f7d8d6ccc-9lkds"
Nov 02 12:58:40.408696 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:40.407922    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b93c19bf-9bc0-4c51-a945-5b196d5dd80e-maas-api-tls\") pod \"maas-api-e2e-explicit-e430c7-6f7d8d6ccc-9lkds\" (UID: \"b93c19bf-9bc0-4c51-a945-5b196d5dd80e\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-e430c7-6f7d8d6ccc-9lkds"
Nov 02 12:58:40.414037 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:40.414005    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b93c19bf-9bc0-4c51-a945-5b196d5dd80e-metrics-certs\") pod \"maas-api-e2e-explicit-e430c7-6f7d8d6ccc-9lkds\" (UID: \"b93c19bf-9bc0-4c51-a945-5b196d5dd80e\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-e430c7-6f7d8d6ccc-9lkds"
Nov 02 12:58:40.416559 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:40.415244    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b93c19bf-9bc0-4c51-a945-5b196d5dd80e-maas-api-tls\") pod \"maas-api-e2e-explicit-e430c7-6f7d8d6ccc-9lkds\" (UID: \"b93c19bf-9bc0-4c51-a945-5b196d5dd80e\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-e430c7-6f7d8d6ccc-9lkds"
Nov 02 12:58:40.428934 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:40.427603    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zddtb\" (UniqueName: \"kubernetes.io/projected/b93c19bf-9bc0-4c51-a945-5b196d5dd80e-kube-api-access-zddtb\") pod \"maas-api-e2e-explicit-e430c7-6f7d8d6ccc-9lkds\" (UID: \"b93c19bf-9bc0-4c51-a945-5b196d5dd80e\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-e430c7-6f7d8d6ccc-9lkds"
Nov 02 12:58:40.447056 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:40.447016    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-e430c7-6f7d8d6ccc-9lkds"
Nov 02 12:58:40.759624 ip-10-0-128-132 kubenswrapper[4559]: W1102 12:58:40.759594    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb93c19bf_9bc0_4c51_a945_5b196d5dd80e.slice/crio-f3e5a94a0342f5fbec882968f2356f7eebc35e89c5850855104386b52c62ef7c WatchSource:0}: Error finding container f3e5a94a0342f5fbec882968f2356f7eebc35e89c5850855104386b52c62ef7c: Status 404 returned error can't find the container with id f3e5a94a0342f5fbec882968f2356f7eebc35e89c5850855104386b52c62ef7c
Nov 02 12:58:40.760915 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:40.760889    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-e430c7-6f7d8d6ccc-9lkds"]
Nov 02 12:58:41.419627 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:41.419593    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-e430c7-6f7d8d6ccc-9lkds" event={"ID":"b93c19bf-9bc0-4c51-a945-5b196d5dd80e","Type":"ContainerStarted","Data":"f3e5a94a0342f5fbec882968f2356f7eebc35e89c5850855104386b52c62ef7c"}
Nov 02 12:58:42.428219 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:42.428173    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-e430c7-6f7d8d6ccc-9lkds" event={"ID":"b93c19bf-9bc0-4c51-a945-5b196d5dd80e","Type":"ContainerStarted","Data":"f46e2199f03ed8d50dec7426dcee381e46a8f0f9478f047397f5913f713e883a"}
Nov 02 12:58:42.428712 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:42.428239    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-e430c7-6f7d8d6ccc-9lkds"
Nov 02 12:58:42.474123 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:42.474072    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-e430c7-6f7d8d6ccc-9lkds" podStartSLOduration=1.992022692 podStartE2EDuration="2.474056894s" totalImagesPullingTime="482.034202ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:58:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:58:42.470731118 +0000 UTC m=+1918.726130214" watchObservedRunningTime="2026-11-02 12:58:42.474056894 +0000 UTC m=+1918.729455989"
Nov 02 12:58:45.093772 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:58:45.093733    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqcmj" containerName="extractor"
Nov 02 12:58:46.279421 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:46.279338    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-fwt4g"]
Nov 02 12:58:46.279907 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:46.279559    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-fwt4g" podUID="91783d5b-f538-43a0-a9d8-29f7270cadcd" containerName="istio-proxy" containerID="cri-o://f75d8cc15890dd03513350f317fa6efc86f60028e165eacc7c22765c06af76ad" gracePeriod=30
Nov 02 12:58:46.293316 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:46.293287    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-cdnsv"]
Nov 02 12:58:46.293526 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:46.293503    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-cdnsv" podUID="be334717-f78a-4113-bf03-15d5c4365107" containerName="istio-proxy" containerID="cri-o://32750de39eedcb157478ffa2aca02da0c43e2efc8a63d644f34bbce8d1ef88b2" gracePeriod=30
Nov 02 12:58:46.298101 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:46.298077    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-52bzl"]
Nov 02 12:58:46.298250 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:46.298232    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-52bzl" podUID="160f0d66-4e21-49eb-9c25-b467bdb56f41" containerName="istio-proxy" containerID="cri-o://a56f8505504b37a1ad5c87e4fd9da9d9f95274c306e46273d0c503e6576f210e" gracePeriod=30
Nov 02 12:58:48.439822 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:48.439788    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-e430c7-6f7d8d6ccc-9lkds"
Nov 02 12:58:49.177279 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:49.177236    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-fbc46"]
Nov 02 12:58:49.177524 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:49.177493    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-fbc46" podUID="988e5461-e95e-408c-b145-7253a1a21056" containerName="istio-proxy" containerID="cri-o://8d70670962ad5c661ab1ecfd443fb8be9b3543ba0a63d44d92a0cac91b173323" gracePeriod=30
Nov 02 12:58:49.188688 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:49.188658    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-mdchl"]
Nov 02 12:58:49.188928 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:49.188890    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-mdchl" podUID="bb1e97a1-f680-4922-920c-ee0a3468cb1f" containerName="istio-proxy" containerID="cri-o://3e0c9b0bc6a05cd6cd0b8e306b2739f4742aaabf4c9a9bd3666c4584dfce9b96" gracePeriod=30
Nov 02 12:58:49.192321 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:49.192296    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-vtpz8"]
Nov 02 12:58:49.192498 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:49.192477    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-vtpz8" podUID="d7f18bc2-015d-41da-a666-912b2e250e2a" containerName="istio-proxy" containerID="cri-o://a49513b40f2e0a5da91fdbeeac25ddfa386465768ff6828922520c816389cb94" gracePeriod=30
Nov 02 12:58:50.555316 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:50.555275    4559 patch_prober.go:29] interesting pod/e2e-worker-w3-068756-openshift-default-84f69c9cf5-cdnsv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 12:58:50.555756 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:50.555341    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-cdnsv" podUID="be334717-f78a-4113-bf03-15d5c4365107" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 12:58:51.498999 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:51.498824    4559 generic.go:356] "Generic (PLEG): container finished" podID="be334717-f78a-4113-bf03-15d5c4365107" containerID="32750de39eedcb157478ffa2aca02da0c43e2efc8a63d644f34bbce8d1ef88b2" exitCode=0
Nov 02 12:58:51.498999 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:51.498911    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-cdnsv" event={"ID":"be334717-f78a-4113-bf03-15d5c4365107","Type":"ContainerDied","Data":"32750de39eedcb157478ffa2aca02da0c43e2efc8a63d644f34bbce8d1ef88b2"}
Nov 02 12:58:51.501674 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:51.501360    4559 generic.go:356] "Generic (PLEG): container finished" podID="91783d5b-f538-43a0-a9d8-29f7270cadcd" containerID="f75d8cc15890dd03513350f317fa6efc86f60028e165eacc7c22765c06af76ad" exitCode=0
Nov 02 12:58:51.501674 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:51.501573    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-fwt4g" event={"ID":"91783d5b-f538-43a0-a9d8-29f7270cadcd","Type":"ContainerDied","Data":"f75d8cc15890dd03513350f317fa6efc86f60028e165eacc7c22765c06af76ad"}
Nov 02 12:58:51.504465 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:51.504329    4559 generic.go:356] "Generic (PLEG): container finished" podID="160f0d66-4e21-49eb-9c25-b467bdb56f41" containerID="a56f8505504b37a1ad5c87e4fd9da9d9f95274c306e46273d0c503e6576f210e" exitCode=0
Nov 02 12:58:51.504465 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:51.504391    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-52bzl" event={"ID":"160f0d66-4e21-49eb-9c25-b467bdb56f41","Type":"ContainerDied","Data":"a56f8505504b37a1ad5c87e4fd9da9d9f95274c306e46273d0c503e6576f210e"}
Nov 02 12:58:51.611885 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:51.611853    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-fwt4g"
Nov 02 12:58:51.658934 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:51.658909    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-cdnsv"
Nov 02 12:58:51.683161 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:51.683125    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-52bzl"
Nov 02 12:58:51.724296 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:51.724263    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be334717-f78a-4113-bf03-15d5c4365107-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be334717-f78a-4113-bf03-15d5c4365107-istio-envoy\") pod \"be334717-f78a-4113-bf03-15d5c4365107\" (UID: \"be334717-f78a-4113-bf03-15d5c4365107\") "
Nov 02 12:58:51.724500 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:51.724315    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91783d5b-f538-43a0-a9d8-29f7270cadcd-istio-token\" (UniqueName: \"kubernetes.io/projected/91783d5b-f538-43a0-a9d8-29f7270cadcd-istio-token\") pod \"91783d5b-f538-43a0-a9d8-29f7270cadcd\" (UID: \"91783d5b-f538-43a0-a9d8-29f7270cadcd\") "
Nov 02 12:58:51.724500 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:51.724362    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91783d5b-f538-43a0-a9d8-29f7270cadcd-kube-api-access-9shkk\" (UniqueName: \"kubernetes.io/projected/91783d5b-f538-43a0-a9d8-29f7270cadcd-kube-api-access-9shkk\") pod \"91783d5b-f538-43a0-a9d8-29f7270cadcd\" (UID: \"91783d5b-f538-43a0-a9d8-29f7270cadcd\") "
Nov 02 12:58:51.724500 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:51.724386    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/160f0d66-4e21-49eb-9c25-b467bdb56f41-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/160f0d66-4e21-49eb-9c25-b467bdb56f41-istiod-ca-cert\") pod \"160f0d66-4e21-49eb-9c25-b467bdb56f41\" (UID: \"160f0d66-4e21-49eb-9c25-b467bdb56f41\") "
Nov 02 12:58:51.724500 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:51.724412    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be334717-f78a-4113-bf03-15d5c4365107-kube-api-access-pt7tk\" (UniqueName: \"kubernetes.io/projected/be334717-f78a-4113-bf03-15d5c4365107-kube-api-access-pt7tk\") pod \"be334717-f78a-4113-bf03-15d5c4365107\" (UID: \"be334717-f78a-4113-bf03-15d5c4365107\") "
Nov 02 12:58:51.724500 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:51.724443    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91783d5b-f538-43a0-a9d8-29f7270cadcd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91783d5b-f538-43a0-a9d8-29f7270cadcd-credential-socket\") pod \"91783d5b-f538-43a0-a9d8-29f7270cadcd\" (UID: \"91783d5b-f538-43a0-a9d8-29f7270cadcd\") "
Nov 02 12:58:51.725286 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:51.724749    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91783d5b-f538-43a0-a9d8-29f7270cadcd-credential-socket" pod "91783d5b-f538-43a0-a9d8-29f7270cadcd" (UID: "91783d5b-f538-43a0-a9d8-29f7270cadcd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:58:51.725286 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:51.724794    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/160f0d66-4e21-49eb-9c25-b467bdb56f41-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/160f0d66-4e21-49eb-9c25-b467bdb56f41-istio-data\") pod \"160f0d66-4e21-49eb-9c25-b467bdb56f41\" (UID: \"160f0d66-4e21-49eb-9c25-b467bdb56f41\") "
Nov 02 12:58:51.725286 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:51.724822    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/160f0d66-4e21-49eb-9c25-b467bdb56f41-istiod-ca-cert" pod "160f0d66-4e21-49eb-9c25-b467bdb56f41" (UID: "160f0d66-4e21-49eb-9c25-b467bdb56f41"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:58:51.725286 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:51.724892    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/160f0d66-4e21-49eb-9c25-b467bdb56f41-istio-token\" (UniqueName: \"kubernetes.io/projected/160f0d66-4e21-49eb-9c25-b467bdb56f41-istio-token\") pod \"160f0d66-4e21-49eb-9c25-b467bdb56f41\" (UID: \"160f0d66-4e21-49eb-9c25-b467bdb56f41\") "
Nov 02 12:58:51.725286 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:51.724921    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/160f0d66-4e21-49eb-9c25-b467bdb56f41-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/160f0d66-4e21-49eb-9c25-b467bdb56f41-workload-socket\") pod \"160f0d66-4e21-49eb-9c25-b467bdb56f41\" (UID: \"160f0d66-4e21-49eb-9c25-b467bdb56f41\") "
Nov 02 12:58:51.725286 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:51.724947    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be334717-f78a-4113-bf03-15d5c4365107-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be334717-f78a-4113-bf03-15d5c4365107-istio-data\") pod \"be334717-f78a-4113-bf03-15d5c4365107\" (UID: \"be334717-f78a-4113-bf03-15d5c4365107\") "
Nov 02 12:58:51.725286 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:51.724978    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be334717-f78a-4113-bf03-15d5c4365107-istio-token\" (UniqueName: \"kubernetes.io/projected/be334717-f78a-4113-bf03-15d5c4365107-istio-token\") pod \"be334717-f78a-4113-bf03-15d5c4365107\" (UID: \"be334717-f78a-4113-bf03-15d5c4365107\") "
Nov 02 12:58:51.725286 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:51.725008    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/160f0d66-4e21-49eb-9c25-b467bdb56f41-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/160f0d66-4e21-49eb-9c25-b467bdb56f41-istio-envoy\") pod \"160f0d66-4e21-49eb-9c25-b467bdb56f41\" (UID: \"160f0d66-4e21-49eb-9c25-b467bdb56f41\") "
Nov 02 12:58:51.725286 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:51.725052    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/160f0d66-4e21-49eb-9c25-b467bdb56f41-kube-api-access-ttrdh\" (UniqueName: \"kubernetes.io/projected/160f0d66-4e21-49eb-9c25-b467bdb56f41-kube-api-access-ttrdh\") pod \"160f0d66-4e21-49eb-9c25-b467bdb56f41\" (UID: \"160f0d66-4e21-49eb-9c25-b467bdb56f41\") "
Nov 02 12:58:51.725286 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:51.725051    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/160f0d66-4e21-49eb-9c25-b467bdb56f41-istio-data" pod "160f0d66-4e21-49eb-9c25-b467bdb56f41" (UID: "160f0d66-4e21-49eb-9c25-b467bdb56f41"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:58:51.725286 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:51.725139    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/160f0d66-4e21-49eb-9c25-b467bdb56f41-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/160f0d66-4e21-49eb-9c25-b467bdb56f41-istio-podinfo\") pod \"160f0d66-4e21-49eb-9c25-b467bdb56f41\" (UID: \"160f0d66-4e21-49eb-9c25-b467bdb56f41\") "
Nov 02 12:58:51.725286 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:51.725210    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be334717-f78a-4113-bf03-15d5c4365107-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be334717-f78a-4113-bf03-15d5c4365107-workload-certs\") pod \"be334717-f78a-4113-bf03-15d5c4365107\" (UID: \"be334717-f78a-4113-bf03-15d5c4365107\") "
Nov 02 12:58:51.725286 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:51.725271    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be334717-f78a-4113-bf03-15d5c4365107-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be334717-f78a-4113-bf03-15d5c4365107-credential-socket\") pod \"be334717-f78a-4113-bf03-15d5c4365107\" (UID: \"be334717-f78a-4113-bf03-15d5c4365107\") "
Nov 02 12:58:51.738513 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:51.725323    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91783d5b-f538-43a0-a9d8-29f7270cadcd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91783d5b-f538-43a0-a9d8-29f7270cadcd-workload-socket\") pod \"91783d5b-f538-43a0-a9d8-29f7270cadcd\" (UID: \"91783d5b-f538-43a0-a9d8-29f7270cadcd\") "
Nov 02 12:58:51.738513 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:51.725368    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/160f0d66-4e21-49eb-9c25-b467bdb56f41-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/160f0d66-4e21-49eb-9c25-b467bdb56f41-credential-socket\") pod \"160f0d66-4e21-49eb-9c25-b467bdb56f41\" (UID: \"160f0d66-4e21-49eb-9c25-b467bdb56f41\") "
Nov 02 12:58:51.738513 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:51.725409    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91783d5b-f538-43a0-a9d8-29f7270cadcd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91783d5b-f538-43a0-a9d8-29f7270cadcd-workload-certs\") pod \"91783d5b-f538-43a0-a9d8-29f7270cadcd\" (UID: \"91783d5b-f538-43a0-a9d8-29f7270cadcd\") "
Nov 02 12:58:51.738513 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:51.725509    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91783d5b-f538-43a0-a9d8-29f7270cadcd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91783d5b-f538-43a0-a9d8-29f7270cadcd-istio-envoy\") pod \"91783d5b-f538-43a0-a9d8-29f7270cadcd\" (UID: \"91783d5b-f538-43a0-a9d8-29f7270cadcd\") "
Nov 02 12:58:51.738513 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:51.725552    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91783d5b-f538-43a0-a9d8-29f7270cadcd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91783d5b-f538-43a0-a9d8-29f7270cadcd-istiod-ca-cert\") pod \"91783d5b-f538-43a0-a9d8-29f7270cadcd\" (UID: \"91783d5b-f538-43a0-a9d8-29f7270cadcd\") "
Nov 02 12:58:51.738513 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:51.725577    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be334717-f78a-4113-bf03-15d5c4365107-workload-certs" pod "be334717-f78a-4113-bf03-15d5c4365107" (UID: "be334717-f78a-4113-bf03-15d5c4365107"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:58:51.738513 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:51.725621    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/be334717-f78a-4113-bf03-15d5c4365107-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be334717-f78a-4113-bf03-15d5c4365107-istio-podinfo\") pod \"be334717-f78a-4113-bf03-15d5c4365107\" (UID: \"be334717-f78a-4113-bf03-15d5c4365107\") "
Nov 02 12:58:51.738513 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:51.725608    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91783d5b-f538-43a0-a9d8-29f7270cadcd-workload-socket" pod "91783d5b-f538-43a0-a9d8-29f7270cadcd" (UID: "91783d5b-f538-43a0-a9d8-29f7270cadcd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:58:51.738513 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:51.725712    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be334717-f78a-4113-bf03-15d5c4365107-istio-data" pod "be334717-f78a-4113-bf03-15d5c4365107" (UID: "be334717-f78a-4113-bf03-15d5c4365107"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:58:51.738513 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:51.725734    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91783d5b-f538-43a0-a9d8-29f7270cadcd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91783d5b-f538-43a0-a9d8-29f7270cadcd-istio-data\") pod \"91783d5b-f538-43a0-a9d8-29f7270cadcd\" (UID: \"91783d5b-f538-43a0-a9d8-29f7270cadcd\") "
Nov 02 12:58:51.738513 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:51.725770    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be334717-f78a-4113-bf03-15d5c4365107-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be334717-f78a-4113-bf03-15d5c4365107-istiod-ca-cert\") pod \"be334717-f78a-4113-bf03-15d5c4365107\" (UID: \"be334717-f78a-4113-bf03-15d5c4365107\") "
Nov 02 12:58:51.738513 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:51.725831    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be334717-f78a-4113-bf03-15d5c4365107-credential-socket" pod "be334717-f78a-4113-bf03-15d5c4365107" (UID: "be334717-f78a-4113-bf03-15d5c4365107"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:58:51.738513 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:51.725833    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/160f0d66-4e21-49eb-9c25-b467bdb56f41-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/160f0d66-4e21-49eb-9c25-b467bdb56f41-workload-certs\") pod \"160f0d66-4e21-49eb-9c25-b467bdb56f41\" (UID: \"160f0d66-4e21-49eb-9c25-b467bdb56f41\") "
Nov 02 12:58:51.738513 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:51.725928    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/160f0d66-4e21-49eb-9c25-b467bdb56f41-workload-socket" pod "160f0d66-4e21-49eb-9c25-b467bdb56f41" (UID: "160f0d66-4e21-49eb-9c25-b467bdb56f41"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:58:51.738513 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:51.726028    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/160f0d66-4e21-49eb-9c25-b467bdb56f41-workload-certs" pod "160f0d66-4e21-49eb-9c25-b467bdb56f41" (UID: "160f0d66-4e21-49eb-9c25-b467bdb56f41"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:58:51.738513 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:51.726144    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91783d5b-f538-43a0-a9d8-29f7270cadcd-istio-data" pod "91783d5b-f538-43a0-a9d8-29f7270cadcd" (UID: "91783d5b-f538-43a0-a9d8-29f7270cadcd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:58:51.738513 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:51.726168    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91783d5b-f538-43a0-a9d8-29f7270cadcd-istiod-ca-cert" pod "91783d5b-f538-43a0-a9d8-29f7270cadcd" (UID: "91783d5b-f538-43a0-a9d8-29f7270cadcd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:58:51.738513 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:51.726366    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be334717-f78a-4113-bf03-15d5c4365107-istiod-ca-cert" pod "be334717-f78a-4113-bf03-15d5c4365107" (UID: "be334717-f78a-4113-bf03-15d5c4365107"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:58:51.738513 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:51.726377    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be334717-f78a-4113-bf03-15d5c4365107-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be334717-f78a-4113-bf03-15d5c4365107-workload-socket\") pod \"be334717-f78a-4113-bf03-15d5c4365107\" (UID: \"be334717-f78a-4113-bf03-15d5c4365107\") "
Nov 02 12:58:51.738513 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:51.726432    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/91783d5b-f538-43a0-a9d8-29f7270cadcd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91783d5b-f538-43a0-a9d8-29f7270cadcd-istio-podinfo\") pod \"91783d5b-f538-43a0-a9d8-29f7270cadcd\" (UID: \"91783d5b-f538-43a0-a9d8-29f7270cadcd\") "
Nov 02 12:58:51.738513 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:51.726499    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91783d5b-f538-43a0-a9d8-29f7270cadcd-workload-certs" pod "91783d5b-f538-43a0-a9d8-29f7270cadcd" (UID: "91783d5b-f538-43a0-a9d8-29f7270cadcd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:58:51.738513 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:51.727279    4559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91783d5b-f538-43a0-a9d8-29f7270cadcd-workload-certs\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:58:51.738513 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:51.727302    4559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91783d5b-f538-43a0-a9d8-29f7270cadcd-istiod-ca-cert\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:58:51.738513 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:51.727315    4559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91783d5b-f538-43a0-a9d8-29f7270cadcd-istio-data\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:58:51.738513 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:51.727327    4559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be334717-f78a-4113-bf03-15d5c4365107-istiod-ca-cert\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:58:51.738513 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:51.727339    4559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/160f0d66-4e21-49eb-9c25-b467bdb56f41-workload-certs\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:58:51.738513 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:51.727351    4559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/160f0d66-4e21-49eb-9c25-b467bdb56f41-istiod-ca-cert\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:58:51.738513 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:51.727364    4559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91783d5b-f538-43a0-a9d8-29f7270cadcd-credential-socket\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:58:51.738513 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:51.727375    4559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/160f0d66-4e21-49eb-9c25-b467bdb56f41-istio-data\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:58:51.738513 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:51.727388    4559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/160f0d66-4e21-49eb-9c25-b467bdb56f41-workload-socket\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:58:51.738513 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:51.727401    4559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be334717-f78a-4113-bf03-15d5c4365107-istio-data\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:58:51.738513 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:51.727414    4559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be334717-f78a-4113-bf03-15d5c4365107-workload-certs\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:58:51.738513 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:51.727426    4559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be334717-f78a-4113-bf03-15d5c4365107-credential-socket\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:58:51.738513 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:51.727439    4559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91783d5b-f538-43a0-a9d8-29f7270cadcd-workload-socket\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:58:51.738513 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:51.727893    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/160f0d66-4e21-49eb-9c25-b467bdb56f41-credential-socket" pod "160f0d66-4e21-49eb-9c25-b467bdb56f41" (UID: "160f0d66-4e21-49eb-9c25-b467bdb56f41"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:58:51.738513 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:51.728316    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be334717-f78a-4113-bf03-15d5c4365107-istio-envoy" pod "be334717-f78a-4113-bf03-15d5c4365107" (UID: "be334717-f78a-4113-bf03-15d5c4365107"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:58:51.738513 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:51.730097    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91783d5b-f538-43a0-a9d8-29f7270cadcd-istio-token" pod "91783d5b-f538-43a0-a9d8-29f7270cadcd" (UID: "91783d5b-f538-43a0-a9d8-29f7270cadcd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:58:51.738513 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:51.730318    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be334717-f78a-4113-bf03-15d5c4365107-workload-socket" pod "be334717-f78a-4113-bf03-15d5c4365107" (UID: "be334717-f78a-4113-bf03-15d5c4365107"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:58:51.745632 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:51.745574    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/160f0d66-4e21-49eb-9c25-b467bdb56f41-istio-envoy" pod "160f0d66-4e21-49eb-9c25-b467bdb56f41" (UID: "160f0d66-4e21-49eb-9c25-b467bdb56f41"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:58:51.745632 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:51.745594    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91783d5b-f538-43a0-a9d8-29f7270cadcd-istio-envoy" pod "91783d5b-f538-43a0-a9d8-29f7270cadcd" (UID: "91783d5b-f538-43a0-a9d8-29f7270cadcd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:58:51.745828 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:51.745699    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be334717-f78a-4113-bf03-15d5c4365107-istio-token" pod "be334717-f78a-4113-bf03-15d5c4365107" (UID: "be334717-f78a-4113-bf03-15d5c4365107"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:58:51.745828 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:51.745777    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/be334717-f78a-4113-bf03-15d5c4365107-istio-podinfo" pod "be334717-f78a-4113-bf03-15d5c4365107" (UID: "be334717-f78a-4113-bf03-15d5c4365107"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 12:58:51.745948 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:51.745879    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/160f0d66-4e21-49eb-9c25-b467bdb56f41-istio-podinfo" pod "160f0d66-4e21-49eb-9c25-b467bdb56f41" (UID: "160f0d66-4e21-49eb-9c25-b467bdb56f41"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 12:58:51.746011 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:51.745971    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be334717-f78a-4113-bf03-15d5c4365107-kube-api-access-pt7tk" pod "be334717-f78a-4113-bf03-15d5c4365107" (UID: "be334717-f78a-4113-bf03-15d5c4365107"). InnerVolumeSpecName "kube-api-access-pt7tk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:58:51.746885 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:51.746816    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/160f0d66-4e21-49eb-9c25-b467bdb56f41-istio-token" pod "160f0d66-4e21-49eb-9c25-b467bdb56f41" (UID: "160f0d66-4e21-49eb-9c25-b467bdb56f41"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:58:51.746885 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:51.746842    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/160f0d66-4e21-49eb-9c25-b467bdb56f41-kube-api-access-ttrdh" pod "160f0d66-4e21-49eb-9c25-b467bdb56f41" (UID: "160f0d66-4e21-49eb-9c25-b467bdb56f41"). InnerVolumeSpecName "kube-api-access-ttrdh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:58:51.746885 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:51.746851    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91783d5b-f538-43a0-a9d8-29f7270cadcd-kube-api-access-9shkk" pod "91783d5b-f538-43a0-a9d8-29f7270cadcd" (UID: "91783d5b-f538-43a0-a9d8-29f7270cadcd"). InnerVolumeSpecName "kube-api-access-9shkk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:58:51.748499 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:51.748474    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/91783d5b-f538-43a0-a9d8-29f7270cadcd-istio-podinfo" pod "91783d5b-f538-43a0-a9d8-29f7270cadcd" (UID: "91783d5b-f538-43a0-a9d8-29f7270cadcd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 12:58:51.828745 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:51.828650    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9shkk\" (UniqueName: \"kubernetes.io/projected/91783d5b-f538-43a0-a9d8-29f7270cadcd-kube-api-access-9shkk\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:58:51.828745 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:51.828684    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pt7tk\" (UniqueName: \"kubernetes.io/projected/be334717-f78a-4113-bf03-15d5c4365107-kube-api-access-pt7tk\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:58:51.828745 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:51.828694    4559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/160f0d66-4e21-49eb-9c25-b467bdb56f41-istio-token\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:58:51.828745 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:51.828705    4559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be334717-f78a-4113-bf03-15d5c4365107-istio-token\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:58:51.828745 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:51.828713    4559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/160f0d66-4e21-49eb-9c25-b467bdb56f41-istio-envoy\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:58:51.828745 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:51.828720    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ttrdh\" (UniqueName: \"kubernetes.io/projected/160f0d66-4e21-49eb-9c25-b467bdb56f41-kube-api-access-ttrdh\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:58:51.828745 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:51.828730    4559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/160f0d66-4e21-49eb-9c25-b467bdb56f41-istio-podinfo\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:58:51.828745 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:51.828739    4559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/160f0d66-4e21-49eb-9c25-b467bdb56f41-credential-socket\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:58:51.828745 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:51.828748    4559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91783d5b-f538-43a0-a9d8-29f7270cadcd-istio-envoy\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:58:51.828745 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:51.828756    4559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be334717-f78a-4113-bf03-15d5c4365107-istio-podinfo\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:58:51.829417 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:51.828764    4559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be334717-f78a-4113-bf03-15d5c4365107-workload-socket\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:58:51.829417 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:51.828773    4559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91783d5b-f538-43a0-a9d8-29f7270cadcd-istio-podinfo\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:58:51.829417 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:51.828782    4559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be334717-f78a-4113-bf03-15d5c4365107-istio-envoy\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:58:51.829417 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:51.828790    4559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91783d5b-f538-43a0-a9d8-29f7270cadcd-istio-token\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:58:52.513324 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:52.513292    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-52bzl"
Nov 02 12:58:52.513324 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:52.513309    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-52bzl" event={"ID":"160f0d66-4e21-49eb-9c25-b467bdb56f41","Type":"ContainerDied","Data":"e476de3840eab8389aa0c3a0e72bb7e77c3fb5039fd0b23669fbb0407b5ad0d2"}
Nov 02 12:58:52.513546 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:52.513406    4559 scope.go:122] "RemoveContainer" containerID="a56f8505504b37a1ad5c87e4fd9da9d9f95274c306e46273d0c503e6576f210e"
Nov 02 12:58:52.514993 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:52.514965    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-cdnsv" event={"ID":"be334717-f78a-4113-bf03-15d5c4365107","Type":"ContainerDied","Data":"4062ef3dd834f9ab73f100bd0c11663d045f8293df7c56f6864749d074fd2d1d"}
Nov 02 12:58:52.515091 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:52.514976    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-cdnsv"
Nov 02 12:58:52.516714 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:52.516687    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-fwt4g" event={"ID":"91783d5b-f538-43a0-a9d8-29f7270cadcd","Type":"ContainerDied","Data":"3dee1d8dc53ff0b6ef36cdf323543a7e9df177d3ecc508f30cbf450919087c2e"}
Nov 02 12:58:52.516811 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:52.516737    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-fwt4g"
Nov 02 12:58:52.524808 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:52.524788    4559 scope.go:122] "RemoveContainer" containerID="32750de39eedcb157478ffa2aca02da0c43e2efc8a63d644f34bbce8d1ef88b2"
Nov 02 12:58:52.535974 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:52.535955    4559 scope.go:122] "RemoveContainer" containerID="f75d8cc15890dd03513350f317fa6efc86f60028e165eacc7c22765c06af76ad"
Nov 02 12:58:52.584050 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:52.583978    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-fwt4g"]
Nov 02 12:58:52.595692 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:52.595658    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-fwt4g"]
Nov 02 12:58:52.631353 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:52.631323    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-cdnsv"]
Nov 02 12:58:52.638133 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:52.638106    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-cdnsv"]
Nov 02 12:58:52.695092 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:52.695056    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-52bzl"]
Nov 02 12:58:52.703194 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:52.703159    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-52bzl"]
Nov 02 12:58:53.095315 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:53.095272    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-znwwt"]
Nov 02 12:58:53.095535 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:53.095510    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-znwwt" podUID="53afdb1f-5ab4-4106-bccd-9ce04d9ced37" containerName="istio-proxy" containerID="cri-o://df2df1e59e30a10900104ef6b717fdfe588a3983fc690b685b499b712a7a0bee" gracePeriod=30
Nov 02 12:58:53.894355 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:53.894323    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="160f0d66-4e21-49eb-9c25-b467bdb56f41" path="/var/lib/kubelet/pods/160f0d66-4e21-49eb-9c25-b467bdb56f41/volumes"
Nov 02 12:58:53.894890 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:53.894866    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91783d5b-f538-43a0-a9d8-29f7270cadcd" path="/var/lib/kubelet/pods/91783d5b-f538-43a0-a9d8-29f7270cadcd/volumes"
Nov 02 12:58:53.895322 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:53.895309    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be334717-f78a-4113-bf03-15d5c4365107" path="/var/lib/kubelet/pods/be334717-f78a-4113-bf03-15d5c4365107/volumes"
Nov 02 12:58:54.496826 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:54.496753    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-fbc46"
Nov 02 12:58:54.504934 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:54.504910    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-mdchl"
Nov 02 12:58:54.537127 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:54.537093    4559 generic.go:356] "Generic (PLEG): container finished" podID="bb1e97a1-f680-4922-920c-ee0a3468cb1f" containerID="3e0c9b0bc6a05cd6cd0b8e306b2739f4742aaabf4c9a9bd3666c4584dfce9b96" exitCode=0
Nov 02 12:58:54.537303 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:54.537155    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-mdchl"
Nov 02 12:58:54.537303 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:54.537170    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-mdchl" event={"ID":"bb1e97a1-f680-4922-920c-ee0a3468cb1f","Type":"ContainerDied","Data":"3e0c9b0bc6a05cd6cd0b8e306b2739f4742aaabf4c9a9bd3666c4584dfce9b96"}
Nov 02 12:58:54.537303 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:54.537228    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-mdchl" event={"ID":"bb1e97a1-f680-4922-920c-ee0a3468cb1f","Type":"ContainerDied","Data":"83c2158a2a6cac15ee3ff57eaddd3b9775ca87d24749ec97cb83a8ae39a02334"}
Nov 02 12:58:54.537303 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:54.537253    4559 scope.go:122] "RemoveContainer" containerID="3e0c9b0bc6a05cd6cd0b8e306b2739f4742aaabf4c9a9bd3666c4584dfce9b96"
Nov 02 12:58:54.538861 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:54.538838    4559 generic.go:356] "Generic (PLEG): container finished" podID="988e5461-e95e-408c-b145-7253a1a21056" containerID="8d70670962ad5c661ab1ecfd443fb8be9b3543ba0a63d44d92a0cac91b173323" exitCode=0
Nov 02 12:58:54.538999 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:54.538883    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-fbc46"
Nov 02 12:58:54.538999 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:54.538930    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-fbc46" event={"ID":"988e5461-e95e-408c-b145-7253a1a21056","Type":"ContainerDied","Data":"8d70670962ad5c661ab1ecfd443fb8be9b3543ba0a63d44d92a0cac91b173323"}
Nov 02 12:58:54.538999 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:54.538972    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-fbc46" event={"ID":"988e5461-e95e-408c-b145-7253a1a21056","Type":"ContainerDied","Data":"ccc27a630aea470d8a537fb23e7a2ae1c995c1300fdda53f2a1fe8fbc3e2a866"}
Nov 02 12:58:54.540835 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:54.540818    4559 generic.go:356] "Generic (PLEG): container finished" podID="d7f18bc2-015d-41da-a666-912b2e250e2a" containerID="a49513b40f2e0a5da91fdbeeac25ddfa386465768ff6828922520c816389cb94" exitCode=0
Nov 02 12:58:54.540944 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:54.540859    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-vtpz8" event={"ID":"d7f18bc2-015d-41da-a666-912b2e250e2a","Type":"ContainerDied","Data":"a49513b40f2e0a5da91fdbeeac25ddfa386465768ff6828922520c816389cb94"}
Nov 02 12:58:54.549672 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:54.549618    4559 scope.go:122] "RemoveContainer" containerID="3e0c9b0bc6a05cd6cd0b8e306b2739f4742aaabf4c9a9bd3666c4584dfce9b96"
Nov 02 12:58:54.550084 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:58:54.550060    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e0c9b0bc6a05cd6cd0b8e306b2739f4742aaabf4c9a9bd3666c4584dfce9b96\": container with ID starting with 3e0c9b0bc6a05cd6cd0b8e306b2739f4742aaabf4c9a9bd3666c4584dfce9b96 not found: ID does not exist" containerID="3e0c9b0bc6a05cd6cd0b8e306b2739f4742aaabf4c9a9bd3666c4584dfce9b96"
Nov 02 12:58:54.550178 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:54.550094    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e0c9b0bc6a05cd6cd0b8e306b2739f4742aaabf4c9a9bd3666c4584dfce9b96"} err="failed to get container status \"3e0c9b0bc6a05cd6cd0b8e306b2739f4742aaabf4c9a9bd3666c4584dfce9b96\": rpc error: code = NotFound desc = could not find container \"3e0c9b0bc6a05cd6cd0b8e306b2739f4742aaabf4c9a9bd3666c4584dfce9b96\": container with ID starting with 3e0c9b0bc6a05cd6cd0b8e306b2739f4742aaabf4c9a9bd3666c4584dfce9b96 not found: ID does not exist"
Nov 02 12:58:54.550178 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:54.550112    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-vtpz8"
Nov 02 12:58:54.550178 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:54.550115    4559 scope.go:122] "RemoveContainer" containerID="8d70670962ad5c661ab1ecfd443fb8be9b3543ba0a63d44d92a0cac91b173323"
Nov 02 12:58:54.560107 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:54.560089    4559 scope.go:122] "RemoveContainer" containerID="8d70670962ad5c661ab1ecfd443fb8be9b3543ba0a63d44d92a0cac91b173323"
Nov 02 12:58:54.560453 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:58:54.560429    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d70670962ad5c661ab1ecfd443fb8be9b3543ba0a63d44d92a0cac91b173323\": container with ID starting with 8d70670962ad5c661ab1ecfd443fb8be9b3543ba0a63d44d92a0cac91b173323 not found: ID does not exist" containerID="8d70670962ad5c661ab1ecfd443fb8be9b3543ba0a63d44d92a0cac91b173323"
Nov 02 12:58:54.560530 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:54.560470    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d70670962ad5c661ab1ecfd443fb8be9b3543ba0a63d44d92a0cac91b173323"} err="failed to get container status \"8d70670962ad5c661ab1ecfd443fb8be9b3543ba0a63d44d92a0cac91b173323\": rpc error: code = NotFound desc = could not find container \"8d70670962ad5c661ab1ecfd443fb8be9b3543ba0a63d44d92a0cac91b173323\": container with ID starting with 8d70670962ad5c661ab1ecfd443fb8be9b3543ba0a63d44d92a0cac91b173323 not found: ID does not exist"
Nov 02 12:58:54.571199 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:54.571175    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/988e5461-e95e-408c-b145-7253a1a21056-kube-api-access-mccs7\" (UniqueName: \"kubernetes.io/projected/988e5461-e95e-408c-b145-7253a1a21056-kube-api-access-mccs7\") pod \"988e5461-e95e-408c-b145-7253a1a21056\" (UID: \"988e5461-e95e-408c-b145-7253a1a21056\") "
Nov 02 12:58:54.571331 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:54.571239    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/988e5461-e95e-408c-b145-7253a1a21056-istio-token\" (UniqueName: \"kubernetes.io/projected/988e5461-e95e-408c-b145-7253a1a21056-istio-token\") pod \"988e5461-e95e-408c-b145-7253a1a21056\" (UID: \"988e5461-e95e-408c-b145-7253a1a21056\") "
Nov 02 12:58:54.571331 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:54.571275    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/988e5461-e95e-408c-b145-7253a1a21056-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/988e5461-e95e-408c-b145-7253a1a21056-istio-envoy\") pod \"988e5461-e95e-408c-b145-7253a1a21056\" (UID: \"988e5461-e95e-408c-b145-7253a1a21056\") "
Nov 02 12:58:54.571331 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:54.571326    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb1e97a1-f680-4922-920c-ee0a3468cb1f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb1e97a1-f680-4922-920c-ee0a3468cb1f-istio-data\") pod \"bb1e97a1-f680-4922-920c-ee0a3468cb1f\" (UID: \"bb1e97a1-f680-4922-920c-ee0a3468cb1f\") "
Nov 02 12:58:54.571517 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:54.571395    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bb1e97a1-f680-4922-920c-ee0a3468cb1f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb1e97a1-f680-4922-920c-ee0a3468cb1f-istio-podinfo\") pod \"bb1e97a1-f680-4922-920c-ee0a3468cb1f\" (UID: \"bb1e97a1-f680-4922-920c-ee0a3468cb1f\") "
Nov 02 12:58:54.571517 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:54.571425    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/988e5461-e95e-408c-b145-7253a1a21056-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/988e5461-e95e-408c-b145-7253a1a21056-istiod-ca-cert\") pod \"988e5461-e95e-408c-b145-7253a1a21056\" (UID: \"988e5461-e95e-408c-b145-7253a1a21056\") "
Nov 02 12:58:54.571517 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:54.571444    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb1e97a1-f680-4922-920c-ee0a3468cb1f-kube-api-access-fcmdk\" (UniqueName: \"kubernetes.io/projected/bb1e97a1-f680-4922-920c-ee0a3468cb1f-kube-api-access-fcmdk\") pod \"bb1e97a1-f680-4922-920c-ee0a3468cb1f\" (UID: \"bb1e97a1-f680-4922-920c-ee0a3468cb1f\") "
Nov 02 12:58:54.571517 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:54.571473    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb1e97a1-f680-4922-920c-ee0a3468cb1f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb1e97a1-f680-4922-920c-ee0a3468cb1f-credential-socket\") pod \"bb1e97a1-f680-4922-920c-ee0a3468cb1f\" (UID: \"bb1e97a1-f680-4922-920c-ee0a3468cb1f\") "
Nov 02 12:58:54.571517 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:54.571494    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb1e97a1-f680-4922-920c-ee0a3468cb1f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb1e97a1-f680-4922-920c-ee0a3468cb1f-istiod-ca-cert\") pod \"bb1e97a1-f680-4922-920c-ee0a3468cb1f\" (UID: \"bb1e97a1-f680-4922-920c-ee0a3468cb1f\") "
Nov 02 12:58:54.571517 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:54.571511    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb1e97a1-f680-4922-920c-ee0a3468cb1f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb1e97a1-f680-4922-920c-ee0a3468cb1f-workload-socket\") pod \"bb1e97a1-f680-4922-920c-ee0a3468cb1f\" (UID: \"bb1e97a1-f680-4922-920c-ee0a3468cb1f\") "
Nov 02 12:58:54.572006 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:54.571534    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb1e97a1-f680-4922-920c-ee0a3468cb1f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb1e97a1-f680-4922-920c-ee0a3468cb1f-istio-envoy\") pod \"bb1e97a1-f680-4922-920c-ee0a3468cb1f\" (UID: \"bb1e97a1-f680-4922-920c-ee0a3468cb1f\") "
Nov 02 12:58:54.572006 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:54.571551    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb1e97a1-f680-4922-920c-ee0a3468cb1f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb1e97a1-f680-4922-920c-ee0a3468cb1f-workload-certs\") pod \"bb1e97a1-f680-4922-920c-ee0a3468cb1f\" (UID: \"bb1e97a1-f680-4922-920c-ee0a3468cb1f\") "
Nov 02 12:58:54.572006 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:54.571581    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/988e5461-e95e-408c-b145-7253a1a21056-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/988e5461-e95e-408c-b145-7253a1a21056-workload-socket\") pod \"988e5461-e95e-408c-b145-7253a1a21056\" (UID: \"988e5461-e95e-408c-b145-7253a1a21056\") "
Nov 02 12:58:54.572006 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:54.571615    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/988e5461-e95e-408c-b145-7253a1a21056-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/988e5461-e95e-408c-b145-7253a1a21056-credential-socket\") pod \"988e5461-e95e-408c-b145-7253a1a21056\" (UID: \"988e5461-e95e-408c-b145-7253a1a21056\") "
Nov 02 12:58:54.572006 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:54.571684    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/988e5461-e95e-408c-b145-7253a1a21056-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/988e5461-e95e-408c-b145-7253a1a21056-istio-data\") pod \"988e5461-e95e-408c-b145-7253a1a21056\" (UID: \"988e5461-e95e-408c-b145-7253a1a21056\") "
Nov 02 12:58:54.572006 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:54.571713    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb1e97a1-f680-4922-920c-ee0a3468cb1f-istio-token\" (UniqueName: \"kubernetes.io/projected/bb1e97a1-f680-4922-920c-ee0a3468cb1f-istio-token\") pod \"bb1e97a1-f680-4922-920c-ee0a3468cb1f\" (UID: \"bb1e97a1-f680-4922-920c-ee0a3468cb1f\") "
Nov 02 12:58:54.572006 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:54.571771    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/988e5461-e95e-408c-b145-7253a1a21056-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/988e5461-e95e-408c-b145-7253a1a21056-workload-certs\") pod \"988e5461-e95e-408c-b145-7253a1a21056\" (UID: \"988e5461-e95e-408c-b145-7253a1a21056\") "
Nov 02 12:58:54.572006 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:54.571832    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/988e5461-e95e-408c-b145-7253a1a21056-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/988e5461-e95e-408c-b145-7253a1a21056-istio-podinfo\") pod \"988e5461-e95e-408c-b145-7253a1a21056\" (UID: \"988e5461-e95e-408c-b145-7253a1a21056\") "
Nov 02 12:58:54.584166 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:54.574227    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/988e5461-e95e-408c-b145-7253a1a21056-istio-token" pod "988e5461-e95e-408c-b145-7253a1a21056" (UID: "988e5461-e95e-408c-b145-7253a1a21056"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:58:54.584166 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:54.574367    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/988e5461-e95e-408c-b145-7253a1a21056-kube-api-access-mccs7" pod "988e5461-e95e-408c-b145-7253a1a21056" (UID: "988e5461-e95e-408c-b145-7253a1a21056"). InnerVolumeSpecName "kube-api-access-mccs7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:58:54.584166 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:54.574967    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/988e5461-e95e-408c-b145-7253a1a21056-istio-podinfo" pod "988e5461-e95e-408c-b145-7253a1a21056" (UID: "988e5461-e95e-408c-b145-7253a1a21056"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 12:58:54.584166 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:54.575392    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb1e97a1-f680-4922-920c-ee0a3468cb1f-workload-socket" pod "bb1e97a1-f680-4922-920c-ee0a3468cb1f" (UID: "bb1e97a1-f680-4922-920c-ee0a3468cb1f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:58:54.584166 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:54.575508    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/988e5461-e95e-408c-b145-7253a1a21056-credential-socket" pod "988e5461-e95e-408c-b145-7253a1a21056" (UID: "988e5461-e95e-408c-b145-7253a1a21056"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:58:54.584166 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:54.575586    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb1e97a1-f680-4922-920c-ee0a3468cb1f-istio-data" pod "bb1e97a1-f680-4922-920c-ee0a3468cb1f" (UID: "bb1e97a1-f680-4922-920c-ee0a3468cb1f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:58:54.584166 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:54.577581    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/988e5461-e95e-408c-b145-7253a1a21056-istio-envoy" pod "988e5461-e95e-408c-b145-7253a1a21056" (UID: "988e5461-e95e-408c-b145-7253a1a21056"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:58:54.584166 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:54.577766    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bb1e97a1-f680-4922-920c-ee0a3468cb1f-istio-podinfo" pod "bb1e97a1-f680-4922-920c-ee0a3468cb1f" (UID: "bb1e97a1-f680-4922-920c-ee0a3468cb1f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 12:58:54.584166 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:54.577858    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/988e5461-e95e-408c-b145-7253a1a21056-istio-data" pod "988e5461-e95e-408c-b145-7253a1a21056" (UID: "988e5461-e95e-408c-b145-7253a1a21056"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:58:54.584166 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:54.578110    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/988e5461-e95e-408c-b145-7253a1a21056-istiod-ca-cert" pod "988e5461-e95e-408c-b145-7253a1a21056" (UID: "988e5461-e95e-408c-b145-7253a1a21056"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:58:54.584166 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:54.578571    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb1e97a1-f680-4922-920c-ee0a3468cb1f-istio-envoy" pod "bb1e97a1-f680-4922-920c-ee0a3468cb1f" (UID: "bb1e97a1-f680-4922-920c-ee0a3468cb1f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:58:54.584166 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:54.578826    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb1e97a1-f680-4922-920c-ee0a3468cb1f-workload-certs" pod "bb1e97a1-f680-4922-920c-ee0a3468cb1f" (UID: "bb1e97a1-f680-4922-920c-ee0a3468cb1f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:58:54.584166 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:54.579008    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/988e5461-e95e-408c-b145-7253a1a21056-workload-socket" pod "988e5461-e95e-408c-b145-7253a1a21056" (UID: "988e5461-e95e-408c-b145-7253a1a21056"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:58:54.584166 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:54.579195    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb1e97a1-f680-4922-920c-ee0a3468cb1f-credential-socket" pod "bb1e97a1-f680-4922-920c-ee0a3468cb1f" (UID: "bb1e97a1-f680-4922-920c-ee0a3468cb1f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:58:54.584166 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:54.579447    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb1e97a1-f680-4922-920c-ee0a3468cb1f-istiod-ca-cert" pod "bb1e97a1-f680-4922-920c-ee0a3468cb1f" (UID: "bb1e97a1-f680-4922-920c-ee0a3468cb1f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:58:54.584166 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:54.579792    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/988e5461-e95e-408c-b145-7253a1a21056-workload-certs" pod "988e5461-e95e-408c-b145-7253a1a21056" (UID: "988e5461-e95e-408c-b145-7253a1a21056"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:58:54.584166 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:54.579929    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb1e97a1-f680-4922-920c-ee0a3468cb1f-istio-token" pod "bb1e97a1-f680-4922-920c-ee0a3468cb1f" (UID: "bb1e97a1-f680-4922-920c-ee0a3468cb1f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:58:54.584166 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:54.581928    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb1e97a1-f680-4922-920c-ee0a3468cb1f-kube-api-access-fcmdk" pod "bb1e97a1-f680-4922-920c-ee0a3468cb1f" (UID: "bb1e97a1-f680-4922-920c-ee0a3468cb1f"). InnerVolumeSpecName "kube-api-access-fcmdk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:58:54.673583 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:54.673548    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7f18bc2-015d-41da-a666-912b2e250e2a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7f18bc2-015d-41da-a666-912b2e250e2a-credential-socket\") pod \"d7f18bc2-015d-41da-a666-912b2e250e2a\" (UID: \"d7f18bc2-015d-41da-a666-912b2e250e2a\") "
Nov 02 12:58:54.673812 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:54.673626    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7f18bc2-015d-41da-a666-912b2e250e2a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7f18bc2-015d-41da-a666-912b2e250e2a-workload-socket\") pod \"d7f18bc2-015d-41da-a666-912b2e250e2a\" (UID: \"d7f18bc2-015d-41da-a666-912b2e250e2a\") "
Nov 02 12:58:54.673812 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:54.673742    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7f18bc2-015d-41da-a666-912b2e250e2a-istio-token\" (UniqueName: \"kubernetes.io/projected/d7f18bc2-015d-41da-a666-912b2e250e2a-istio-token\") pod \"d7f18bc2-015d-41da-a666-912b2e250e2a\" (UID: \"d7f18bc2-015d-41da-a666-912b2e250e2a\") "
Nov 02 12:58:54.673955 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:54.673824    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7f18bc2-015d-41da-a666-912b2e250e2a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7f18bc2-015d-41da-a666-912b2e250e2a-istiod-ca-cert\") pod \"d7f18bc2-015d-41da-a666-912b2e250e2a\" (UID: \"d7f18bc2-015d-41da-a666-912b2e250e2a\") "
Nov 02 12:58:54.673955 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:54.673890    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7f18bc2-015d-41da-a666-912b2e250e2a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7f18bc2-015d-41da-a666-912b2e250e2a-istio-data\") pod \"d7f18bc2-015d-41da-a666-912b2e250e2a\" (UID: \"d7f18bc2-015d-41da-a666-912b2e250e2a\") "
Nov 02 12:58:54.673955 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:54.673916    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7f18bc2-015d-41da-a666-912b2e250e2a-credential-socket" pod "d7f18bc2-015d-41da-a666-912b2e250e2a" (UID: "d7f18bc2-015d-41da-a666-912b2e250e2a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:58:54.674132 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:54.673982    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d7f18bc2-015d-41da-a666-912b2e250e2a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7f18bc2-015d-41da-a666-912b2e250e2a-istio-podinfo\") pod \"d7f18bc2-015d-41da-a666-912b2e250e2a\" (UID: \"d7f18bc2-015d-41da-a666-912b2e250e2a\") "
Nov 02 12:58:54.674132 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:54.674006    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7f18bc2-015d-41da-a666-912b2e250e2a-workload-socket" pod "d7f18bc2-015d-41da-a666-912b2e250e2a" (UID: "d7f18bc2-015d-41da-a666-912b2e250e2a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:58:54.674132 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:54.674036    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7f18bc2-015d-41da-a666-912b2e250e2a-kube-api-access-dkcq6\" (UniqueName: \"kubernetes.io/projected/d7f18bc2-015d-41da-a666-912b2e250e2a-kube-api-access-dkcq6\") pod \"d7f18bc2-015d-41da-a666-912b2e250e2a\" (UID: \"d7f18bc2-015d-41da-a666-912b2e250e2a\") "
Nov 02 12:58:54.674132 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:54.674071    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7f18bc2-015d-41da-a666-912b2e250e2a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7f18bc2-015d-41da-a666-912b2e250e2a-workload-certs\") pod \"d7f18bc2-015d-41da-a666-912b2e250e2a\" (UID: \"d7f18bc2-015d-41da-a666-912b2e250e2a\") "
Nov 02 12:58:54.674132 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:54.674105    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7f18bc2-015d-41da-a666-912b2e250e2a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7f18bc2-015d-41da-a666-912b2e250e2a-istio-envoy\") pod \"d7f18bc2-015d-41da-a666-912b2e250e2a\" (UID: \"d7f18bc2-015d-41da-a666-912b2e250e2a\") "
Nov 02 12:58:54.674413 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:54.674298    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7f18bc2-015d-41da-a666-912b2e250e2a-workload-certs" pod "d7f18bc2-015d-41da-a666-912b2e250e2a" (UID: "d7f18bc2-015d-41da-a666-912b2e250e2a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:58:54.674477 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:54.674442    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7f18bc2-015d-41da-a666-912b2e250e2a-istio-data" pod "d7f18bc2-015d-41da-a666-912b2e250e2a" (UID: "d7f18bc2-015d-41da-a666-912b2e250e2a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:58:54.674602 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:54.674581    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7f18bc2-015d-41da-a666-912b2e250e2a-istiod-ca-cert" pod "d7f18bc2-015d-41da-a666-912b2e250e2a" (UID: "d7f18bc2-015d-41da-a666-912b2e250e2a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:58:54.675054 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:54.675029    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mccs7\" (UniqueName: \"kubernetes.io/projected/988e5461-e95e-408c-b145-7253a1a21056-kube-api-access-mccs7\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:58:54.675155 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:54.675060    4559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7f18bc2-015d-41da-a666-912b2e250e2a-workload-certs\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:58:54.675155 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:54.675078    4559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/988e5461-e95e-408c-b145-7253a1a21056-istio-token\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:58:54.675155 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:54.675092    4559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/988e5461-e95e-408c-b145-7253a1a21056-istio-envoy\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:58:54.675155 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:54.675105    4559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7f18bc2-015d-41da-a666-912b2e250e2a-credential-socket\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:58:54.675155 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:54.675120    4559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb1e97a1-f680-4922-920c-ee0a3468cb1f-istio-data\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:58:54.675155 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:54.675135    4559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7f18bc2-015d-41da-a666-912b2e250e2a-workload-socket\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:58:54.675155 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:54.675147    4559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb1e97a1-f680-4922-920c-ee0a3468cb1f-istio-podinfo\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:58:54.675554 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:54.675160    4559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/988e5461-e95e-408c-b145-7253a1a21056-istiod-ca-cert\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:58:54.675554 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:54.675176    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fcmdk\" (UniqueName: \"kubernetes.io/projected/bb1e97a1-f680-4922-920c-ee0a3468cb1f-kube-api-access-fcmdk\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:58:54.675554 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:54.675190    4559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb1e97a1-f680-4922-920c-ee0a3468cb1f-credential-socket\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:58:54.675554 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:54.675204    4559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb1e97a1-f680-4922-920c-ee0a3468cb1f-istiod-ca-cert\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:58:54.675554 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:54.675218    4559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb1e97a1-f680-4922-920c-ee0a3468cb1f-workload-socket\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:58:54.675554 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:54.675230    4559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb1e97a1-f680-4922-920c-ee0a3468cb1f-istio-envoy\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:58:54.675554 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:54.675244    4559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb1e97a1-f680-4922-920c-ee0a3468cb1f-workload-certs\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:58:54.675554 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:54.675259    4559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/988e5461-e95e-408c-b145-7253a1a21056-workload-socket\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:58:54.675554 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:54.675273    4559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/988e5461-e95e-408c-b145-7253a1a21056-credential-socket\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:58:54.675554 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:54.675286    4559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/988e5461-e95e-408c-b145-7253a1a21056-istio-data\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:58:54.675554 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:54.675298    4559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb1e97a1-f680-4922-920c-ee0a3468cb1f-istio-token\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:58:54.675554 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:54.675310    4559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7f18bc2-015d-41da-a666-912b2e250e2a-istiod-ca-cert\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:58:54.675554 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:54.675323    4559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/988e5461-e95e-408c-b145-7253a1a21056-workload-certs\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:58:54.675554 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:54.675335    4559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/988e5461-e95e-408c-b145-7253a1a21056-istio-podinfo\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:58:54.675554 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:54.675347    4559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7f18bc2-015d-41da-a666-912b2e250e2a-istio-data\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:58:54.678314 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:54.678285    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7f18bc2-015d-41da-a666-912b2e250e2a-istio-token" pod "d7f18bc2-015d-41da-a666-912b2e250e2a" (UID: "d7f18bc2-015d-41da-a666-912b2e250e2a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:58:54.678427 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:54.678330    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7f18bc2-015d-41da-a666-912b2e250e2a-istio-envoy" pod "d7f18bc2-015d-41da-a666-912b2e250e2a" (UID: "d7f18bc2-015d-41da-a666-912b2e250e2a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:58:54.678485 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:54.678423    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d7f18bc2-015d-41da-a666-912b2e250e2a-istio-podinfo" pod "d7f18bc2-015d-41da-a666-912b2e250e2a" (UID: "d7f18bc2-015d-41da-a666-912b2e250e2a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 12:58:54.678525 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:54.678477    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7f18bc2-015d-41da-a666-912b2e250e2a-kube-api-access-dkcq6" pod "d7f18bc2-015d-41da-a666-912b2e250e2a" (UID: "d7f18bc2-015d-41da-a666-912b2e250e2a"). InnerVolumeSpecName "kube-api-access-dkcq6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:58:54.783125 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:54.783028    4559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7f18bc2-015d-41da-a666-912b2e250e2a-istio-envoy\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:58:54.783125 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:54.783057    4559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7f18bc2-015d-41da-a666-912b2e250e2a-istio-token\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:58:54.783125 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:54.783067    4559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7f18bc2-015d-41da-a666-912b2e250e2a-istio-podinfo\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:58:54.783125 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:54.783076    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dkcq6\" (UniqueName: \"kubernetes.io/projected/d7f18bc2-015d-41da-a666-912b2e250e2a-kube-api-access-dkcq6\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:58:54.888224 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:54.888194    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-mdchl"]
Nov 02 12:58:54.893825 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:54.893768    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-mdchl"]
Nov 02 12:58:54.929900 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:54.929868    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-fbc46"]
Nov 02 12:58:54.933229 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:54.933208    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-fbc46"]
Nov 02 12:58:55.549481 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:55.549396    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-vtpz8"
Nov 02 12:58:55.549481 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:55.549417    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-vtpz8" event={"ID":"d7f18bc2-015d-41da-a666-912b2e250e2a","Type":"ContainerDied","Data":"1fc462bf450a25de16439d40abc49c09bba0bc08d2b35592a47a07b56068abd9"}
Nov 02 12:58:55.549728 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:55.549519    4559 scope.go:122] "RemoveContainer" containerID="a49513b40f2e0a5da91fdbeeac25ddfa386465768ff6828922520c816389cb94"
Nov 02 12:58:55.617483 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:55.617450    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-vtpz8"]
Nov 02 12:58:55.630769 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:55.630738    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-vtpz8"]
Nov 02 12:58:55.894899 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:55.894822    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="988e5461-e95e-408c-b145-7253a1a21056" path="/var/lib/kubelet/pods/988e5461-e95e-408c-b145-7253a1a21056/volumes"
Nov 02 12:58:55.895367 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:55.895353    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb1e97a1-f680-4922-920c-ee0a3468cb1f" path="/var/lib/kubelet/pods/bb1e97a1-f680-4922-920c-ee0a3468cb1f/volumes"
Nov 02 12:58:55.895808 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:55.895795    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7f18bc2-015d-41da-a666-912b2e250e2a" path="/var/lib/kubelet/pods/d7f18bc2-015d-41da-a666-912b2e250e2a/volumes"
Nov 02 12:58:58.428101 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:58.428075    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-znwwt"
Nov 02 12:58:58.544001 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:58.543964    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53afdb1f-5ab4-4106-bccd-9ce04d9ced37-kube-api-access-fkbjj\" (UniqueName: \"kubernetes.io/projected/53afdb1f-5ab4-4106-bccd-9ce04d9ced37-kube-api-access-fkbjj\") pod \"53afdb1f-5ab4-4106-bccd-9ce04d9ced37\" (UID: \"53afdb1f-5ab4-4106-bccd-9ce04d9ced37\") "
Nov 02 12:58:58.544170 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:58.544010    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53afdb1f-5ab4-4106-bccd-9ce04d9ced37-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53afdb1f-5ab4-4106-bccd-9ce04d9ced37-credential-socket\") pod \"53afdb1f-5ab4-4106-bccd-9ce04d9ced37\" (UID: \"53afdb1f-5ab4-4106-bccd-9ce04d9ced37\") "
Nov 02 12:58:58.544170 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:58.544086    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53afdb1f-5ab4-4106-bccd-9ce04d9ced37-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53afdb1f-5ab4-4106-bccd-9ce04d9ced37-workload-socket\") pod \"53afdb1f-5ab4-4106-bccd-9ce04d9ced37\" (UID: \"53afdb1f-5ab4-4106-bccd-9ce04d9ced37\") "
Nov 02 12:58:58.544170 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:58.544152    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53afdb1f-5ab4-4106-bccd-9ce04d9ced37-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53afdb1f-5ab4-4106-bccd-9ce04d9ced37-workload-certs\") pod \"53afdb1f-5ab4-4106-bccd-9ce04d9ced37\" (UID: \"53afdb1f-5ab4-4106-bccd-9ce04d9ced37\") "
Nov 02 12:58:58.544293 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:58.544187    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/53afdb1f-5ab4-4106-bccd-9ce04d9ced37-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53afdb1f-5ab4-4106-bccd-9ce04d9ced37-istio-podinfo\") pod \"53afdb1f-5ab4-4106-bccd-9ce04d9ced37\" (UID: \"53afdb1f-5ab4-4106-bccd-9ce04d9ced37\") "
Nov 02 12:58:58.544358 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:58.544289    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53afdb1f-5ab4-4106-bccd-9ce04d9ced37-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53afdb1f-5ab4-4106-bccd-9ce04d9ced37-istio-envoy\") pod \"53afdb1f-5ab4-4106-bccd-9ce04d9ced37\" (UID: \"53afdb1f-5ab4-4106-bccd-9ce04d9ced37\") "
Nov 02 12:58:58.544420 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:58.544404    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53afdb1f-5ab4-4106-bccd-9ce04d9ced37-istio-token\" (UniqueName: \"kubernetes.io/projected/53afdb1f-5ab4-4106-bccd-9ce04d9ced37-istio-token\") pod \"53afdb1f-5ab4-4106-bccd-9ce04d9ced37\" (UID: \"53afdb1f-5ab4-4106-bccd-9ce04d9ced37\") "
Nov 02 12:58:58.544420 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:58.544410    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53afdb1f-5ab4-4106-bccd-9ce04d9ced37-workload-socket" pod "53afdb1f-5ab4-4106-bccd-9ce04d9ced37" (UID: "53afdb1f-5ab4-4106-bccd-9ce04d9ced37"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:58:58.544748 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:58.544388    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53afdb1f-5ab4-4106-bccd-9ce04d9ced37-credential-socket" pod "53afdb1f-5ab4-4106-bccd-9ce04d9ced37" (UID: "53afdb1f-5ab4-4106-bccd-9ce04d9ced37"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:58:58.544748 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:58.544445    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53afdb1f-5ab4-4106-bccd-9ce04d9ced37-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53afdb1f-5ab4-4106-bccd-9ce04d9ced37-istio-data\") pod \"53afdb1f-5ab4-4106-bccd-9ce04d9ced37\" (UID: \"53afdb1f-5ab4-4106-bccd-9ce04d9ced37\") "
Nov 02 12:58:58.544748 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:58.544434    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53afdb1f-5ab4-4106-bccd-9ce04d9ced37-workload-certs" pod "53afdb1f-5ab4-4106-bccd-9ce04d9ced37" (UID: "53afdb1f-5ab4-4106-bccd-9ce04d9ced37"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:58:58.544748 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:58.544491    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53afdb1f-5ab4-4106-bccd-9ce04d9ced37-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53afdb1f-5ab4-4106-bccd-9ce04d9ced37-istiod-ca-cert\") pod \"53afdb1f-5ab4-4106-bccd-9ce04d9ced37\" (UID: \"53afdb1f-5ab4-4106-bccd-9ce04d9ced37\") "
Nov 02 12:58:58.544748 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:58.544680    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53afdb1f-5ab4-4106-bccd-9ce04d9ced37-istio-data" pod "53afdb1f-5ab4-4106-bccd-9ce04d9ced37" (UID: "53afdb1f-5ab4-4106-bccd-9ce04d9ced37"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:58:58.546404 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:58.544951    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53afdb1f-5ab4-4106-bccd-9ce04d9ced37-istiod-ca-cert" pod "53afdb1f-5ab4-4106-bccd-9ce04d9ced37" (UID: "53afdb1f-5ab4-4106-bccd-9ce04d9ced37"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:58:58.546404 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:58.545179    4559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53afdb1f-5ab4-4106-bccd-9ce04d9ced37-istio-data\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:58:58.546404 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:58.545204    4559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53afdb1f-5ab4-4106-bccd-9ce04d9ced37-istiod-ca-cert\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:58:58.546404 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:58.545218    4559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53afdb1f-5ab4-4106-bccd-9ce04d9ced37-credential-socket\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:58:58.546404 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:58.545233    4559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53afdb1f-5ab4-4106-bccd-9ce04d9ced37-workload-socket\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:58:58.546404 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:58.545247    4559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53afdb1f-5ab4-4106-bccd-9ce04d9ced37-workload-certs\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:58:58.546667 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:58.546630    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53afdb1f-5ab4-4106-bccd-9ce04d9ced37-istio-envoy" pod "53afdb1f-5ab4-4106-bccd-9ce04d9ced37" (UID: "53afdb1f-5ab4-4106-bccd-9ce04d9ced37"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:58:58.546667 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:58.546627    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53afdb1f-5ab4-4106-bccd-9ce04d9ced37-istio-token" pod "53afdb1f-5ab4-4106-bccd-9ce04d9ced37" (UID: "53afdb1f-5ab4-4106-bccd-9ce04d9ced37"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:58:58.546773 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:58.546691    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53afdb1f-5ab4-4106-bccd-9ce04d9ced37-kube-api-access-fkbjj" pod "53afdb1f-5ab4-4106-bccd-9ce04d9ced37" (UID: "53afdb1f-5ab4-4106-bccd-9ce04d9ced37"). InnerVolumeSpecName "kube-api-access-fkbjj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:58:58.546773 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:58.546719    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/53afdb1f-5ab4-4106-bccd-9ce04d9ced37-istio-podinfo" pod "53afdb1f-5ab4-4106-bccd-9ce04d9ced37" (UID: "53afdb1f-5ab4-4106-bccd-9ce04d9ced37"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 12:58:58.575698 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:58.575631    4559 generic.go:356] "Generic (PLEG): container finished" podID="53afdb1f-5ab4-4106-bccd-9ce04d9ced37" containerID="df2df1e59e30a10900104ef6b717fdfe588a3983fc690b685b499b712a7a0bee" exitCode=0
Nov 02 12:58:58.575698 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:58.575694    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-znwwt"
Nov 02 12:58:58.575883 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:58.575714    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-znwwt" event={"ID":"53afdb1f-5ab4-4106-bccd-9ce04d9ced37","Type":"ContainerDied","Data":"df2df1e59e30a10900104ef6b717fdfe588a3983fc690b685b499b712a7a0bee"}
Nov 02 12:58:58.575883 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:58.575766    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-znwwt" event={"ID":"53afdb1f-5ab4-4106-bccd-9ce04d9ced37","Type":"ContainerDied","Data":"f8c1ca967a05566d23a2d20a929cd2d90b5c2cf613cb9ba529972bfddeaa74cc"}
Nov 02 12:58:58.575883 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:58.575789    4559 scope.go:122] "RemoveContainer" containerID="df2df1e59e30a10900104ef6b717fdfe588a3983fc690b685b499b712a7a0bee"
Nov 02 12:58:58.586276 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:58.586260    4559 scope.go:122] "RemoveContainer" containerID="df2df1e59e30a10900104ef6b717fdfe588a3983fc690b685b499b712a7a0bee"
Nov 02 12:58:58.586558 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:58:58.586540    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df2df1e59e30a10900104ef6b717fdfe588a3983fc690b685b499b712a7a0bee\": container with ID starting with df2df1e59e30a10900104ef6b717fdfe588a3983fc690b685b499b712a7a0bee not found: ID does not exist" containerID="df2df1e59e30a10900104ef6b717fdfe588a3983fc690b685b499b712a7a0bee"
Nov 02 12:58:58.586633 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:58.586566    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df2df1e59e30a10900104ef6b717fdfe588a3983fc690b685b499b712a7a0bee"} err="failed to get container status \"df2df1e59e30a10900104ef6b717fdfe588a3983fc690b685b499b712a7a0bee\": rpc error: code = NotFound desc = could not find container \"df2df1e59e30a10900104ef6b717fdfe588a3983fc690b685b499b712a7a0bee\": container with ID starting with df2df1e59e30a10900104ef6b717fdfe588a3983fc690b685b499b712a7a0bee not found: ID does not exist"
Nov 02 12:58:58.630921 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:58.630891    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-znwwt"]
Nov 02 12:58:58.636314 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:58.636284    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-znwwt"]
Nov 02 12:58:58.647146 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:58.647116    4559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53afdb1f-5ab4-4106-bccd-9ce04d9ced37-istio-token\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:58:58.647146 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:58.647145    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fkbjj\" (UniqueName: \"kubernetes.io/projected/53afdb1f-5ab4-4106-bccd-9ce04d9ced37-kube-api-access-fkbjj\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:58:58.647347 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:58.647159    4559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53afdb1f-5ab4-4106-bccd-9ce04d9ced37-istio-podinfo\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:58:58.647347 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:58.647175    4559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53afdb1f-5ab4-4106-bccd-9ce04d9ced37-istio-envoy\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:58:59.894326 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:58:59.894286    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53afdb1f-5ab4-4106-bccd-9ce04d9ced37" path="/var/lib/kubelet/pods/53afdb1f-5ab4-4106-bccd-9ce04d9ced37/volumes"
Nov 02 12:59:07.285714 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:07.283795    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"]
Nov 02 12:59:10.012974 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:10.012936    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-e430c7/explicit-model-e430c7-kserve-66569595-ws8hn"]
Nov 02 12:59:10.013606 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:10.013585    4559 cpu_manager.go:414] "containerMap: removing container" podUID="d7f18bc2-015d-41da-a666-912b2e250e2a" containerName="istio-proxy"
Nov 02 12:59:10.013606 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:10.013604    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7f18bc2-015d-41da-a666-912b2e250e2a" containerName="istio-proxy"
Nov 02 12:59:10.013841 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:10.013625    4559 cpu_manager.go:414] "containerMap: removing container" podUID="be334717-f78a-4113-bf03-15d5c4365107" containerName="istio-proxy"
Nov 02 12:59:10.013841 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:10.013631    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be334717-f78a-4113-bf03-15d5c4365107" containerName="istio-proxy"
Nov 02 12:59:10.013841 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:10.013654    4559 cpu_manager.go:414] "containerMap: removing container" podUID="160f0d66-4e21-49eb-9c25-b467bdb56f41" containerName="istio-proxy"
Nov 02 12:59:10.013841 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:10.013659    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="160f0d66-4e21-49eb-9c25-b467bdb56f41" containerName="istio-proxy"
Nov 02 12:59:10.013841 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:10.013669    4559 cpu_manager.go:414] "containerMap: removing container" podUID="bb1e97a1-f680-4922-920c-ee0a3468cb1f" containerName="istio-proxy"
Nov 02 12:59:10.013841 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:10.013674    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb1e97a1-f680-4922-920c-ee0a3468cb1f" containerName="istio-proxy"
Nov 02 12:59:10.013841 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:10.013700    4559 cpu_manager.go:414] "containerMap: removing container" podUID="53afdb1f-5ab4-4106-bccd-9ce04d9ced37" containerName="istio-proxy"
Nov 02 12:59:10.013841 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:10.013707    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53afdb1f-5ab4-4106-bccd-9ce04d9ced37" containerName="istio-proxy"
Nov 02 12:59:10.013841 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:10.013735    4559 cpu_manager.go:414] "containerMap: removing container" podUID="988e5461-e95e-408c-b145-7253a1a21056" containerName="istio-proxy"
Nov 02 12:59:10.013841 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:10.013743    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="988e5461-e95e-408c-b145-7253a1a21056" containerName="istio-proxy"
Nov 02 12:59:10.013841 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:10.013762    4559 cpu_manager.go:414] "containerMap: removing container" podUID="91783d5b-f538-43a0-a9d8-29f7270cadcd" containerName="istio-proxy"
Nov 02 12:59:10.013841 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:10.013768    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91783d5b-f538-43a0-a9d8-29f7270cadcd" containerName="istio-proxy"
Nov 02 12:59:10.014445 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:10.013911    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="988e5461-e95e-408c-b145-7253a1a21056" containerName="istio-proxy"
Nov 02 12:59:10.014445 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:10.013927    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="91783d5b-f538-43a0-a9d8-29f7270cadcd" containerName="istio-proxy"
Nov 02 12:59:10.014445 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:10.013942    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7f18bc2-015d-41da-a666-912b2e250e2a" containerName="istio-proxy"
Nov 02 12:59:10.014445 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:10.013954    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="be334717-f78a-4113-bf03-15d5c4365107" containerName="istio-proxy"
Nov 02 12:59:10.014445 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:10.013963    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="160f0d66-4e21-49eb-9c25-b467bdb56f41" containerName="istio-proxy"
Nov 02 12:59:10.014445 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:10.013972    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb1e97a1-f680-4922-920c-ee0a3468cb1f" containerName="istio-proxy"
Nov 02 12:59:10.014445 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:10.013985    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="53afdb1f-5ab4-4106-bccd-9ce04d9ced37" containerName="istio-proxy"
Nov 02 12:59:10.018774 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:10.018750    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-e430c7/explicit-model-e430c7-kserve-66569595-ws8hn"
Nov 02 12:59:10.038298 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:10.038267    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-e430c7/explicit-model-e430c7-kserve-66569595-ws8hn"]
Nov 02 12:59:10.045477 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:10.045446    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sd7mm\" (UniqueName: \"kubernetes.io/projected/7e9360ac-a67e-4617-b345-110bcfa57381-kube-api-access-sd7mm\") pod \"explicit-model-e430c7-kserve-66569595-ws8hn\" (UID: \"7e9360ac-a67e-4617-b345-110bcfa57381\") " pod="ai-tenant-e2e-explicit-e430c7/explicit-model-e430c7-kserve-66569595-ws8hn"
Nov 02 12:59:10.045813 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:10.045788    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e9360ac-a67e-4617-b345-110bcfa57381-tls-certs\") pod \"explicit-model-e430c7-kserve-66569595-ws8hn\" (UID: \"7e9360ac-a67e-4617-b345-110bcfa57381\") " pod="ai-tenant-e2e-explicit-e430c7/explicit-model-e430c7-kserve-66569595-ws8hn"
Nov 02 12:59:10.045997 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:10.045963    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e9360ac-a67e-4617-b345-110bcfa57381-tmp-dir\") pod \"explicit-model-e430c7-kserve-66569595-ws8hn\" (UID: \"7e9360ac-a67e-4617-b345-110bcfa57381\") " pod="ai-tenant-e2e-explicit-e430c7/explicit-model-e430c7-kserve-66569595-ws8hn"
Nov 02 12:59:10.046102 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:10.046039    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e9360ac-a67e-4617-b345-110bcfa57381-model-cache\") pod \"explicit-model-e430c7-kserve-66569595-ws8hn\" (UID: \"7e9360ac-a67e-4617-b345-110bcfa57381\") " pod="ai-tenant-e2e-explicit-e430c7/explicit-model-e430c7-kserve-66569595-ws8hn"
Nov 02 12:59:10.046178 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:10.046114    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-e430c7\"/\"openshift-service-ca.crt\""
Nov 02 12:59:10.046178 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:10.046131    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e9360ac-a67e-4617-b345-110bcfa57381-home\") pod \"explicit-model-e430c7-kserve-66569595-ws8hn\" (UID: \"7e9360ac-a67e-4617-b345-110bcfa57381\") " pod="ai-tenant-e2e-explicit-e430c7/explicit-model-e430c7-kserve-66569595-ws8hn"
Nov 02 12:59:10.046272 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:10.046247    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e9360ac-a67e-4617-b345-110bcfa57381-dshm\") pod \"explicit-model-e430c7-kserve-66569595-ws8hn\" (UID: \"7e9360ac-a67e-4617-b345-110bcfa57381\") " pod="ai-tenant-e2e-explicit-e430c7/explicit-model-e430c7-kserve-66569595-ws8hn"
Nov 02 12:59:10.054321 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:10.054294    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-e430c7\"/\"kube-root-ca.crt\""
Nov 02 12:59:10.054451 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:10.054369    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-e430c7\"/\"explicit-model-e430c7-kserve-self-signed-certs\""
Nov 02 12:59:10.054521 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:10.054492    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-e430c7\"/\"default-dockercfg-tnlkl\""
Nov 02 12:59:10.148250 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:10.148210    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e9360ac-a67e-4617-b345-110bcfa57381-model-cache\") pod \"explicit-model-e430c7-kserve-66569595-ws8hn\" (UID: \"7e9360ac-a67e-4617-b345-110bcfa57381\") " pod="ai-tenant-e2e-explicit-e430c7/explicit-model-e430c7-kserve-66569595-ws8hn"
Nov 02 12:59:10.148457 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:10.148272    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e9360ac-a67e-4617-b345-110bcfa57381-home\") pod \"explicit-model-e430c7-kserve-66569595-ws8hn\" (UID: \"7e9360ac-a67e-4617-b345-110bcfa57381\") " pod="ai-tenant-e2e-explicit-e430c7/explicit-model-e430c7-kserve-66569595-ws8hn"
Nov 02 12:59:10.148457 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:10.148350    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e9360ac-a67e-4617-b345-110bcfa57381-dshm\") pod \"explicit-model-e430c7-kserve-66569595-ws8hn\" (UID: \"7e9360ac-a67e-4617-b345-110bcfa57381\") " pod="ai-tenant-e2e-explicit-e430c7/explicit-model-e430c7-kserve-66569595-ws8hn"
Nov 02 12:59:10.148584 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:10.148479    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sd7mm\" (UniqueName: \"kubernetes.io/projected/7e9360ac-a67e-4617-b345-110bcfa57381-kube-api-access-sd7mm\") pod \"explicit-model-e430c7-kserve-66569595-ws8hn\" (UID: \"7e9360ac-a67e-4617-b345-110bcfa57381\") " pod="ai-tenant-e2e-explicit-e430c7/explicit-model-e430c7-kserve-66569595-ws8hn"
Nov 02 12:59:10.148683 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:10.148600    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e9360ac-a67e-4617-b345-110bcfa57381-tls-certs\") pod \"explicit-model-e430c7-kserve-66569595-ws8hn\" (UID: \"7e9360ac-a67e-4617-b345-110bcfa57381\") " pod="ai-tenant-e2e-explicit-e430c7/explicit-model-e430c7-kserve-66569595-ws8hn"
Nov 02 12:59:10.148683 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:10.148655    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e9360ac-a67e-4617-b345-110bcfa57381-tmp-dir\") pod \"explicit-model-e430c7-kserve-66569595-ws8hn\" (UID: \"7e9360ac-a67e-4617-b345-110bcfa57381\") " pod="ai-tenant-e2e-explicit-e430c7/explicit-model-e430c7-kserve-66569595-ws8hn"
Nov 02 12:59:10.148806 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:10.148681    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e9360ac-a67e-4617-b345-110bcfa57381-model-cache\") pod \"explicit-model-e430c7-kserve-66569595-ws8hn\" (UID: \"7e9360ac-a67e-4617-b345-110bcfa57381\") " pod="ai-tenant-e2e-explicit-e430c7/explicit-model-e430c7-kserve-66569595-ws8hn"
Nov 02 12:59:10.148806 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:10.148708    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e9360ac-a67e-4617-b345-110bcfa57381-home\") pod \"explicit-model-e430c7-kserve-66569595-ws8hn\" (UID: \"7e9360ac-a67e-4617-b345-110bcfa57381\") " pod="ai-tenant-e2e-explicit-e430c7/explicit-model-e430c7-kserve-66569595-ws8hn"
Nov 02 12:59:10.148995 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:10.148976    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e9360ac-a67e-4617-b345-110bcfa57381-tmp-dir\") pod \"explicit-model-e430c7-kserve-66569595-ws8hn\" (UID: \"7e9360ac-a67e-4617-b345-110bcfa57381\") " pod="ai-tenant-e2e-explicit-e430c7/explicit-model-e430c7-kserve-66569595-ws8hn"
Nov 02 12:59:10.150729 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:10.150707    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e9360ac-a67e-4617-b345-110bcfa57381-dshm\") pod \"explicit-model-e430c7-kserve-66569595-ws8hn\" (UID: \"7e9360ac-a67e-4617-b345-110bcfa57381\") " pod="ai-tenant-e2e-explicit-e430c7/explicit-model-e430c7-kserve-66569595-ws8hn"
Nov 02 12:59:10.150977 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:10.150959    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e9360ac-a67e-4617-b345-110bcfa57381-tls-certs\") pod \"explicit-model-e430c7-kserve-66569595-ws8hn\" (UID: \"7e9360ac-a67e-4617-b345-110bcfa57381\") " pod="ai-tenant-e2e-explicit-e430c7/explicit-model-e430c7-kserve-66569595-ws8hn"
Nov 02 12:59:10.173368 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:10.173341    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sd7mm\" (UniqueName: \"kubernetes.io/projected/7e9360ac-a67e-4617-b345-110bcfa57381-kube-api-access-sd7mm\") pod \"explicit-model-e430c7-kserve-66569595-ws8hn\" (UID: \"7e9360ac-a67e-4617-b345-110bcfa57381\") " pod="ai-tenant-e2e-explicit-e430c7/explicit-model-e430c7-kserve-66569595-ws8hn"
Nov 02 12:59:10.338718 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:10.337940    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-e430c7/explicit-model-e430c7-kserve-66569595-ws8hn"
Nov 02 12:59:10.738820 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:10.738484    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-e430c7/explicit-model-e430c7-kserve-66569595-ws8hn"]
Nov 02 12:59:11.718601 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:11.718562    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-e430c7/explicit-model-e430c7-kserve-66569595-ws8hn" event={"ID":"7e9360ac-a67e-4617-b345-110bcfa57381","Type":"ContainerStarted","Data":"0d7d044b4144783b963a37d8cebd953b11082c846636abb87fc0de978f285a04"}
Nov 02 12:59:11.719151 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:11.718612    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-e430c7/explicit-model-e430c7-kserve-66569595-ws8hn" event={"ID":"7e9360ac-a67e-4617-b345-110bcfa57381","Type":"ContainerStarted","Data":"7443849b0e3aea244a1b603e5bb352b01d8bad3f75656cf667a0a8fe0b894bfb"}
Nov 02 12:59:11.762356 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:11.762306    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-e430c7/explicit-model-e430c7-kserve-66569595-ws8hn" podStartSLOduration=2.762292312 podStartE2EDuration="2.762292312s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:59:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:59:11.760948468 +0000 UTC m=+1948.016347567" watchObservedRunningTime="2026-11-02 12:59:11.762292312 +0000 UTC m=+1948.017691408"
Nov 02 12:59:12.339087 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:12.339049    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-e430c7/explicit-model-e430c7-kserve-66569595-ws8hn"
Nov 02 12:59:15.087932 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:59:15.087887    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqcmj" containerName="extractor"
Nov 02 12:59:17.202809 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:17.202759    4559 patch_prober.go:29] interesting pod/e2e-explicit-e430c7-openshift-default-8649b6f6fc-vl6ql container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.198:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 12:59:17.203360 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:17.202843    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-explicit-e430c7-openshift-default-8649b6f6fc-vl6ql" podUID="6f8849e4-f808-45f0-b24a-e1e1c8b99974" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.198:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:59:18.774030 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:18.773998    4559 generic.go:356] "Generic (PLEG): container finished" podID="6f8849e4-f808-45f0-b24a-e1e1c8b99974" containerID="01e7c9f818f941d68525708b9d7b5448400bc31b9777538360f08d5a61d40e6d" exitCode=0
Nov 02 12:59:18.774598 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:18.774074    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-e430c7-openshift-default-8649b6f6fc-vl6ql" event={"ID":"6f8849e4-f808-45f0-b24a-e1e1c8b99974","Type":"ContainerDied","Data":"01e7c9f818f941d68525708b9d7b5448400bc31b9777538360f08d5a61d40e6d"}
Nov 02 12:59:18.774717 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:18.774618    4559 scope.go:122] "RemoveContainer" containerID="01e7c9f818f941d68525708b9d7b5448400bc31b9777538360f08d5a61d40e6d"
Nov 02 12:59:18.776439 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:18.776410    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:59:18.776518 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:18.776486    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:59:18.776574 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:18.776520    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 12:59:19.811046 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:19.811005    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-e430c7-openshift-default-8649b6f6fc-vl6ql" event={"ID":"6f8849e4-f808-45f0-b24a-e1e1c8b99974","Type":"ContainerStarted","Data":"36e7f4b54929fd9b7406a9c8575aec0290af27998119ca3607b55489f4d601ec"}
Nov 02 12:59:20.339199 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:20.339155    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-explicit-e430c7/explicit-model-e430c7-kserve-66569595-ws8hn"
Nov 02 12:59:20.352169 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:20.352144    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-explicit-e430c7/explicit-model-e430c7-kserve-66569595-ws8hn"
Nov 02 12:59:20.791677 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:20.790996    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-e430c7-openshift-default-8649b6f6fc-vl6ql"
Nov 02 12:59:20.791677 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:20.791116    4559 patch_prober.go:29] interesting pod/e2e-explicit-e430c7-openshift-default-8649b6f6fc-vl6ql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused" start-of-body=
Nov 02 12:59:20.791677 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:20.791166    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-e430c7-openshift-default-8649b6f6fc-vl6ql" podUID="6f8849e4-f808-45f0-b24a-e1e1c8b99974" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused"
Nov 02 12:59:20.829744 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:20.829710    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-explicit-e430c7/explicit-model-e430c7-kserve-66569595-ws8hn"
Nov 02 12:59:21.801664 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:21.788062    4559 patch_prober.go:29] interesting pod/e2e-explicit-e430c7-openshift-default-8649b6f6fc-vl6ql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused" start-of-body=
Nov 02 12:59:21.801664 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:21.788139    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-e430c7-openshift-default-8649b6f6fc-vl6ql" podUID="6f8849e4-f808-45f0-b24a-e1e1c8b99974" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused"
Nov 02 12:59:22.796673 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:22.796015    4559 patch_prober.go:29] interesting pod/e2e-explicit-e430c7-openshift-default-8649b6f6fc-vl6ql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused" start-of-body=
Nov 02 12:59:22.796673 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:22.796083    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-e430c7-openshift-default-8649b6f6fc-vl6ql" podUID="6f8849e4-f808-45f0-b24a-e1e1c8b99974" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused"
Nov 02 12:59:23.793663 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:23.791090    4559 patch_prober.go:29] interesting pod/e2e-explicit-e430c7-openshift-default-8649b6f6fc-vl6ql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused" start-of-body=
Nov 02 12:59:23.793663 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:23.791157    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-e430c7-openshift-default-8649b6f6fc-vl6ql" podUID="6f8849e4-f808-45f0-b24a-e1e1c8b99974" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused"
Nov 02 12:59:24.796119 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:24.796076    4559 patch_prober.go:29] interesting pod/e2e-explicit-e430c7-openshift-default-8649b6f6fc-vl6ql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused" start-of-body=
Nov 02 12:59:24.796745 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:24.796712    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-e430c7-openshift-default-8649b6f6fc-vl6ql" podUID="6f8849e4-f808-45f0-b24a-e1e1c8b99974" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused"
Nov 02 12:59:25.802997 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:25.802961    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-e430c7-openshift-default-8649b6f6fc-vl6ql"
Nov 02 12:59:25.805206 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:25.805111    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-e430c7-openshift-default-8649b6f6fc-vl6ql"
Nov 02 12:59:25.806972 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:25.806949    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-e430c7-openshift-default-8649b6f6fc-vl6ql"
Nov 02 12:59:28.262987 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:28.262950    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-6992-7f0817"]
Nov 02 12:59:28.272804 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:28.271035    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6992-7f0817"
Nov 02 12:59:28.283877 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:28.283839    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6992-7f0817"]
Nov 02 12:59:28.289709 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:28.289683    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-hfcns\""
Nov 02 12:59:28.354810 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:28.354773    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjdfg\" (UniqueName: \"kubernetes.io/projected/a3622e27-7879-4e6d-ae5f-7ba184fdf1d7-kube-api-access-hjdfg\") pod \"test-curl-6992-7f0817\" (UID: \"a3622e27-7879-4e6d-ae5f-7ba184fdf1d7\") " pod="opendatahub/test-curl-6992-7f0817"
Nov 02 12:59:28.456523 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:28.456468    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjdfg\" (UniqueName: \"kubernetes.io/projected/a3622e27-7879-4e6d-ae5f-7ba184fdf1d7-kube-api-access-hjdfg\") pod \"test-curl-6992-7f0817\" (UID: \"a3622e27-7879-4e6d-ae5f-7ba184fdf1d7\") " pod="opendatahub/test-curl-6992-7f0817"
Nov 02 12:59:28.480756 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:28.480723    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjdfg\" (UniqueName: \"kubernetes.io/projected/a3622e27-7879-4e6d-ae5f-7ba184fdf1d7-kube-api-access-hjdfg\") pod \"test-curl-6992-7f0817\" (UID: \"a3622e27-7879-4e6d-ae5f-7ba184fdf1d7\") " pod="opendatahub/test-curl-6992-7f0817"
Nov 02 12:59:28.593435 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:28.593360    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6992-7f0817"
Nov 02 12:59:28.897484 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:28.897433    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6992-7f0817"]
Nov 02 12:59:29.881522 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:29.881464    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6992-7f0817" event={"ID":"a3622e27-7879-4e6d-ae5f-7ba184fdf1d7","Type":"ContainerStarted","Data":"deee633bca263e23949d8a2c1b15a17956e4af46e89d9bbd5b4729c163536c74"}
Nov 02 12:59:29.881522 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:29.881518    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6992-7f0817" event={"ID":"a3622e27-7879-4e6d-ae5f-7ba184fdf1d7","Type":"ContainerStarted","Data":"03823ac6444eccd774fd335d3027a74e6009ca49777ed311d76950a269d88942"}
Nov 02 12:59:29.921825 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:29.921773    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-6992-7f0817" podStartSLOduration=1.921759234 podStartE2EDuration="1.921759234s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:59:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:59:29.920171497 +0000 UTC m=+1966.175570593" watchObservedRunningTime="2026-11-02 12:59:29.921759234 +0000 UTC m=+1966.177158329"
Nov 02 12:59:30.735432 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:30.735397    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-qdrs6"]
Nov 02 12:59:30.735682 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:30.735619    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-qdrs6" podUID="3aabb9b6-9382-42d9-8c52-4d89e361c25b" containerName="istio-proxy" containerID="cri-o://45af072c4ede5a3117cf04d4c7314ff8c012723495d29dd227e1391ed307b0a9" gracePeriod=30
Nov 02 12:59:30.818152 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:30.818122    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-6992-7f0817"]
Nov 02 12:59:30.825201 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:30.825164    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-6992-7f0817"]
Nov 02 12:59:31.898241 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:31.898201    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-6992-7f0817" podUID="a3622e27-7879-4e6d-ae5f-7ba184fdf1d7" containerName="test-curl-6992-7f0817" containerID="cri-o://deee633bca263e23949d8a2c1b15a17956e4af46e89d9bbd5b4729c163536c74" gracePeriod=2
Nov 02 12:59:34.110860 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:34.110833    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6992-7f0817"
Nov 02 12:59:34.240263 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:34.240232    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3622e27-7879-4e6d-ae5f-7ba184fdf1d7-kube-api-access-hjdfg\" (UniqueName: \"kubernetes.io/projected/a3622e27-7879-4e6d-ae5f-7ba184fdf1d7-kube-api-access-hjdfg\") pod \"a3622e27-7879-4e6d-ae5f-7ba184fdf1d7\" (UID: \"a3622e27-7879-4e6d-ae5f-7ba184fdf1d7\") "
Nov 02 12:59:34.242361 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:34.242329    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3622e27-7879-4e6d-ae5f-7ba184fdf1d7-kube-api-access-hjdfg" pod "a3622e27-7879-4e6d-ae5f-7ba184fdf1d7" (UID: "a3622e27-7879-4e6d-ae5f-7ba184fdf1d7"). InnerVolumeSpecName "kube-api-access-hjdfg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:59:34.342947 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:34.342903    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hjdfg\" (UniqueName: \"kubernetes.io/projected/a3622e27-7879-4e6d-ae5f-7ba184fdf1d7-kube-api-access-hjdfg\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:59:34.920570 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:34.920532    4559 generic.go:356] "Generic (PLEG): container finished" podID="a3622e27-7879-4e6d-ae5f-7ba184fdf1d7" containerID="deee633bca263e23949d8a2c1b15a17956e4af46e89d9bbd5b4729c163536c74" exitCode=137
Nov 02 12:59:34.920772 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:34.920578    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6992-7f0817"
Nov 02 12:59:34.920772 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:34.920619    4559 scope.go:122] "RemoveContainer" containerID="deee633bca263e23949d8a2c1b15a17956e4af46e89d9bbd5b4729c163536c74"
Nov 02 12:59:34.930420 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:34.930402    4559 scope.go:122] "RemoveContainer" containerID="deee633bca263e23949d8a2c1b15a17956e4af46e89d9bbd5b4729c163536c74"
Nov 02 12:59:34.930696 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:59:34.930677    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"deee633bca263e23949d8a2c1b15a17956e4af46e89d9bbd5b4729c163536c74\": container with ID starting with deee633bca263e23949d8a2c1b15a17956e4af46e89d9bbd5b4729c163536c74 not found: ID does not exist" containerID="deee633bca263e23949d8a2c1b15a17956e4af46e89d9bbd5b4729c163536c74"
Nov 02 12:59:34.930758 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:34.930703    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"deee633bca263e23949d8a2c1b15a17956e4af46e89d9bbd5b4729c163536c74"} err="failed to get container status \"deee633bca263e23949d8a2c1b15a17956e4af46e89d9bbd5b4729c163536c74\": rpc error: code = NotFound desc = could not find container \"deee633bca263e23949d8a2c1b15a17956e4af46e89d9bbd5b4729c163536c74\": container with ID starting with deee633bca263e23949d8a2c1b15a17956e4af46e89d9bbd5b4729c163536c74 not found: ID does not exist"
Nov 02 12:59:34.943069 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:59:34.943042    4559 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6992-7f0817\" is forbidden: User \"system:node:ip-10-0-128-132.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-132.ec2.internal' and this object" podUID="a3622e27-7879-4e6d-ae5f-7ba184fdf1d7" pod="opendatahub/test-curl-6992-7f0817"
Nov 02 12:59:35.136029 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:35.135978    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-e430c7/explicit-model-e430c7-kserve-66569595-ws8hn"]
Nov 02 12:59:35.150620 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:59:35.150583    4559 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6992-7f0817\" is forbidden: User \"system:node:ip-10-0-128-132.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-132.ec2.internal' and this object" podUID="a3622e27-7879-4e6d-ae5f-7ba184fdf1d7" pod="opendatahub/test-curl-6992-7f0817"
Nov 02 12:59:35.902796 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:35.902761    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3622e27-7879-4e6d-ae5f-7ba184fdf1d7" path="/var/lib/kubelet/pods/a3622e27-7879-4e6d-ae5f-7ba184fdf1d7/volumes"
Nov 02 12:59:35.929604 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:35.929571    4559 generic.go:356] "Generic (PLEG): container finished" podID="3aabb9b6-9382-42d9-8c52-4d89e361c25b" containerID="45af072c4ede5a3117cf04d4c7314ff8c012723495d29dd227e1391ed307b0a9" exitCode=0
Nov 02 12:59:35.929782 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:35.929629    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-qdrs6" event={"ID":"3aabb9b6-9382-42d9-8c52-4d89e361c25b","Type":"ContainerDied","Data":"45af072c4ede5a3117cf04d4c7314ff8c012723495d29dd227e1391ed307b0a9"}
Nov 02 12:59:36.418390 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:36.418365    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-qdrs6"
Nov 02 12:59:36.499392 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:36.499359    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3aabb9b6-9382-42d9-8c52-4d89e361c25b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3aabb9b6-9382-42d9-8c52-4d89e361c25b-workload-certs\") pod \"3aabb9b6-9382-42d9-8c52-4d89e361c25b\" (UID: \"3aabb9b6-9382-42d9-8c52-4d89e361c25b\") "
Nov 02 12:59:36.499580 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:36.499417    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3aabb9b6-9382-42d9-8c52-4d89e361c25b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3aabb9b6-9382-42d9-8c52-4d89e361c25b-credential-socket\") pod \"3aabb9b6-9382-42d9-8c52-4d89e361c25b\" (UID: \"3aabb9b6-9382-42d9-8c52-4d89e361c25b\") "
Nov 02 12:59:36.499580 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:36.499495    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3aabb9b6-9382-42d9-8c52-4d89e361c25b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3aabb9b6-9382-42d9-8c52-4d89e361c25b-istio-data\") pod \"3aabb9b6-9382-42d9-8c52-4d89e361c25b\" (UID: \"3aabb9b6-9382-42d9-8c52-4d89e361c25b\") "
Nov 02 12:59:36.499721 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:36.499611    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3aabb9b6-9382-42d9-8c52-4d89e361c25b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3aabb9b6-9382-42d9-8c52-4d89e361c25b-istio-podinfo\") pod \"3aabb9b6-9382-42d9-8c52-4d89e361c25b\" (UID: \"3aabb9b6-9382-42d9-8c52-4d89e361c25b\") "
Nov 02 12:59:36.499721 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:36.499692    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3aabb9b6-9382-42d9-8c52-4d89e361c25b-kube-api-access-rtsvd\" (UniqueName: \"kubernetes.io/projected/3aabb9b6-9382-42d9-8c52-4d89e361c25b-kube-api-access-rtsvd\") pod \"3aabb9b6-9382-42d9-8c52-4d89e361c25b\" (UID: \"3aabb9b6-9382-42d9-8c52-4d89e361c25b\") "
Nov 02 12:59:36.499841 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:36.499725    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3aabb9b6-9382-42d9-8c52-4d89e361c25b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3aabb9b6-9382-42d9-8c52-4d89e361c25b-istiod-ca-cert\") pod \"3aabb9b6-9382-42d9-8c52-4d89e361c25b\" (UID: \"3aabb9b6-9382-42d9-8c52-4d89e361c25b\") "
Nov 02 12:59:36.499841 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:36.499717    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3aabb9b6-9382-42d9-8c52-4d89e361c25b-workload-certs" pod "3aabb9b6-9382-42d9-8c52-4d89e361c25b" (UID: "3aabb9b6-9382-42d9-8c52-4d89e361c25b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:59:36.499841 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:36.499779    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3aabb9b6-9382-42d9-8c52-4d89e361c25b-credential-socket" pod "3aabb9b6-9382-42d9-8c52-4d89e361c25b" (UID: "3aabb9b6-9382-42d9-8c52-4d89e361c25b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:59:36.499989 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:36.499852    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3aabb9b6-9382-42d9-8c52-4d89e361c25b-istio-token\" (UniqueName: \"kubernetes.io/projected/3aabb9b6-9382-42d9-8c52-4d89e361c25b-istio-token\") pod \"3aabb9b6-9382-42d9-8c52-4d89e361c25b\" (UID: \"3aabb9b6-9382-42d9-8c52-4d89e361c25b\") "
Nov 02 12:59:36.499989 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:36.499888    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3aabb9b6-9382-42d9-8c52-4d89e361c25b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3aabb9b6-9382-42d9-8c52-4d89e361c25b-istio-envoy\") pod \"3aabb9b6-9382-42d9-8c52-4d89e361c25b\" (UID: \"3aabb9b6-9382-42d9-8c52-4d89e361c25b\") "
Nov 02 12:59:36.499989 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:36.499928    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3aabb9b6-9382-42d9-8c52-4d89e361c25b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3aabb9b6-9382-42d9-8c52-4d89e361c25b-workload-socket\") pod \"3aabb9b6-9382-42d9-8c52-4d89e361c25b\" (UID: \"3aabb9b6-9382-42d9-8c52-4d89e361c25b\") "
Nov 02 12:59:36.499989 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:36.499945    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3aabb9b6-9382-42d9-8c52-4d89e361c25b-istio-data" pod "3aabb9b6-9382-42d9-8c52-4d89e361c25b" (UID: "3aabb9b6-9382-42d9-8c52-4d89e361c25b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:59:36.500282 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:36.500133    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3aabb9b6-9382-42d9-8c52-4d89e361c25b-workload-socket" pod "3aabb9b6-9382-42d9-8c52-4d89e361c25b" (UID: "3aabb9b6-9382-42d9-8c52-4d89e361c25b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:59:36.500282 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:36.500180    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3aabb9b6-9382-42d9-8c52-4d89e361c25b-istiod-ca-cert" pod "3aabb9b6-9382-42d9-8c52-4d89e361c25b" (UID: "3aabb9b6-9382-42d9-8c52-4d89e361c25b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:59:36.500786 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:36.500758    4559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3aabb9b6-9382-42d9-8c52-4d89e361c25b-workload-socket\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:59:36.500918 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:36.500831    4559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3aabb9b6-9382-42d9-8c52-4d89e361c25b-workload-certs\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:59:36.500918 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:36.500849    4559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3aabb9b6-9382-42d9-8c52-4d89e361c25b-credential-socket\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:59:36.500918 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:36.500863    4559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3aabb9b6-9382-42d9-8c52-4d89e361c25b-istio-data\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:59:36.500918 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:36.500904    4559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3aabb9b6-9382-42d9-8c52-4d89e361c25b-istiod-ca-cert\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:59:36.502345 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:36.502314    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3aabb9b6-9382-42d9-8c52-4d89e361c25b-istio-token" pod "3aabb9b6-9382-42d9-8c52-4d89e361c25b" (UID: "3aabb9b6-9382-42d9-8c52-4d89e361c25b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:59:36.502544 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:36.502516    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3aabb9b6-9382-42d9-8c52-4d89e361c25b-istio-podinfo" pod "3aabb9b6-9382-42d9-8c52-4d89e361c25b" (UID: "3aabb9b6-9382-42d9-8c52-4d89e361c25b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 12:59:36.502621 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:36.502590    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3aabb9b6-9382-42d9-8c52-4d89e361c25b-kube-api-access-rtsvd" pod "3aabb9b6-9382-42d9-8c52-4d89e361c25b" (UID: "3aabb9b6-9382-42d9-8c52-4d89e361c25b"). InnerVolumeSpecName "kube-api-access-rtsvd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:59:36.502718 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:36.502699    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3aabb9b6-9382-42d9-8c52-4d89e361c25b-istio-envoy" pod "3aabb9b6-9382-42d9-8c52-4d89e361c25b" (UID: "3aabb9b6-9382-42d9-8c52-4d89e361c25b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:59:36.602277 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:36.602240    4559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3aabb9b6-9382-42d9-8c52-4d89e361c25b-istio-podinfo\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:59:36.602277 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:36.602274    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rtsvd\" (UniqueName: \"kubernetes.io/projected/3aabb9b6-9382-42d9-8c52-4d89e361c25b-kube-api-access-rtsvd\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:59:36.602277 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:36.602285    4559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3aabb9b6-9382-42d9-8c52-4d89e361c25b-istio-token\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:59:36.602520 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:36.602296    4559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3aabb9b6-9382-42d9-8c52-4d89e361c25b-istio-envoy\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:59:36.938017 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:36.937987    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-qdrs6"
Nov 02 12:59:36.938215 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:36.937982    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-qdrs6" event={"ID":"3aabb9b6-9382-42d9-8c52-4d89e361c25b","Type":"ContainerDied","Data":"5eb4183d3e04fda9ccba08abfff666b95b3b17bbae5094e2f706bcd2f17b46c4"}
Nov 02 12:59:36.938215 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:36.938130    4559 scope.go:122] "RemoveContainer" containerID="45af072c4ede5a3117cf04d4c7314ff8c012723495d29dd227e1391ed307b0a9"
Nov 02 12:59:36.996488 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:36.996460    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-qdrs6"]
Nov 02 12:59:37.008559 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:37.008533    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-qdrs6"]
Nov 02 12:59:37.894311 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:37.894277    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3aabb9b6-9382-42d9-8c52-4d89e361c25b" path="/var/lib/kubelet/pods/3aabb9b6-9382-42d9-8c52-4d89e361c25b/volumes"
Nov 02 12:59:45.087818 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:59:45.086980    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqcmj" containerName="extractor"
Nov 02 12:59:45.977233 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:45.977197    4559 scope.go:122] "RemoveContainer" containerID="f755ee04d01c7f3d58b850f2df7e42492d23991c912151676e95a1c41a82da52"
Nov 02 12:59:50.157925 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:50.157884    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-e430c7/explicit-model-e430c7-kserve-66569595-ws8hn" podUID="7e9360ac-a67e-4617-b345-110bcfa57381" containerName="main" containerID="cri-o://0d7d044b4144783b963a37d8cebd953b11082c846636abb87fc0de978f285a04" gracePeriod=45
Nov 02 12:59:50.423587 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:50.423562    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-e430c7/explicit-model-e430c7-kserve-66569595-ws8hn"
Nov 02 12:59:50.526348 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:50.526308    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-d7jbc"]
Nov 02 12:59:50.526579 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:50.526553    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-d7jbc" podUID="39c7e6bf-f8ae-4913-aa8a-5e8325a379d4" containerName="istio-proxy" containerID="cri-o://81e74f357cf88f7e21ad34989b7f895f1b557fb3fa711c1e63cdf5ac9859b2e3" gracePeriod=30
Nov 02 12:59:50.529537 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:50.529506    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7e9360ac-a67e-4617-b345-110bcfa57381-tls-certs\" (UniqueName: \"kubernetes.io/secret/7e9360ac-a67e-4617-b345-110bcfa57381-tls-certs\") pod \"7e9360ac-a67e-4617-b345-110bcfa57381\" (UID: \"7e9360ac-a67e-4617-b345-110bcfa57381\") "
Nov 02 12:59:50.529732 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:50.529701    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e9360ac-a67e-4617-b345-110bcfa57381-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e9360ac-a67e-4617-b345-110bcfa57381-model-cache\") pod \"7e9360ac-a67e-4617-b345-110bcfa57381\" (UID: \"7e9360ac-a67e-4617-b345-110bcfa57381\") "
Nov 02 12:59:50.529812 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:50.529746    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e9360ac-a67e-4617-b345-110bcfa57381-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e9360ac-a67e-4617-b345-110bcfa57381-tmp-dir\") pod \"7e9360ac-a67e-4617-b345-110bcfa57381\" (UID: \"7e9360ac-a67e-4617-b345-110bcfa57381\") "
Nov 02 12:59:50.529812 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:50.529795    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e9360ac-a67e-4617-b345-110bcfa57381-kube-api-access-sd7mm\" (UniqueName: \"kubernetes.io/projected/7e9360ac-a67e-4617-b345-110bcfa57381-kube-api-access-sd7mm\") pod \"7e9360ac-a67e-4617-b345-110bcfa57381\" (UID: \"7e9360ac-a67e-4617-b345-110bcfa57381\") "
Nov 02 12:59:50.529942 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:50.529915    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e9360ac-a67e-4617-b345-110bcfa57381-home\" (UniqueName: \"kubernetes.io/empty-dir/7e9360ac-a67e-4617-b345-110bcfa57381-home\") pod \"7e9360ac-a67e-4617-b345-110bcfa57381\" (UID: \"7e9360ac-a67e-4617-b345-110bcfa57381\") "
Nov 02 12:59:50.530010 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:50.529957    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e9360ac-a67e-4617-b345-110bcfa57381-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e9360ac-a67e-4617-b345-110bcfa57381-dshm\") pod \"7e9360ac-a67e-4617-b345-110bcfa57381\" (UID: \"7e9360ac-a67e-4617-b345-110bcfa57381\") "
Nov 02 12:59:50.530069 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:50.529995    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e9360ac-a67e-4617-b345-110bcfa57381-model-cache" pod "7e9360ac-a67e-4617-b345-110bcfa57381" (UID: "7e9360ac-a67e-4617-b345-110bcfa57381"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:59:50.530128 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:50.530082    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e9360ac-a67e-4617-b345-110bcfa57381-tmp-dir" pod "7e9360ac-a67e-4617-b345-110bcfa57381" (UID: "7e9360ac-a67e-4617-b345-110bcfa57381"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:59:50.530443 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:50.530412    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e9360ac-a67e-4617-b345-110bcfa57381-home" pod "7e9360ac-a67e-4617-b345-110bcfa57381" (UID: "7e9360ac-a67e-4617-b345-110bcfa57381"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:59:50.530633 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:50.530608    4559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e9360ac-a67e-4617-b345-110bcfa57381-model-cache\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:59:50.530774 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:50.530690    4559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e9360ac-a67e-4617-b345-110bcfa57381-tmp-dir\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:59:50.530774 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:50.530707    4559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e9360ac-a67e-4617-b345-110bcfa57381-home\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:59:50.532860 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:50.532825    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e9360ac-a67e-4617-b345-110bcfa57381-tls-certs" pod "7e9360ac-a67e-4617-b345-110bcfa57381" (UID: "7e9360ac-a67e-4617-b345-110bcfa57381"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:59:50.537574 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:50.537545    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e9360ac-a67e-4617-b345-110bcfa57381-dshm" pod "7e9360ac-a67e-4617-b345-110bcfa57381" (UID: "7e9360ac-a67e-4617-b345-110bcfa57381"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:59:50.537767 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:50.537748    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e9360ac-a67e-4617-b345-110bcfa57381-kube-api-access-sd7mm" pod "7e9360ac-a67e-4617-b345-110bcfa57381" (UID: "7e9360ac-a67e-4617-b345-110bcfa57381"). InnerVolumeSpecName "kube-api-access-sd7mm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:59:50.631980 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:50.631947    4559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e9360ac-a67e-4617-b345-110bcfa57381-tls-certs\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:59:50.631980 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:50.631974    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sd7mm\" (UniqueName: \"kubernetes.io/projected/7e9360ac-a67e-4617-b345-110bcfa57381-kube-api-access-sd7mm\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:59:50.631980 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:50.631984    4559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e9360ac-a67e-4617-b345-110bcfa57381-dshm\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:59:51.057071 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:51.057039    4559 generic.go:356] "Generic (PLEG): container finished" podID="7e9360ac-a67e-4617-b345-110bcfa57381" containerID="0d7d044b4144783b963a37d8cebd953b11082c846636abb87fc0de978f285a04" exitCode=0
Nov 02 12:59:51.057071 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:51.057073    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-e430c7/explicit-model-e430c7-kserve-66569595-ws8hn" event={"ID":"7e9360ac-a67e-4617-b345-110bcfa57381","Type":"ContainerDied","Data":"0d7d044b4144783b963a37d8cebd953b11082c846636abb87fc0de978f285a04"}
Nov 02 12:59:51.057331 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:51.057091    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-e430c7/explicit-model-e430c7-kserve-66569595-ws8hn"
Nov 02 12:59:51.057331 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:51.057110    4559 scope.go:122] "RemoveContainer" containerID="0d7d044b4144783b963a37d8cebd953b11082c846636abb87fc0de978f285a04"
Nov 02 12:59:51.057331 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:51.057100    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-e430c7/explicit-model-e430c7-kserve-66569595-ws8hn" event={"ID":"7e9360ac-a67e-4617-b345-110bcfa57381","Type":"ContainerDied","Data":"7443849b0e3aea244a1b603e5bb352b01d8bad3f75656cf667a0a8fe0b894bfb"}
Nov 02 12:59:51.068408 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:51.068387    4559 scope.go:122] "RemoveContainer" containerID="0d7d044b4144783b963a37d8cebd953b11082c846636abb87fc0de978f285a04"
Nov 02 12:59:51.068717 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:59:51.068698    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d7d044b4144783b963a37d8cebd953b11082c846636abb87fc0de978f285a04\": container with ID starting with 0d7d044b4144783b963a37d8cebd953b11082c846636abb87fc0de978f285a04 not found: ID does not exist" containerID="0d7d044b4144783b963a37d8cebd953b11082c846636abb87fc0de978f285a04"
Nov 02 12:59:51.068786 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:51.068725    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d7d044b4144783b963a37d8cebd953b11082c846636abb87fc0de978f285a04"} err="failed to get container status \"0d7d044b4144783b963a37d8cebd953b11082c846636abb87fc0de978f285a04\": rpc error: code = NotFound desc = could not find container \"0d7d044b4144783b963a37d8cebd953b11082c846636abb87fc0de978f285a04\": container with ID starting with 0d7d044b4144783b963a37d8cebd953b11082c846636abb87fc0de978f285a04 not found: ID does not exist"
Nov 02 12:59:51.111973 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:51.111944    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-e430c7/explicit-model-e430c7-kserve-66569595-ws8hn"]
Nov 02 12:59:51.124536 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:51.124512    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-e430c7/explicit-model-e430c7-kserve-66569595-ws8hn"]
Nov 02 12:59:51.893900 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:51.893869    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e9360ac-a67e-4617-b345-110bcfa57381" path="/var/lib/kubelet/pods/7e9360ac-a67e-4617-b345-110bcfa57381/volumes"
Nov 02 12:59:54.786101 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:54.786069    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"]
Nov 02 12:59:55.459315 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:55.459275    4559 patch_prober.go:29] interesting pod/e2e-worker-w4-843978-openshift-default-7965f48c7f-d7jbc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 12:59:55.459492 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:55.459341    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-d7jbc" podUID="39c7e6bf-f8ae-4913-aa8a-5e8325a379d4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 12:59:56.015665 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:56.002830    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-d7jbc"
Nov 02 12:59:56.141482 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:56.139881    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39c7e6bf-f8ae-4913-aa8a-5e8325a379d4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39c7e6bf-f8ae-4913-aa8a-5e8325a379d4-workload-certs\") pod \"39c7e6bf-f8ae-4913-aa8a-5e8325a379d4\" (UID: \"39c7e6bf-f8ae-4913-aa8a-5e8325a379d4\") "
Nov 02 12:59:56.141482 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:56.139993    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39c7e6bf-f8ae-4913-aa8a-5e8325a379d4-istio-token\" (UniqueName: \"kubernetes.io/projected/39c7e6bf-f8ae-4913-aa8a-5e8325a379d4-istio-token\") pod \"39c7e6bf-f8ae-4913-aa8a-5e8325a379d4\" (UID: \"39c7e6bf-f8ae-4913-aa8a-5e8325a379d4\") "
Nov 02 12:59:56.141482 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:56.140057    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39c7e6bf-f8ae-4913-aa8a-5e8325a379d4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39c7e6bf-f8ae-4913-aa8a-5e8325a379d4-workload-socket\") pod \"39c7e6bf-f8ae-4913-aa8a-5e8325a379d4\" (UID: \"39c7e6bf-f8ae-4913-aa8a-5e8325a379d4\") "
Nov 02 12:59:56.141482 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:56.140092    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39c7e6bf-f8ae-4913-aa8a-5e8325a379d4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39c7e6bf-f8ae-4913-aa8a-5e8325a379d4-credential-socket\") pod \"39c7e6bf-f8ae-4913-aa8a-5e8325a379d4\" (UID: \"39c7e6bf-f8ae-4913-aa8a-5e8325a379d4\") "
Nov 02 12:59:56.141482 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:56.140157    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39c7e6bf-f8ae-4913-aa8a-5e8325a379d4-kube-api-access-d6bxz\" (UniqueName: \"kubernetes.io/projected/39c7e6bf-f8ae-4913-aa8a-5e8325a379d4-kube-api-access-d6bxz\") pod \"39c7e6bf-f8ae-4913-aa8a-5e8325a379d4\" (UID: \"39c7e6bf-f8ae-4913-aa8a-5e8325a379d4\") "
Nov 02 12:59:56.141482 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:56.140297    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/39c7e6bf-f8ae-4913-aa8a-5e8325a379d4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39c7e6bf-f8ae-4913-aa8a-5e8325a379d4-istio-podinfo\") pod \"39c7e6bf-f8ae-4913-aa8a-5e8325a379d4\" (UID: \"39c7e6bf-f8ae-4913-aa8a-5e8325a379d4\") "
Nov 02 12:59:56.141482 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:56.140367    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39c7e6bf-f8ae-4913-aa8a-5e8325a379d4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39c7e6bf-f8ae-4913-aa8a-5e8325a379d4-istio-data\") pod \"39c7e6bf-f8ae-4913-aa8a-5e8325a379d4\" (UID: \"39c7e6bf-f8ae-4913-aa8a-5e8325a379d4\") "
Nov 02 12:59:56.141482 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:56.140398    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39c7e6bf-f8ae-4913-aa8a-5e8325a379d4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39c7e6bf-f8ae-4913-aa8a-5e8325a379d4-istiod-ca-cert\") pod \"39c7e6bf-f8ae-4913-aa8a-5e8325a379d4\" (UID: \"39c7e6bf-f8ae-4913-aa8a-5e8325a379d4\") "
Nov 02 12:59:56.141482 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:56.140453    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39c7e6bf-f8ae-4913-aa8a-5e8325a379d4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39c7e6bf-f8ae-4913-aa8a-5e8325a379d4-istio-envoy\") pod \"39c7e6bf-f8ae-4913-aa8a-5e8325a379d4\" (UID: \"39c7e6bf-f8ae-4913-aa8a-5e8325a379d4\") "
Nov 02 12:59:56.145789 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:56.145755    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39c7e6bf-f8ae-4913-aa8a-5e8325a379d4-credential-socket" pod "39c7e6bf-f8ae-4913-aa8a-5e8325a379d4" (UID: "39c7e6bf-f8ae-4913-aa8a-5e8325a379d4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:59:56.146268 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:56.146243    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39c7e6bf-f8ae-4913-aa8a-5e8325a379d4-workload-socket" pod "39c7e6bf-f8ae-4913-aa8a-5e8325a379d4" (UID: "39c7e6bf-f8ae-4913-aa8a-5e8325a379d4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:59:56.198519 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:56.198467    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39c7e6bf-f8ae-4913-aa8a-5e8325a379d4-istio-data" pod "39c7e6bf-f8ae-4913-aa8a-5e8325a379d4" (UID: "39c7e6bf-f8ae-4913-aa8a-5e8325a379d4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:59:56.274700 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:56.211257    4559 generic.go:356] "Generic (PLEG): container finished" podID="39c7e6bf-f8ae-4913-aa8a-5e8325a379d4" containerID="81e74f357cf88f7e21ad34989b7f895f1b557fb3fa711c1e63cdf5ac9859b2e3" exitCode=0
Nov 02 12:59:56.274700 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:56.211474    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-d7jbc" event={"ID":"39c7e6bf-f8ae-4913-aa8a-5e8325a379d4","Type":"ContainerDied","Data":"81e74f357cf88f7e21ad34989b7f895f1b557fb3fa711c1e63cdf5ac9859b2e3"}
Nov 02 12:59:56.274700 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:56.211533    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-d7jbc" event={"ID":"39c7e6bf-f8ae-4913-aa8a-5e8325a379d4","Type":"ContainerDied","Data":"b8dbb35b0062baf75b1c50b32b9bfac7a534dee9cf6ed09268fcae974dff5a7f"}
Nov 02 12:59:56.274700 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:56.211565    4559 scope.go:122] "RemoveContainer" containerID="81e74f357cf88f7e21ad34989b7f895f1b557fb3fa711c1e63cdf5ac9859b2e3"
Nov 02 12:59:56.274700 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:56.241938    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-d7jbc"
Nov 02 12:59:56.281412 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:56.281377    4559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39c7e6bf-f8ae-4913-aa8a-5e8325a379d4-istio-data\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:59:56.289751 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:56.289718    4559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39c7e6bf-f8ae-4913-aa8a-5e8325a379d4-workload-socket\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:59:56.290719 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:56.290692    4559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39c7e6bf-f8ae-4913-aa8a-5e8325a379d4-credential-socket\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:59:56.332672 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:56.331923    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39c7e6bf-f8ae-4913-aa8a-5e8325a379d4-istiod-ca-cert" pod "39c7e6bf-f8ae-4913-aa8a-5e8325a379d4" (UID: "39c7e6bf-f8ae-4913-aa8a-5e8325a379d4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:59:56.343262 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:56.343224    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39c7e6bf-f8ae-4913-aa8a-5e8325a379d4-workload-certs" pod "39c7e6bf-f8ae-4913-aa8a-5e8325a379d4" (UID: "39c7e6bf-f8ae-4913-aa8a-5e8325a379d4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:59:56.424824 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:56.424350    4559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39c7e6bf-f8ae-4913-aa8a-5e8325a379d4-istiod-ca-cert\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:59:56.425397 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:56.425374    4559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39c7e6bf-f8ae-4913-aa8a-5e8325a379d4-workload-certs\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:59:56.484105 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:56.481688    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39c7e6bf-f8ae-4913-aa8a-5e8325a379d4-istio-token" pod "39c7e6bf-f8ae-4913-aa8a-5e8325a379d4" (UID: "39c7e6bf-f8ae-4913-aa8a-5e8325a379d4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:59:56.484105 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:56.481824    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39c7e6bf-f8ae-4913-aa8a-5e8325a379d4-kube-api-access-d6bxz" pod "39c7e6bf-f8ae-4913-aa8a-5e8325a379d4" (UID: "39c7e6bf-f8ae-4913-aa8a-5e8325a379d4"). InnerVolumeSpecName "kube-api-access-d6bxz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:59:56.486711 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:56.484735    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/39c7e6bf-f8ae-4913-aa8a-5e8325a379d4-istio-podinfo" pod "39c7e6bf-f8ae-4913-aa8a-5e8325a379d4" (UID: "39c7e6bf-f8ae-4913-aa8a-5e8325a379d4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 12:59:56.486711 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:56.485150    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39c7e6bf-f8ae-4913-aa8a-5e8325a379d4-istio-envoy" pod "39c7e6bf-f8ae-4913-aa8a-5e8325a379d4" (UID: "39c7e6bf-f8ae-4913-aa8a-5e8325a379d4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:59:56.516002 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:56.515970    4559 scope.go:122] "RemoveContainer" containerID="81e74f357cf88f7e21ad34989b7f895f1b557fb3fa711c1e63cdf5ac9859b2e3"
Nov 02 12:59:56.516317 ip-10-0-128-132 kubenswrapper[4559]: E1102 12:59:56.516287    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81e74f357cf88f7e21ad34989b7f895f1b557fb3fa711c1e63cdf5ac9859b2e3\": container with ID starting with 81e74f357cf88f7e21ad34989b7f895f1b557fb3fa711c1e63cdf5ac9859b2e3 not found: ID does not exist" containerID="81e74f357cf88f7e21ad34989b7f895f1b557fb3fa711c1e63cdf5ac9859b2e3"
Nov 02 12:59:56.516416 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:56.516331    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81e74f357cf88f7e21ad34989b7f895f1b557fb3fa711c1e63cdf5ac9859b2e3"} err="failed to get container status \"81e74f357cf88f7e21ad34989b7f895f1b557fb3fa711c1e63cdf5ac9859b2e3\": rpc error: code = NotFound desc = could not find container \"81e74f357cf88f7e21ad34989b7f895f1b557fb3fa711c1e63cdf5ac9859b2e3\": container with ID starting with 81e74f357cf88f7e21ad34989b7f895f1b557fb3fa711c1e63cdf5ac9859b2e3 not found: ID does not exist"
Nov 02 12:59:56.526823 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:56.526794    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d6bxz\" (UniqueName: \"kubernetes.io/projected/39c7e6bf-f8ae-4913-aa8a-5e8325a379d4-kube-api-access-d6bxz\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:59:56.526933 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:56.526828    4559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39c7e6bf-f8ae-4913-aa8a-5e8325a379d4-istio-podinfo\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:59:56.526933 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:56.526848    4559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39c7e6bf-f8ae-4913-aa8a-5e8325a379d4-istio-envoy\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:59:56.526933 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:56.526862    4559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39c7e6bf-f8ae-4913-aa8a-5e8325a379d4-istio-token\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 12:59:56.600675 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:56.600633    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-d7jbc"]
Nov 02 12:59:56.604109 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:56.604079    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-d7jbc"]
Nov 02 12:59:57.899585 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:57.899529    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39c7e6bf-f8ae-4913-aa8a-5e8325a379d4" path="/var/lib/kubelet/pods/39c7e6bf-f8ae-4913-aa8a-5e8325a379d4/volumes"
Nov 02 12:59:58.117127 ip-10-0-128-132 kubenswrapper[4559]: I1102 12:59:58.117093    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-8032f3/auth-test-model-8032f3-kserve-675786644f-2dmj4"]
Nov 02 13:00:00.177380 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:00.177346    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5514e5-29893740-s9p6j"]
Nov 02 13:00:00.178292 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:00.178266    4559 cpu_manager.go:414] "containerMap: removing container" podUID="39c7e6bf-f8ae-4913-aa8a-5e8325a379d4" containerName="istio-proxy"
Nov 02 13:00:00.178456 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:00.178293    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39c7e6bf-f8ae-4913-aa8a-5e8325a379d4" containerName="istio-proxy"
Nov 02 13:00:00.178456 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:00.178320    4559 cpu_manager.go:414] "containerMap: removing container" podUID="a3622e27-7879-4e6d-ae5f-7ba184fdf1d7" containerName="test-curl-6992-7f0817"
Nov 02 13:00:00.178456 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:00.178327    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3622e27-7879-4e6d-ae5f-7ba184fdf1d7" containerName="test-curl-6992-7f0817"
Nov 02 13:00:00.178456 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:00.178370    4559 cpu_manager.go:414] "containerMap: removing container" podUID="3aabb9b6-9382-42d9-8c52-4d89e361c25b" containerName="istio-proxy"
Nov 02 13:00:00.178456 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:00.178377    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3aabb9b6-9382-42d9-8c52-4d89e361c25b" containerName="istio-proxy"
Nov 02 13:00:00.178456 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:00.178421    4559 cpu_manager.go:414] "containerMap: removing container" podUID="7e9360ac-a67e-4617-b345-110bcfa57381" containerName="main"
Nov 02 13:00:00.178456 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:00.178427    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e9360ac-a67e-4617-b345-110bcfa57381" containerName="main"
Nov 02 13:00:00.178901 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:00.178554    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e9360ac-a67e-4617-b345-110bcfa57381" containerName="main"
Nov 02 13:00:00.178901 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:00.178565    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="39c7e6bf-f8ae-4913-aa8a-5e8325a379d4" containerName="istio-proxy"
Nov 02 13:00:00.178901 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:00.178576    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3622e27-7879-4e6d-ae5f-7ba184fdf1d7" containerName="test-curl-6992-7f0817"
Nov 02 13:00:00.178901 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:00.178595    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="3aabb9b6-9382-42d9-8c52-4d89e361c25b" containerName="istio-proxy"
Nov 02 13:00:00.184290 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:00.184265    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5514e5-29893740-s9p6j"
Nov 02 13:00:00.202856 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:00.202816    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893740-ljp9t"]
Nov 02 13:00:00.211772 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:00.211736    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5514e5-29893740-s9p6j"]
Nov 02 13:00:00.211944 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:00.211868    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893740-ljp9t"
Nov 02 13:00:00.215066 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:00.215037    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e7561a-29893740-blswv"]
Nov 02 13:00:00.220283 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:00.220256    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e7561a-29893740-blswv"
Nov 02 13:00:00.230758 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:00.229764    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893740-ljp9t"]
Nov 02 13:00:00.240044 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:00.240015    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e7561a-29893740-blswv"]
Nov 02 13:00:00.277443 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:00.277414    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w9cwp\""
Nov 02 13:00:00.337822 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:00.337785    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3974db10-eea9-47c6-bbd4-557b4dca531f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-5514e5-29893740-s9p6j\" (UID: \"3974db10-eea9-47c6-bbd4-557b4dca531f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5514e5-29893740-s9p6j"
Nov 02 13:00:00.338012 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:00.337945    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a74304bb-3fb2-4dcd-be61-8299061bff15-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-e7561a-29893740-blswv\" (UID: \"a74304bb-3fb2-4dcd-be61-8299061bff15\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e7561a-29893740-blswv"
Nov 02 13:00:00.338084 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:00.338016    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mk5v\" (UniqueName: \"kubernetes.io/projected/3974db10-eea9-47c6-bbd4-557b4dca531f-kube-api-access-4mk5v\") pod \"maas-api-key-cleanup-e2e-worker-w0-5514e5-29893740-s9p6j\" (UID: \"3974db10-eea9-47c6-bbd4-557b4dca531f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5514e5-29893740-s9p6j"
Nov 02 13:00:00.338180 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:00.338159    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7eddad9-7bbd-4ef1-8d9c-3db6e4a58961-maas-api-service-ca\") pod \"maas-api-key-cleanup-29893740-ljp9t\" (UID: \"c7eddad9-7bbd-4ef1-8d9c-3db6e4a58961\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893740-ljp9t"
Nov 02 13:00:00.338294 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:00.338275    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5q6g\" (UniqueName: \"kubernetes.io/projected/c7eddad9-7bbd-4ef1-8d9c-3db6e4a58961-kube-api-access-j5q6g\") pod \"maas-api-key-cleanup-29893740-ljp9t\" (UID: \"c7eddad9-7bbd-4ef1-8d9c-3db6e4a58961\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893740-ljp9t"
Nov 02 13:00:00.338340 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:00.338312    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvvgt\" (UniqueName: \"kubernetes.io/projected/a74304bb-3fb2-4dcd-be61-8299061bff15-kube-api-access-rvvgt\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-e7561a-29893740-blswv\" (UID: \"a74304bb-3fb2-4dcd-be61-8299061bff15\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e7561a-29893740-blswv"
Nov 02 13:00:00.439576 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:00.439486    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7eddad9-7bbd-4ef1-8d9c-3db6e4a58961-maas-api-service-ca\") pod \"maas-api-key-cleanup-29893740-ljp9t\" (UID: \"c7eddad9-7bbd-4ef1-8d9c-3db6e4a58961\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893740-ljp9t"
Nov 02 13:00:00.439576 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:00.439557    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5q6g\" (UniqueName: \"kubernetes.io/projected/c7eddad9-7bbd-4ef1-8d9c-3db6e4a58961-kube-api-access-j5q6g\") pod \"maas-api-key-cleanup-29893740-ljp9t\" (UID: \"c7eddad9-7bbd-4ef1-8d9c-3db6e4a58961\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893740-ljp9t"
Nov 02 13:00:00.439867 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:00.439585    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvvgt\" (UniqueName: \"kubernetes.io/projected/a74304bb-3fb2-4dcd-be61-8299061bff15-kube-api-access-rvvgt\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-e7561a-29893740-blswv\" (UID: \"a74304bb-3fb2-4dcd-be61-8299061bff15\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e7561a-29893740-blswv"
Nov 02 13:00:00.439867 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:00.439607    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3974db10-eea9-47c6-bbd4-557b4dca531f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-5514e5-29893740-s9p6j\" (UID: \"3974db10-eea9-47c6-bbd4-557b4dca531f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5514e5-29893740-s9p6j"
Nov 02 13:00:00.439867 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:00.439702    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a74304bb-3fb2-4dcd-be61-8299061bff15-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-e7561a-29893740-blswv\" (UID: \"a74304bb-3fb2-4dcd-be61-8299061bff15\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e7561a-29893740-blswv"
Nov 02 13:00:00.439867 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:00.439768    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mk5v\" (UniqueName: \"kubernetes.io/projected/3974db10-eea9-47c6-bbd4-557b4dca531f-kube-api-access-4mk5v\") pod \"maas-api-key-cleanup-e2e-worker-w0-5514e5-29893740-s9p6j\" (UID: \"3974db10-eea9-47c6-bbd4-557b4dca531f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5514e5-29893740-s9p6j"
Nov 02 13:00:00.440328 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:00.440302    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7eddad9-7bbd-4ef1-8d9c-3db6e4a58961-maas-api-service-ca\") pod \"maas-api-key-cleanup-29893740-ljp9t\" (UID: \"c7eddad9-7bbd-4ef1-8d9c-3db6e4a58961\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893740-ljp9t"
Nov 02 13:00:00.440416 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:00.440355    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a74304bb-3fb2-4dcd-be61-8299061bff15-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-e7561a-29893740-blswv\" (UID: \"a74304bb-3fb2-4dcd-be61-8299061bff15\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e7561a-29893740-blswv"
Nov 02 13:00:00.440416 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:00.440391    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3974db10-eea9-47c6-bbd4-557b4dca531f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-5514e5-29893740-s9p6j\" (UID: \"3974db10-eea9-47c6-bbd4-557b4dca531f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5514e5-29893740-s9p6j"
Nov 02 13:00:00.464117 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:00.464088    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvvgt\" (UniqueName: \"kubernetes.io/projected/a74304bb-3fb2-4dcd-be61-8299061bff15-kube-api-access-rvvgt\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-e7561a-29893740-blswv\" (UID: \"a74304bb-3fb2-4dcd-be61-8299061bff15\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e7561a-29893740-blswv"
Nov 02 13:00:00.464243 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:00.464180    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5q6g\" (UniqueName: \"kubernetes.io/projected/c7eddad9-7bbd-4ef1-8d9c-3db6e4a58961-kube-api-access-j5q6g\") pod \"maas-api-key-cleanup-29893740-ljp9t\" (UID: \"c7eddad9-7bbd-4ef1-8d9c-3db6e4a58961\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893740-ljp9t"
Nov 02 13:00:00.464243 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:00.464178    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mk5v\" (UniqueName: \"kubernetes.io/projected/3974db10-eea9-47c6-bbd4-557b4dca531f-kube-api-access-4mk5v\") pod \"maas-api-key-cleanup-e2e-worker-w0-5514e5-29893740-s9p6j\" (UID: \"3974db10-eea9-47c6-bbd4-557b4dca531f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5514e5-29893740-s9p6j"
Nov 02 13:00:00.510105 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:00.510080    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5514e5-29893740-s9p6j"
Nov 02 13:00:00.543159 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:00.543106    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893740-ljp9t"
Nov 02 13:00:00.550566 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:00.550534    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e7561a-29893740-blswv"
Nov 02 13:00:00.823655 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:00.823610    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5514e5-29893740-s9p6j"]
Nov 02 13:00:00.823948 ip-10-0-128-132 kubenswrapper[4559]: W1102 13:00:00.823926    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3974db10_eea9_47c6_bbd4_557b4dca531f.slice/crio-98ed8c1f631a7804f8426ec6e22caeb071da2ef91f62609c76807e88c3d6c308 WatchSource:0}: Error finding container 98ed8c1f631a7804f8426ec6e22caeb071da2ef91f62609c76807e88c3d6c308: Status 404 returned error can't find the container with id 98ed8c1f631a7804f8426ec6e22caeb071da2ef91f62609c76807e88c3d6c308
Nov 02 13:00:00.873676 ip-10-0-128-132 kubenswrapper[4559]: W1102 13:00:00.873619    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7eddad9_7bbd_4ef1_8d9c_3db6e4a58961.slice/crio-ac4b30a474d6ffc2f8ad961eaa78c14295c3e7efbf2d33a6d02071d311576d0e WatchSource:0}: Error finding container ac4b30a474d6ffc2f8ad961eaa78c14295c3e7efbf2d33a6d02071d311576d0e: Status 404 returned error can't find the container with id ac4b30a474d6ffc2f8ad961eaa78c14295c3e7efbf2d33a6d02071d311576d0e
Nov 02 13:00:00.874629 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:00.874609    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893740-ljp9t"]
Nov 02 13:00:00.883294 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:00.883268    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e7561a-29893740-blswv"]
Nov 02 13:00:00.887180 ip-10-0-128-132 kubenswrapper[4559]: W1102 13:00:00.887155    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda74304bb_3fb2_4dcd_be61_8299061bff15.slice/crio-0359b137d8d4a0b63632e56b4b84056223168a4b7d2e8506cdbd8461f810c8e7 WatchSource:0}: Error finding container 0359b137d8d4a0b63632e56b4b84056223168a4b7d2e8506cdbd8461f810c8e7: Status 404 returned error can't find the container with id 0359b137d8d4a0b63632e56b4b84056223168a4b7d2e8506cdbd8461f810c8e7
Nov 02 13:00:01.290143 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:01.289924    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e7561a-29893740-blswv" event={"ID":"a74304bb-3fb2-4dcd-be61-8299061bff15","Type":"ContainerStarted","Data":"0359b137d8d4a0b63632e56b4b84056223168a4b7d2e8506cdbd8461f810c8e7"}
Nov 02 13:00:01.294216 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:01.294181    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5514e5-29893740-s9p6j" event={"ID":"3974db10-eea9-47c6-bbd4-557b4dca531f","Type":"ContainerStarted","Data":"c0ad6ca9bec1fdf056ae0ca88db650fe7d495e30290a4fe4c98175a7e4ed5203"}
Nov 02 13:00:01.294321 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:01.294248    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5514e5-29893740-s9p6j" event={"ID":"3974db10-eea9-47c6-bbd4-557b4dca531f","Type":"ContainerStarted","Data":"98ed8c1f631a7804f8426ec6e22caeb071da2ef91f62609c76807e88c3d6c308"}
Nov 02 13:00:01.296051 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:01.296024    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893740-ljp9t" event={"ID":"c7eddad9-7bbd-4ef1-8d9c-3db6e4a58961","Type":"ContainerStarted","Data":"ac4b30a474d6ffc2f8ad961eaa78c14295c3e7efbf2d33a6d02071d311576d0e"}
Nov 02 13:00:01.334931 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:01.334879    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5514e5-29893740-s9p6j" podStartSLOduration=1.334860549 podStartE2EDuration="1.334860549s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:00:01.334369043 +0000 UTC m=+1997.589768139" watchObservedRunningTime="2026-11-02 13:00:01.334860549 +0000 UTC m=+1997.590259645"
Nov 02 13:00:02.307223 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:02.307185    4559 generic.go:356] "Generic (PLEG): container finished" podID="3974db10-eea9-47c6-bbd4-557b4dca531f" containerID="c0ad6ca9bec1fdf056ae0ca88db650fe7d495e30290a4fe4c98175a7e4ed5203" exitCode=60
Nov 02 13:00:02.307722 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:02.307255    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5514e5-29893740-s9p6j" event={"ID":"3974db10-eea9-47c6-bbd4-557b4dca531f","Type":"ContainerDied","Data":"c0ad6ca9bec1fdf056ae0ca88db650fe7d495e30290a4fe4c98175a7e4ed5203"}
Nov 02 13:00:02.307722 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:02.307668    4559 scope.go:122] "RemoveContainer" containerID="c0ad6ca9bec1fdf056ae0ca88db650fe7d495e30290a4fe4c98175a7e4ed5203"
Nov 02 13:00:02.309253 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:02.309230    4559 generic.go:356] "Generic (PLEG): container finished" podID="c7eddad9-7bbd-4ef1-8d9c-3db6e4a58961" containerID="88fb31c19b3d476779c052dd0aff544c9501f6c39117483900429e189d435d87" exitCode=60
Nov 02 13:00:02.309382 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:02.309306    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893740-ljp9t" event={"ID":"c7eddad9-7bbd-4ef1-8d9c-3db6e4a58961","Type":"ContainerDied","Data":"88fb31c19b3d476779c052dd0aff544c9501f6c39117483900429e189d435d87"}
Nov 02 13:00:02.309555 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:02.309537    4559 scope.go:122] "RemoveContainer" containerID="88fb31c19b3d476779c052dd0aff544c9501f6c39117483900429e189d435d87"
Nov 02 13:00:02.311181 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:02.311166    4559 generic.go:356] "Generic (PLEG): container finished" podID="a74304bb-3fb2-4dcd-be61-8299061bff15" containerID="3c3b2202b01b1f30ffec2067bbdd3f1633dfc1cddf134e7b3e8fd4adb020e325" exitCode=60
Nov 02 13:00:02.311236 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:02.311194    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e7561a-29893740-blswv" event={"ID":"a74304bb-3fb2-4dcd-be61-8299061bff15","Type":"ContainerDied","Data":"3c3b2202b01b1f30ffec2067bbdd3f1633dfc1cddf134e7b3e8fd4adb020e325"}
Nov 02 13:00:02.311462 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:02.311446    4559 scope.go:122] "RemoveContainer" containerID="3c3b2202b01b1f30ffec2067bbdd3f1633dfc1cddf134e7b3e8fd4adb020e325"
Nov 02 13:00:03.321562 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:03.321528    4559 generic.go:356] "Generic (PLEG): container finished" podID="a74304bb-3fb2-4dcd-be61-8299061bff15" containerID="5a51efe84d618a4ed68bef14f7278b7d29fc10c61954df31e314d47defc5851d" exitCode=60
Nov 02 13:00:03.322056 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:03.321631    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e7561a-29893740-blswv" event={"ID":"a74304bb-3fb2-4dcd-be61-8299061bff15","Type":"ContainerDied","Data":"5a51efe84d618a4ed68bef14f7278b7d29fc10c61954df31e314d47defc5851d"}
Nov 02 13:00:03.322056 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:03.321701    4559 scope.go:122] "RemoveContainer" containerID="3c3b2202b01b1f30ffec2067bbdd3f1633dfc1cddf134e7b3e8fd4adb020e325"
Nov 02 13:00:03.322056 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:03.321939    4559 scope.go:122] "RemoveContainer" containerID="5a51efe84d618a4ed68bef14f7278b7d29fc10c61954df31e314d47defc5851d"
Nov 02 13:00:03.322299 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:00:03.322279    4559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-e7561a-29893740-blswv_odh-ai-gateway-infra(a74304bb-3fb2-4dcd-be61-8299061bff15)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e7561a-29893740-blswv" podUID="a74304bb-3fb2-4dcd-be61-8299061bff15"
Nov 02 13:00:03.323698 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:03.323677    4559 generic.go:356] "Generic (PLEG): container finished" podID="3974db10-eea9-47c6-bbd4-557b4dca531f" containerID="ce9ee9c889881421b7c43eaa242b88531b79a0e3177a96c0374f33358292570c" exitCode=60
Nov 02 13:00:03.323820 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:03.323800    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5514e5-29893740-s9p6j" event={"ID":"3974db10-eea9-47c6-bbd4-557b4dca531f","Type":"ContainerDied","Data":"ce9ee9c889881421b7c43eaa242b88531b79a0e3177a96c0374f33358292570c"}
Nov 02 13:00:03.324091 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:03.324032    4559 scope.go:122] "RemoveContainer" containerID="ce9ee9c889881421b7c43eaa242b88531b79a0e3177a96c0374f33358292570c"
Nov 02 13:00:03.324347 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:00:03.324327    4559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-5514e5-29893740-s9p6j_odh-ai-gateway-infra(3974db10-eea9-47c6-bbd4-557b4dca531f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5514e5-29893740-s9p6j" podUID="3974db10-eea9-47c6-bbd4-557b4dca531f"
Nov 02 13:00:03.325535 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:03.325514    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893740-ljp9t" event={"ID":"c7eddad9-7bbd-4ef1-8d9c-3db6e4a58961","Type":"ContainerStarted","Data":"a1f0d028ade8119dc00e4f45d1b9c06698653b852404aa5d037ede8d130d8dc3"}
Nov 02 13:00:03.445633 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:03.445534    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893740-ljp9t" podStartSLOduration=3.44552096 podStartE2EDuration="3.44552096s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:00:03.443994495 +0000 UTC m=+1999.699393590" watchObservedRunningTime="2026-11-02 13:00:03.44552096 +0000 UTC m=+1999.700920055"
Nov 02 13:00:03.460396 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:03.460375    4559 scope.go:122] "RemoveContainer" containerID="c0ad6ca9bec1fdf056ae0ca88db650fe7d495e30290a4fe4c98175a7e4ed5203"
Nov 02 13:00:04.336181 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:04.336151    4559 scope.go:122] "RemoveContainer" containerID="5a51efe84d618a4ed68bef14f7278b7d29fc10c61954df31e314d47defc5851d"
Nov 02 13:00:04.336600 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:00:04.336502    4559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-e7561a-29893740-blswv_odh-ai-gateway-infra(a74304bb-3fb2-4dcd-be61-8299061bff15)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e7561a-29893740-blswv" podUID="a74304bb-3fb2-4dcd-be61-8299061bff15"
Nov 02 13:00:04.337779 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:04.337755    4559 scope.go:122] "RemoveContainer" containerID="ce9ee9c889881421b7c43eaa242b88531b79a0e3177a96c0374f33358292570c"
Nov 02 13:00:04.338115 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:00:04.338094    4559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-5514e5-29893740-s9p6j_odh-ai-gateway-infra(3974db10-eea9-47c6-bbd4-557b4dca531f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5514e5-29893740-s9p6j" podUID="3974db10-eea9-47c6-bbd4-557b4dca531f"
Nov 02 13:00:04.339328 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:04.339304    4559 generic.go:356] "Generic (PLEG): container finished" podID="c7eddad9-7bbd-4ef1-8d9c-3db6e4a58961" containerID="a1f0d028ade8119dc00e4f45d1b9c06698653b852404aa5d037ede8d130d8dc3" exitCode=60
Nov 02 13:00:04.339442 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:04.339337    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893740-ljp9t" event={"ID":"c7eddad9-7bbd-4ef1-8d9c-3db6e4a58961","Type":"ContainerDied","Data":"a1f0d028ade8119dc00e4f45d1b9c06698653b852404aa5d037ede8d130d8dc3"}
Nov 02 13:00:04.339442 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:04.339378    4559 scope.go:122] "RemoveContainer" containerID="88fb31c19b3d476779c052dd0aff544c9501f6c39117483900429e189d435d87"
Nov 02 13:00:04.339679 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:04.339629    4559 scope.go:122] "RemoveContainer" containerID="a1f0d028ade8119dc00e4f45d1b9c06698653b852404aa5d037ede8d130d8dc3"
Nov 02 13:00:04.339920 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:00:04.339901    4559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29893740-ljp9t_odh-ai-gateway-infra(c7eddad9-7bbd-4ef1-8d9c-3db6e4a58961)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893740-ljp9t" podUID="c7eddad9-7bbd-4ef1-8d9c-3db6e4a58961"
Nov 02 13:00:05.348436 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:05.348409    4559 scope.go:122] "RemoveContainer" containerID="a1f0d028ade8119dc00e4f45d1b9c06698653b852404aa5d037ede8d130d8dc3"
Nov 02 13:00:05.348849 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:00:05.348732    4559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29893740-ljp9t_odh-ai-gateway-infra(c7eddad9-7bbd-4ef1-8d9c-3db6e4a58961)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893740-ljp9t" podUID="c7eddad9-7bbd-4ef1-8d9c-3db6e4a58961"
Nov 02 13:00:08.303962 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:08.303922    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-7qxgl"]
Nov 02 13:00:08.304433 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:08.304127    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-7qxgl" podUID="13f8bac2-10c1-40aa-a968-3a9b64a6676f" containerName="istio-proxy" containerID="cri-o://5a797e1223770184135d924d845ace05020d8abd016fbd85d030252a6e4bd07e" gracePeriod=30
Nov 02 13:00:10.349113 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:10.349073    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-43be82/auth-test-model-43be82-kserve-5c6dfd4dfc-n4dl7"]
Nov 02 13:00:11.419849 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:11.419810    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-ecec44-openshift-default-6d94dccbcf-thqw7"]
Nov 02 13:00:11.425198 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:11.425173    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-ecec44-openshift-default-6d94dccbcf-thqw7"
Nov 02 13:00:11.447866 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:11.447833    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-ecec44-openshift-default-6d94dccbcf-thqw7"]
Nov 02 13:00:11.454220 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:11.454157    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-ecec44-openshift-default-dockercfg-s57mf\""
Nov 02 13:00:11.504759 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:11.504713    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abcacc36-3628-44e7-a103-fdf1027634cf-istio-token\") pod \"e2e-autores-ecec44-openshift-default-6d94dccbcf-thqw7\" (UID: \"abcacc36-3628-44e7-a103-fdf1027634cf\") " pod="openshift-ingress/e2e-autores-ecec44-openshift-default-6d94dccbcf-thqw7"
Nov 02 13:00:11.504967 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:11.504793    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abcacc36-3628-44e7-a103-fdf1027634cf-istio-podinfo\") pod \"e2e-autores-ecec44-openshift-default-6d94dccbcf-thqw7\" (UID: \"abcacc36-3628-44e7-a103-fdf1027634cf\") " pod="openshift-ingress/e2e-autores-ecec44-openshift-default-6d94dccbcf-thqw7"
Nov 02 13:00:11.504967 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:11.504864    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abcacc36-3628-44e7-a103-fdf1027634cf-workload-certs\") pod \"e2e-autores-ecec44-openshift-default-6d94dccbcf-thqw7\" (UID: \"abcacc36-3628-44e7-a103-fdf1027634cf\") " pod="openshift-ingress/e2e-autores-ecec44-openshift-default-6d94dccbcf-thqw7"
Nov 02 13:00:11.504967 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:11.504900    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abcacc36-3628-44e7-a103-fdf1027634cf-istiod-ca-cert\") pod \"e2e-autores-ecec44-openshift-default-6d94dccbcf-thqw7\" (UID: \"abcacc36-3628-44e7-a103-fdf1027634cf\") " pod="openshift-ingress/e2e-autores-ecec44-openshift-default-6d94dccbcf-thqw7"
Nov 02 13:00:11.505080 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:11.504981    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zknzt\" (UniqueName: \"kubernetes.io/projected/abcacc36-3628-44e7-a103-fdf1027634cf-kube-api-access-zknzt\") pod \"e2e-autores-ecec44-openshift-default-6d94dccbcf-thqw7\" (UID: \"abcacc36-3628-44e7-a103-fdf1027634cf\") " pod="openshift-ingress/e2e-autores-ecec44-openshift-default-6d94dccbcf-thqw7"
Nov 02 13:00:11.505080 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:11.505048    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abcacc36-3628-44e7-a103-fdf1027634cf-workload-socket\") pod \"e2e-autores-ecec44-openshift-default-6d94dccbcf-thqw7\" (UID: \"abcacc36-3628-44e7-a103-fdf1027634cf\") " pod="openshift-ingress/e2e-autores-ecec44-openshift-default-6d94dccbcf-thqw7"
Nov 02 13:00:11.505204 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:11.505187    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abcacc36-3628-44e7-a103-fdf1027634cf-credential-socket\") pod \"e2e-autores-ecec44-openshift-default-6d94dccbcf-thqw7\" (UID: \"abcacc36-3628-44e7-a103-fdf1027634cf\") " pod="openshift-ingress/e2e-autores-ecec44-openshift-default-6d94dccbcf-thqw7"
Nov 02 13:00:11.505245 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:11.505224    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abcacc36-3628-44e7-a103-fdf1027634cf-istio-data\") pod \"e2e-autores-ecec44-openshift-default-6d94dccbcf-thqw7\" (UID: \"abcacc36-3628-44e7-a103-fdf1027634cf\") " pod="openshift-ingress/e2e-autores-ecec44-openshift-default-6d94dccbcf-thqw7"
Nov 02 13:00:11.505298 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:11.505259    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abcacc36-3628-44e7-a103-fdf1027634cf-istio-envoy\") pod \"e2e-autores-ecec44-openshift-default-6d94dccbcf-thqw7\" (UID: \"abcacc36-3628-44e7-a103-fdf1027634cf\") " pod="openshift-ingress/e2e-autores-ecec44-openshift-default-6d94dccbcf-thqw7"
Nov 02 13:00:11.643827 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:11.607622    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abcacc36-3628-44e7-a103-fdf1027634cf-istio-token\") pod \"e2e-autores-ecec44-openshift-default-6d94dccbcf-thqw7\" (UID: \"abcacc36-3628-44e7-a103-fdf1027634cf\") " pod="openshift-ingress/e2e-autores-ecec44-openshift-default-6d94dccbcf-thqw7"
Nov 02 13:00:11.643827 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:11.607743    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abcacc36-3628-44e7-a103-fdf1027634cf-istio-podinfo\") pod \"e2e-autores-ecec44-openshift-default-6d94dccbcf-thqw7\" (UID: \"abcacc36-3628-44e7-a103-fdf1027634cf\") " pod="openshift-ingress/e2e-autores-ecec44-openshift-default-6d94dccbcf-thqw7"
Nov 02 13:00:11.643827 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:11.607776    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abcacc36-3628-44e7-a103-fdf1027634cf-workload-certs\") pod \"e2e-autores-ecec44-openshift-default-6d94dccbcf-thqw7\" (UID: \"abcacc36-3628-44e7-a103-fdf1027634cf\") " pod="openshift-ingress/e2e-autores-ecec44-openshift-default-6d94dccbcf-thqw7"
Nov 02 13:00:11.643827 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:11.607807    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abcacc36-3628-44e7-a103-fdf1027634cf-istiod-ca-cert\") pod \"e2e-autores-ecec44-openshift-default-6d94dccbcf-thqw7\" (UID: \"abcacc36-3628-44e7-a103-fdf1027634cf\") " pod="openshift-ingress/e2e-autores-ecec44-openshift-default-6d94dccbcf-thqw7"
Nov 02 13:00:11.643827 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:11.607891    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zknzt\" (UniqueName: \"kubernetes.io/projected/abcacc36-3628-44e7-a103-fdf1027634cf-kube-api-access-zknzt\") pod \"e2e-autores-ecec44-openshift-default-6d94dccbcf-thqw7\" (UID: \"abcacc36-3628-44e7-a103-fdf1027634cf\") " pod="openshift-ingress/e2e-autores-ecec44-openshift-default-6d94dccbcf-thqw7"
Nov 02 13:00:11.643827 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:11.607992    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abcacc36-3628-44e7-a103-fdf1027634cf-workload-socket\") pod \"e2e-autores-ecec44-openshift-default-6d94dccbcf-thqw7\" (UID: \"abcacc36-3628-44e7-a103-fdf1027634cf\") " pod="openshift-ingress/e2e-autores-ecec44-openshift-default-6d94dccbcf-thqw7"
Nov 02 13:00:11.643827 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:11.608103    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abcacc36-3628-44e7-a103-fdf1027634cf-credential-socket\") pod \"e2e-autores-ecec44-openshift-default-6d94dccbcf-thqw7\" (UID: \"abcacc36-3628-44e7-a103-fdf1027634cf\") " pod="openshift-ingress/e2e-autores-ecec44-openshift-default-6d94dccbcf-thqw7"
Nov 02 13:00:11.643827 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:11.608149    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abcacc36-3628-44e7-a103-fdf1027634cf-istio-data\") pod \"e2e-autores-ecec44-openshift-default-6d94dccbcf-thqw7\" (UID: \"abcacc36-3628-44e7-a103-fdf1027634cf\") " pod="openshift-ingress/e2e-autores-ecec44-openshift-default-6d94dccbcf-thqw7"
Nov 02 13:00:11.643827 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:11.608189    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abcacc36-3628-44e7-a103-fdf1027634cf-istio-envoy\") pod \"e2e-autores-ecec44-openshift-default-6d94dccbcf-thqw7\" (UID: \"abcacc36-3628-44e7-a103-fdf1027634cf\") " pod="openshift-ingress/e2e-autores-ecec44-openshift-default-6d94dccbcf-thqw7"
Nov 02 13:00:11.643827 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:11.615286    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abcacc36-3628-44e7-a103-fdf1027634cf-istio-envoy\") pod \"e2e-autores-ecec44-openshift-default-6d94dccbcf-thqw7\" (UID: \"abcacc36-3628-44e7-a103-fdf1027634cf\") " pod="openshift-ingress/e2e-autores-ecec44-openshift-default-6d94dccbcf-thqw7"
Nov 02 13:00:11.643827 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:11.630319    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abcacc36-3628-44e7-a103-fdf1027634cf-workload-socket\") pod \"e2e-autores-ecec44-openshift-default-6d94dccbcf-thqw7\" (UID: \"abcacc36-3628-44e7-a103-fdf1027634cf\") " pod="openshift-ingress/e2e-autores-ecec44-openshift-default-6d94dccbcf-thqw7"
Nov 02 13:00:11.643827 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:11.630630    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abcacc36-3628-44e7-a103-fdf1027634cf-workload-certs\") pod \"e2e-autores-ecec44-openshift-default-6d94dccbcf-thqw7\" (UID: \"abcacc36-3628-44e7-a103-fdf1027634cf\") " pod="openshift-ingress/e2e-autores-ecec44-openshift-default-6d94dccbcf-thqw7"
Nov 02 13:00:11.643827 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:11.631251    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abcacc36-3628-44e7-a103-fdf1027634cf-istiod-ca-cert\") pod \"e2e-autores-ecec44-openshift-default-6d94dccbcf-thqw7\" (UID: \"abcacc36-3628-44e7-a103-fdf1027634cf\") " pod="openshift-ingress/e2e-autores-ecec44-openshift-default-6d94dccbcf-thqw7"
Nov 02 13:00:11.643827 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:11.631824    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abcacc36-3628-44e7-a103-fdf1027634cf-istio-data\") pod \"e2e-autores-ecec44-openshift-default-6d94dccbcf-thqw7\" (UID: \"abcacc36-3628-44e7-a103-fdf1027634cf\") " pod="openshift-ingress/e2e-autores-ecec44-openshift-default-6d94dccbcf-thqw7"
Nov 02 13:00:11.643827 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:11.632052    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abcacc36-3628-44e7-a103-fdf1027634cf-credential-socket\") pod \"e2e-autores-ecec44-openshift-default-6d94dccbcf-thqw7\" (UID: \"abcacc36-3628-44e7-a103-fdf1027634cf\") " pod="openshift-ingress/e2e-autores-ecec44-openshift-default-6d94dccbcf-thqw7"
Nov 02 13:00:11.669725 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:11.652158    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abcacc36-3628-44e7-a103-fdf1027634cf-istio-token\") pod \"e2e-autores-ecec44-openshift-default-6d94dccbcf-thqw7\" (UID: \"abcacc36-3628-44e7-a103-fdf1027634cf\") " pod="openshift-ingress/e2e-autores-ecec44-openshift-default-6d94dccbcf-thqw7"
Nov 02 13:00:11.669725 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:11.662562    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zknzt\" (UniqueName: \"kubernetes.io/projected/abcacc36-3628-44e7-a103-fdf1027634cf-kube-api-access-zknzt\") pod \"e2e-autores-ecec44-openshift-default-6d94dccbcf-thqw7\" (UID: \"abcacc36-3628-44e7-a103-fdf1027634cf\") " pod="openshift-ingress/e2e-autores-ecec44-openshift-default-6d94dccbcf-thqw7"
Nov 02 13:00:11.685327 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:11.685203    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abcacc36-3628-44e7-a103-fdf1027634cf-istio-podinfo\") pod \"e2e-autores-ecec44-openshift-default-6d94dccbcf-thqw7\" (UID: \"abcacc36-3628-44e7-a103-fdf1027634cf\") " pod="openshift-ingress/e2e-autores-ecec44-openshift-default-6d94dccbcf-thqw7"
Nov 02 13:00:11.743311 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:11.742919    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-ecec44-openshift-default-6d94dccbcf-thqw7"
Nov 02 13:00:12.140059 ip-10-0-128-132 kubenswrapper[4559]: W1102 13:00:12.140030    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabcacc36_3628_44e7_a103_fdf1027634cf.slice/crio-f9a7168a19074010c8064fd5f01cda6aab0d4aba9bbe2510e214e745e979a0e3 WatchSource:0}: Error finding container f9a7168a19074010c8064fd5f01cda6aab0d4aba9bbe2510e214e745e979a0e3: Status 404 returned error can't find the container with id f9a7168a19074010c8064fd5f01cda6aab0d4aba9bbe2510e214e745e979a0e3
Nov 02 13:00:12.140905 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:12.140880    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-ecec44-openshift-default-6d94dccbcf-thqw7"]
Nov 02 13:00:12.142618 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:12.142588    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 13:00:12.142694 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:12.142677    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 13:00:12.142740 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:12.142714    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 13:00:12.399340 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:12.399252    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-ecec44-openshift-default-6d94dccbcf-thqw7" event={"ID":"abcacc36-3628-44e7-a103-fdf1027634cf","Type":"ContainerStarted","Data":"ae925b4b0b98226777ab79fddf3d1314563e478a87f64c3170ac1de0d48ab2bf"}
Nov 02 13:00:12.399340 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:12.399308    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-ecec44-openshift-default-6d94dccbcf-thqw7" event={"ID":"abcacc36-3628-44e7-a103-fdf1027634cf","Type":"ContainerStarted","Data":"f9a7168a19074010c8064fd5f01cda6aab0d4aba9bbe2510e214e745e979a0e3"}
Nov 02 13:00:12.449802 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:12.449747    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-ecec44-openshift-default-6d94dccbcf-thqw7" podStartSLOduration=1.449734212 podStartE2EDuration="1.449734212s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:00:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:00:12.44646938 +0000 UTC m=+2008.701868475" watchObservedRunningTime="2026-11-02 13:00:12.449734212 +0000 UTC m=+2008.705133306"
Nov 02 13:00:12.743723 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:12.743685    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-ecec44-openshift-default-6d94dccbcf-thqw7"
Nov 02 13:00:13.385188 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:13.385149    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w0-8032f3/auth-test-model-8032f3-kserve-675786644f-2dmj4" podUID="e8024fcc-5300-4b6c-84c0-7c2c30a0370a" containerName="main" containerID="cri-o://7783ba0607706736796a59dcfae865bf0505f812f9c5dd2e56d7f63d6c008209" gracePeriod=45
Nov 02 13:00:13.411392 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:13.411365    4559 generic.go:356] "Generic (PLEG): container finished" podID="13f8bac2-10c1-40aa-a968-3a9b64a6676f" containerID="5a797e1223770184135d924d845ace05020d8abd016fbd85d030252a6e4bd07e" exitCode=0
Nov 02 13:00:13.411516 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:13.411440    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-7qxgl" event={"ID":"13f8bac2-10c1-40aa-a968-3a9b64a6676f","Type":"ContainerDied","Data":"5a797e1223770184135d924d845ace05020d8abd016fbd85d030252a6e4bd07e"}
Nov 02 13:00:13.659715 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:13.659688    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-8032f3/auth-test-model-8032f3-kserve-675786644f-2dmj4"
Nov 02 13:00:13.750775 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:13.750739    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-ecec44-openshift-default-6d94dccbcf-thqw7"
Nov 02 13:00:13.754050 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:13.754027    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8024fcc-5300-4b6c-84c0-7c2c30a0370a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8024fcc-5300-4b6c-84c0-7c2c30a0370a-dshm\") pod \"e8024fcc-5300-4b6c-84c0-7c2c30a0370a\" (UID: \"e8024fcc-5300-4b6c-84c0-7c2c30a0370a\") "
Nov 02 13:00:13.754157 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:13.754067    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8024fcc-5300-4b6c-84c0-7c2c30a0370a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8024fcc-5300-4b6c-84c0-7c2c30a0370a-tmp-dir\") pod \"e8024fcc-5300-4b6c-84c0-7c2c30a0370a\" (UID: \"e8024fcc-5300-4b6c-84c0-7c2c30a0370a\") "
Nov 02 13:00:13.754157 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:13.754114    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e8024fcc-5300-4b6c-84c0-7c2c30a0370a-tls-certs\" (UniqueName: \"kubernetes.io/secret/e8024fcc-5300-4b6c-84c0-7c2c30a0370a-tls-certs\") pod \"e8024fcc-5300-4b6c-84c0-7c2c30a0370a\" (UID: \"e8024fcc-5300-4b6c-84c0-7c2c30a0370a\") "
Nov 02 13:00:13.754157 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:13.754141    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8024fcc-5300-4b6c-84c0-7c2c30a0370a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8024fcc-5300-4b6c-84c0-7c2c30a0370a-model-cache\") pod \"e8024fcc-5300-4b6c-84c0-7c2c30a0370a\" (UID: \"e8024fcc-5300-4b6c-84c0-7c2c30a0370a\") "
Nov 02 13:00:13.754335 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:13.754208    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8024fcc-5300-4b6c-84c0-7c2c30a0370a-home\" (UniqueName: \"kubernetes.io/empty-dir/e8024fcc-5300-4b6c-84c0-7c2c30a0370a-home\") pod \"e8024fcc-5300-4b6c-84c0-7c2c30a0370a\" (UID: \"e8024fcc-5300-4b6c-84c0-7c2c30a0370a\") "
Nov 02 13:00:13.754403 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:13.754330    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8024fcc-5300-4b6c-84c0-7c2c30a0370a-kube-api-access-pcdsv\" (UniqueName: \"kubernetes.io/projected/e8024fcc-5300-4b6c-84c0-7c2c30a0370a-kube-api-access-pcdsv\") pod \"e8024fcc-5300-4b6c-84c0-7c2c30a0370a\" (UID: \"e8024fcc-5300-4b6c-84c0-7c2c30a0370a\") "
Nov 02 13:00:13.754462 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:13.754419    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8024fcc-5300-4b6c-84c0-7c2c30a0370a-tmp-dir" pod "e8024fcc-5300-4b6c-84c0-7c2c30a0370a" (UID: "e8024fcc-5300-4b6c-84c0-7c2c30a0370a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:00:13.754517 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:13.754464    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8024fcc-5300-4b6c-84c0-7c2c30a0370a-model-cache" pod "e8024fcc-5300-4b6c-84c0-7c2c30a0370a" (UID: "e8024fcc-5300-4b6c-84c0-7c2c30a0370a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:00:13.754587 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:13.754531    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8024fcc-5300-4b6c-84c0-7c2c30a0370a-home" pod "e8024fcc-5300-4b6c-84c0-7c2c30a0370a" (UID: "e8024fcc-5300-4b6c-84c0-7c2c30a0370a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:00:13.755008 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:13.754985    4559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8024fcc-5300-4b6c-84c0-7c2c30a0370a-tmp-dir\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:00:13.755008 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:13.755010    4559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8024fcc-5300-4b6c-84c0-7c2c30a0370a-model-cache\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:00:13.755196 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:13.755022    4559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8024fcc-5300-4b6c-84c0-7c2c30a0370a-home\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:00:13.756398 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:13.756378    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8024fcc-5300-4b6c-84c0-7c2c30a0370a-dshm" pod "e8024fcc-5300-4b6c-84c0-7c2c30a0370a" (UID: "e8024fcc-5300-4b6c-84c0-7c2c30a0370a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:00:13.756493 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:13.756448    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8024fcc-5300-4b6c-84c0-7c2c30a0370a-tls-certs" pod "e8024fcc-5300-4b6c-84c0-7c2c30a0370a" (UID: "e8024fcc-5300-4b6c-84c0-7c2c30a0370a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:00:13.756543 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:13.756523    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8024fcc-5300-4b6c-84c0-7c2c30a0370a-kube-api-access-pcdsv" pod "e8024fcc-5300-4b6c-84c0-7c2c30a0370a" (UID: "e8024fcc-5300-4b6c-84c0-7c2c30a0370a"). InnerVolumeSpecName "kube-api-access-pcdsv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:00:13.856895 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:13.856864    4559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8024fcc-5300-4b6c-84c0-7c2c30a0370a-tls-certs\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:00:13.856895 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:13.856892    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pcdsv\" (UniqueName: \"kubernetes.io/projected/e8024fcc-5300-4b6c-84c0-7c2c30a0370a-kube-api-access-pcdsv\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:00:13.857186 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:13.856904    4559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8024fcc-5300-4b6c-84c0-7c2c30a0370a-dshm\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:00:13.987617 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:13.987589    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-7qxgl"
Nov 02 13:00:14.059774 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:14.059739    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13f8bac2-10c1-40aa-a968-3a9b64a6676f-istio-token\" (UniqueName: \"kubernetes.io/projected/13f8bac2-10c1-40aa-a968-3a9b64a6676f-istio-token\") pod \"13f8bac2-10c1-40aa-a968-3a9b64a6676f\" (UID: \"13f8bac2-10c1-40aa-a968-3a9b64a6676f\") "
Nov 02 13:00:14.059970 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:14.059840    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13f8bac2-10c1-40aa-a968-3a9b64a6676f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13f8bac2-10c1-40aa-a968-3a9b64a6676f-istio-data\") pod \"13f8bac2-10c1-40aa-a968-3a9b64a6676f\" (UID: \"13f8bac2-10c1-40aa-a968-3a9b64a6676f\") "
Nov 02 13:00:14.059970 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:14.059894    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13f8bac2-10c1-40aa-a968-3a9b64a6676f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13f8bac2-10c1-40aa-a968-3a9b64a6676f-istio-envoy\") pod \"13f8bac2-10c1-40aa-a968-3a9b64a6676f\" (UID: \"13f8bac2-10c1-40aa-a968-3a9b64a6676f\") "
Nov 02 13:00:14.059970 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:14.059941    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13f8bac2-10c1-40aa-a968-3a9b64a6676f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13f8bac2-10c1-40aa-a968-3a9b64a6676f-credential-socket\") pod \"13f8bac2-10c1-40aa-a968-3a9b64a6676f\" (UID: \"13f8bac2-10c1-40aa-a968-3a9b64a6676f\") "
Nov 02 13:00:14.060162 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:14.059984    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/13f8bac2-10c1-40aa-a968-3a9b64a6676f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13f8bac2-10c1-40aa-a968-3a9b64a6676f-istio-podinfo\") pod \"13f8bac2-10c1-40aa-a968-3a9b64a6676f\" (UID: \"13f8bac2-10c1-40aa-a968-3a9b64a6676f\") "
Nov 02 13:00:14.060162 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:14.060038    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13f8bac2-10c1-40aa-a968-3a9b64a6676f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13f8bac2-10c1-40aa-a968-3a9b64a6676f-workload-certs\") pod \"13f8bac2-10c1-40aa-a968-3a9b64a6676f\" (UID: \"13f8bac2-10c1-40aa-a968-3a9b64a6676f\") "
Nov 02 13:00:14.060162 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:14.060134    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13f8bac2-10c1-40aa-a968-3a9b64a6676f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13f8bac2-10c1-40aa-a968-3a9b64a6676f-workload-socket\") pod \"13f8bac2-10c1-40aa-a968-3a9b64a6676f\" (UID: \"13f8bac2-10c1-40aa-a968-3a9b64a6676f\") "
Nov 02 13:00:14.060328 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:14.060193    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13f8bac2-10c1-40aa-a968-3a9b64a6676f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13f8bac2-10c1-40aa-a968-3a9b64a6676f-istiod-ca-cert\") pod \"13f8bac2-10c1-40aa-a968-3a9b64a6676f\" (UID: \"13f8bac2-10c1-40aa-a968-3a9b64a6676f\") "
Nov 02 13:00:14.060328 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:14.060262    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13f8bac2-10c1-40aa-a968-3a9b64a6676f-kube-api-access-bkfb6\" (UniqueName: \"kubernetes.io/projected/13f8bac2-10c1-40aa-a968-3a9b64a6676f-kube-api-access-bkfb6\") pod \"13f8bac2-10c1-40aa-a968-3a9b64a6676f\" (UID: \"13f8bac2-10c1-40aa-a968-3a9b64a6676f\") "
Nov 02 13:00:14.060487 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:14.060457    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13f8bac2-10c1-40aa-a968-3a9b64a6676f-credential-socket" pod "13f8bac2-10c1-40aa-a968-3a9b64a6676f" (UID: "13f8bac2-10c1-40aa-a968-3a9b64a6676f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:00:14.060850 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:14.060824    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13f8bac2-10c1-40aa-a968-3a9b64a6676f-istio-data" pod "13f8bac2-10c1-40aa-a968-3a9b64a6676f" (UID: "13f8bac2-10c1-40aa-a968-3a9b64a6676f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:00:14.061061 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:14.060999    4559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13f8bac2-10c1-40aa-a968-3a9b64a6676f-credential-socket\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:00:14.061382 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:14.061361    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13f8bac2-10c1-40aa-a968-3a9b64a6676f-workload-certs" pod "13f8bac2-10c1-40aa-a968-3a9b64a6676f" (UID: "13f8bac2-10c1-40aa-a968-3a9b64a6676f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:00:14.061801 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:14.061776    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13f8bac2-10c1-40aa-a968-3a9b64a6676f-workload-socket" pod "13f8bac2-10c1-40aa-a968-3a9b64a6676f" (UID: "13f8bac2-10c1-40aa-a968-3a9b64a6676f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:00:14.062077 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:14.062055    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13f8bac2-10c1-40aa-a968-3a9b64a6676f-istiod-ca-cert" pod "13f8bac2-10c1-40aa-a968-3a9b64a6676f" (UID: "13f8bac2-10c1-40aa-a968-3a9b64a6676f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:00:14.065229 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:14.065204    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13f8bac2-10c1-40aa-a968-3a9b64a6676f-kube-api-access-bkfb6" pod "13f8bac2-10c1-40aa-a968-3a9b64a6676f" (UID: "13f8bac2-10c1-40aa-a968-3a9b64a6676f"). InnerVolumeSpecName "kube-api-access-bkfb6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:00:14.065338 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:14.065296    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13f8bac2-10c1-40aa-a968-3a9b64a6676f-istio-envoy" pod "13f8bac2-10c1-40aa-a968-3a9b64a6676f" (UID: "13f8bac2-10c1-40aa-a968-3a9b64a6676f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:00:14.065444 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:14.065424    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/13f8bac2-10c1-40aa-a968-3a9b64a6676f-istio-podinfo" pod "13f8bac2-10c1-40aa-a968-3a9b64a6676f" (UID: "13f8bac2-10c1-40aa-a968-3a9b64a6676f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 13:00:14.065606 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:14.065587    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13f8bac2-10c1-40aa-a968-3a9b64a6676f-istio-token" pod "13f8bac2-10c1-40aa-a968-3a9b64a6676f" (UID: "13f8bac2-10c1-40aa-a968-3a9b64a6676f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:00:14.162513 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:14.162433    4559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13f8bac2-10c1-40aa-a968-3a9b64a6676f-istio-token\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:00:14.162513 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:14.162461    4559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13f8bac2-10c1-40aa-a968-3a9b64a6676f-istio-data\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:00:14.162513 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:14.162469    4559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13f8bac2-10c1-40aa-a968-3a9b64a6676f-istio-envoy\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:00:14.162513 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:14.162477    4559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13f8bac2-10c1-40aa-a968-3a9b64a6676f-istio-podinfo\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:00:14.162513 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:14.162485    4559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13f8bac2-10c1-40aa-a968-3a9b64a6676f-workload-certs\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:00:14.162513 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:14.162493    4559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13f8bac2-10c1-40aa-a968-3a9b64a6676f-workload-socket\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:00:14.162513 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:14.162502    4559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13f8bac2-10c1-40aa-a968-3a9b64a6676f-istiod-ca-cert\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:00:14.162513 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:14.162510    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bkfb6\" (UniqueName: \"kubernetes.io/projected/13f8bac2-10c1-40aa-a968-3a9b64a6676f-kube-api-access-bkfb6\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:00:14.419526 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:14.419499    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-7qxgl"
Nov 02 13:00:14.419748 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:14.419541    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-7qxgl" event={"ID":"13f8bac2-10c1-40aa-a968-3a9b64a6676f","Type":"ContainerDied","Data":"e2551fb75e560198534bc03c43845072c09f4b3c37dd35f68e551e82a4383f63"}
Nov 02 13:00:14.419748 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:14.419603    4559 scope.go:122] "RemoveContainer" containerID="5a797e1223770184135d924d845ace05020d8abd016fbd85d030252a6e4bd07e"
Nov 02 13:00:14.421147 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:14.421128    4559 generic.go:356] "Generic (PLEG): container finished" podID="e8024fcc-5300-4b6c-84c0-7c2c30a0370a" containerID="7783ba0607706736796a59dcfae865bf0505f812f9c5dd2e56d7f63d6c008209" exitCode=0
Nov 02 13:00:14.421244 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:14.421180    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-8032f3/auth-test-model-8032f3-kserve-675786644f-2dmj4"
Nov 02 13:00:14.421244 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:14.421217    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-8032f3/auth-test-model-8032f3-kserve-675786644f-2dmj4" event={"ID":"e8024fcc-5300-4b6c-84c0-7c2c30a0370a","Type":"ContainerDied","Data":"7783ba0607706736796a59dcfae865bf0505f812f9c5dd2e56d7f63d6c008209"}
Nov 02 13:00:14.421327 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:14.421259    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-8032f3/auth-test-model-8032f3-kserve-675786644f-2dmj4" event={"ID":"e8024fcc-5300-4b6c-84c0-7c2c30a0370a","Type":"ContainerDied","Data":"ce7084a959e080e4a560c3456a2cd49f4c0dff45c5001d8222d0dbb425b5bfc2"}
Nov 02 13:00:14.421713 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:14.421684    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-ecec44-openshift-default-6d94dccbcf-thqw7"
Nov 02 13:00:14.423364 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:14.423342    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-ecec44-openshift-default-6d94dccbcf-thqw7"
Nov 02 13:00:14.430747 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:14.430731    4559 scope.go:122] "RemoveContainer" containerID="7783ba0607706736796a59dcfae865bf0505f812f9c5dd2e56d7f63d6c008209"
Nov 02 13:00:14.439870 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:14.439854    4559 scope.go:122] "RemoveContainer" containerID="7783ba0607706736796a59dcfae865bf0505f812f9c5dd2e56d7f63d6c008209"
Nov 02 13:00:14.440166 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:00:14.440147    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7783ba0607706736796a59dcfae865bf0505f812f9c5dd2e56d7f63d6c008209\": container with ID starting with 7783ba0607706736796a59dcfae865bf0505f812f9c5dd2e56d7f63d6c008209 not found: ID does not exist" containerID="7783ba0607706736796a59dcfae865bf0505f812f9c5dd2e56d7f63d6c008209"
Nov 02 13:00:14.440211 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:14.440175    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7783ba0607706736796a59dcfae865bf0505f812f9c5dd2e56d7f63d6c008209"} err="failed to get container status \"7783ba0607706736796a59dcfae865bf0505f812f9c5dd2e56d7f63d6c008209\": rpc error: code = NotFound desc = could not find container \"7783ba0607706736796a59dcfae865bf0505f812f9c5dd2e56d7f63d6c008209\": container with ID starting with 7783ba0607706736796a59dcfae865bf0505f812f9c5dd2e56d7f63d6c008209 not found: ID does not exist"
Nov 02 13:00:14.479954 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:14.479929    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-8032f3/auth-test-model-8032f3-kserve-675786644f-2dmj4"]
Nov 02 13:00:14.485350 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:14.485325    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w0-8032f3/auth-test-model-8032f3-kserve-675786644f-2dmj4"]
Nov 02 13:00:14.567111 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:14.567066    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-7qxgl"]
Nov 02 13:00:14.571164 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:14.571137    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-7qxgl"]
Nov 02 13:00:14.886359 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:14.886287    4559 scope.go:122] "RemoveContainer" containerID="a1f0d028ade8119dc00e4f45d1b9c06698653b852404aa5d037ede8d130d8dc3"
Nov 02 13:00:14.886784 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:14.886408    4559 scope.go:122] "RemoveContainer" containerID="ce9ee9c889881421b7c43eaa242b88531b79a0e3177a96c0374f33358292570c"
Nov 02 13:00:14.886784 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:14.886450    4559 scope.go:122] "RemoveContainer" containerID="5a51efe84d618a4ed68bef14f7278b7d29fc10c61954df31e314d47defc5851d"
Nov 02 13:00:15.151301 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:00:15.151220    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqcmj" containerName="extractor"
Nov 02 13:00:15.429696 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:15.429666    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893740-ljp9t" event={"ID":"c7eddad9-7bbd-4ef1-8d9c-3db6e4a58961","Type":"ContainerStarted","Data":"0e0870e2d28c0a44d6ecd086b9dd927368d2a61aa86820cf96255cdd143bb484"}
Nov 02 13:00:15.433250 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:15.433226    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e7561a-29893740-blswv" event={"ID":"a74304bb-3fb2-4dcd-be61-8299061bff15","Type":"ContainerStarted","Data":"cf69d14af140878f2f36ba5403a92dd0171d541008aee6e0b6000e2b48b34958"}
Nov 02 13:00:15.435093 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:15.435074    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5514e5-29893740-s9p6j" event={"ID":"3974db10-eea9-47c6-bbd4-557b4dca531f","Type":"ContainerStarted","Data":"91981bb17e83c749de96e932ee42f13e5bbf171b325eb8ce7a0062a2764dfd42"}
Nov 02 13:00:15.616587 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:15.616530    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e7561a-29893740-blswv" podStartSLOduration=15.616514291 podStartE2EDuration="15.616514291s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:00:15.61449555 +0000 UTC m=+2011.869894635" watchObservedRunningTime="2026-11-02 13:00:15.616514291 +0000 UTC m=+2011.871913383"
Nov 02 13:00:15.834469 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:15.834434    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-1e14d4-openshift-default-6dfdf7978c-jdj7d"]
Nov 02 13:00:15.835208 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:15.835184    4559 cpu_manager.go:414] "containerMap: removing container" podUID="13f8bac2-10c1-40aa-a968-3a9b64a6676f" containerName="istio-proxy"
Nov 02 13:00:15.835208 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:15.835205    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13f8bac2-10c1-40aa-a968-3a9b64a6676f" containerName="istio-proxy"
Nov 02 13:00:15.835365 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:15.835222    4559 cpu_manager.go:414] "containerMap: removing container" podUID="e8024fcc-5300-4b6c-84c0-7c2c30a0370a" containerName="main"
Nov 02 13:00:15.835365 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:15.835227    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8024fcc-5300-4b6c-84c0-7c2c30a0370a" containerName="main"
Nov 02 13:00:15.835434 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:15.835376    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="13f8bac2-10c1-40aa-a968-3a9b64a6676f" containerName="istio-proxy"
Nov 02 13:00:15.835434 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:15.835384    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8024fcc-5300-4b6c-84c0-7c2c30a0370a" containerName="main"
Nov 02 13:00:15.838876 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:15.838858    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-1e14d4-openshift-default-6dfdf7978c-jdj7d"
Nov 02 13:00:15.866281 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:15.866241    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-1e14d4-openshift-default-6dfdf7978c-jdj7d"]
Nov 02 13:00:15.918053 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:15.915998    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13f8bac2-10c1-40aa-a968-3a9b64a6676f" path="/var/lib/kubelet/pods/13f8bac2-10c1-40aa-a968-3a9b64a6676f/volumes"
Nov 02 13:00:15.918053 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:15.916903    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8024fcc-5300-4b6c-84c0-7c2c30a0370a" path="/var/lib/kubelet/pods/e8024fcc-5300-4b6c-84c0-7c2c30a0370a/volumes"
Nov 02 13:00:15.918053 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:15.917078    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e103054-2a54-43b1-b73d-a50a44c6492a-istiod-ca-cert\") pod \"e2e-shared-b-w2-1e14d4-openshift-default-6dfdf7978c-jdj7d\" (UID: \"4e103054-2a54-43b1-b73d-a50a44c6492a\") " pod="openshift-ingress/e2e-shared-b-w2-1e14d4-openshift-default-6dfdf7978c-jdj7d"
Nov 02 13:00:15.918053 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:15.917165    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e103054-2a54-43b1-b73d-a50a44c6492a-credential-socket\") pod \"e2e-shared-b-w2-1e14d4-openshift-default-6dfdf7978c-jdj7d\" (UID: \"4e103054-2a54-43b1-b73d-a50a44c6492a\") " pod="openshift-ingress/e2e-shared-b-w2-1e14d4-openshift-default-6dfdf7978c-jdj7d"
Nov 02 13:00:15.918053 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:15.917208    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e103054-2a54-43b1-b73d-a50a44c6492a-workload-socket\") pod \"e2e-shared-b-w2-1e14d4-openshift-default-6dfdf7978c-jdj7d\" (UID: \"4e103054-2a54-43b1-b73d-a50a44c6492a\") " pod="openshift-ingress/e2e-shared-b-w2-1e14d4-openshift-default-6dfdf7978c-jdj7d"
Nov 02 13:00:15.918053 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:15.917278    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e103054-2a54-43b1-b73d-a50a44c6492a-workload-certs\") pod \"e2e-shared-b-w2-1e14d4-openshift-default-6dfdf7978c-jdj7d\" (UID: \"4e103054-2a54-43b1-b73d-a50a44c6492a\") " pod="openshift-ingress/e2e-shared-b-w2-1e14d4-openshift-default-6dfdf7978c-jdj7d"
Nov 02 13:00:15.918053 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:15.917454    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e103054-2a54-43b1-b73d-a50a44c6492a-istio-podinfo\") pod \"e2e-shared-b-w2-1e14d4-openshift-default-6dfdf7978c-jdj7d\" (UID: \"4e103054-2a54-43b1-b73d-a50a44c6492a\") " pod="openshift-ingress/e2e-shared-b-w2-1e14d4-openshift-default-6dfdf7978c-jdj7d"
Nov 02 13:00:15.918053 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:15.917502    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e103054-2a54-43b1-b73d-a50a44c6492a-istio-envoy\") pod \"e2e-shared-b-w2-1e14d4-openshift-default-6dfdf7978c-jdj7d\" (UID: \"4e103054-2a54-43b1-b73d-a50a44c6492a\") " pod="openshift-ingress/e2e-shared-b-w2-1e14d4-openshift-default-6dfdf7978c-jdj7d"
Nov 02 13:00:15.918053 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:15.917559    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkj2c\" (UniqueName: \"kubernetes.io/projected/4e103054-2a54-43b1-b73d-a50a44c6492a-kube-api-access-nkj2c\") pod \"e2e-shared-b-w2-1e14d4-openshift-default-6dfdf7978c-jdj7d\" (UID: \"4e103054-2a54-43b1-b73d-a50a44c6492a\") " pod="openshift-ingress/e2e-shared-b-w2-1e14d4-openshift-default-6dfdf7978c-jdj7d"
Nov 02 13:00:15.918053 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:15.917624    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e103054-2a54-43b1-b73d-a50a44c6492a-istio-token\") pod \"e2e-shared-b-w2-1e14d4-openshift-default-6dfdf7978c-jdj7d\" (UID: \"4e103054-2a54-43b1-b73d-a50a44c6492a\") " pod="openshift-ingress/e2e-shared-b-w2-1e14d4-openshift-default-6dfdf7978c-jdj7d"
Nov 02 13:00:15.918053 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:15.917684    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e103054-2a54-43b1-b73d-a50a44c6492a-istio-data\") pod \"e2e-shared-b-w2-1e14d4-openshift-default-6dfdf7978c-jdj7d\" (UID: \"4e103054-2a54-43b1-b73d-a50a44c6492a\") " pod="openshift-ingress/e2e-shared-b-w2-1e14d4-openshift-default-6dfdf7978c-jdj7d"
Nov 02 13:00:15.935172 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:15.935125    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-1e14d4-openshift-default-dockercfg-9n6bp\""
Nov 02 13:00:16.019585 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:16.019549    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkj2c\" (UniqueName: \"kubernetes.io/projected/4e103054-2a54-43b1-b73d-a50a44c6492a-kube-api-access-nkj2c\") pod \"e2e-shared-b-w2-1e14d4-openshift-default-6dfdf7978c-jdj7d\" (UID: \"4e103054-2a54-43b1-b73d-a50a44c6492a\") " pod="openshift-ingress/e2e-shared-b-w2-1e14d4-openshift-default-6dfdf7978c-jdj7d"
Nov 02 13:00:16.019909 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:16.019605    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e103054-2a54-43b1-b73d-a50a44c6492a-istio-token\") pod \"e2e-shared-b-w2-1e14d4-openshift-default-6dfdf7978c-jdj7d\" (UID: \"4e103054-2a54-43b1-b73d-a50a44c6492a\") " pod="openshift-ingress/e2e-shared-b-w2-1e14d4-openshift-default-6dfdf7978c-jdj7d"
Nov 02 13:00:16.019909 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:16.019678    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e103054-2a54-43b1-b73d-a50a44c6492a-istio-data\") pod \"e2e-shared-b-w2-1e14d4-openshift-default-6dfdf7978c-jdj7d\" (UID: \"4e103054-2a54-43b1-b73d-a50a44c6492a\") " pod="openshift-ingress/e2e-shared-b-w2-1e14d4-openshift-default-6dfdf7978c-jdj7d"
Nov 02 13:00:16.019909 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:16.019792    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e103054-2a54-43b1-b73d-a50a44c6492a-istiod-ca-cert\") pod \"e2e-shared-b-w2-1e14d4-openshift-default-6dfdf7978c-jdj7d\" (UID: \"4e103054-2a54-43b1-b73d-a50a44c6492a\") " pod="openshift-ingress/e2e-shared-b-w2-1e14d4-openshift-default-6dfdf7978c-jdj7d"
Nov 02 13:00:16.019909 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:16.019861    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e103054-2a54-43b1-b73d-a50a44c6492a-credential-socket\") pod \"e2e-shared-b-w2-1e14d4-openshift-default-6dfdf7978c-jdj7d\" (UID: \"4e103054-2a54-43b1-b73d-a50a44c6492a\") " pod="openshift-ingress/e2e-shared-b-w2-1e14d4-openshift-default-6dfdf7978c-jdj7d"
Nov 02 13:00:16.020142 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:16.019900    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e103054-2a54-43b1-b73d-a50a44c6492a-workload-socket\") pod \"e2e-shared-b-w2-1e14d4-openshift-default-6dfdf7978c-jdj7d\" (UID: \"4e103054-2a54-43b1-b73d-a50a44c6492a\") " pod="openshift-ingress/e2e-shared-b-w2-1e14d4-openshift-default-6dfdf7978c-jdj7d"
Nov 02 13:00:16.020142 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:16.019935    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e103054-2a54-43b1-b73d-a50a44c6492a-workload-certs\") pod \"e2e-shared-b-w2-1e14d4-openshift-default-6dfdf7978c-jdj7d\" (UID: \"4e103054-2a54-43b1-b73d-a50a44c6492a\") " pod="openshift-ingress/e2e-shared-b-w2-1e14d4-openshift-default-6dfdf7978c-jdj7d"
Nov 02 13:00:16.020142 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:16.020059    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e103054-2a54-43b1-b73d-a50a44c6492a-istio-podinfo\") pod \"e2e-shared-b-w2-1e14d4-openshift-default-6dfdf7978c-jdj7d\" (UID: \"4e103054-2a54-43b1-b73d-a50a44c6492a\") " pod="openshift-ingress/e2e-shared-b-w2-1e14d4-openshift-default-6dfdf7978c-jdj7d"
Nov 02 13:00:16.020142 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:16.020101    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e103054-2a54-43b1-b73d-a50a44c6492a-istio-envoy\") pod \"e2e-shared-b-w2-1e14d4-openshift-default-6dfdf7978c-jdj7d\" (UID: \"4e103054-2a54-43b1-b73d-a50a44c6492a\") " pod="openshift-ingress/e2e-shared-b-w2-1e14d4-openshift-default-6dfdf7978c-jdj7d"
Nov 02 13:00:16.020349 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:16.020268    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e103054-2a54-43b1-b73d-a50a44c6492a-credential-socket\") pod \"e2e-shared-b-w2-1e14d4-openshift-default-6dfdf7978c-jdj7d\" (UID: \"4e103054-2a54-43b1-b73d-a50a44c6492a\") " pod="openshift-ingress/e2e-shared-b-w2-1e14d4-openshift-default-6dfdf7978c-jdj7d"
Nov 02 13:00:16.020349 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:16.020330    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e103054-2a54-43b1-b73d-a50a44c6492a-workload-certs\") pod \"e2e-shared-b-w2-1e14d4-openshift-default-6dfdf7978c-jdj7d\" (UID: \"4e103054-2a54-43b1-b73d-a50a44c6492a\") " pod="openshift-ingress/e2e-shared-b-w2-1e14d4-openshift-default-6dfdf7978c-jdj7d"
Nov 02 13:00:16.020493 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:16.020360    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e103054-2a54-43b1-b73d-a50a44c6492a-workload-socket\") pod \"e2e-shared-b-w2-1e14d4-openshift-default-6dfdf7978c-jdj7d\" (UID: \"4e103054-2a54-43b1-b73d-a50a44c6492a\") " pod="openshift-ingress/e2e-shared-b-w2-1e14d4-openshift-default-6dfdf7978c-jdj7d"
Nov 02 13:00:16.020493 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:16.020436    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e103054-2a54-43b1-b73d-a50a44c6492a-istio-data\") pod \"e2e-shared-b-w2-1e14d4-openshift-default-6dfdf7978c-jdj7d\" (UID: \"4e103054-2a54-43b1-b73d-a50a44c6492a\") " pod="openshift-ingress/e2e-shared-b-w2-1e14d4-openshift-default-6dfdf7978c-jdj7d"
Nov 02 13:00:16.020616 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:16.020597    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e103054-2a54-43b1-b73d-a50a44c6492a-istiod-ca-cert\") pod \"e2e-shared-b-w2-1e14d4-openshift-default-6dfdf7978c-jdj7d\" (UID: \"4e103054-2a54-43b1-b73d-a50a44c6492a\") " pod="openshift-ingress/e2e-shared-b-w2-1e14d4-openshift-default-6dfdf7978c-jdj7d"
Nov 02 13:00:16.022528 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:16.022507    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e103054-2a54-43b1-b73d-a50a44c6492a-istio-envoy\") pod \"e2e-shared-b-w2-1e14d4-openshift-default-6dfdf7978c-jdj7d\" (UID: \"4e103054-2a54-43b1-b73d-a50a44c6492a\") " pod="openshift-ingress/e2e-shared-b-w2-1e14d4-openshift-default-6dfdf7978c-jdj7d"
Nov 02 13:00:16.022681 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:16.022660    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e103054-2a54-43b1-b73d-a50a44c6492a-istio-podinfo\") pod \"e2e-shared-b-w2-1e14d4-openshift-default-6dfdf7978c-jdj7d\" (UID: \"4e103054-2a54-43b1-b73d-a50a44c6492a\") " pod="openshift-ingress/e2e-shared-b-w2-1e14d4-openshift-default-6dfdf7978c-jdj7d"
Nov 02 13:00:16.047474 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:16.047444    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkj2c\" (UniqueName: \"kubernetes.io/projected/4e103054-2a54-43b1-b73d-a50a44c6492a-kube-api-access-nkj2c\") pod \"e2e-shared-b-w2-1e14d4-openshift-default-6dfdf7978c-jdj7d\" (UID: \"4e103054-2a54-43b1-b73d-a50a44c6492a\") " pod="openshift-ingress/e2e-shared-b-w2-1e14d4-openshift-default-6dfdf7978c-jdj7d"
Nov 02 13:00:16.047681 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:16.047451    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e103054-2a54-43b1-b73d-a50a44c6492a-istio-token\") pod \"e2e-shared-b-w2-1e14d4-openshift-default-6dfdf7978c-jdj7d\" (UID: \"4e103054-2a54-43b1-b73d-a50a44c6492a\") " pod="openshift-ingress/e2e-shared-b-w2-1e14d4-openshift-default-6dfdf7978c-jdj7d"
Nov 02 13:00:16.153976 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:16.153941    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-1e14d4-openshift-default-6dfdf7978c-jdj7d"
Nov 02 13:00:16.445563 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:16.445529    4559 generic.go:356] "Generic (PLEG): container finished" podID="a74304bb-3fb2-4dcd-be61-8299061bff15" containerID="cf69d14af140878f2f36ba5403a92dd0171d541008aee6e0b6000e2b48b34958" exitCode=60
Nov 02 13:00:16.445778 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:16.445603    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e7561a-29893740-blswv" event={"ID":"a74304bb-3fb2-4dcd-be61-8299061bff15","Type":"ContainerDied","Data":"cf69d14af140878f2f36ba5403a92dd0171d541008aee6e0b6000e2b48b34958"}
Nov 02 13:00:16.445778 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:16.445689    4559 scope.go:122] "RemoveContainer" containerID="5a51efe84d618a4ed68bef14f7278b7d29fc10c61954df31e314d47defc5851d"
Nov 02 13:00:16.446075 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:16.446060    4559 scope.go:122] "RemoveContainer" containerID="cf69d14af140878f2f36ba5403a92dd0171d541008aee6e0b6000e2b48b34958"
Nov 02 13:00:16.446363 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:00:16.446342    4559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-e7561a-29893740-blswv_odh-ai-gateway-infra(a74304bb-3fb2-4dcd-be61-8299061bff15)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e7561a-29893740-blswv" podUID="a74304bb-3fb2-4dcd-be61-8299061bff15"
Nov 02 13:00:16.447525 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:16.447502    4559 generic.go:356] "Generic (PLEG): container finished" podID="3974db10-eea9-47c6-bbd4-557b4dca531f" containerID="91981bb17e83c749de96e932ee42f13e5bbf171b325eb8ce7a0062a2764dfd42" exitCode=60
Nov 02 13:00:16.447650 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:16.447535    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5514e5-29893740-s9p6j" event={"ID":"3974db10-eea9-47c6-bbd4-557b4dca531f","Type":"ContainerDied","Data":"91981bb17e83c749de96e932ee42f13e5bbf171b325eb8ce7a0062a2764dfd42"}
Nov 02 13:00:16.447919 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:16.447905    4559 scope.go:122] "RemoveContainer" containerID="91981bb17e83c749de96e932ee42f13e5bbf171b325eb8ce7a0062a2764dfd42"
Nov 02 13:00:16.448229 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:00:16.448207    4559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-5514e5-29893740-s9p6j_odh-ai-gateway-infra(3974db10-eea9-47c6-bbd4-557b4dca531f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5514e5-29893740-s9p6j" podUID="3974db10-eea9-47c6-bbd4-557b4dca531f"
Nov 02 13:00:16.449514 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:16.449496    4559 generic.go:356] "Generic (PLEG): container finished" podID="c7eddad9-7bbd-4ef1-8d9c-3db6e4a58961" containerID="0e0870e2d28c0a44d6ecd086b9dd927368d2a61aa86820cf96255cdd143bb484" exitCode=60
Nov 02 13:00:16.449615 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:16.449575    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893740-ljp9t" event={"ID":"c7eddad9-7bbd-4ef1-8d9c-3db6e4a58961","Type":"ContainerDied","Data":"0e0870e2d28c0a44d6ecd086b9dd927368d2a61aa86820cf96255cdd143bb484"}
Nov 02 13:00:16.450046 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:16.450029    4559 scope.go:122] "RemoveContainer" containerID="0e0870e2d28c0a44d6ecd086b9dd927368d2a61aa86820cf96255cdd143bb484"
Nov 02 13:00:16.450298 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:00:16.450277    4559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29893740-ljp9t_odh-ai-gateway-infra(c7eddad9-7bbd-4ef1-8d9c-3db6e4a58961)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893740-ljp9t" podUID="c7eddad9-7bbd-4ef1-8d9c-3db6e4a58961"
Nov 02 13:00:16.459702 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:16.459681    4559 scope.go:122] "RemoveContainer" containerID="ce9ee9c889881421b7c43eaa242b88531b79a0e3177a96c0374f33358292570c"
Nov 02 13:00:16.476485 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:16.476433    4559 scope.go:122] "RemoveContainer" containerID="a1f0d028ade8119dc00e4f45d1b9c06698653b852404aa5d037ede8d130d8dc3"
Nov 02 13:00:16.481186 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:16.481163    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-1e14d4-openshift-default-6dfdf7978c-jdj7d"]
Nov 02 13:00:16.482038 ip-10-0-128-132 kubenswrapper[4559]: W1102 13:00:16.482011    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e103054_2a54_43b1_b73d_a50a44c6492a.slice/crio-295b1c068adc8c3eac7ec798eed201ae82f7c62bc31eb86c14cb0e9518b139df WatchSource:0}: Error finding container 295b1c068adc8c3eac7ec798eed201ae82f7c62bc31eb86c14cb0e9518b139df: Status 404 returned error can't find the container with id 295b1c068adc8c3eac7ec798eed201ae82f7c62bc31eb86c14cb0e9518b139df
Nov 02 13:00:16.484732 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:16.484703    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 13:00:16.484814 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:16.484765    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 13:00:16.484814 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:16.484801    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 13:00:16.499590 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:16.499562    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893740-ljp9t"]
Nov 02 13:00:16.541555 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:16.541301    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5514e5-29893740-s9p6j"]
Nov 02 13:00:16.629663 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:16.629607    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e7561a-29893740-blswv"]
Nov 02 13:00:17.470798 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:17.470761    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-1e14d4-openshift-default-6dfdf7978c-jdj7d" event={"ID":"4e103054-2a54-43b1-b73d-a50a44c6492a","Type":"ContainerStarted","Data":"31e13065cd3b65f6ca84a37466fe3229b7b91b31ce7ee35d6c1ef1b655cabfe6"}
Nov 02 13:00:17.471179 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:17.470828    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-1e14d4-openshift-default-6dfdf7978c-jdj7d" event={"ID":"4e103054-2a54-43b1-b73d-a50a44c6492a","Type":"ContainerStarted","Data":"295b1c068adc8c3eac7ec798eed201ae82f7c62bc31eb86c14cb0e9518b139df"}
Nov 02 13:00:17.668811 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:17.668719    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-1e14d4-openshift-default-6dfdf7978c-jdj7d" podStartSLOduration=2.668697892 podStartE2EDuration="2.668697892s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:00:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:00:17.665717299 +0000 UTC m=+2013.921116397" watchObservedRunningTime="2026-11-02 13:00:17.668697892 +0000 UTC m=+2013.924096989"
Nov 02 13:00:17.840336 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:17.840301    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5514e5-29893740-s9p6j"
Nov 02 13:00:17.872535 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:17.872505    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893740-ljp9t"
Nov 02 13:00:17.875434 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:17.875412    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e7561a-29893740-blswv"
Nov 02 13:00:17.954931 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:17.954894    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7eddad9-7bbd-4ef1-8d9c-3db6e4a58961-kube-api-access-j5q6g\" (UniqueName: \"kubernetes.io/projected/c7eddad9-7bbd-4ef1-8d9c-3db6e4a58961-kube-api-access-j5q6g\") pod \"c7eddad9-7bbd-4ef1-8d9c-3db6e4a58961\" (UID: \"c7eddad9-7bbd-4ef1-8d9c-3db6e4a58961\") "
Nov 02 13:00:17.955134 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:17.954944    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3974db10-eea9-47c6-bbd4-557b4dca531f-kube-api-access-4mk5v\" (UniqueName: \"kubernetes.io/projected/3974db10-eea9-47c6-bbd4-557b4dca531f-kube-api-access-4mk5v\") pod \"3974db10-eea9-47c6-bbd4-557b4dca531f\" (UID: \"3974db10-eea9-47c6-bbd4-557b4dca531f\") "
Nov 02 13:00:17.955134 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:17.955127    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a74304bb-3fb2-4dcd-be61-8299061bff15-kube-api-access-rvvgt\" (UniqueName: \"kubernetes.io/projected/a74304bb-3fb2-4dcd-be61-8299061bff15-kube-api-access-rvvgt\") pod \"a74304bb-3fb2-4dcd-be61-8299061bff15\" (UID: \"a74304bb-3fb2-4dcd-be61-8299061bff15\") "
Nov 02 13:00:17.955269 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:17.955220    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7eddad9-7bbd-4ef1-8d9c-3db6e4a58961-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7eddad9-7bbd-4ef1-8d9c-3db6e4a58961-maas-api-service-ca\") pod \"c7eddad9-7bbd-4ef1-8d9c-3db6e4a58961\" (UID: \"c7eddad9-7bbd-4ef1-8d9c-3db6e4a58961\") "
Nov 02 13:00:17.955269 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:17.955242    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3974db10-eea9-47c6-bbd4-557b4dca531f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3974db10-eea9-47c6-bbd4-557b4dca531f-maas-api-service-ca\") pod \"3974db10-eea9-47c6-bbd4-557b4dca531f\" (UID: \"3974db10-eea9-47c6-bbd4-557b4dca531f\") "
Nov 02 13:00:17.955269 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:17.955265    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a74304bb-3fb2-4dcd-be61-8299061bff15-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a74304bb-3fb2-4dcd-be61-8299061bff15-maas-api-service-ca\") pod \"a74304bb-3fb2-4dcd-be61-8299061bff15\" (UID: \"a74304bb-3fb2-4dcd-be61-8299061bff15\") "
Nov 02 13:00:17.955990 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:17.955962    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a74304bb-3fb2-4dcd-be61-8299061bff15-maas-api-service-ca" pod "a74304bb-3fb2-4dcd-be61-8299061bff15" (UID: "a74304bb-3fb2-4dcd-be61-8299061bff15"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:00:17.956125 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:17.955958    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3974db10-eea9-47c6-bbd4-557b4dca531f-maas-api-service-ca" pod "3974db10-eea9-47c6-bbd4-557b4dca531f" (UID: "3974db10-eea9-47c6-bbd4-557b4dca531f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:00:17.956125 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:17.955970    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7eddad9-7bbd-4ef1-8d9c-3db6e4a58961-maas-api-service-ca" pod "c7eddad9-7bbd-4ef1-8d9c-3db6e4a58961" (UID: "c7eddad9-7bbd-4ef1-8d9c-3db6e4a58961"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:00:17.958036 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:17.958013    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a74304bb-3fb2-4dcd-be61-8299061bff15-kube-api-access-rvvgt" pod "a74304bb-3fb2-4dcd-be61-8299061bff15" (UID: "a74304bb-3fb2-4dcd-be61-8299061bff15"). InnerVolumeSpecName "kube-api-access-rvvgt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:00:17.961674 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:17.958184    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3974db10-eea9-47c6-bbd4-557b4dca531f-kube-api-access-4mk5v" pod "3974db10-eea9-47c6-bbd4-557b4dca531f" (UID: "3974db10-eea9-47c6-bbd4-557b4dca531f"). InnerVolumeSpecName "kube-api-access-4mk5v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:00:17.961674 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:17.959075    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7eddad9-7bbd-4ef1-8d9c-3db6e4a58961-kube-api-access-j5q6g" pod "c7eddad9-7bbd-4ef1-8d9c-3db6e4a58961" (UID: "c7eddad9-7bbd-4ef1-8d9c-3db6e4a58961"). InnerVolumeSpecName "kube-api-access-j5q6g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:00:18.057094 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:18.057004    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7eddad9-7bbd-4ef1-8d9c-3db6e4a58961-maas-api-service-ca\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:00:18.057094 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:18.057039    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3974db10-eea9-47c6-bbd4-557b4dca531f-maas-api-service-ca\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:00:18.057094 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:18.057057    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a74304bb-3fb2-4dcd-be61-8299061bff15-maas-api-service-ca\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:00:18.057094 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:18.057072    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j5q6g\" (UniqueName: \"kubernetes.io/projected/c7eddad9-7bbd-4ef1-8d9c-3db6e4a58961-kube-api-access-j5q6g\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:00:18.057094 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:18.057087    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4mk5v\" (UniqueName: \"kubernetes.io/projected/3974db10-eea9-47c6-bbd4-557b4dca531f-kube-api-access-4mk5v\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:00:18.057386 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:18.057101    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvvgt\" (UniqueName: \"kubernetes.io/projected/a74304bb-3fb2-4dcd-be61-8299061bff15-kube-api-access-rvvgt\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:00:18.154954 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:18.154914    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-1e14d4-openshift-default-6dfdf7978c-jdj7d"
Nov 02 13:00:18.156789 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:18.156759    4559 patch_prober.go:29] interesting pod/e2e-shared-b-w2-1e14d4-openshift-default-6dfdf7978c-jdj7d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body=
Nov 02 13:00:18.156958 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:18.156809    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-1e14d4-openshift-default-6dfdf7978c-jdj7d" podUID="4e103054-2a54-43b1-b73d-a50a44c6492a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused"
Nov 02 13:00:18.481805 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:18.481772    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e7561a-29893740-blswv"
Nov 02 13:00:18.482256 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:18.481768    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e7561a-29893740-blswv" event={"ID":"a74304bb-3fb2-4dcd-be61-8299061bff15","Type":"ContainerDied","Data":"0359b137d8d4a0b63632e56b4b84056223168a4b7d2e8506cdbd8461f810c8e7"}
Nov 02 13:00:18.482256 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:18.481898    4559 scope.go:122] "RemoveContainer" containerID="cf69d14af140878f2f36ba5403a92dd0171d541008aee6e0b6000e2b48b34958"
Nov 02 13:00:18.508999 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:18.508968    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5514e5-29893740-s9p6j"
Nov 02 13:00:18.508999 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:18.508978    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5514e5-29893740-s9p6j" event={"ID":"3974db10-eea9-47c6-bbd4-557b4dca531f","Type":"ContainerDied","Data":"98ed8c1f631a7804f8426ec6e22caeb071da2ef91f62609c76807e88c3d6c308"}
Nov 02 13:00:18.511174 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:18.511131    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893740-ljp9t"
Nov 02 13:00:18.511297 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:18.511146    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893740-ljp9t" event={"ID":"c7eddad9-7bbd-4ef1-8d9c-3db6e4a58961","Type":"ContainerDied","Data":"ac4b30a474d6ffc2f8ad961eaa78c14295c3e7efbf2d33a6d02071d311576d0e"}
Nov 02 13:00:18.521985 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:18.521962    4559 scope.go:122] "RemoveContainer" containerID="91981bb17e83c749de96e932ee42f13e5bbf171b325eb8ce7a0062a2764dfd42"
Nov 02 13:00:18.532069 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:18.532054    4559 scope.go:122] "RemoveContainer" containerID="0e0870e2d28c0a44d6ecd086b9dd927368d2a61aa86820cf96255cdd143bb484"
Nov 02 13:00:18.563935 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:18.563905    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e7561a-29893740-blswv"]
Nov 02 13:00:18.576403 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:18.576363    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e7561a-29893740-blswv"]
Nov 02 13:00:18.614911 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:18.614878    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5514e5-29893740-s9p6j"]
Nov 02 13:00:18.618267 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:18.618231    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5514e5-29893740-s9p6j"]
Nov 02 13:00:18.652029 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:18.651998    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893740-ljp9t"]
Nov 02 13:00:18.664650 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:18.664608    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893740-ljp9t"]
Nov 02 13:00:18.819726 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:18.819628    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-e430c7-75f68d8dd5-68xr4"]
Nov 02 13:00:18.861753 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:18.861722    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-e430c7-85548db555-r7xkb"]
Nov 02 13:00:19.160730 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:19.158933    4559 patch_prober.go:29] interesting pod/e2e-shared-b-w2-1e14d4-openshift-default-6dfdf7978c-jdj7d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body=
Nov 02 13:00:19.160730 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:19.159032    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-1e14d4-openshift-default-6dfdf7978c-jdj7d" podUID="4e103054-2a54-43b1-b73d-a50a44c6492a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused"
Nov 02 13:00:19.916362 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:19.916327    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3974db10-eea9-47c6-bbd4-557b4dca531f" path="/var/lib/kubelet/pods/3974db10-eea9-47c6-bbd4-557b4dca531f/volumes"
Nov 02 13:00:19.916882 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:19.916866    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a74304bb-3fb2-4dcd-be61-8299061bff15" path="/var/lib/kubelet/pods/a74304bb-3fb2-4dcd-be61-8299061bff15/volumes"
Nov 02 13:00:19.917262 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:19.917250    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7eddad9-7bbd-4ef1-8d9c-3db6e4a58961" path="/var/lib/kubelet/pods/c7eddad9-7bbd-4ef1-8d9c-3db6e4a58961/volumes"
Nov 02 13:00:20.154899 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:20.154857    4559 patch_prober.go:29] interesting pod/e2e-shared-b-w2-1e14d4-openshift-default-6dfdf7978c-jdj7d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body=
Nov 02 13:00:20.155098 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:20.154944    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-1e14d4-openshift-default-6dfdf7978c-jdj7d" podUID="4e103054-2a54-43b1-b73d-a50a44c6492a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused"
Nov 02 13:00:21.155111 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:21.155070    4559 patch_prober.go:29] interesting pod/e2e-shared-b-w2-1e14d4-openshift-default-6dfdf7978c-jdj7d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body=
Nov 02 13:00:21.155626 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:21.155135    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-1e14d4-openshift-default-6dfdf7978c-jdj7d" podUID="4e103054-2a54-43b1-b73d-a50a44c6492a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused"
Nov 02 13:00:22.155102 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:22.155058    4559 patch_prober.go:29] interesting pod/e2e-shared-b-w2-1e14d4-openshift-default-6dfdf7978c-jdj7d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body=
Nov 02 13:00:22.155441 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:22.155401    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-1e14d4-openshift-default-6dfdf7978c-jdj7d" podUID="4e103054-2a54-43b1-b73d-a50a44c6492a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused"
Nov 02 13:00:23.154811 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:23.154768    4559 patch_prober.go:29] interesting pod/e2e-shared-b-w2-1e14d4-openshift-default-6dfdf7978c-jdj7d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body=
Nov 02 13:00:23.155012 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:23.154850    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-1e14d4-openshift-default-6dfdf7978c-jdj7d" podUID="4e103054-2a54-43b1-b73d-a50a44c6492a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused"
Nov 02 13:00:23.820289 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:23.820240    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-e430c7-75f68d8dd5-68xr4" podUID="0cf4374b-846c-4a69-872f-6c03ed423b41" containerName="payload-processing" containerID="cri-o://d7178ca335fa285f4abfbc31c99174acc5d52716d84961cf94c43e6febf849bd" gracePeriod=25
Nov 02 13:00:23.862371 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:23.862325    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-e430c7-85548db555-r7xkb" podUID="d9e9675e-a96b-4f43-a253-608bcd468854" containerName="payload-pre-processing" containerID="cri-o://dc249e4d6aa08c3984d95bc80dc2d9aff90d393b2425501d611b652a5ac230e8" gracePeriod=25
Nov 02 13:00:24.158314 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:24.158282    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-e430c7-75f68d8dd5-68xr4"
Nov 02 13:00:24.160230 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:24.160204    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-1e14d4-openshift-default-6dfdf7978c-jdj7d"
Nov 02 13:00:24.160584 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:24.160538    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-1e14d4-openshift-default-6dfdf7978c-jdj7d"
Nov 02 13:00:24.161625 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:24.161592    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-1e14d4-openshift-default-6dfdf7978c-jdj7d"
Nov 02 13:00:24.193978 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:24.193956    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-e430c7-85548db555-r7xkb"
Nov 02 13:00:24.338472 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:24.338348    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0cf4374b-846c-4a69-872f-6c03ed423b41-kube-api-access-cbdv6\" (UniqueName: \"kubernetes.io/projected/0cf4374b-846c-4a69-872f-6c03ed423b41-kube-api-access-cbdv6\") pod \"0cf4374b-846c-4a69-872f-6c03ed423b41\" (UID: \"0cf4374b-846c-4a69-872f-6c03ed423b41\") "
Nov 02 13:00:24.338472 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:24.338436    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9e9675e-a96b-4f43-a253-608bcd468854-kube-api-access-c4j7h\" (UniqueName: \"kubernetes.io/projected/d9e9675e-a96b-4f43-a253-608bcd468854-kube-api-access-c4j7h\") pod \"d9e9675e-a96b-4f43-a253-608bcd468854\" (UID: \"d9e9675e-a96b-4f43-a253-608bcd468854\") "
Nov 02 13:00:24.339043 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:24.338555    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0cf4374b-846c-4a69-872f-6c03ed423b41-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0cf4374b-846c-4a69-872f-6c03ed423b41-plugins-config-volume\") pod \"0cf4374b-846c-4a69-872f-6c03ed423b41\" (UID: \"0cf4374b-846c-4a69-872f-6c03ed423b41\") "
Nov 02 13:00:24.339043 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:24.338798    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9e9675e-a96b-4f43-a253-608bcd468854-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9e9675e-a96b-4f43-a253-608bcd468854-plugins-config-volume\") pod \"d9e9675e-a96b-4f43-a253-608bcd468854\" (UID: \"d9e9675e-a96b-4f43-a253-608bcd468854\") "
Nov 02 13:00:24.364977 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:24.360897    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cf4374b-846c-4a69-872f-6c03ed423b41-kube-api-access-cbdv6" pod "0cf4374b-846c-4a69-872f-6c03ed423b41" (UID: "0cf4374b-846c-4a69-872f-6c03ed423b41"). InnerVolumeSpecName "kube-api-access-cbdv6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:00:24.364977 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:24.361093    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cf4374b-846c-4a69-872f-6c03ed423b41-plugins-config-volume" pod "0cf4374b-846c-4a69-872f-6c03ed423b41" (UID: "0cf4374b-846c-4a69-872f-6c03ed423b41"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:00:24.364977 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:24.361149    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9e9675e-a96b-4f43-a253-608bcd468854-plugins-config-volume" pod "d9e9675e-a96b-4f43-a253-608bcd468854" (UID: "d9e9675e-a96b-4f43-a253-608bcd468854"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:00:24.364977 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:24.361582    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9e9675e-a96b-4f43-a253-608bcd468854-kube-api-access-c4j7h" pod "d9e9675e-a96b-4f43-a253-608bcd468854" (UID: "d9e9675e-a96b-4f43-a253-608bcd468854"). InnerVolumeSpecName "kube-api-access-c4j7h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:00:24.441201 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:24.441166    4559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9e9675e-a96b-4f43-a253-608bcd468854-plugins-config-volume\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:00:24.441201 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:24.441195    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cbdv6\" (UniqueName: \"kubernetes.io/projected/0cf4374b-846c-4a69-872f-6c03ed423b41-kube-api-access-cbdv6\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:00:24.441201 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:24.441206    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c4j7h\" (UniqueName: \"kubernetes.io/projected/d9e9675e-a96b-4f43-a253-608bcd468854-kube-api-access-c4j7h\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:00:24.441451 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:24.441216    4559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0cf4374b-846c-4a69-872f-6c03ed423b41-plugins-config-volume\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:00:24.615743 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:24.615629    4559 generic.go:356] "Generic (PLEG): container finished" podID="0cf4374b-846c-4a69-872f-6c03ed423b41" containerID="d7178ca335fa285f4abfbc31c99174acc5d52716d84961cf94c43e6febf849bd" exitCode=0
Nov 02 13:00:24.615743 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:24.615670    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-e430c7-75f68d8dd5-68xr4" event={"ID":"0cf4374b-846c-4a69-872f-6c03ed423b41","Type":"ContainerDied","Data":"d7178ca335fa285f4abfbc31c99174acc5d52716d84961cf94c43e6febf849bd"}
Nov 02 13:00:24.615743 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:24.615695    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-e430c7-75f68d8dd5-68xr4"
Nov 02 13:00:24.615743 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:24.615725    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-e430c7-75f68d8dd5-68xr4" event={"ID":"0cf4374b-846c-4a69-872f-6c03ed423b41","Type":"ContainerDied","Data":"0f25d9420c18418724580c85e491d1831255c142430efe2acb2442a6ea120d40"}
Nov 02 13:00:24.616088 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:24.615747    4559 scope.go:122] "RemoveContainer" containerID="d7178ca335fa285f4abfbc31c99174acc5d52716d84961cf94c43e6febf849bd"
Nov 02 13:00:24.617377 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:24.617358    4559 generic.go:356] "Generic (PLEG): container finished" podID="d9e9675e-a96b-4f43-a253-608bcd468854" containerID="dc249e4d6aa08c3984d95bc80dc2d9aff90d393b2425501d611b652a5ac230e8" exitCode=0
Nov 02 13:00:24.617510 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:24.617406    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-e430c7-85548db555-r7xkb"
Nov 02 13:00:24.617510 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:24.617443    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-e430c7-85548db555-r7xkb" event={"ID":"d9e9675e-a96b-4f43-a253-608bcd468854","Type":"ContainerDied","Data":"dc249e4d6aa08c3984d95bc80dc2d9aff90d393b2425501d611b652a5ac230e8"}
Nov 02 13:00:24.617510 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:24.617487    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-e430c7-85548db555-r7xkb" event={"ID":"d9e9675e-a96b-4f43-a253-608bcd468854","Type":"ContainerDied","Data":"410caee7de487c59578d762cb4384d45def6a64b3ef92b71bf44fffcebb25178"}
Nov 02 13:00:24.630424 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:24.630310    4559 scope.go:122] "RemoveContainer" containerID="d7178ca335fa285f4abfbc31c99174acc5d52716d84961cf94c43e6febf849bd"
Nov 02 13:00:24.630724 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:00:24.630701    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7178ca335fa285f4abfbc31c99174acc5d52716d84961cf94c43e6febf849bd\": container with ID starting with d7178ca335fa285f4abfbc31c99174acc5d52716d84961cf94c43e6febf849bd not found: ID does not exist" containerID="d7178ca335fa285f4abfbc31c99174acc5d52716d84961cf94c43e6febf849bd"
Nov 02 13:00:24.630812 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:24.630732    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7178ca335fa285f4abfbc31c99174acc5d52716d84961cf94c43e6febf849bd"} err="failed to get container status \"d7178ca335fa285f4abfbc31c99174acc5d52716d84961cf94c43e6febf849bd\": rpc error: code = NotFound desc = could not find container \"d7178ca335fa285f4abfbc31c99174acc5d52716d84961cf94c43e6febf849bd\": container with ID starting with d7178ca335fa285f4abfbc31c99174acc5d52716d84961cf94c43e6febf849bd not found: ID does not exist"
Nov 02 13:00:24.630812 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:24.630752    4559 scope.go:122] "RemoveContainer" containerID="dc249e4d6aa08c3984d95bc80dc2d9aff90d393b2425501d611b652a5ac230e8"
Nov 02 13:00:24.641014 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:24.640987    4559 scope.go:122] "RemoveContainer" containerID="dc249e4d6aa08c3984d95bc80dc2d9aff90d393b2425501d611b652a5ac230e8"
Nov 02 13:00:24.641278 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:00:24.641259    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc249e4d6aa08c3984d95bc80dc2d9aff90d393b2425501d611b652a5ac230e8\": container with ID starting with dc249e4d6aa08c3984d95bc80dc2d9aff90d393b2425501d611b652a5ac230e8 not found: ID does not exist" containerID="dc249e4d6aa08c3984d95bc80dc2d9aff90d393b2425501d611b652a5ac230e8"
Nov 02 13:00:24.641327 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:24.641284    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc249e4d6aa08c3984d95bc80dc2d9aff90d393b2425501d611b652a5ac230e8"} err="failed to get container status \"dc249e4d6aa08c3984d95bc80dc2d9aff90d393b2425501d611b652a5ac230e8\": rpc error: code = NotFound desc = could not find container \"dc249e4d6aa08c3984d95bc80dc2d9aff90d393b2425501d611b652a5ac230e8\": container with ID starting with dc249e4d6aa08c3984d95bc80dc2d9aff90d393b2425501d611b652a5ac230e8 not found: ID does not exist"
Nov 02 13:00:24.687867 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:24.687837    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-e430c7-75f68d8dd5-68xr4"]
Nov 02 13:00:24.693337 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:24.693308    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-e430c7-75f68d8dd5-68xr4"]
Nov 02 13:00:24.728753 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:24.728721    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-e430c7-85548db555-r7xkb"]
Nov 02 13:00:24.731605 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:24.731581    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-e430c7-85548db555-r7xkb"]
Nov 02 13:00:25.372420 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:25.372374    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w0-43be82/auth-test-model-43be82-kserve-5c6dfd4dfc-n4dl7" podUID="750b979e-ef5c-4c06-b456-d1c6cdce27e5" containerName="main" containerID="cri-o://7c04f3ca2a3fe91f92a0ed51e71dd8fca1238eb7ac2fbe84f9813f0388d570cb" gracePeriod=45
Nov 02 13:00:25.638404 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:25.638370    4559 generic.go:356] "Generic (PLEG): container finished" podID="750b979e-ef5c-4c06-b456-d1c6cdce27e5" containerID="7c04f3ca2a3fe91f92a0ed51e71dd8fca1238eb7ac2fbe84f9813f0388d570cb" exitCode=0
Nov 02 13:00:25.638698 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:25.638439    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-43be82/auth-test-model-43be82-kserve-5c6dfd4dfc-n4dl7" event={"ID":"750b979e-ef5c-4c06-b456-d1c6cdce27e5","Type":"ContainerDied","Data":"7c04f3ca2a3fe91f92a0ed51e71dd8fca1238eb7ac2fbe84f9813f0388d570cb"}
Nov 02 13:00:25.656897 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:25.656868    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-43be82/auth-test-model-43be82-kserve-5c6dfd4dfc-n4dl7"
Nov 02 13:00:25.758316 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:25.758281    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/750b979e-ef5c-4c06-b456-d1c6cdce27e5-home\" (UniqueName: \"kubernetes.io/empty-dir/750b979e-ef5c-4c06-b456-d1c6cdce27e5-home\") pod \"750b979e-ef5c-4c06-b456-d1c6cdce27e5\" (UID: \"750b979e-ef5c-4c06-b456-d1c6cdce27e5\") "
Nov 02 13:00:25.758498 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:25.758329    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/750b979e-ef5c-4c06-b456-d1c6cdce27e5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/750b979e-ef5c-4c06-b456-d1c6cdce27e5-dshm\") pod \"750b979e-ef5c-4c06-b456-d1c6cdce27e5\" (UID: \"750b979e-ef5c-4c06-b456-d1c6cdce27e5\") "
Nov 02 13:00:25.758498 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:25.758381    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/750b979e-ef5c-4c06-b456-d1c6cdce27e5-tls-certs\" (UniqueName: \"kubernetes.io/secret/750b979e-ef5c-4c06-b456-d1c6cdce27e5-tls-certs\") pod \"750b979e-ef5c-4c06-b456-d1c6cdce27e5\" (UID: \"750b979e-ef5c-4c06-b456-d1c6cdce27e5\") "
Nov 02 13:00:25.758498 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:25.758402    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/750b979e-ef5c-4c06-b456-d1c6cdce27e5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/750b979e-ef5c-4c06-b456-d1c6cdce27e5-model-cache\") pod \"750b979e-ef5c-4c06-b456-d1c6cdce27e5\" (UID: \"750b979e-ef5c-4c06-b456-d1c6cdce27e5\") "
Nov 02 13:00:25.758498 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:25.758482    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/750b979e-ef5c-4c06-b456-d1c6cdce27e5-kube-api-access-kbmf5\" (UniqueName: \"kubernetes.io/projected/750b979e-ef5c-4c06-b456-d1c6cdce27e5-kube-api-access-kbmf5\") pod \"750b979e-ef5c-4c06-b456-d1c6cdce27e5\" (UID: \"750b979e-ef5c-4c06-b456-d1c6cdce27e5\") "
Nov 02 13:00:25.758794 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:25.758592    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/750b979e-ef5c-4c06-b456-d1c6cdce27e5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/750b979e-ef5c-4c06-b456-d1c6cdce27e5-tmp-dir\") pod \"750b979e-ef5c-4c06-b456-d1c6cdce27e5\" (UID: \"750b979e-ef5c-4c06-b456-d1c6cdce27e5\") "
Nov 02 13:00:25.758794 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:25.758602    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/750b979e-ef5c-4c06-b456-d1c6cdce27e5-home" pod "750b979e-ef5c-4c06-b456-d1c6cdce27e5" (UID: "750b979e-ef5c-4c06-b456-d1c6cdce27e5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:00:25.758794 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:25.758784    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/750b979e-ef5c-4c06-b456-d1c6cdce27e5-model-cache" pod "750b979e-ef5c-4c06-b456-d1c6cdce27e5" (UID: "750b979e-ef5c-4c06-b456-d1c6cdce27e5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:00:25.758977 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:25.758927    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/750b979e-ef5c-4c06-b456-d1c6cdce27e5-tmp-dir" pod "750b979e-ef5c-4c06-b456-d1c6cdce27e5" (UID: "750b979e-ef5c-4c06-b456-d1c6cdce27e5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:00:25.759468 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:25.759443    4559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/750b979e-ef5c-4c06-b456-d1c6cdce27e5-home\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:00:25.759602 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:25.759469    4559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/750b979e-ef5c-4c06-b456-d1c6cdce27e5-model-cache\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:00:25.759602 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:25.759485    4559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/750b979e-ef5c-4c06-b456-d1c6cdce27e5-tmp-dir\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:00:25.760868 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:25.760843    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/750b979e-ef5c-4c06-b456-d1c6cdce27e5-tls-certs" pod "750b979e-ef5c-4c06-b456-d1c6cdce27e5" (UID: "750b979e-ef5c-4c06-b456-d1c6cdce27e5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:00:25.760970 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:25.760907    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/750b979e-ef5c-4c06-b456-d1c6cdce27e5-kube-api-access-kbmf5" pod "750b979e-ef5c-4c06-b456-d1c6cdce27e5" (UID: "750b979e-ef5c-4c06-b456-d1c6cdce27e5"). InnerVolumeSpecName "kube-api-access-kbmf5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:00:25.760970 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:25.760911    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/750b979e-ef5c-4c06-b456-d1c6cdce27e5-dshm" pod "750b979e-ef5c-4c06-b456-d1c6cdce27e5" (UID: "750b979e-ef5c-4c06-b456-d1c6cdce27e5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:00:25.861216 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:25.861187    4559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/750b979e-ef5c-4c06-b456-d1c6cdce27e5-dshm\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:00:25.861216 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:25.861212    4559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/750b979e-ef5c-4c06-b456-d1c6cdce27e5-tls-certs\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:00:25.861416 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:25.861224    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kbmf5\" (UniqueName: \"kubernetes.io/projected/750b979e-ef5c-4c06-b456-d1c6cdce27e5-kube-api-access-kbmf5\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:00:25.895182 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:25.895098    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0cf4374b-846c-4a69-872f-6c03ed423b41" path="/var/lib/kubelet/pods/0cf4374b-846c-4a69-872f-6c03ed423b41/volumes"
Nov 02 13:00:25.895672 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:25.895651    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9e9675e-a96b-4f43-a253-608bcd468854" path="/var/lib/kubelet/pods/d9e9675e-a96b-4f43-a253-608bcd468854/volumes"
Nov 02 13:00:26.647542 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:26.647507    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-43be82/auth-test-model-43be82-kserve-5c6dfd4dfc-n4dl7" event={"ID":"750b979e-ef5c-4c06-b456-d1c6cdce27e5","Type":"ContainerDied","Data":"f4f229d9c68ad689d14a1e88d4520b1975f429e264a5855ee470d6ec5a0d79c5"}
Nov 02 13:00:26.647542 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:26.647523    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-43be82/auth-test-model-43be82-kserve-5c6dfd4dfc-n4dl7"
Nov 02 13:00:26.648110 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:26.647560    4559 scope.go:122] "RemoveContainer" containerID="7c04f3ca2a3fe91f92a0ed51e71dd8fca1238eb7ac2fbe84f9813f0388d570cb"
Nov 02 13:00:26.701348 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:26.701316    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-43be82/auth-test-model-43be82-kserve-5c6dfd4dfc-n4dl7"]
Nov 02 13:00:26.705295 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:26.705272    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w0-43be82/auth-test-model-43be82-kserve-5c6dfd4dfc-n4dl7"]
Nov 02 13:00:27.065656 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:27.065599    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-035dac-openshift-default-6d5cb5658f-8kmlt"]
Nov 02 13:00:27.066291 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:27.066269    4559 cpu_manager.go:414] "containerMap: removing container" podUID="3974db10-eea9-47c6-bbd4-557b4dca531f" containerName="cleanup"
Nov 02 13:00:27.066412 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:27.066291    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3974db10-eea9-47c6-bbd4-557b4dca531f" containerName="cleanup"
Nov 02 13:00:27.066412 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:27.066312    4559 cpu_manager.go:414] "containerMap: removing container" podUID="d9e9675e-a96b-4f43-a253-608bcd468854" containerName="payload-pre-processing"
Nov 02 13:00:27.066412 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:27.066319    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9e9675e-a96b-4f43-a253-608bcd468854" containerName="payload-pre-processing"
Nov 02 13:00:27.066412 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:27.066341    4559 cpu_manager.go:414] "containerMap: removing container" podUID="c7eddad9-7bbd-4ef1-8d9c-3db6e4a58961" containerName="cleanup"
Nov 02 13:00:27.066412 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:27.066348    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7eddad9-7bbd-4ef1-8d9c-3db6e4a58961" containerName="cleanup"
Nov 02 13:00:27.066412 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:27.066362    4559 cpu_manager.go:414] "containerMap: removing container" podUID="0cf4374b-846c-4a69-872f-6c03ed423b41" containerName="payload-processing"
Nov 02 13:00:27.066412 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:27.066368    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0cf4374b-846c-4a69-872f-6c03ed423b41" containerName="payload-processing"
Nov 02 13:00:27.066412 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:27.066384    4559 cpu_manager.go:414] "containerMap: removing container" podUID="a74304bb-3fb2-4dcd-be61-8299061bff15" containerName="cleanup"
Nov 02 13:00:27.066412 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:27.066390    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a74304bb-3fb2-4dcd-be61-8299061bff15" containerName="cleanup"
Nov 02 13:00:27.066412 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:27.066401    4559 cpu_manager.go:414] "containerMap: removing container" podUID="a74304bb-3fb2-4dcd-be61-8299061bff15" containerName="cleanup"
Nov 02 13:00:27.066412 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:27.066408    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a74304bb-3fb2-4dcd-be61-8299061bff15" containerName="cleanup"
Nov 02 13:00:27.066984 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:27.066420    4559 cpu_manager.go:414] "containerMap: removing container" podUID="750b979e-ef5c-4c06-b456-d1c6cdce27e5" containerName="main"
Nov 02 13:00:27.066984 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:27.066427    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="750b979e-ef5c-4c06-b456-d1c6cdce27e5" containerName="main"
Nov 02 13:00:27.066984 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:27.066442    4559 cpu_manager.go:414] "containerMap: removing container" podUID="a74304bb-3fb2-4dcd-be61-8299061bff15" containerName="cleanup"
Nov 02 13:00:27.066984 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:27.066451    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a74304bb-3fb2-4dcd-be61-8299061bff15" containerName="cleanup"
Nov 02 13:00:27.066984 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:27.066491    4559 cpu_manager.go:414] "containerMap: removing container" podUID="3974db10-eea9-47c6-bbd4-557b4dca531f" containerName="cleanup"
Nov 02 13:00:27.066984 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:27.066498    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3974db10-eea9-47c6-bbd4-557b4dca531f" containerName="cleanup"
Nov 02 13:00:27.066984 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:27.066516    4559 cpu_manager.go:414] "containerMap: removing container" podUID="c7eddad9-7bbd-4ef1-8d9c-3db6e4a58961" containerName="cleanup"
Nov 02 13:00:27.066984 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:27.066523    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7eddad9-7bbd-4ef1-8d9c-3db6e4a58961" containerName="cleanup"
Nov 02 13:00:27.066984 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:27.066673    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="a74304bb-3fb2-4dcd-be61-8299061bff15" containerName="cleanup"
Nov 02 13:00:27.066984 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:27.066692    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="3974db10-eea9-47c6-bbd4-557b4dca531f" containerName="cleanup"
Nov 02 13:00:27.066984 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:27.066704    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7eddad9-7bbd-4ef1-8d9c-3db6e4a58961" containerName="cleanup"
Nov 02 13:00:27.066984 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:27.066712    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="3974db10-eea9-47c6-bbd4-557b4dca531f" containerName="cleanup"
Nov 02 13:00:27.066984 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:27.066723    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7eddad9-7bbd-4ef1-8d9c-3db6e4a58961" containerName="cleanup"
Nov 02 13:00:27.066984 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:27.066731    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="3974db10-eea9-47c6-bbd4-557b4dca531f" containerName="cleanup"
Nov 02 13:00:27.066984 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:27.066743    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9e9675e-a96b-4f43-a253-608bcd468854" containerName="payload-pre-processing"
Nov 02 13:00:27.066984 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:27.066756    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="0cf4374b-846c-4a69-872f-6c03ed423b41" containerName="payload-processing"
Nov 02 13:00:27.066984 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:27.066768    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="a74304bb-3fb2-4dcd-be61-8299061bff15" containerName="cleanup"
Nov 02 13:00:27.066984 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:27.066776    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="750b979e-ef5c-4c06-b456-d1c6cdce27e5" containerName="main"
Nov 02 13:00:27.071912 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:27.071888    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-035dac-openshift-default-6d5cb5658f-8kmlt"
Nov 02 13:00:27.094505 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:27.094007    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-035dac-openshift-default-dockercfg-zznkp\""
Nov 02 13:00:27.098489 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:27.098466    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-035dac-openshift-default-6d5cb5658f-8kmlt"]
Nov 02 13:00:27.177759 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:27.177716    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a47a44e4-6226-4ce7-9760-58ee5eaf65bf-istio-podinfo\") pod \"e2e-extmod-a-035dac-openshift-default-6d5cb5658f-8kmlt\" (UID: \"a47a44e4-6226-4ce7-9760-58ee5eaf65bf\") " pod="openshift-ingress/e2e-extmod-a-035dac-openshift-default-6d5cb5658f-8kmlt"
Nov 02 13:00:27.177938 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:27.177816    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a47a44e4-6226-4ce7-9760-58ee5eaf65bf-workload-socket\") pod \"e2e-extmod-a-035dac-openshift-default-6d5cb5658f-8kmlt\" (UID: \"a47a44e4-6226-4ce7-9760-58ee5eaf65bf\") " pod="openshift-ingress/e2e-extmod-a-035dac-openshift-default-6d5cb5658f-8kmlt"
Nov 02 13:00:27.177938 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:27.177843    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghmvm\" (UniqueName: \"kubernetes.io/projected/a47a44e4-6226-4ce7-9760-58ee5eaf65bf-kube-api-access-ghmvm\") pod \"e2e-extmod-a-035dac-openshift-default-6d5cb5658f-8kmlt\" (UID: \"a47a44e4-6226-4ce7-9760-58ee5eaf65bf\") " pod="openshift-ingress/e2e-extmod-a-035dac-openshift-default-6d5cb5658f-8kmlt"
Nov 02 13:00:27.177938 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:27.177921    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a47a44e4-6226-4ce7-9760-58ee5eaf65bf-credential-socket\") pod \"e2e-extmod-a-035dac-openshift-default-6d5cb5658f-8kmlt\" (UID: \"a47a44e4-6226-4ce7-9760-58ee5eaf65bf\") " pod="openshift-ingress/e2e-extmod-a-035dac-openshift-default-6d5cb5658f-8kmlt"
Nov 02 13:00:27.178051 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:27.177965    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a47a44e4-6226-4ce7-9760-58ee5eaf65bf-istio-envoy\") pod \"e2e-extmod-a-035dac-openshift-default-6d5cb5658f-8kmlt\" (UID: \"a47a44e4-6226-4ce7-9760-58ee5eaf65bf\") " pod="openshift-ingress/e2e-extmod-a-035dac-openshift-default-6d5cb5658f-8kmlt"
Nov 02 13:00:27.178114 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:27.178091    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a47a44e4-6226-4ce7-9760-58ee5eaf65bf-workload-certs\") pod \"e2e-extmod-a-035dac-openshift-default-6d5cb5658f-8kmlt\" (UID: \"a47a44e4-6226-4ce7-9760-58ee5eaf65bf\") " pod="openshift-ingress/e2e-extmod-a-035dac-openshift-default-6d5cb5658f-8kmlt"
Nov 02 13:00:27.178148 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:27.178138    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a47a44e4-6226-4ce7-9760-58ee5eaf65bf-istiod-ca-cert\") pod \"e2e-extmod-a-035dac-openshift-default-6d5cb5658f-8kmlt\" (UID: \"a47a44e4-6226-4ce7-9760-58ee5eaf65bf\") " pod="openshift-ingress/e2e-extmod-a-035dac-openshift-default-6d5cb5658f-8kmlt"
Nov 02 13:00:27.178293 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:27.178279    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a47a44e4-6226-4ce7-9760-58ee5eaf65bf-istio-token\") pod \"e2e-extmod-a-035dac-openshift-default-6d5cb5658f-8kmlt\" (UID: \"a47a44e4-6226-4ce7-9760-58ee5eaf65bf\") " pod="openshift-ingress/e2e-extmod-a-035dac-openshift-default-6d5cb5658f-8kmlt"
Nov 02 13:00:27.178330 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:27.178304    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a47a44e4-6226-4ce7-9760-58ee5eaf65bf-istio-data\") pod \"e2e-extmod-a-035dac-openshift-default-6d5cb5658f-8kmlt\" (UID: \"a47a44e4-6226-4ce7-9760-58ee5eaf65bf\") " pod="openshift-ingress/e2e-extmod-a-035dac-openshift-default-6d5cb5658f-8kmlt"
Nov 02 13:00:27.283671 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:27.283591    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a47a44e4-6226-4ce7-9760-58ee5eaf65bf-istio-token\") pod \"e2e-extmod-a-035dac-openshift-default-6d5cb5658f-8kmlt\" (UID: \"a47a44e4-6226-4ce7-9760-58ee5eaf65bf\") " pod="openshift-ingress/e2e-extmod-a-035dac-openshift-default-6d5cb5658f-8kmlt"
Nov 02 13:00:27.283671 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:27.283670    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a47a44e4-6226-4ce7-9760-58ee5eaf65bf-istio-data\") pod \"e2e-extmod-a-035dac-openshift-default-6d5cb5658f-8kmlt\" (UID: \"a47a44e4-6226-4ce7-9760-58ee5eaf65bf\") " pod="openshift-ingress/e2e-extmod-a-035dac-openshift-default-6d5cb5658f-8kmlt"
Nov 02 13:00:27.283952 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:27.283722    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a47a44e4-6226-4ce7-9760-58ee5eaf65bf-istio-podinfo\") pod \"e2e-extmod-a-035dac-openshift-default-6d5cb5658f-8kmlt\" (UID: \"a47a44e4-6226-4ce7-9760-58ee5eaf65bf\") " pod="openshift-ingress/e2e-extmod-a-035dac-openshift-default-6d5cb5658f-8kmlt"
Nov 02 13:00:27.283952 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:27.283804    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a47a44e4-6226-4ce7-9760-58ee5eaf65bf-workload-socket\") pod \"e2e-extmod-a-035dac-openshift-default-6d5cb5658f-8kmlt\" (UID: \"a47a44e4-6226-4ce7-9760-58ee5eaf65bf\") " pod="openshift-ingress/e2e-extmod-a-035dac-openshift-default-6d5cb5658f-8kmlt"
Nov 02 13:00:27.283952 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:27.283838    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghmvm\" (UniqueName: \"kubernetes.io/projected/a47a44e4-6226-4ce7-9760-58ee5eaf65bf-kube-api-access-ghmvm\") pod \"e2e-extmod-a-035dac-openshift-default-6d5cb5658f-8kmlt\" (UID: \"a47a44e4-6226-4ce7-9760-58ee5eaf65bf\") " pod="openshift-ingress/e2e-extmod-a-035dac-openshift-default-6d5cb5658f-8kmlt"
Nov 02 13:00:27.283952 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:27.283900    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a47a44e4-6226-4ce7-9760-58ee5eaf65bf-credential-socket\") pod \"e2e-extmod-a-035dac-openshift-default-6d5cb5658f-8kmlt\" (UID: \"a47a44e4-6226-4ce7-9760-58ee5eaf65bf\") " pod="openshift-ingress/e2e-extmod-a-035dac-openshift-default-6d5cb5658f-8kmlt"
Nov 02 13:00:27.284156 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:27.283963    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a47a44e4-6226-4ce7-9760-58ee5eaf65bf-istio-envoy\") pod \"e2e-extmod-a-035dac-openshift-default-6d5cb5658f-8kmlt\" (UID: \"a47a44e4-6226-4ce7-9760-58ee5eaf65bf\") " pod="openshift-ingress/e2e-extmod-a-035dac-openshift-default-6d5cb5658f-8kmlt"
Nov 02 13:00:27.284219 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:27.284173    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a47a44e4-6226-4ce7-9760-58ee5eaf65bf-workload-certs\") pod \"e2e-extmod-a-035dac-openshift-default-6d5cb5658f-8kmlt\" (UID: \"a47a44e4-6226-4ce7-9760-58ee5eaf65bf\") " pod="openshift-ingress/e2e-extmod-a-035dac-openshift-default-6d5cb5658f-8kmlt"
Nov 02 13:00:27.284275 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:27.284229    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a47a44e4-6226-4ce7-9760-58ee5eaf65bf-istiod-ca-cert\") pod \"e2e-extmod-a-035dac-openshift-default-6d5cb5658f-8kmlt\" (UID: \"a47a44e4-6226-4ce7-9760-58ee5eaf65bf\") " pod="openshift-ingress/e2e-extmod-a-035dac-openshift-default-6d5cb5658f-8kmlt"
Nov 02 13:00:27.285669 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:27.285157    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a47a44e4-6226-4ce7-9760-58ee5eaf65bf-istiod-ca-cert\") pod \"e2e-extmod-a-035dac-openshift-default-6d5cb5658f-8kmlt\" (UID: \"a47a44e4-6226-4ce7-9760-58ee5eaf65bf\") " pod="openshift-ingress/e2e-extmod-a-035dac-openshift-default-6d5cb5658f-8kmlt"
Nov 02 13:00:27.289663 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:27.288972    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a47a44e4-6226-4ce7-9760-58ee5eaf65bf-credential-socket\") pod \"e2e-extmod-a-035dac-openshift-default-6d5cb5658f-8kmlt\" (UID: \"a47a44e4-6226-4ce7-9760-58ee5eaf65bf\") " pod="openshift-ingress/e2e-extmod-a-035dac-openshift-default-6d5cb5658f-8kmlt"
Nov 02 13:00:27.289663 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:27.289221    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a47a44e4-6226-4ce7-9760-58ee5eaf65bf-workload-socket\") pod \"e2e-extmod-a-035dac-openshift-default-6d5cb5658f-8kmlt\" (UID: \"a47a44e4-6226-4ce7-9760-58ee5eaf65bf\") " pod="openshift-ingress/e2e-extmod-a-035dac-openshift-default-6d5cb5658f-8kmlt"
Nov 02 13:00:27.289663 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:27.289447    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a47a44e4-6226-4ce7-9760-58ee5eaf65bf-workload-certs\") pod \"e2e-extmod-a-035dac-openshift-default-6d5cb5658f-8kmlt\" (UID: \"a47a44e4-6226-4ce7-9760-58ee5eaf65bf\") " pod="openshift-ingress/e2e-extmod-a-035dac-openshift-default-6d5cb5658f-8kmlt"
Nov 02 13:00:27.303670 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:27.298081    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a47a44e4-6226-4ce7-9760-58ee5eaf65bf-istio-envoy\") pod \"e2e-extmod-a-035dac-openshift-default-6d5cb5658f-8kmlt\" (UID: \"a47a44e4-6226-4ce7-9760-58ee5eaf65bf\") " pod="openshift-ingress/e2e-extmod-a-035dac-openshift-default-6d5cb5658f-8kmlt"
Nov 02 13:00:27.307662 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:27.307311    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a47a44e4-6226-4ce7-9760-58ee5eaf65bf-istio-podinfo\") pod \"e2e-extmod-a-035dac-openshift-default-6d5cb5658f-8kmlt\" (UID: \"a47a44e4-6226-4ce7-9760-58ee5eaf65bf\") " pod="openshift-ingress/e2e-extmod-a-035dac-openshift-default-6d5cb5658f-8kmlt"
Nov 02 13:00:27.325714 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:27.308995    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a47a44e4-6226-4ce7-9760-58ee5eaf65bf-istio-data\") pod \"e2e-extmod-a-035dac-openshift-default-6d5cb5658f-8kmlt\" (UID: \"a47a44e4-6226-4ce7-9760-58ee5eaf65bf\") " pod="openshift-ingress/e2e-extmod-a-035dac-openshift-default-6d5cb5658f-8kmlt"
Nov 02 13:00:27.334340 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:27.329719    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a47a44e4-6226-4ce7-9760-58ee5eaf65bf-istio-token\") pod \"e2e-extmod-a-035dac-openshift-default-6d5cb5658f-8kmlt\" (UID: \"a47a44e4-6226-4ce7-9760-58ee5eaf65bf\") " pod="openshift-ingress/e2e-extmod-a-035dac-openshift-default-6d5cb5658f-8kmlt"
Nov 02 13:00:27.334340 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:27.332208    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghmvm\" (UniqueName: \"kubernetes.io/projected/a47a44e4-6226-4ce7-9760-58ee5eaf65bf-kube-api-access-ghmvm\") pod \"e2e-extmod-a-035dac-openshift-default-6d5cb5658f-8kmlt\" (UID: \"a47a44e4-6226-4ce7-9760-58ee5eaf65bf\") " pod="openshift-ingress/e2e-extmod-a-035dac-openshift-default-6d5cb5658f-8kmlt"
Nov 02 13:00:27.403931 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:27.400107    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-035dac-openshift-default-6d5cb5658f-8kmlt"
Nov 02 13:00:27.734012 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:27.733982    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-035dac-openshift-default-6d5cb5658f-8kmlt"]
Nov 02 13:00:27.735403 ip-10-0-128-132 kubenswrapper[4559]: W1102 13:00:27.735371    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda47a44e4_6226_4ce7_9760_58ee5eaf65bf.slice/crio-88e9ca2c9abc2d5a75a7978f6f13377320be40b38745dda93599a990ec042371 WatchSource:0}: Error finding container 88e9ca2c9abc2d5a75a7978f6f13377320be40b38745dda93599a990ec042371: Status 404 returned error can't find the container with id 88e9ca2c9abc2d5a75a7978f6f13377320be40b38745dda93599a990ec042371
Nov 02 13:00:27.738553 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:27.738518    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 13:00:27.738668 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:27.738589    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 13:00:27.738668 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:27.738622    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 13:00:27.893787 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:27.893755    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="750b979e-ef5c-4c06-b456-d1c6cdce27e5" path="/var/lib/kubelet/pods/750b979e-ef5c-4c06-b456-d1c6cdce27e5/volumes"
Nov 02 13:00:28.667148 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:28.667102    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-035dac-openshift-default-6d5cb5658f-8kmlt" event={"ID":"a47a44e4-6226-4ce7-9760-58ee5eaf65bf","Type":"ContainerStarted","Data":"215bfa85b115960b75bd02c0ea3213a462ea4cb0494f640b969ee693e9c7d93c"}
Nov 02 13:00:28.667366 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:28.667221    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-035dac-openshift-default-6d5cb5658f-8kmlt" event={"ID":"a47a44e4-6226-4ce7-9760-58ee5eaf65bf","Type":"ContainerStarted","Data":"88e9ca2c9abc2d5a75a7978f6f13377320be40b38745dda93599a990ec042371"}
Nov 02 13:00:28.764212 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:28.764159    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-035dac-openshift-default-6d5cb5658f-8kmlt" podStartSLOduration=1.764146926 podStartE2EDuration="1.764146926s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:00:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:00:28.759997836 +0000 UTC m=+2025.015396933" watchObservedRunningTime="2026-11-02 13:00:28.764146926 +0000 UTC m=+2025.019546097"
Nov 02 13:00:29.401309 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:29.401271    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-035dac-openshift-default-6d5cb5658f-8kmlt"
Nov 02 13:00:29.406930 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:29.406904    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-035dac-openshift-default-6d5cb5658f-8kmlt"
Nov 02 13:00:29.673504 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:29.673477    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-035dac-openshift-default-6d5cb5658f-8kmlt"
Nov 02 13:00:29.674392 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:29.674374    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-035dac-openshift-default-6d5cb5658f-8kmlt"
Nov 02 13:00:41.219582 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:41.219546    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"]
Nov 02 13:00:41.378732 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:41.378691    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-1e14d4/auth-test-model-1e14d4-kserve-97cc85bb9-5jjh5"]
Nov 02 13:00:45.086710 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:00:45.086666    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqcmj" containerName="extractor"
Nov 02 13:00:45.768453 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:45.768414    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-e430c7-6f7d8d6ccc-9lkds"]
Nov 02 13:00:45.768723 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:45.768694    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-e430c7-6f7d8d6ccc-9lkds" podUID="b93c19bf-9bc0-4c51-a945-5b196d5dd80e" containerName="maas-api" containerID="cri-o://f46e2199f03ed8d50dec7426dcee381e46a8f0f9478f047397f5913f713e883a" gracePeriod=30
Nov 02 13:00:45.917217 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:45.917185    4559 generic.go:356] "Generic (PLEG): container finished" podID="b93c19bf-9bc0-4c51-a945-5b196d5dd80e" containerID="f46e2199f03ed8d50dec7426dcee381e46a8f0f9478f047397f5913f713e883a" exitCode=0
Nov 02 13:00:45.917393 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:45.917260    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-e430c7-6f7d8d6ccc-9lkds" event={"ID":"b93c19bf-9bc0-4c51-a945-5b196d5dd80e","Type":"ContainerDied","Data":"f46e2199f03ed8d50dec7426dcee381e46a8f0f9478f047397f5913f713e883a"}
Nov 02 13:00:46.165285 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:46.160616    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-e430c7-6f7d8d6ccc-9lkds"
Nov 02 13:00:46.174046 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:46.173423    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b93c19bf-9bc0-4c51-a945-5b196d5dd80e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b93c19bf-9bc0-4c51-a945-5b196d5dd80e-metrics-certs\") pod \"b93c19bf-9bc0-4c51-a945-5b196d5dd80e\" (UID: \"b93c19bf-9bc0-4c51-a945-5b196d5dd80e\") "
Nov 02 13:00:46.174046 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:46.173475    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b93c19bf-9bc0-4c51-a945-5b196d5dd80e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b93c19bf-9bc0-4c51-a945-5b196d5dd80e-openshift-service-ca\") pod \"b93c19bf-9bc0-4c51-a945-5b196d5dd80e\" (UID: \"b93c19bf-9bc0-4c51-a945-5b196d5dd80e\") "
Nov 02 13:00:46.174046 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:46.173553    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b93c19bf-9bc0-4c51-a945-5b196d5dd80e-kube-api-access-zddtb\" (UniqueName: \"kubernetes.io/projected/b93c19bf-9bc0-4c51-a945-5b196d5dd80e-kube-api-access-zddtb\") pod \"b93c19bf-9bc0-4c51-a945-5b196d5dd80e\" (UID: \"b93c19bf-9bc0-4c51-a945-5b196d5dd80e\") "
Nov 02 13:00:46.174046 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:46.173590    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b93c19bf-9bc0-4c51-a945-5b196d5dd80e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b93c19bf-9bc0-4c51-a945-5b196d5dd80e-maas-api-tls\") pod \"b93c19bf-9bc0-4c51-a945-5b196d5dd80e\" (UID: \"b93c19bf-9bc0-4c51-a945-5b196d5dd80e\") "
Nov 02 13:00:46.177164 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:46.177136    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b93c19bf-9bc0-4c51-a945-5b196d5dd80e-openshift-service-ca" pod "b93c19bf-9bc0-4c51-a945-5b196d5dd80e" (UID: "b93c19bf-9bc0-4c51-a945-5b196d5dd80e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:00:46.193023 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:46.192765    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b93c19bf-9bc0-4c51-a945-5b196d5dd80e-metrics-certs" pod "b93c19bf-9bc0-4c51-a945-5b196d5dd80e" (UID: "b93c19bf-9bc0-4c51-a945-5b196d5dd80e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:00:46.240006 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:46.239833    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b93c19bf-9bc0-4c51-a945-5b196d5dd80e-maas-api-tls" pod "b93c19bf-9bc0-4c51-a945-5b196d5dd80e" (UID: "b93c19bf-9bc0-4c51-a945-5b196d5dd80e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:00:46.240443 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:46.240336    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b93c19bf-9bc0-4c51-a945-5b196d5dd80e-kube-api-access-zddtb" pod "b93c19bf-9bc0-4c51-a945-5b196d5dd80e" (UID: "b93c19bf-9bc0-4c51-a945-5b196d5dd80e"). InnerVolumeSpecName "kube-api-access-zddtb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:00:46.313937 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:46.304700    4559 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b93c19bf-9bc0-4c51-a945-5b196d5dd80e-metrics-certs\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:00:46.313937 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:46.304733    4559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b93c19bf-9bc0-4c51-a945-5b196d5dd80e-openshift-service-ca\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:00:46.313937 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:46.304749    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zddtb\" (UniqueName: \"kubernetes.io/projected/b93c19bf-9bc0-4c51-a945-5b196d5dd80e-kube-api-access-zddtb\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:00:46.313937 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:46.304761    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b93c19bf-9bc0-4c51-a945-5b196d5dd80e-maas-api-tls\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:00:46.927030 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:46.926981    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-e430c7-6f7d8d6ccc-9lkds" event={"ID":"b93c19bf-9bc0-4c51-a945-5b196d5dd80e","Type":"ContainerDied","Data":"f3e5a94a0342f5fbec882968f2356f7eebc35e89c5850855104386b52c62ef7c"}
Nov 02 13:00:46.927341 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:46.927091    4559 scope.go:122] "RemoveContainer" containerID="f46e2199f03ed8d50dec7426dcee381e46a8f0f9478f047397f5913f713e883a"
Nov 02 13:00:46.927463 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:46.927427    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-e430c7-6f7d8d6ccc-9lkds"
Nov 02 13:00:46.984148 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:46.984111    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-e430c7-6f7d8d6ccc-9lkds"]
Nov 02 13:00:46.996583 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:46.996554    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-e430c7-6f7d8d6ccc-9lkds"]
Nov 02 13:00:47.893887 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:47.893851    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b93c19bf-9bc0-4c51-a945-5b196d5dd80e" path="/var/lib/kubelet/pods/b93c19bf-9bc0-4c51-a945-5b196d5dd80e/volumes"
Nov 02 13:00:53.706268 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:53.706227    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-740f09-openshift-default-786896ffcc-lmqlz"]
Nov 02 13:00:53.707141 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:53.707107    4559 cpu_manager.go:414] "containerMap: removing container" podUID="b93c19bf-9bc0-4c51-a945-5b196d5dd80e" containerName="maas-api"
Nov 02 13:00:53.707141 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:53.707128    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b93c19bf-9bc0-4c51-a945-5b196d5dd80e" containerName="maas-api"
Nov 02 13:00:53.707368 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:53.707153    4559 cpu_manager.go:414] "containerMap: removing container" podUID="c7eddad9-7bbd-4ef1-8d9c-3db6e4a58961" containerName="cleanup"
Nov 02 13:00:53.707368 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:53.707163    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7eddad9-7bbd-4ef1-8d9c-3db6e4a58961" containerName="cleanup"
Nov 02 13:00:53.707368 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:53.707179    4559 cpu_manager.go:414] "containerMap: removing container" podUID="3974db10-eea9-47c6-bbd4-557b4dca531f" containerName="cleanup"
Nov 02 13:00:53.707368 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:53.707189    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3974db10-eea9-47c6-bbd4-557b4dca531f" containerName="cleanup"
Nov 02 13:00:53.707368 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:53.707342    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="a74304bb-3fb2-4dcd-be61-8299061bff15" containerName="cleanup"
Nov 02 13:00:53.707368 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:53.707358    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="b93c19bf-9bc0-4c51-a945-5b196d5dd80e" containerName="maas-api"
Nov 02 13:00:53.707368 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:53.707369    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7eddad9-7bbd-4ef1-8d9c-3db6e4a58961" containerName="cleanup"
Nov 02 13:00:53.718901 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:53.712965    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-740f09-openshift-default-786896ffcc-lmqlz"
Nov 02 13:00:53.733450 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:53.733424    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-740f09-openshift-default-786896ffcc-lmqlz"]
Nov 02 13:00:53.741822 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:53.741790    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-740f09-openshift-default-dockercfg-dxrm4\""
Nov 02 13:00:53.822597 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:53.822559    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f2bb155-8114-4c86-a74d-479522616e10-workload-certs\") pod \"e2e-extmod-a-740f09-openshift-default-786896ffcc-lmqlz\" (UID: \"8f2bb155-8114-4c86-a74d-479522616e10\") " pod="openshift-ingress/e2e-extmod-a-740f09-openshift-default-786896ffcc-lmqlz"
Nov 02 13:00:53.822792 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:53.822616    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f2bb155-8114-4c86-a74d-479522616e10-workload-socket\") pod \"e2e-extmod-a-740f09-openshift-default-786896ffcc-lmqlz\" (UID: \"8f2bb155-8114-4c86-a74d-479522616e10\") " pod="openshift-ingress/e2e-extmod-a-740f09-openshift-default-786896ffcc-lmqlz"
Nov 02 13:00:53.822792 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:53.822756    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f2bb155-8114-4c86-a74d-479522616e10-istio-data\") pod \"e2e-extmod-a-740f09-openshift-default-786896ffcc-lmqlz\" (UID: \"8f2bb155-8114-4c86-a74d-479522616e10\") " pod="openshift-ingress/e2e-extmod-a-740f09-openshift-default-786896ffcc-lmqlz"
Nov 02 13:00:53.822792 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:53.822785    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f2bb155-8114-4c86-a74d-479522616e10-istio-envoy\") pod \"e2e-extmod-a-740f09-openshift-default-786896ffcc-lmqlz\" (UID: \"8f2bb155-8114-4c86-a74d-479522616e10\") " pod="openshift-ingress/e2e-extmod-a-740f09-openshift-default-786896ffcc-lmqlz"
Nov 02 13:00:53.822911 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:53.822854    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q99sk\" (UniqueName: \"kubernetes.io/projected/8f2bb155-8114-4c86-a74d-479522616e10-kube-api-access-q99sk\") pod \"e2e-extmod-a-740f09-openshift-default-786896ffcc-lmqlz\" (UID: \"8f2bb155-8114-4c86-a74d-479522616e10\") " pod="openshift-ingress/e2e-extmod-a-740f09-openshift-default-786896ffcc-lmqlz"
Nov 02 13:00:53.822948 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:53.822908    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f2bb155-8114-4c86-a74d-479522616e10-credential-socket\") pod \"e2e-extmod-a-740f09-openshift-default-786896ffcc-lmqlz\" (UID: \"8f2bb155-8114-4c86-a74d-479522616e10\") " pod="openshift-ingress/e2e-extmod-a-740f09-openshift-default-786896ffcc-lmqlz"
Nov 02 13:00:53.822948 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:53.822926    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f2bb155-8114-4c86-a74d-479522616e10-istiod-ca-cert\") pod \"e2e-extmod-a-740f09-openshift-default-786896ffcc-lmqlz\" (UID: \"8f2bb155-8114-4c86-a74d-479522616e10\") " pod="openshift-ingress/e2e-extmod-a-740f09-openshift-default-786896ffcc-lmqlz"
Nov 02 13:00:53.823054 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:53.823039    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f2bb155-8114-4c86-a74d-479522616e10-istio-podinfo\") pod \"e2e-extmod-a-740f09-openshift-default-786896ffcc-lmqlz\" (UID: \"8f2bb155-8114-4c86-a74d-479522616e10\") " pod="openshift-ingress/e2e-extmod-a-740f09-openshift-default-786896ffcc-lmqlz"
Nov 02 13:00:53.823091 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:53.823076    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f2bb155-8114-4c86-a74d-479522616e10-istio-token\") pod \"e2e-extmod-a-740f09-openshift-default-786896ffcc-lmqlz\" (UID: \"8f2bb155-8114-4c86-a74d-479522616e10\") " pod="openshift-ingress/e2e-extmod-a-740f09-openshift-default-786896ffcc-lmqlz"
Nov 02 13:00:53.945668 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:53.944834    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f2bb155-8114-4c86-a74d-479522616e10-istio-envoy\") pod \"e2e-extmod-a-740f09-openshift-default-786896ffcc-lmqlz\" (UID: \"8f2bb155-8114-4c86-a74d-479522616e10\") " pod="openshift-ingress/e2e-extmod-a-740f09-openshift-default-786896ffcc-lmqlz"
Nov 02 13:00:53.945668 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:53.944913    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q99sk\" (UniqueName: \"kubernetes.io/projected/8f2bb155-8114-4c86-a74d-479522616e10-kube-api-access-q99sk\") pod \"e2e-extmod-a-740f09-openshift-default-786896ffcc-lmqlz\" (UID: \"8f2bb155-8114-4c86-a74d-479522616e10\") " pod="openshift-ingress/e2e-extmod-a-740f09-openshift-default-786896ffcc-lmqlz"
Nov 02 13:00:53.945668 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:53.944984    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f2bb155-8114-4c86-a74d-479522616e10-credential-socket\") pod \"e2e-extmod-a-740f09-openshift-default-786896ffcc-lmqlz\" (UID: \"8f2bb155-8114-4c86-a74d-479522616e10\") " pod="openshift-ingress/e2e-extmod-a-740f09-openshift-default-786896ffcc-lmqlz"
Nov 02 13:00:53.945668 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:53.945011    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f2bb155-8114-4c86-a74d-479522616e10-istiod-ca-cert\") pod \"e2e-extmod-a-740f09-openshift-default-786896ffcc-lmqlz\" (UID: \"8f2bb155-8114-4c86-a74d-479522616e10\") " pod="openshift-ingress/e2e-extmod-a-740f09-openshift-default-786896ffcc-lmqlz"
Nov 02 13:00:53.945668 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:53.945076    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f2bb155-8114-4c86-a74d-479522616e10-istio-podinfo\") pod \"e2e-extmod-a-740f09-openshift-default-786896ffcc-lmqlz\" (UID: \"8f2bb155-8114-4c86-a74d-479522616e10\") " pod="openshift-ingress/e2e-extmod-a-740f09-openshift-default-786896ffcc-lmqlz"
Nov 02 13:00:53.945668 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:53.945125    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f2bb155-8114-4c86-a74d-479522616e10-istio-token\") pod \"e2e-extmod-a-740f09-openshift-default-786896ffcc-lmqlz\" (UID: \"8f2bb155-8114-4c86-a74d-479522616e10\") " pod="openshift-ingress/e2e-extmod-a-740f09-openshift-default-786896ffcc-lmqlz"
Nov 02 13:00:53.945668 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:53.945287    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f2bb155-8114-4c86-a74d-479522616e10-workload-certs\") pod \"e2e-extmod-a-740f09-openshift-default-786896ffcc-lmqlz\" (UID: \"8f2bb155-8114-4c86-a74d-479522616e10\") " pod="openshift-ingress/e2e-extmod-a-740f09-openshift-default-786896ffcc-lmqlz"
Nov 02 13:00:53.945668 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:53.945347    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f2bb155-8114-4c86-a74d-479522616e10-workload-socket\") pod \"e2e-extmod-a-740f09-openshift-default-786896ffcc-lmqlz\" (UID: \"8f2bb155-8114-4c86-a74d-479522616e10\") " pod="openshift-ingress/e2e-extmod-a-740f09-openshift-default-786896ffcc-lmqlz"
Nov 02 13:00:53.945668 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:53.945406    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f2bb155-8114-4c86-a74d-479522616e10-istio-data\") pod \"e2e-extmod-a-740f09-openshift-default-786896ffcc-lmqlz\" (UID: \"8f2bb155-8114-4c86-a74d-479522616e10\") " pod="openshift-ingress/e2e-extmod-a-740f09-openshift-default-786896ffcc-lmqlz"
Nov 02 13:00:53.946304 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:53.945917    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f2bb155-8114-4c86-a74d-479522616e10-istio-data\") pod \"e2e-extmod-a-740f09-openshift-default-786896ffcc-lmqlz\" (UID: \"8f2bb155-8114-4c86-a74d-479522616e10\") " pod="openshift-ingress/e2e-extmod-a-740f09-openshift-default-786896ffcc-lmqlz"
Nov 02 13:00:53.976707 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:53.970144    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f2bb155-8114-4c86-a74d-479522616e10-workload-socket\") pod \"e2e-extmod-a-740f09-openshift-default-786896ffcc-lmqlz\" (UID: \"8f2bb155-8114-4c86-a74d-479522616e10\") " pod="openshift-ingress/e2e-extmod-a-740f09-openshift-default-786896ffcc-lmqlz"
Nov 02 13:00:53.976707 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:53.970818    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f2bb155-8114-4c86-a74d-479522616e10-istiod-ca-cert\") pod \"e2e-extmod-a-740f09-openshift-default-786896ffcc-lmqlz\" (UID: \"8f2bb155-8114-4c86-a74d-479522616e10\") " pod="openshift-ingress/e2e-extmod-a-740f09-openshift-default-786896ffcc-lmqlz"
Nov 02 13:00:53.996665 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:53.992520    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f2bb155-8114-4c86-a74d-479522616e10-credential-socket\") pod \"e2e-extmod-a-740f09-openshift-default-786896ffcc-lmqlz\" (UID: \"8f2bb155-8114-4c86-a74d-479522616e10\") " pod="openshift-ingress/e2e-extmod-a-740f09-openshift-default-786896ffcc-lmqlz"
Nov 02 13:00:53.996665 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:53.994015    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f2bb155-8114-4c86-a74d-479522616e10-workload-certs\") pod \"e2e-extmod-a-740f09-openshift-default-786896ffcc-lmqlz\" (UID: \"8f2bb155-8114-4c86-a74d-479522616e10\") " pod="openshift-ingress/e2e-extmod-a-740f09-openshift-default-786896ffcc-lmqlz"
Nov 02 13:00:54.020065 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:54.012597    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f2bb155-8114-4c86-a74d-479522616e10-istio-envoy\") pod \"e2e-extmod-a-740f09-openshift-default-786896ffcc-lmqlz\" (UID: \"8f2bb155-8114-4c86-a74d-479522616e10\") " pod="openshift-ingress/e2e-extmod-a-740f09-openshift-default-786896ffcc-lmqlz"
Nov 02 13:00:54.029661 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:54.023300    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f2bb155-8114-4c86-a74d-479522616e10-istio-podinfo\") pod \"e2e-extmod-a-740f09-openshift-default-786896ffcc-lmqlz\" (UID: \"8f2bb155-8114-4c86-a74d-479522616e10\") " pod="openshift-ingress/e2e-extmod-a-740f09-openshift-default-786896ffcc-lmqlz"
Nov 02 13:00:54.100060 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:54.098527    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q99sk\" (UniqueName: \"kubernetes.io/projected/8f2bb155-8114-4c86-a74d-479522616e10-kube-api-access-q99sk\") pod \"e2e-extmod-a-740f09-openshift-default-786896ffcc-lmqlz\" (UID: \"8f2bb155-8114-4c86-a74d-479522616e10\") " pod="openshift-ingress/e2e-extmod-a-740f09-openshift-default-786896ffcc-lmqlz"
Nov 02 13:00:54.111664 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:54.100695    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f2bb155-8114-4c86-a74d-479522616e10-istio-token\") pod \"e2e-extmod-a-740f09-openshift-default-786896ffcc-lmqlz\" (UID: \"8f2bb155-8114-4c86-a74d-479522616e10\") " pod="openshift-ingress/e2e-extmod-a-740f09-openshift-default-786896ffcc-lmqlz"
Nov 02 13:00:54.115664 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:54.112991    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-e430c7-openshift-default-8649b6f6fc-vl6ql"]
Nov 02 13:00:54.147101 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:54.146820    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-explicit-e430c7-openshift-default-8649b6f6fc-vl6ql" podUID="6f8849e4-f808-45f0-b24a-e1e1c8b99974" containerName="istio-proxy" containerID="cri-o://36e7f4b54929fd9b7406a9c8575aec0290af27998119ca3607b55489f4d601ec" gracePeriod=30
Nov 02 13:00:54.352612 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:54.346159    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-740f09-openshift-default-786896ffcc-lmqlz"
Nov 02 13:00:54.703168 ip-10-0-128-132 kubenswrapper[4559]: W1102 13:00:54.703138    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f2bb155_8114_4c86_a74d_479522616e10.slice/crio-176b8adc4a7afe836783302c0db1184b60fdf7c5cab538e063e1080c0b6f3b08 WatchSource:0}: Error finding container 176b8adc4a7afe836783302c0db1184b60fdf7c5cab538e063e1080c0b6f3b08: Status 404 returned error can't find the container with id 176b8adc4a7afe836783302c0db1184b60fdf7c5cab538e063e1080c0b6f3b08
Nov 02 13:00:54.703697 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:54.703668    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-740f09-openshift-default-786896ffcc-lmqlz"]
Nov 02 13:00:54.706162 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:54.706128    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 13:00:54.706263 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:54.706194    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 13:00:54.706263 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:54.706226    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 13:00:55.192310 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:55.192273    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-noten-7aca74-openshift-default-5df749499f-n9t77"]
Nov 02 13:00:55.197169 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:55.197144    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-7aca74-openshift-default-5df749499f-n9t77"
Nov 02 13:00:55.213843 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:55.213817    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-noten-7aca74-openshift-default-dockercfg-6l9mk\""
Nov 02 13:00:55.231381 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:55.231348    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-7aca74-openshift-default-5df749499f-n9t77"]
Nov 02 13:00:55.240415 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:55.240360    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-740f09-openshift-default-786896ffcc-lmqlz" event={"ID":"8f2bb155-8114-4c86-a74d-479522616e10","Type":"ContainerStarted","Data":"815680021cae23ccb7232840479478c9c9425105ef0b77eee0ee0e05076164ef"}
Nov 02 13:00:55.240868 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:55.240429    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-740f09-openshift-default-786896ffcc-lmqlz" event={"ID":"8f2bb155-8114-4c86-a74d-479522616e10","Type":"ContainerStarted","Data":"176b8adc4a7afe836783302c0db1184b60fdf7c5cab538e063e1080c0b6f3b08"}
Nov 02 13:00:55.278419 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:55.278377    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxc7k\" (UniqueName: \"kubernetes.io/projected/e398716c-4a76-4c4e-881f-415c51600a83-kube-api-access-xxc7k\") pod \"e2e-noten-7aca74-openshift-default-5df749499f-n9t77\" (UID: \"e398716c-4a76-4c4e-881f-415c51600a83\") " pod="openshift-ingress/e2e-noten-7aca74-openshift-default-5df749499f-n9t77"
Nov 02 13:00:55.278672 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:55.278427    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e398716c-4a76-4c4e-881f-415c51600a83-istio-token\") pod \"e2e-noten-7aca74-openshift-default-5df749499f-n9t77\" (UID: \"e398716c-4a76-4c4e-881f-415c51600a83\") " pod="openshift-ingress/e2e-noten-7aca74-openshift-default-5df749499f-n9t77"
Nov 02 13:00:55.278672 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:55.278449    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e398716c-4a76-4c4e-881f-415c51600a83-istio-envoy\") pod \"e2e-noten-7aca74-openshift-default-5df749499f-n9t77\" (UID: \"e398716c-4a76-4c4e-881f-415c51600a83\") " pod="openshift-ingress/e2e-noten-7aca74-openshift-default-5df749499f-n9t77"
Nov 02 13:00:55.278672 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:55.278549    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e398716c-4a76-4c4e-881f-415c51600a83-istiod-ca-cert\") pod \"e2e-noten-7aca74-openshift-default-5df749499f-n9t77\" (UID: \"e398716c-4a76-4c4e-881f-415c51600a83\") " pod="openshift-ingress/e2e-noten-7aca74-openshift-default-5df749499f-n9t77"
Nov 02 13:00:55.278832 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:55.278709    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e398716c-4a76-4c4e-881f-415c51600a83-istio-podinfo\") pod \"e2e-noten-7aca74-openshift-default-5df749499f-n9t77\" (UID: \"e398716c-4a76-4c4e-881f-415c51600a83\") " pod="openshift-ingress/e2e-noten-7aca74-openshift-default-5df749499f-n9t77"
Nov 02 13:00:55.278832 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:55.278736    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e398716c-4a76-4c4e-881f-415c51600a83-workload-socket\") pod \"e2e-noten-7aca74-openshift-default-5df749499f-n9t77\" (UID: \"e398716c-4a76-4c4e-881f-415c51600a83\") " pod="openshift-ingress/e2e-noten-7aca74-openshift-default-5df749499f-n9t77"
Nov 02 13:00:55.278832 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:55.278783    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e398716c-4a76-4c4e-881f-415c51600a83-istio-data\") pod \"e2e-noten-7aca74-openshift-default-5df749499f-n9t77\" (UID: \"e398716c-4a76-4c4e-881f-415c51600a83\") " pod="openshift-ingress/e2e-noten-7aca74-openshift-default-5df749499f-n9t77"
Nov 02 13:00:55.279032 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:55.278865    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e398716c-4a76-4c4e-881f-415c51600a83-workload-certs\") pod \"e2e-noten-7aca74-openshift-default-5df749499f-n9t77\" (UID: \"e398716c-4a76-4c4e-881f-415c51600a83\") " pod="openshift-ingress/e2e-noten-7aca74-openshift-default-5df749499f-n9t77"
Nov 02 13:00:55.279032 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:55.278918    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e398716c-4a76-4c4e-881f-415c51600a83-credential-socket\") pod \"e2e-noten-7aca74-openshift-default-5df749499f-n9t77\" (UID: \"e398716c-4a76-4c4e-881f-415c51600a83\") " pod="openshift-ingress/e2e-noten-7aca74-openshift-default-5df749499f-n9t77"
Nov 02 13:00:55.287519 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:55.287460    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-740f09-openshift-default-786896ffcc-lmqlz" podStartSLOduration=2.287442334 podStartE2EDuration="2.287442334s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:00:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:00:55.286900181 +0000 UTC m=+2051.542299279" watchObservedRunningTime="2026-11-02 13:00:55.287442334 +0000 UTC m=+2051.542841430"
Nov 02 13:00:55.353065 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:55.353033    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-740f09-openshift-default-786896ffcc-lmqlz"
Nov 02 13:00:55.360189 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:55.360163    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-740f09-openshift-default-786896ffcc-lmqlz"
Nov 02 13:00:55.380156 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:55.380125    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e398716c-4a76-4c4e-881f-415c51600a83-istio-podinfo\") pod \"e2e-noten-7aca74-openshift-default-5df749499f-n9t77\" (UID: \"e398716c-4a76-4c4e-881f-415c51600a83\") " pod="openshift-ingress/e2e-noten-7aca74-openshift-default-5df749499f-n9t77"
Nov 02 13:00:55.380334 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:55.380164    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e398716c-4a76-4c4e-881f-415c51600a83-workload-socket\") pod \"e2e-noten-7aca74-openshift-default-5df749499f-n9t77\" (UID: \"e398716c-4a76-4c4e-881f-415c51600a83\") " pod="openshift-ingress/e2e-noten-7aca74-openshift-default-5df749499f-n9t77"
Nov 02 13:00:55.380334 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:55.380302    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e398716c-4a76-4c4e-881f-415c51600a83-istio-data\") pod \"e2e-noten-7aca74-openshift-default-5df749499f-n9t77\" (UID: \"e398716c-4a76-4c4e-881f-415c51600a83\") " pod="openshift-ingress/e2e-noten-7aca74-openshift-default-5df749499f-n9t77"
Nov 02 13:00:55.380473 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:55.380354    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e398716c-4a76-4c4e-881f-415c51600a83-workload-certs\") pod \"e2e-noten-7aca74-openshift-default-5df749499f-n9t77\" (UID: \"e398716c-4a76-4c4e-881f-415c51600a83\") " pod="openshift-ingress/e2e-noten-7aca74-openshift-default-5df749499f-n9t77"
Nov 02 13:00:55.380473 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:55.380406    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e398716c-4a76-4c4e-881f-415c51600a83-credential-socket\") pod \"e2e-noten-7aca74-openshift-default-5df749499f-n9t77\" (UID: \"e398716c-4a76-4c4e-881f-415c51600a83\") " pod="openshift-ingress/e2e-noten-7aca74-openshift-default-5df749499f-n9t77"
Nov 02 13:00:55.380615 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:55.380595    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxc7k\" (UniqueName: \"kubernetes.io/projected/e398716c-4a76-4c4e-881f-415c51600a83-kube-api-access-xxc7k\") pod \"e2e-noten-7aca74-openshift-default-5df749499f-n9t77\" (UID: \"e398716c-4a76-4c4e-881f-415c51600a83\") " pod="openshift-ingress/e2e-noten-7aca74-openshift-default-5df749499f-n9t77"
Nov 02 13:00:55.380717 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:55.380662    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e398716c-4a76-4c4e-881f-415c51600a83-istio-token\") pod \"e2e-noten-7aca74-openshift-default-5df749499f-n9t77\" (UID: \"e398716c-4a76-4c4e-881f-415c51600a83\") " pod="openshift-ingress/e2e-noten-7aca74-openshift-default-5df749499f-n9t77"
Nov 02 13:00:55.380717 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:55.380672    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e398716c-4a76-4c4e-881f-415c51600a83-workload-certs\") pod \"e2e-noten-7aca74-openshift-default-5df749499f-n9t77\" (UID: \"e398716c-4a76-4c4e-881f-415c51600a83\") " pod="openshift-ingress/e2e-noten-7aca74-openshift-default-5df749499f-n9t77"
Nov 02 13:00:55.380717 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:55.380697    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e398716c-4a76-4c4e-881f-415c51600a83-istio-envoy\") pod \"e2e-noten-7aca74-openshift-default-5df749499f-n9t77\" (UID: \"e398716c-4a76-4c4e-881f-415c51600a83\") " pod="openshift-ingress/e2e-noten-7aca74-openshift-default-5df749499f-n9t77"
Nov 02 13:00:55.380717 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:55.380599    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e398716c-4a76-4c4e-881f-415c51600a83-workload-socket\") pod \"e2e-noten-7aca74-openshift-default-5df749499f-n9t77\" (UID: \"e398716c-4a76-4c4e-881f-415c51600a83\") " pod="openshift-ingress/e2e-noten-7aca74-openshift-default-5df749499f-n9t77"
Nov 02 13:00:55.380942 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:55.380725    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e398716c-4a76-4c4e-881f-415c51600a83-istio-data\") pod \"e2e-noten-7aca74-openshift-default-5df749499f-n9t77\" (UID: \"e398716c-4a76-4c4e-881f-415c51600a83\") " pod="openshift-ingress/e2e-noten-7aca74-openshift-default-5df749499f-n9t77"
Nov 02 13:00:55.380942 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:55.380747    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e398716c-4a76-4c4e-881f-415c51600a83-istiod-ca-cert\") pod \"e2e-noten-7aca74-openshift-default-5df749499f-n9t77\" (UID: \"e398716c-4a76-4c4e-881f-415c51600a83\") " pod="openshift-ingress/e2e-noten-7aca74-openshift-default-5df749499f-n9t77"
Nov 02 13:00:55.380942 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:55.380814    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e398716c-4a76-4c4e-881f-415c51600a83-credential-socket\") pod \"e2e-noten-7aca74-openshift-default-5df749499f-n9t77\" (UID: \"e398716c-4a76-4c4e-881f-415c51600a83\") " pod="openshift-ingress/e2e-noten-7aca74-openshift-default-5df749499f-n9t77"
Nov 02 13:00:55.381366 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:55.381346    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e398716c-4a76-4c4e-881f-415c51600a83-istiod-ca-cert\") pod \"e2e-noten-7aca74-openshift-default-5df749499f-n9t77\" (UID: \"e398716c-4a76-4c4e-881f-415c51600a83\") " pod="openshift-ingress/e2e-noten-7aca74-openshift-default-5df749499f-n9t77"
Nov 02 13:00:55.383155 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:55.383132    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e398716c-4a76-4c4e-881f-415c51600a83-istio-envoy\") pod \"e2e-noten-7aca74-openshift-default-5df749499f-n9t77\" (UID: \"e398716c-4a76-4c4e-881f-415c51600a83\") " pod="openshift-ingress/e2e-noten-7aca74-openshift-default-5df749499f-n9t77"
Nov 02 13:00:55.383358 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:55.383338    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e398716c-4a76-4c4e-881f-415c51600a83-istio-podinfo\") pod \"e2e-noten-7aca74-openshift-default-5df749499f-n9t77\" (UID: \"e398716c-4a76-4c4e-881f-415c51600a83\") " pod="openshift-ingress/e2e-noten-7aca74-openshift-default-5df749499f-n9t77"
Nov 02 13:00:55.406674 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:55.406523    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e398716c-4a76-4c4e-881f-415c51600a83-istio-token\") pod \"e2e-noten-7aca74-openshift-default-5df749499f-n9t77\" (UID: \"e398716c-4a76-4c4e-881f-415c51600a83\") " pod="openshift-ingress/e2e-noten-7aca74-openshift-default-5df749499f-n9t77"
Nov 02 13:00:55.406852 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:55.406809    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxc7k\" (UniqueName: \"kubernetes.io/projected/e398716c-4a76-4c4e-881f-415c51600a83-kube-api-access-xxc7k\") pod \"e2e-noten-7aca74-openshift-default-5df749499f-n9t77\" (UID: \"e398716c-4a76-4c4e-881f-415c51600a83\") " pod="openshift-ingress/e2e-noten-7aca74-openshift-default-5df749499f-n9t77"
Nov 02 13:00:55.516250 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:55.515919    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-7aca74-openshift-default-5df749499f-n9t77"
Nov 02 13:00:55.808036 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:55.807957    4559 patch_prober.go:29] interesting pod/e2e-explicit-e430c7-openshift-default-8649b6f6fc-vl6ql container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 13:00:55.808036 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:55.808014    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-explicit-e430c7-openshift-default-8649b6f6fc-vl6ql" podUID="6f8849e4-f808-45f0-b24a-e1e1c8b99974" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 13:00:55.991095 ip-10-0-128-132 kubenswrapper[4559]: W1102 13:00:55.991067    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode398716c_4a76_4c4e_881f_415c51600a83.slice/crio-b55bf127e73698a573434da2a29b3119a2c32560dba9bcd7524bc1e9dc8acc3f WatchSource:0}: Error finding container b55bf127e73698a573434da2a29b3119a2c32560dba9bcd7524bc1e9dc8acc3f: Status 404 returned error can't find the container with id b55bf127e73698a573434da2a29b3119a2c32560dba9bcd7524bc1e9dc8acc3f
Nov 02 13:00:55.991374 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:55.991352    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-7aca74-openshift-default-5df749499f-n9t77"]
Nov 02 13:00:55.994124 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:55.994092    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 13:00:55.994207 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:55.994155    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 13:00:55.994207 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:55.994188    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 13:00:56.249254 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:56.249210    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-7aca74-openshift-default-5df749499f-n9t77" event={"ID":"e398716c-4a76-4c4e-881f-415c51600a83","Type":"ContainerStarted","Data":"53b372e8a46893c77498b89edcd4add5a74c5700156e921a0359889920aad8e1"}
Nov 02 13:00:56.249470 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:56.249318    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-7aca74-openshift-default-5df749499f-n9t77" event={"ID":"e398716c-4a76-4c4e-881f-415c51600a83","Type":"ContainerStarted","Data":"b55bf127e73698a573434da2a29b3119a2c32560dba9bcd7524bc1e9dc8acc3f"}
Nov 02 13:00:56.249538 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:56.249487    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-740f09-openshift-default-786896ffcc-lmqlz"
Nov 02 13:00:56.250440 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:56.250419    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-740f09-openshift-default-786896ffcc-lmqlz"
Nov 02 13:00:56.298121 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:56.298071    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-noten-7aca74-openshift-default-5df749499f-n9t77" podStartSLOduration=1.298056249 podStartE2EDuration="1.298056249s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:00:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:00:56.294670479 +0000 UTC m=+2052.550069567" watchObservedRunningTime="2026-11-02 13:00:56.298056249 +0000 UTC m=+2052.553455344"
Nov 02 13:00:56.401799 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:56.401702    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w2-1e14d4/auth-test-model-1e14d4-kserve-97cc85bb9-5jjh5" podUID="9664d93b-e25a-430b-bef1-f4570d950694" containerName="main" containerID="cri-o://16fcf1f4c720cb22e6d39df2a40cfddb2dc550fbf9ea980193de9f432f67177d" gracePeriod=45
Nov 02 13:00:56.519954 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:56.517048    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-noten-7aca74-openshift-default-5df749499f-n9t77"
Nov 02 13:00:56.673000 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:56.672974    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-1e14d4/auth-test-model-1e14d4-kserve-97cc85bb9-5jjh5"
Nov 02 13:00:56.821157 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:56.821123    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9664d93b-e25a-430b-bef1-f4570d950694-tls-certs\" (UniqueName: \"kubernetes.io/secret/9664d93b-e25a-430b-bef1-f4570d950694-tls-certs\") pod \"9664d93b-e25a-430b-bef1-f4570d950694\" (UID: \"9664d93b-e25a-430b-bef1-f4570d950694\") "
Nov 02 13:00:56.821596 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:56.821226    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9664d93b-e25a-430b-bef1-f4570d950694-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9664d93b-e25a-430b-bef1-f4570d950694-model-cache\") pod \"9664d93b-e25a-430b-bef1-f4570d950694\" (UID: \"9664d93b-e25a-430b-bef1-f4570d950694\") "
Nov 02 13:00:56.821596 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:56.821265    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9664d93b-e25a-430b-bef1-f4570d950694-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9664d93b-e25a-430b-bef1-f4570d950694-tmp-dir\") pod \"9664d93b-e25a-430b-bef1-f4570d950694\" (UID: \"9664d93b-e25a-430b-bef1-f4570d950694\") "
Nov 02 13:00:56.821596 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:56.821294    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9664d93b-e25a-430b-bef1-f4570d950694-home\" (UniqueName: \"kubernetes.io/empty-dir/9664d93b-e25a-430b-bef1-f4570d950694-home\") pod \"9664d93b-e25a-430b-bef1-f4570d950694\" (UID: \"9664d93b-e25a-430b-bef1-f4570d950694\") "
Nov 02 13:00:56.821596 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:56.821324    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9664d93b-e25a-430b-bef1-f4570d950694-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9664d93b-e25a-430b-bef1-f4570d950694-dshm\") pod \"9664d93b-e25a-430b-bef1-f4570d950694\" (UID: \"9664d93b-e25a-430b-bef1-f4570d950694\") "
Nov 02 13:00:56.821596 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:56.821439    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9664d93b-e25a-430b-bef1-f4570d950694-kube-api-access-rp4vg\" (UniqueName: \"kubernetes.io/projected/9664d93b-e25a-430b-bef1-f4570d950694-kube-api-access-rp4vg\") pod \"9664d93b-e25a-430b-bef1-f4570d950694\" (UID: \"9664d93b-e25a-430b-bef1-f4570d950694\") "
Nov 02 13:00:56.821596 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:56.821509    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9664d93b-e25a-430b-bef1-f4570d950694-model-cache" pod "9664d93b-e25a-430b-bef1-f4570d950694" (UID: "9664d93b-e25a-430b-bef1-f4570d950694"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:00:56.821596 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:56.821527    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9664d93b-e25a-430b-bef1-f4570d950694-home" pod "9664d93b-e25a-430b-bef1-f4570d950694" (UID: "9664d93b-e25a-430b-bef1-f4570d950694"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:00:56.821596 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:56.821558    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9664d93b-e25a-430b-bef1-f4570d950694-tmp-dir" pod "9664d93b-e25a-430b-bef1-f4570d950694" (UID: "9664d93b-e25a-430b-bef1-f4570d950694"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:00:56.822084 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:56.822064    4559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9664d93b-e25a-430b-bef1-f4570d950694-model-cache\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:00:56.822130 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:56.822092    4559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9664d93b-e25a-430b-bef1-f4570d950694-tmp-dir\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:00:56.822130 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:56.822107    4559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9664d93b-e25a-430b-bef1-f4570d950694-home\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:00:56.823491 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:56.823467    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9664d93b-e25a-430b-bef1-f4570d950694-dshm" pod "9664d93b-e25a-430b-bef1-f4570d950694" (UID: "9664d93b-e25a-430b-bef1-f4570d950694"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:00:56.823576 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:56.823517    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9664d93b-e25a-430b-bef1-f4570d950694-tls-certs" pod "9664d93b-e25a-430b-bef1-f4570d950694" (UID: "9664d93b-e25a-430b-bef1-f4570d950694"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:00:56.823614 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:56.823578    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9664d93b-e25a-430b-bef1-f4570d950694-kube-api-access-rp4vg" pod "9664d93b-e25a-430b-bef1-f4570d950694" (UID: "9664d93b-e25a-430b-bef1-f4570d950694"). InnerVolumeSpecName "kube-api-access-rp4vg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:00:56.923766 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:56.923732    4559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9664d93b-e25a-430b-bef1-f4570d950694-tls-certs\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:00:56.923766 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:56.923767    4559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9664d93b-e25a-430b-bef1-f4570d950694-dshm\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:00:56.924003 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:56.923781    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rp4vg\" (UniqueName: \"kubernetes.io/projected/9664d93b-e25a-430b-bef1-f4570d950694-kube-api-access-rp4vg\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:00:56.926819 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:56.926795    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-035dac-7bf5549cc6-hd2vb"]
Nov 02 13:00:56.927429 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:56.927415    4559 cpu_manager.go:414] "containerMap: removing container" podUID="9664d93b-e25a-430b-bef1-f4570d950694" containerName="main"
Nov 02 13:00:56.927468 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:56.927431    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9664d93b-e25a-430b-bef1-f4570d950694" containerName="main"
Nov 02 13:00:56.927529 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:56.927520    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="9664d93b-e25a-430b-bef1-f4570d950694" containerName="main"
Nov 02 13:00:56.931171 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:56.931149    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-035dac-7bf5549cc6-hd2vb"
Nov 02 13:00:56.952379 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:56.952347    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-035dac-7bf5549cc6-hd2vb"]
Nov 02 13:00:57.025036 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:57.025003    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc6f8750-45f1-4dfe-b2a0-f381d0dde490-maas-api-tls\") pod \"maas-api-e2e-extmod-a-035dac-7bf5549cc6-hd2vb\" (UID: \"cc6f8750-45f1-4dfe-b2a0-f381d0dde490\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-035dac-7bf5549cc6-hd2vb"
Nov 02 13:00:57.025213 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:57.025075    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79z7q\" (UniqueName: \"kubernetes.io/projected/cc6f8750-45f1-4dfe-b2a0-f381d0dde490-kube-api-access-79z7q\") pod \"maas-api-e2e-extmod-a-035dac-7bf5549cc6-hd2vb\" (UID: \"cc6f8750-45f1-4dfe-b2a0-f381d0dde490\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-035dac-7bf5549cc6-hd2vb"
Nov 02 13:00:57.025279 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:57.025256    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc6f8750-45f1-4dfe-b2a0-f381d0dde490-metrics-certs\") pod \"maas-api-e2e-extmod-a-035dac-7bf5549cc6-hd2vb\" (UID: \"cc6f8750-45f1-4dfe-b2a0-f381d0dde490\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-035dac-7bf5549cc6-hd2vb"
Nov 02 13:00:57.025322 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:57.025307    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc6f8750-45f1-4dfe-b2a0-f381d0dde490-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-035dac-7bf5549cc6-hd2vb\" (UID: \"cc6f8750-45f1-4dfe-b2a0-f381d0dde490\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-035dac-7bf5549cc6-hd2vb"
Nov 02 13:00:57.030652 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:57.030617    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-035dac-serving-cert\""
Nov 02 13:00:57.126623 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:57.126587    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc6f8750-45f1-4dfe-b2a0-f381d0dde490-maas-api-tls\") pod \"maas-api-e2e-extmod-a-035dac-7bf5549cc6-hd2vb\" (UID: \"cc6f8750-45f1-4dfe-b2a0-f381d0dde490\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-035dac-7bf5549cc6-hd2vb"
Nov 02 13:00:57.126819 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:57.126702    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79z7q\" (UniqueName: \"kubernetes.io/projected/cc6f8750-45f1-4dfe-b2a0-f381d0dde490-kube-api-access-79z7q\") pod \"maas-api-e2e-extmod-a-035dac-7bf5549cc6-hd2vb\" (UID: \"cc6f8750-45f1-4dfe-b2a0-f381d0dde490\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-035dac-7bf5549cc6-hd2vb"
Nov 02 13:00:57.126819 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:57.126810    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc6f8750-45f1-4dfe-b2a0-f381d0dde490-metrics-certs\") pod \"maas-api-e2e-extmod-a-035dac-7bf5549cc6-hd2vb\" (UID: \"cc6f8750-45f1-4dfe-b2a0-f381d0dde490\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-035dac-7bf5549cc6-hd2vb"
Nov 02 13:00:57.126926 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:57.126838    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc6f8750-45f1-4dfe-b2a0-f381d0dde490-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-035dac-7bf5549cc6-hd2vb\" (UID: \"cc6f8750-45f1-4dfe-b2a0-f381d0dde490\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-035dac-7bf5549cc6-hd2vb"
Nov 02 13:00:57.127470 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:57.127439    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc6f8750-45f1-4dfe-b2a0-f381d0dde490-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-035dac-7bf5549cc6-hd2vb\" (UID: \"cc6f8750-45f1-4dfe-b2a0-f381d0dde490\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-035dac-7bf5549cc6-hd2vb"
Nov 02 13:00:57.129162 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:57.129138    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc6f8750-45f1-4dfe-b2a0-f381d0dde490-maas-api-tls\") pod \"maas-api-e2e-extmod-a-035dac-7bf5549cc6-hd2vb\" (UID: \"cc6f8750-45f1-4dfe-b2a0-f381d0dde490\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-035dac-7bf5549cc6-hd2vb"
Nov 02 13:00:57.129272 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:57.129186    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc6f8750-45f1-4dfe-b2a0-f381d0dde490-metrics-certs\") pod \"maas-api-e2e-extmod-a-035dac-7bf5549cc6-hd2vb\" (UID: \"cc6f8750-45f1-4dfe-b2a0-f381d0dde490\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-035dac-7bf5549cc6-hd2vb"
Nov 02 13:00:57.187732 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:57.187580    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79z7q\" (UniqueName: \"kubernetes.io/projected/cc6f8750-45f1-4dfe-b2a0-f381d0dde490-kube-api-access-79z7q\") pod \"maas-api-e2e-extmod-a-035dac-7bf5549cc6-hd2vb\" (UID: \"cc6f8750-45f1-4dfe-b2a0-f381d0dde490\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-035dac-7bf5549cc6-hd2vb"
Nov 02 13:00:57.245888 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:57.245852    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-035dac-7bf5549cc6-hd2vb"
Nov 02 13:00:57.257883 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:57.257857    4559 generic.go:356] "Generic (PLEG): container finished" podID="9664d93b-e25a-430b-bef1-f4570d950694" containerID="16fcf1f4c720cb22e6d39df2a40cfddb2dc550fbf9ea980193de9f432f67177d" exitCode=0
Nov 02 13:00:57.258008 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:57.257905    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-1e14d4/auth-test-model-1e14d4-kserve-97cc85bb9-5jjh5"
Nov 02 13:00:57.258008 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:57.257953    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-1e14d4/auth-test-model-1e14d4-kserve-97cc85bb9-5jjh5" event={"ID":"9664d93b-e25a-430b-bef1-f4570d950694","Type":"ContainerDied","Data":"16fcf1f4c720cb22e6d39df2a40cfddb2dc550fbf9ea980193de9f432f67177d"}
Nov 02 13:00:57.258008 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:57.257993    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-1e14d4/auth-test-model-1e14d4-kserve-97cc85bb9-5jjh5" event={"ID":"9664d93b-e25a-430b-bef1-f4570d950694","Type":"ContainerDied","Data":"3f61eb7514d12af7dd4ef9eaea666c24d538e02ab2a8b0b5d35a127864d2e85f"}
Nov 02 13:00:57.258171 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:57.258011    4559 scope.go:122] "RemoveContainer" containerID="16fcf1f4c720cb22e6d39df2a40cfddb2dc550fbf9ea980193de9f432f67177d"
Nov 02 13:00:57.275151 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:57.275130    4559 scope.go:122] "RemoveContainer" containerID="16fcf1f4c720cb22e6d39df2a40cfddb2dc550fbf9ea980193de9f432f67177d"
Nov 02 13:00:57.275478 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:00:57.275456    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16fcf1f4c720cb22e6d39df2a40cfddb2dc550fbf9ea980193de9f432f67177d\": container with ID starting with 16fcf1f4c720cb22e6d39df2a40cfddb2dc550fbf9ea980193de9f432f67177d not found: ID does not exist" containerID="16fcf1f4c720cb22e6d39df2a40cfddb2dc550fbf9ea980193de9f432f67177d"
Nov 02 13:00:57.275558 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:57.275490    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16fcf1f4c720cb22e6d39df2a40cfddb2dc550fbf9ea980193de9f432f67177d"} err="failed to get container status \"16fcf1f4c720cb22e6d39df2a40cfddb2dc550fbf9ea980193de9f432f67177d\": rpc error: code = NotFound desc = could not find container \"16fcf1f4c720cb22e6d39df2a40cfddb2dc550fbf9ea980193de9f432f67177d\": container with ID starting with 16fcf1f4c720cb22e6d39df2a40cfddb2dc550fbf9ea980193de9f432f67177d not found: ID does not exist"
Nov 02 13:00:57.316241 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:57.316131    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-1e14d4/auth-test-model-1e14d4-kserve-97cc85bb9-5jjh5"]
Nov 02 13:00:57.319563 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:57.319540    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w2-1e14d4/auth-test-model-1e14d4-kserve-97cc85bb9-5jjh5"]
Nov 02 13:00:57.523331 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:57.523256    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-noten-7aca74-openshift-default-5df749499f-n9t77"
Nov 02 13:00:57.568833 ip-10-0-128-132 kubenswrapper[4559]: W1102 13:00:57.568801    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc6f8750_45f1_4dfe_b2a0_f381d0dde490.slice/crio-bdcbf9f676bdeab5d7274e8e1c2d1c3deefdb53ddc940219fa4ca256d59e7f79 WatchSource:0}: Error finding container bdcbf9f676bdeab5d7274e8e1c2d1c3deefdb53ddc940219fa4ca256d59e7f79: Status 404 returned error can't find the container with id bdcbf9f676bdeab5d7274e8e1c2d1c3deefdb53ddc940219fa4ca256d59e7f79
Nov 02 13:00:57.569823 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:57.569565    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-035dac-7bf5549cc6-hd2vb"]
Nov 02 13:00:57.570961 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:57.570940    4559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 13:00:58.005663 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:58.005082    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9664d93b-e25a-430b-bef1-f4570d950694" path="/var/lib/kubelet/pods/9664d93b-e25a-430b-bef1-f4570d950694/volumes"
Nov 02 13:00:58.302911 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:58.302813    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-035dac-7bf5549cc6-hd2vb" event={"ID":"cc6f8750-45f1-4dfe-b2a0-f381d0dde490","Type":"ContainerStarted","Data":"f816912a6917249d7fbfe5103d545f57c66a0a443764fa6d03da2d7666b4f066"}
Nov 02 13:00:58.304178 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:58.304151    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-035dac-7bf5549cc6-hd2vb" event={"ID":"cc6f8750-45f1-4dfe-b2a0-f381d0dde490","Type":"ContainerStarted","Data":"bdcbf9f676bdeab5d7274e8e1c2d1c3deefdb53ddc940219fa4ca256d59e7f79"}
Nov 02 13:00:58.304243 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:58.304229    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-035dac-7bf5549cc6-hd2vb"
Nov 02 13:00:58.304288 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:58.304257    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-noten-7aca74-openshift-default-5df749499f-n9t77"
Nov 02 13:00:58.304496 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:58.304480    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-noten-7aca74-openshift-default-5df749499f-n9t77"
Nov 02 13:00:58.365331 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:58.365274    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-035dac-7bf5549cc6-hd2vb" podStartSLOduration=1.864389317 podStartE2EDuration="2.365255413s" totalImagesPullingTime="500.866096ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:00:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:00:58.363683977 +0000 UTC m=+2054.619083072" watchObservedRunningTime="2026-11-02 13:00:58.365255413 +0000 UTC m=+2054.620654510"
Nov 02 13:00:59.637034 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:59.637007    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-e430c7-openshift-default-8649b6f6fc-vl6ql"
Nov 02 13:00:59.787871 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:59.787782    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f8849e4-f808-45f0-b24a-e1e1c8b99974-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f8849e4-f808-45f0-b24a-e1e1c8b99974-workload-socket\") pod \"6f8849e4-f808-45f0-b24a-e1e1c8b99974\" (UID: \"6f8849e4-f808-45f0-b24a-e1e1c8b99974\") "
Nov 02 13:00:59.787871 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:59.787845    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f8849e4-f808-45f0-b24a-e1e1c8b99974-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f8849e4-f808-45f0-b24a-e1e1c8b99974-istio-envoy\") pod \"6f8849e4-f808-45f0-b24a-e1e1c8b99974\" (UID: \"6f8849e4-f808-45f0-b24a-e1e1c8b99974\") "
Nov 02 13:00:59.788092 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:59.787884    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f8849e4-f808-45f0-b24a-e1e1c8b99974-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f8849e4-f808-45f0-b24a-e1e1c8b99974-credential-socket\") pod \"6f8849e4-f808-45f0-b24a-e1e1c8b99974\" (UID: \"6f8849e4-f808-45f0-b24a-e1e1c8b99974\") "
Nov 02 13:00:59.788092 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:59.787916    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f8849e4-f808-45f0-b24a-e1e1c8b99974-kube-api-access-97dkk\" (UniqueName: \"kubernetes.io/projected/6f8849e4-f808-45f0-b24a-e1e1c8b99974-kube-api-access-97dkk\") pod \"6f8849e4-f808-45f0-b24a-e1e1c8b99974\" (UID: \"6f8849e4-f808-45f0-b24a-e1e1c8b99974\") "
Nov 02 13:00:59.788092 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:59.787965    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f8849e4-f808-45f0-b24a-e1e1c8b99974-istio-token\" (UniqueName: \"kubernetes.io/projected/6f8849e4-f808-45f0-b24a-e1e1c8b99974-istio-token\") pod \"6f8849e4-f808-45f0-b24a-e1e1c8b99974\" (UID: \"6f8849e4-f808-45f0-b24a-e1e1c8b99974\") "
Nov 02 13:00:59.788092 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:59.788082    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f8849e4-f808-45f0-b24a-e1e1c8b99974-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f8849e4-f808-45f0-b24a-e1e1c8b99974-workload-certs\") pod \"6f8849e4-f808-45f0-b24a-e1e1c8b99974\" (UID: \"6f8849e4-f808-45f0-b24a-e1e1c8b99974\") "
Nov 02 13:00:59.788312 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:59.788163    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6f8849e4-f808-45f0-b24a-e1e1c8b99974-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f8849e4-f808-45f0-b24a-e1e1c8b99974-istio-podinfo\") pod \"6f8849e4-f808-45f0-b24a-e1e1c8b99974\" (UID: \"6f8849e4-f808-45f0-b24a-e1e1c8b99974\") "
Nov 02 13:00:59.788312 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:59.788074    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f8849e4-f808-45f0-b24a-e1e1c8b99974-workload-socket" pod "6f8849e4-f808-45f0-b24a-e1e1c8b99974" (UID: "6f8849e4-f808-45f0-b24a-e1e1c8b99974"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:00:59.788312 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:59.788254    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f8849e4-f808-45f0-b24a-e1e1c8b99974-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f8849e4-f808-45f0-b24a-e1e1c8b99974-istio-data\") pod \"6f8849e4-f808-45f0-b24a-e1e1c8b99974\" (UID: \"6f8849e4-f808-45f0-b24a-e1e1c8b99974\") "
Nov 02 13:00:59.788312 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:59.788255    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f8849e4-f808-45f0-b24a-e1e1c8b99974-credential-socket" pod "6f8849e4-f808-45f0-b24a-e1e1c8b99974" (UID: "6f8849e4-f808-45f0-b24a-e1e1c8b99974"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:00:59.788559 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:59.788320    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f8849e4-f808-45f0-b24a-e1e1c8b99974-workload-certs" pod "6f8849e4-f808-45f0-b24a-e1e1c8b99974" (UID: "6f8849e4-f808-45f0-b24a-e1e1c8b99974"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:00:59.788559 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:59.788328    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f8849e4-f808-45f0-b24a-e1e1c8b99974-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f8849e4-f808-45f0-b24a-e1e1c8b99974-istiod-ca-cert\") pod \"6f8849e4-f808-45f0-b24a-e1e1c8b99974\" (UID: \"6f8849e4-f808-45f0-b24a-e1e1c8b99974\") "
Nov 02 13:00:59.788687 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:59.788571    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f8849e4-f808-45f0-b24a-e1e1c8b99974-istio-data" pod "6f8849e4-f808-45f0-b24a-e1e1c8b99974" (UID: "6f8849e4-f808-45f0-b24a-e1e1c8b99974"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:00:59.788687 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:59.788633    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f8849e4-f808-45f0-b24a-e1e1c8b99974-istiod-ca-cert" pod "6f8849e4-f808-45f0-b24a-e1e1c8b99974" (UID: "6f8849e4-f808-45f0-b24a-e1e1c8b99974"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:00:59.789006 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:59.788985    4559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f8849e4-f808-45f0-b24a-e1e1c8b99974-workload-certs\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:00:59.789091 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:59.789008    4559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f8849e4-f808-45f0-b24a-e1e1c8b99974-istio-data\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:00:59.789091 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:59.789024    4559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f8849e4-f808-45f0-b24a-e1e1c8b99974-istiod-ca-cert\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:00:59.789091 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:59.789035    4559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f8849e4-f808-45f0-b24a-e1e1c8b99974-workload-socket\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:00:59.789091 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:59.789050    4559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f8849e4-f808-45f0-b24a-e1e1c8b99974-credential-socket\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:00:59.790331 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:59.790303    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f8849e4-f808-45f0-b24a-e1e1c8b99974-kube-api-access-97dkk" pod "6f8849e4-f808-45f0-b24a-e1e1c8b99974" (UID: "6f8849e4-f808-45f0-b24a-e1e1c8b99974"). InnerVolumeSpecName "kube-api-access-97dkk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:00:59.790431 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:59.790351    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6f8849e4-f808-45f0-b24a-e1e1c8b99974-istio-podinfo" pod "6f8849e4-f808-45f0-b24a-e1e1c8b99974" (UID: "6f8849e4-f808-45f0-b24a-e1e1c8b99974"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 13:00:59.790520 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:59.790501    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f8849e4-f808-45f0-b24a-e1e1c8b99974-istio-envoy" pod "6f8849e4-f808-45f0-b24a-e1e1c8b99974" (UID: "6f8849e4-f808-45f0-b24a-e1e1c8b99974"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:00:59.790572 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:59.790546    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f8849e4-f808-45f0-b24a-e1e1c8b99974-istio-token" pod "6f8849e4-f808-45f0-b24a-e1e1c8b99974" (UID: "6f8849e4-f808-45f0-b24a-e1e1c8b99974"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:00:59.890843 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:59.890810    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-97dkk\" (UniqueName: \"kubernetes.io/projected/6f8849e4-f808-45f0-b24a-e1e1c8b99974-kube-api-access-97dkk\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:00:59.890843 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:59.890838    4559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f8849e4-f808-45f0-b24a-e1e1c8b99974-istio-token\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:00:59.890843 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:59.890847    4559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f8849e4-f808-45f0-b24a-e1e1c8b99974-istio-podinfo\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:00:59.891091 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:00:59.890855    4559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f8849e4-f808-45f0-b24a-e1e1c8b99974-istio-envoy\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:01:00.325427 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:00.325383    4559 generic.go:356] "Generic (PLEG): container finished" podID="6f8849e4-f808-45f0-b24a-e1e1c8b99974" containerID="36e7f4b54929fd9b7406a9c8575aec0290af27998119ca3607b55489f4d601ec" exitCode=0
Nov 02 13:01:00.325659 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:00.325436    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-e430c7-openshift-default-8649b6f6fc-vl6ql"
Nov 02 13:01:00.325659 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:00.325466    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-e430c7-openshift-default-8649b6f6fc-vl6ql" event={"ID":"6f8849e4-f808-45f0-b24a-e1e1c8b99974","Type":"ContainerDied","Data":"36e7f4b54929fd9b7406a9c8575aec0290af27998119ca3607b55489f4d601ec"}
Nov 02 13:01:00.325659 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:00.325521    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-e430c7-openshift-default-8649b6f6fc-vl6ql" event={"ID":"6f8849e4-f808-45f0-b24a-e1e1c8b99974","Type":"ContainerDied","Data":"5abbbdcf6276879d5b8a643acac07f0b397f93eec4d335020ab869837795884a"}
Nov 02 13:01:00.325659 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:00.325545    4559 scope.go:122] "RemoveContainer" containerID="36e7f4b54929fd9b7406a9c8575aec0290af27998119ca3607b55489f4d601ec"
Nov 02 13:01:00.335898 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:00.335869    4559 scope.go:122] "RemoveContainer" containerID="01e7c9f818f941d68525708b9d7b5448400bc31b9777538360f08d5a61d40e6d"
Nov 02 13:01:00.344933 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:00.344911    4559 scope.go:122] "RemoveContainer" containerID="36e7f4b54929fd9b7406a9c8575aec0290af27998119ca3607b55489f4d601ec"
Nov 02 13:01:00.345210 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:01:00.345189    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36e7f4b54929fd9b7406a9c8575aec0290af27998119ca3607b55489f4d601ec\": container with ID starting with 36e7f4b54929fd9b7406a9c8575aec0290af27998119ca3607b55489f4d601ec not found: ID does not exist" containerID="36e7f4b54929fd9b7406a9c8575aec0290af27998119ca3607b55489f4d601ec"
Nov 02 13:01:00.345266 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:00.345219    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36e7f4b54929fd9b7406a9c8575aec0290af27998119ca3607b55489f4d601ec"} err="failed to get container status \"36e7f4b54929fd9b7406a9c8575aec0290af27998119ca3607b55489f4d601ec\": rpc error: code = NotFound desc = could not find container \"36e7f4b54929fd9b7406a9c8575aec0290af27998119ca3607b55489f4d601ec\": container with ID starting with 36e7f4b54929fd9b7406a9c8575aec0290af27998119ca3607b55489f4d601ec not found: ID does not exist"
Nov 02 13:01:00.345266 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:00.345240    4559 scope.go:122] "RemoveContainer" containerID="01e7c9f818f941d68525708b9d7b5448400bc31b9777538360f08d5a61d40e6d"
Nov 02 13:01:00.345450 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:01:00.345433    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01e7c9f818f941d68525708b9d7b5448400bc31b9777538360f08d5a61d40e6d\": container with ID starting with 01e7c9f818f941d68525708b9d7b5448400bc31b9777538360f08d5a61d40e6d not found: ID does not exist" containerID="01e7c9f818f941d68525708b9d7b5448400bc31b9777538360f08d5a61d40e6d"
Nov 02 13:01:00.345490 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:00.345456    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01e7c9f818f941d68525708b9d7b5448400bc31b9777538360f08d5a61d40e6d"} err="failed to get container status \"01e7c9f818f941d68525708b9d7b5448400bc31b9777538360f08d5a61d40e6d\": rpc error: code = NotFound desc = could not find container \"01e7c9f818f941d68525708b9d7b5448400bc31b9777538360f08d5a61d40e6d\": container with ID starting with 01e7c9f818f941d68525708b9d7b5448400bc31b9777538360f08d5a61d40e6d not found: ID does not exist"
Nov 02 13:01:00.396049 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:00.396020    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-e430c7-openshift-default-8649b6f6fc-vl6ql"]
Nov 02 13:01:00.408557 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:00.408528    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-explicit-e430c7-openshift-default-8649b6f6fc-vl6ql"]
Nov 02 13:01:01.937680 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:01.937628    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f8849e4-f808-45f0-b24a-e1e1c8b99974" path="/var/lib/kubelet/pods/6f8849e4-f808-45f0-b24a-e1e1c8b99974/volumes"
Nov 02 13:01:05.332201 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:05.332170    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-035dac-7bf5549cc6-hd2vb"
Nov 02 13:01:14.033673 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:14.029629    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-25f7a6-595fd6dd77-72r7x"]
Nov 02 13:01:14.033673 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:14.030661    4559 cpu_manager.go:414] "containerMap: removing container" podUID="6f8849e4-f808-45f0-b24a-e1e1c8b99974" containerName="istio-proxy"
Nov 02 13:01:14.033673 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:14.030683    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f8849e4-f808-45f0-b24a-e1e1c8b99974" containerName="istio-proxy"
Nov 02 13:01:14.033673 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:14.030717    4559 cpu_manager.go:414] "containerMap: removing container" podUID="6f8849e4-f808-45f0-b24a-e1e1c8b99974" containerName="istio-proxy"
Nov 02 13:01:14.033673 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:14.030725    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f8849e4-f808-45f0-b24a-e1e1c8b99974" containerName="istio-proxy"
Nov 02 13:01:14.033673 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:14.030899    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f8849e4-f808-45f0-b24a-e1e1c8b99974" containerName="istio-proxy"
Nov 02 13:01:14.033673 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:14.030917    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f8849e4-f808-45f0-b24a-e1e1c8b99974" containerName="istio-proxy"
Nov 02 13:01:14.070484 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:14.067677    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-25f7a6-595fd6dd77-72r7x"
Nov 02 13:01:14.077762 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:14.071120    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-25f7a6-595fd6dd77-72r7x"]
Nov 02 13:01:14.086663 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:14.084864    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-25f7a6-serving-cert\""
Nov 02 13:01:14.285747 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:14.281657    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e97a314b-6891-4a14-9f4f-c7131a59e231-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-25f7a6-595fd6dd77-72r7x\" (UID: \"e97a314b-6891-4a14-9f4f-c7131a59e231\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-25f7a6-595fd6dd77-72r7x"
Nov 02 13:01:14.285747 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:14.281741    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e97a314b-6891-4a14-9f4f-c7131a59e231-metrics-certs\") pod \"maas-api-e2e-extmod-a-25f7a6-595fd6dd77-72r7x\" (UID: \"e97a314b-6891-4a14-9f4f-c7131a59e231\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-25f7a6-595fd6dd77-72r7x"
Nov 02 13:01:14.285747 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:14.281797    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e97a314b-6891-4a14-9f4f-c7131a59e231-maas-api-tls\") pod \"maas-api-e2e-extmod-a-25f7a6-595fd6dd77-72r7x\" (UID: \"e97a314b-6891-4a14-9f4f-c7131a59e231\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-25f7a6-595fd6dd77-72r7x"
Nov 02 13:01:14.285747 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:14.281998    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dk7p5\" (UniqueName: \"kubernetes.io/projected/e97a314b-6891-4a14-9f4f-c7131a59e231-kube-api-access-dk7p5\") pod \"maas-api-e2e-extmod-a-25f7a6-595fd6dd77-72r7x\" (UID: \"e97a314b-6891-4a14-9f4f-c7131a59e231\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-25f7a6-595fd6dd77-72r7x"
Nov 02 13:01:14.395678 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:14.387771    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dk7p5\" (UniqueName: \"kubernetes.io/projected/e97a314b-6891-4a14-9f4f-c7131a59e231-kube-api-access-dk7p5\") pod \"maas-api-e2e-extmod-a-25f7a6-595fd6dd77-72r7x\" (UID: \"e97a314b-6891-4a14-9f4f-c7131a59e231\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-25f7a6-595fd6dd77-72r7x"
Nov 02 13:01:14.395678 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:14.387919    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e97a314b-6891-4a14-9f4f-c7131a59e231-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-25f7a6-595fd6dd77-72r7x\" (UID: \"e97a314b-6891-4a14-9f4f-c7131a59e231\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-25f7a6-595fd6dd77-72r7x"
Nov 02 13:01:14.395678 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:14.387968    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e97a314b-6891-4a14-9f4f-c7131a59e231-metrics-certs\") pod \"maas-api-e2e-extmod-a-25f7a6-595fd6dd77-72r7x\" (UID: \"e97a314b-6891-4a14-9f4f-c7131a59e231\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-25f7a6-595fd6dd77-72r7x"
Nov 02 13:01:14.395678 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:14.388019    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e97a314b-6891-4a14-9f4f-c7131a59e231-maas-api-tls\") pod \"maas-api-e2e-extmod-a-25f7a6-595fd6dd77-72r7x\" (UID: \"e97a314b-6891-4a14-9f4f-c7131a59e231\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-25f7a6-595fd6dd77-72r7x"
Nov 02 13:01:14.395678 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:01:14.388292    4559 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-a-25f7a6-serving-cert: secret "maas-api-e2e-extmod-a-25f7a6-serving-cert" not found
Nov 02 13:01:14.395678 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:01:14.388365    4559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e97a314b-6891-4a14-9f4f-c7131a59e231-maas-api-tls podName:e97a314b-6891-4a14-9f4f-c7131a59e231 nodeName:}" failed. No retries permitted until 2026-11-02 13:01:14.8883436 +0000 UTC m=+2071.143742679 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e97a314b-6891-4a14-9f4f-c7131a59e231-maas-api-tls") pod "maas-api-e2e-extmod-a-25f7a6-595fd6dd77-72r7x" (UID: "e97a314b-6891-4a14-9f4f-c7131a59e231") : secret "maas-api-e2e-extmod-a-25f7a6-serving-cert" not found
Nov 02 13:01:14.395678 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:14.389574    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e97a314b-6891-4a14-9f4f-c7131a59e231-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-25f7a6-595fd6dd77-72r7x\" (UID: \"e97a314b-6891-4a14-9f4f-c7131a59e231\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-25f7a6-595fd6dd77-72r7x"
Nov 02 13:01:14.445416 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:14.445373    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dk7p5\" (UniqueName: \"kubernetes.io/projected/e97a314b-6891-4a14-9f4f-c7131a59e231-kube-api-access-dk7p5\") pod \"maas-api-e2e-extmod-a-25f7a6-595fd6dd77-72r7x\" (UID: \"e97a314b-6891-4a14-9f4f-c7131a59e231\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-25f7a6-595fd6dd77-72r7x"
Nov 02 13:01:14.445626 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:14.445448    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e97a314b-6891-4a14-9f4f-c7131a59e231-metrics-certs\") pod \"maas-api-e2e-extmod-a-25f7a6-595fd6dd77-72r7x\" (UID: \"e97a314b-6891-4a14-9f4f-c7131a59e231\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-25f7a6-595fd6dd77-72r7x"
Nov 02 13:01:14.910960 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:14.910915    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e97a314b-6891-4a14-9f4f-c7131a59e231-maas-api-tls\") pod \"maas-api-e2e-extmod-a-25f7a6-595fd6dd77-72r7x\" (UID: \"e97a314b-6891-4a14-9f4f-c7131a59e231\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-25f7a6-595fd6dd77-72r7x"
Nov 02 13:01:14.937220 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:14.936660    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e97a314b-6891-4a14-9f4f-c7131a59e231-maas-api-tls\") pod \"maas-api-e2e-extmod-a-25f7a6-595fd6dd77-72r7x\" (UID: \"e97a314b-6891-4a14-9f4f-c7131a59e231\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-25f7a6-595fd6dd77-72r7x"
Nov 02 13:01:15.242667 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:01:15.241523    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqcmj" containerName="extractor"
Nov 02 13:01:15.242667 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:15.241599    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-25f7a6-595fd6dd77-72r7x"
Nov 02 13:01:15.638924 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:15.634838    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-25f7a6-595fd6dd77-72r7x"]
Nov 02 13:01:16.479162 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:16.479064    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-25f7a6-595fd6dd77-72r7x" event={"ID":"e97a314b-6891-4a14-9f4f-c7131a59e231","Type":"ContainerStarted","Data":"d20c487cb4f07ea33ee2373c66b22491e0e2a889fc56abf45850d777988d848f"}
Nov 02 13:01:16.479162 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:16.479128    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-25f7a6-595fd6dd77-72r7x" event={"ID":"e97a314b-6891-4a14-9f4f-c7131a59e231","Type":"ContainerStarted","Data":"d4cf84fb4166b0af9c3dc52ab9e69fbe3de435e97b9a51c26b9934f85506671a"}
Nov 02 13:01:16.479783 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:16.479181    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-25f7a6-595fd6dd77-72r7x"
Nov 02 13:01:16.528633 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:16.528557    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-25f7a6-595fd6dd77-72r7x" podStartSLOduration=3.000450221 podStartE2EDuration="3.528535699s" totalImagesPullingTime="528.085478ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:01:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:01:16.526102584 +0000 UTC m=+2072.781501680" watchObservedRunningTime="2026-11-02 13:01:16.528535699 +0000 UTC m=+2072.783934796"
Nov 02 13:01:19.238629 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:19.238587    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-7aca74-openshift-default-5df749499f-n9t77"]
Nov 02 13:01:19.240103 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:19.240050    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-noten-7aca74-openshift-default-5df749499f-n9t77" podUID="e398716c-4a76-4c4e-881f-415c51600a83" containerName="istio-proxy" containerID="cri-o://53b372e8a46893c77498b89edcd4add5a74c5700156e921a0359889920aad8e1" gracePeriod=30
Nov 02 13:01:22.492549 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:22.492520    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-25f7a6-595fd6dd77-72r7x"
Nov 02 13:01:23.539998 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:23.539957    4559 generic.go:356] "Generic (PLEG): container finished" podID="abcacc36-3628-44e7-a103-fdf1027634cf" containerID="ae925b4b0b98226777ab79fddf3d1314563e478a87f64c3170ac1de0d48ab2bf" exitCode=0
Nov 02 13:01:23.540417 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:23.540026    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-ecec44-openshift-default-6d94dccbcf-thqw7" event={"ID":"abcacc36-3628-44e7-a103-fdf1027634cf","Type":"ContainerDied","Data":"ae925b4b0b98226777ab79fddf3d1314563e478a87f64c3170ac1de0d48ab2bf"}
Nov 02 13:01:23.540463 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:23.540454    4559 scope.go:122] "RemoveContainer" containerID="ae925b4b0b98226777ab79fddf3d1314563e478a87f64c3170ac1de0d48ab2bf"
Nov 02 13:01:23.542212 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:23.542170    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 13:01:23.542333 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:23.542257    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 13:01:23.542333 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:23.542310    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 13:01:24.548406 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:24.548378    4559 generic.go:356] "Generic (PLEG): container finished" podID="e398716c-4a76-4c4e-881f-415c51600a83" containerID="53b372e8a46893c77498b89edcd4add5a74c5700156e921a0359889920aad8e1" exitCode=0
Nov 02 13:01:24.548875 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:24.548455    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-7aca74-openshift-default-5df749499f-n9t77" event={"ID":"e398716c-4a76-4c4e-881f-415c51600a83","Type":"ContainerDied","Data":"53b372e8a46893c77498b89edcd4add5a74c5700156e921a0359889920aad8e1"}
Nov 02 13:01:24.548875 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:24.548524    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-7aca74-openshift-default-5df749499f-n9t77" event={"ID":"e398716c-4a76-4c4e-881f-415c51600a83","Type":"ContainerDied","Data":"b55bf127e73698a573434da2a29b3119a2c32560dba9bcd7524bc1e9dc8acc3f"}
Nov 02 13:01:24.548875 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:24.548556    4559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b55bf127e73698a573434da2a29b3119a2c32560dba9bcd7524bc1e9dc8acc3f"
Nov 02 13:01:24.550746 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:24.550723    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-ecec44-openshift-default-6d94dccbcf-thqw7" event={"ID":"abcacc36-3628-44e7-a103-fdf1027634cf","Type":"ContainerStarted","Data":"91b6c8bf5e03dee916a17312abd00deed23b155b5f4959bf629aa3627856e1a9"}
Nov 02 13:01:24.552346 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:24.552325    4559 generic.go:356] "Generic (PLEG): container finished" podID="a47a44e4-6226-4ce7-9760-58ee5eaf65bf" containerID="215bfa85b115960b75bd02c0ea3213a462ea4cb0494f640b969ee693e9c7d93c" exitCode=0
Nov 02 13:01:24.552477 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:24.552403    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-035dac-openshift-default-6d5cb5658f-8kmlt" event={"ID":"a47a44e4-6226-4ce7-9760-58ee5eaf65bf","Type":"ContainerDied","Data":"215bfa85b115960b75bd02c0ea3213a462ea4cb0494f640b969ee693e9c7d93c"}
Nov 02 13:01:24.552827 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:24.552812    4559 scope.go:122] "RemoveContainer" containerID="215bfa85b115960b75bd02c0ea3213a462ea4cb0494f640b969ee693e9c7d93c"
Nov 02 13:01:24.554678 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:24.554634    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 13:01:24.554746 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:24.554717    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 13:01:24.554796 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:24.554749    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 13:01:24.559696 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:24.559679    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-7aca74-openshift-default-5df749499f-n9t77"
Nov 02 13:01:24.673238 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:24.673208    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e398716c-4a76-4c4e-881f-415c51600a83-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e398716c-4a76-4c4e-881f-415c51600a83-istio-data\") pod \"e398716c-4a76-4c4e-881f-415c51600a83\" (UID: \"e398716c-4a76-4c4e-881f-415c51600a83\") "
Nov 02 13:01:24.674995 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:24.673278    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e398716c-4a76-4c4e-881f-415c51600a83-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e398716c-4a76-4c4e-881f-415c51600a83-workload-socket\") pod \"e398716c-4a76-4c4e-881f-415c51600a83\" (UID: \"e398716c-4a76-4c4e-881f-415c51600a83\") "
Nov 02 13:01:24.674995 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:24.673317    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e398716c-4a76-4c4e-881f-415c51600a83-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e398716c-4a76-4c4e-881f-415c51600a83-istiod-ca-cert\") pod \"e398716c-4a76-4c4e-881f-415c51600a83\" (UID: \"e398716c-4a76-4c4e-881f-415c51600a83\") "
Nov 02 13:01:24.674995 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:24.673377    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e398716c-4a76-4c4e-881f-415c51600a83-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e398716c-4a76-4c4e-881f-415c51600a83-istio-envoy\") pod \"e398716c-4a76-4c4e-881f-415c51600a83\" (UID: \"e398716c-4a76-4c4e-881f-415c51600a83\") "
Nov 02 13:01:24.674995 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:24.673448    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e398716c-4a76-4c4e-881f-415c51600a83-kube-api-access-xxc7k\" (UniqueName: \"kubernetes.io/projected/e398716c-4a76-4c4e-881f-415c51600a83-kube-api-access-xxc7k\") pod \"e398716c-4a76-4c4e-881f-415c51600a83\" (UID: \"e398716c-4a76-4c4e-881f-415c51600a83\") "
Nov 02 13:01:24.674995 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:24.673507    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e398716c-4a76-4c4e-881f-415c51600a83-istio-token\" (UniqueName: \"kubernetes.io/projected/e398716c-4a76-4c4e-881f-415c51600a83-istio-token\") pod \"e398716c-4a76-4c4e-881f-415c51600a83\" (UID: \"e398716c-4a76-4c4e-881f-415c51600a83\") "
Nov 02 13:01:24.674995 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:24.673536    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e398716c-4a76-4c4e-881f-415c51600a83-istio-data" pod "e398716c-4a76-4c4e-881f-415c51600a83" (UID: "e398716c-4a76-4c4e-881f-415c51600a83"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:01:24.674995 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:24.673571    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e398716c-4a76-4c4e-881f-415c51600a83-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e398716c-4a76-4c4e-881f-415c51600a83-credential-socket\") pod \"e398716c-4a76-4c4e-881f-415c51600a83\" (UID: \"e398716c-4a76-4c4e-881f-415c51600a83\") "
Nov 02 13:01:24.674995 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:24.673627    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e398716c-4a76-4c4e-881f-415c51600a83-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e398716c-4a76-4c4e-881f-415c51600a83-workload-certs\") pod \"e398716c-4a76-4c4e-881f-415c51600a83\" (UID: \"e398716c-4a76-4c4e-881f-415c51600a83\") "
Nov 02 13:01:24.674995 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:24.673718    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e398716c-4a76-4c4e-881f-415c51600a83-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e398716c-4a76-4c4e-881f-415c51600a83-istio-podinfo\") pod \"e398716c-4a76-4c4e-881f-415c51600a83\" (UID: \"e398716c-4a76-4c4e-881f-415c51600a83\") "
Nov 02 13:01:24.674995 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:24.674003    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e398716c-4a76-4c4e-881f-415c51600a83-workload-socket" pod "e398716c-4a76-4c4e-881f-415c51600a83" (UID: "e398716c-4a76-4c4e-881f-415c51600a83"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:01:24.674995 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:24.674399    4559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e398716c-4a76-4c4e-881f-415c51600a83-istio-data\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:01:24.674995 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:24.674418    4559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e398716c-4a76-4c4e-881f-415c51600a83-workload-socket\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:01:24.677501 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:24.676525    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e398716c-4a76-4c4e-881f-415c51600a83-istio-podinfo" pod "e398716c-4a76-4c4e-881f-415c51600a83" (UID: "e398716c-4a76-4c4e-881f-415c51600a83"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 13:01:24.677501 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:24.677456    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e398716c-4a76-4c4e-881f-415c51600a83-credential-socket" pod "e398716c-4a76-4c4e-881f-415c51600a83" (UID: "e398716c-4a76-4c4e-881f-415c51600a83"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:01:24.705532 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:24.678955    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e398716c-4a76-4c4e-881f-415c51600a83-istiod-ca-cert" pod "e398716c-4a76-4c4e-881f-415c51600a83" (UID: "e398716c-4a76-4c4e-881f-415c51600a83"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:01:24.705532 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:24.679137    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e398716c-4a76-4c4e-881f-415c51600a83-workload-certs" pod "e398716c-4a76-4c4e-881f-415c51600a83" (UID: "e398716c-4a76-4c4e-881f-415c51600a83"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:01:24.726105 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:24.725862    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e398716c-4a76-4c4e-881f-415c51600a83-istio-envoy" pod "e398716c-4a76-4c4e-881f-415c51600a83" (UID: "e398716c-4a76-4c4e-881f-415c51600a83"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:01:24.727700 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:24.727302    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e398716c-4a76-4c4e-881f-415c51600a83-kube-api-access-xxc7k" pod "e398716c-4a76-4c4e-881f-415c51600a83" (UID: "e398716c-4a76-4c4e-881f-415c51600a83"). InnerVolumeSpecName "kube-api-access-xxc7k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:01:24.727872 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:24.727834    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e398716c-4a76-4c4e-881f-415c51600a83-istio-token" pod "e398716c-4a76-4c4e-881f-415c51600a83" (UID: "e398716c-4a76-4c4e-881f-415c51600a83"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:01:24.744238 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:24.744152    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-ecec44-openshift-default-6d94dccbcf-thqw7"
Nov 02 13:01:24.748795 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:24.748434    4559 patch_prober.go:29] interesting pod/e2e-autores-ecec44-openshift-default-6d94dccbcf-thqw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused" start-of-body=
Nov 02 13:01:24.748795 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:24.748482    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-ecec44-openshift-default-6d94dccbcf-thqw7" podUID="abcacc36-3628-44e7-a103-fdf1027634cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused"
Nov 02 13:01:24.776728 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:24.775269    4559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e398716c-4a76-4c4e-881f-415c51600a83-istiod-ca-cert\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:01:24.776728 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:24.775295    4559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e398716c-4a76-4c4e-881f-415c51600a83-istio-envoy\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:01:24.776728 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:24.775304    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xxc7k\" (UniqueName: \"kubernetes.io/projected/e398716c-4a76-4c4e-881f-415c51600a83-kube-api-access-xxc7k\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:01:24.776728 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:24.775315    4559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e398716c-4a76-4c4e-881f-415c51600a83-istio-token\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:01:24.776728 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:24.775324    4559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e398716c-4a76-4c4e-881f-415c51600a83-credential-socket\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:01:24.776728 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:24.775332    4559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e398716c-4a76-4c4e-881f-415c51600a83-workload-certs\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:01:24.776728 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:24.775341    4559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e398716c-4a76-4c4e-881f-415c51600a83-istio-podinfo\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:01:25.413718 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:25.413238    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-740f09-776cf8d88b-gtp87"]
Nov 02 13:01:25.415668 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:25.414243    4559 cpu_manager.go:414] "containerMap: removing container" podUID="e398716c-4a76-4c4e-881f-415c51600a83" containerName="istio-proxy"
Nov 02 13:01:25.415668 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:25.414303    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e398716c-4a76-4c4e-881f-415c51600a83" containerName="istio-proxy"
Nov 02 13:01:25.415668 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:25.414454    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="e398716c-4a76-4c4e-881f-415c51600a83" containerName="istio-proxy"
Nov 02 13:01:25.439978 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:25.436985    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-740f09-776cf8d88b-gtp87"
Nov 02 13:01:25.454739 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:25.452718    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-740f09-serving-cert\""
Nov 02 13:01:25.465097 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:25.456381    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-740f09-776cf8d88b-gtp87"]
Nov 02 13:01:25.561798 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:25.561756    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-035dac-openshift-default-6d5cb5658f-8kmlt" event={"ID":"a47a44e4-6226-4ce7-9760-58ee5eaf65bf","Type":"ContainerStarted","Data":"1e8ff1f9ec03ea2de2cb7c3549205a4866938f527053c6caf9b6492e9438362e"}
Nov 02 13:01:25.562272 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:25.561812    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-7aca74-openshift-default-5df749499f-n9t77"
Nov 02 13:01:25.608672 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:25.608596    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5k8z\" (UniqueName: \"kubernetes.io/projected/c482bbc5-fff9-42e7-a147-f9664750fc8a-kube-api-access-n5k8z\") pod \"maas-api-e2e-extmod-a-740f09-776cf8d88b-gtp87\" (UID: \"c482bbc5-fff9-42e7-a147-f9664750fc8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-740f09-776cf8d88b-gtp87"
Nov 02 13:01:25.609147 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:25.609101    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c482bbc5-fff9-42e7-a147-f9664750fc8a-metrics-certs\") pod \"maas-api-e2e-extmod-a-740f09-776cf8d88b-gtp87\" (UID: \"c482bbc5-fff9-42e7-a147-f9664750fc8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-740f09-776cf8d88b-gtp87"
Nov 02 13:01:25.612845 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:25.612816    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c482bbc5-fff9-42e7-a147-f9664750fc8a-maas-api-tls\") pod \"maas-api-e2e-extmod-a-740f09-776cf8d88b-gtp87\" (UID: \"c482bbc5-fff9-42e7-a147-f9664750fc8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-740f09-776cf8d88b-gtp87"
Nov 02 13:01:25.698719 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:25.672138    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c482bbc5-fff9-42e7-a147-f9664750fc8a-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-740f09-776cf8d88b-gtp87\" (UID: \"c482bbc5-fff9-42e7-a147-f9664750fc8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-740f09-776cf8d88b-gtp87"
Nov 02 13:01:25.701330 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:25.701292    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-7aca74-openshift-default-5df749499f-n9t77"]
Nov 02 13:01:25.748658 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:25.735693    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-noten-7aca74-openshift-default-5df749499f-n9t77"]
Nov 02 13:01:25.749657 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:25.749597    4559 patch_prober.go:29] interesting pod/e2e-autores-ecec44-openshift-default-6d94dccbcf-thqw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused" start-of-body=
Nov 02 13:01:25.759719 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:25.758570    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-ecec44-openshift-default-6d94dccbcf-thqw7" podUID="abcacc36-3628-44e7-a103-fdf1027634cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused"
Nov 02 13:01:25.773952 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:25.773917    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c482bbc5-fff9-42e7-a147-f9664750fc8a-metrics-certs\") pod \"maas-api-e2e-extmod-a-740f09-776cf8d88b-gtp87\" (UID: \"c482bbc5-fff9-42e7-a147-f9664750fc8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-740f09-776cf8d88b-gtp87"
Nov 02 13:01:25.774127 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:25.773977    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c482bbc5-fff9-42e7-a147-f9664750fc8a-maas-api-tls\") pod \"maas-api-e2e-extmod-a-740f09-776cf8d88b-gtp87\" (UID: \"c482bbc5-fff9-42e7-a147-f9664750fc8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-740f09-776cf8d88b-gtp87"
Nov 02 13:01:25.774197 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:01:25.774153    4559 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-a-740f09-serving-cert: secret "maas-api-e2e-extmod-a-740f09-serving-cert" not found
Nov 02 13:01:25.774267 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:01:25.774227    4559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c482bbc5-fff9-42e7-a147-f9664750fc8a-maas-api-tls podName:c482bbc5-fff9-42e7-a147-f9664750fc8a nodeName:}" failed. No retries permitted until 2026-11-02 13:01:26.274205732 +0000 UTC m=+2082.529604839 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c482bbc5-fff9-42e7-a147-f9664750fc8a-maas-api-tls") pod "maas-api-e2e-extmod-a-740f09-776cf8d88b-gtp87" (UID: "c482bbc5-fff9-42e7-a147-f9664750fc8a") : secret "maas-api-e2e-extmod-a-740f09-serving-cert" not found
Nov 02 13:01:25.774366 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:25.774348    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c482bbc5-fff9-42e7-a147-f9664750fc8a-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-740f09-776cf8d88b-gtp87\" (UID: \"c482bbc5-fff9-42e7-a147-f9664750fc8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-740f09-776cf8d88b-gtp87"
Nov 02 13:01:25.775125 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:25.774444    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5k8z\" (UniqueName: \"kubernetes.io/projected/c482bbc5-fff9-42e7-a147-f9664750fc8a-kube-api-access-n5k8z\") pod \"maas-api-e2e-extmod-a-740f09-776cf8d88b-gtp87\" (UID: \"c482bbc5-fff9-42e7-a147-f9664750fc8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-740f09-776cf8d88b-gtp87"
Nov 02 13:01:25.775476 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:25.775449    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c482bbc5-fff9-42e7-a147-f9664750fc8a-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-740f09-776cf8d88b-gtp87\" (UID: \"c482bbc5-fff9-42e7-a147-f9664750fc8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-740f09-776cf8d88b-gtp87"
Nov 02 13:01:25.777775 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:25.777753    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c482bbc5-fff9-42e7-a147-f9664750fc8a-metrics-certs\") pod \"maas-api-e2e-extmod-a-740f09-776cf8d88b-gtp87\" (UID: \"c482bbc5-fff9-42e7-a147-f9664750fc8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-740f09-776cf8d88b-gtp87"
Nov 02 13:01:25.832911 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:25.832861    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5k8z\" (UniqueName: \"kubernetes.io/projected/c482bbc5-fff9-42e7-a147-f9664750fc8a-kube-api-access-n5k8z\") pod \"maas-api-e2e-extmod-a-740f09-776cf8d88b-gtp87\" (UID: \"c482bbc5-fff9-42e7-a147-f9664750fc8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-740f09-776cf8d88b-gtp87"
Nov 02 13:01:25.941418 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:25.941382    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e398716c-4a76-4c4e-881f-415c51600a83" path="/var/lib/kubelet/pods/e398716c-4a76-4c4e-881f-415c51600a83/volumes"
Nov 02 13:01:26.281423 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:26.281380    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c482bbc5-fff9-42e7-a147-f9664750fc8a-maas-api-tls\") pod \"maas-api-e2e-extmod-a-740f09-776cf8d88b-gtp87\" (UID: \"c482bbc5-fff9-42e7-a147-f9664750fc8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-740f09-776cf8d88b-gtp87"
Nov 02 13:01:26.286681 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:26.285213    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c482bbc5-fff9-42e7-a147-f9664750fc8a-maas-api-tls\") pod \"maas-api-e2e-extmod-a-740f09-776cf8d88b-gtp87\" (UID: \"c482bbc5-fff9-42e7-a147-f9664750fc8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-740f09-776cf8d88b-gtp87"
Nov 02 13:01:26.369888 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:26.369848    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-740f09-776cf8d88b-gtp87"
Nov 02 13:01:26.405672 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:26.403879    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-035dac-openshift-default-6d5cb5658f-8kmlt"
Nov 02 13:01:26.423523 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:26.409784    4559 patch_prober.go:29] interesting pod/e2e-extmod-a-035dac-openshift-default-6d5cb5658f-8kmlt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" start-of-body=
Nov 02 13:01:26.423523 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:26.409877    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-035dac-openshift-default-6d5cb5658f-8kmlt" podUID="a47a44e4-6226-4ce7-9760-58ee5eaf65bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused"
Nov 02 13:01:26.764119 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:26.745793    4559 patch_prober.go:29] interesting pod/e2e-autores-ecec44-openshift-default-6d94dccbcf-thqw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused" start-of-body=
Nov 02 13:01:26.764119 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:26.746471    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-ecec44-openshift-default-6d94dccbcf-thqw7" podUID="abcacc36-3628-44e7-a103-fdf1027634cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused"
Nov 02 13:01:26.951889 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:26.951863    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-740f09-776cf8d88b-gtp87"]
Nov 02 13:01:26.955109 ip-10-0-128-132 kubenswrapper[4559]: W1102 13:01:26.955072    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc482bbc5_fff9_42e7_a147_f9664750fc8a.slice/crio-b13839c1504f582db3c517a85f90c1b8be688d24d794dc7a15f263881084ae40 WatchSource:0}: Error finding container b13839c1504f582db3c517a85f90c1b8be688d24d794dc7a15f263881084ae40: Status 404 returned error can't find the container with id b13839c1504f582db3c517a85f90c1b8be688d24d794dc7a15f263881084ae40
Nov 02 13:01:27.430665 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:27.414603    4559 patch_prober.go:29] interesting pod/e2e-extmod-a-035dac-openshift-default-6d5cb5658f-8kmlt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" start-of-body=
Nov 02 13:01:27.430665 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:27.414709    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-035dac-openshift-default-6d5cb5658f-8kmlt" podUID="a47a44e4-6226-4ce7-9760-58ee5eaf65bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused"
Nov 02 13:01:27.745114 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:27.745026    4559 patch_prober.go:29] interesting pod/e2e-autores-ecec44-openshift-default-6d94dccbcf-thqw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused" start-of-body=
Nov 02 13:01:27.745349 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:27.745321    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-ecec44-openshift-default-6d94dccbcf-thqw7" podUID="abcacc36-3628-44e7-a103-fdf1027634cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused"
Nov 02 13:01:27.781249 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:27.781210    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-740f09-776cf8d88b-gtp87" event={"ID":"c482bbc5-fff9-42e7-a147-f9664750fc8a","Type":"ContainerStarted","Data":"6a4bb787b18fac6145882b8b6aad289a5dd481a8a1429ca200260f503be59a45"}
Nov 02 13:01:27.781683 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:27.781274    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-740f09-776cf8d88b-gtp87" event={"ID":"c482bbc5-fff9-42e7-a147-f9664750fc8a","Type":"ContainerStarted","Data":"b13839c1504f582db3c517a85f90c1b8be688d24d794dc7a15f263881084ae40"}
Nov 02 13:01:27.781683 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:27.781420    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-740f09-776cf8d88b-gtp87"
Nov 02 13:01:27.845012 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:27.844956    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-740f09-776cf8d88b-gtp87" podStartSLOduration=2.366530923 podStartE2EDuration="2.84494147s" totalImagesPullingTime="478.410547ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:01:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:01:27.843890947 +0000 UTC m=+2084.099290044" watchObservedRunningTime="2026-11-02 13:01:27.84494147 +0000 UTC m=+2084.100340593"
Nov 02 13:01:28.404515 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:28.404072    4559 patch_prober.go:29] interesting pod/e2e-extmod-a-035dac-openshift-default-6d5cb5658f-8kmlt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" start-of-body=
Nov 02 13:01:28.404515 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:28.404134    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-035dac-openshift-default-6d5cb5658f-8kmlt" podUID="a47a44e4-6226-4ce7-9760-58ee5eaf65bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused"
Nov 02 13:01:28.754339 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:28.754300    4559 patch_prober.go:29] interesting pod/e2e-autores-ecec44-openshift-default-6d94dccbcf-thqw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused" start-of-body=
Nov 02 13:01:28.754929 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:28.754895    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-ecec44-openshift-default-6d94dccbcf-thqw7" podUID="abcacc36-3628-44e7-a103-fdf1027634cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused"
Nov 02 13:01:29.403101 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:29.403057    4559 patch_prober.go:29] interesting pod/e2e-extmod-a-035dac-openshift-default-6d5cb5658f-8kmlt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" start-of-body=
Nov 02 13:01:29.403722 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:29.403689    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-035dac-openshift-default-6d5cb5658f-8kmlt" podUID="a47a44e4-6226-4ce7-9760-58ee5eaf65bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused"
Nov 02 13:01:29.424303 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:29.424270    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-ecec44-openshift-default-6d94dccbcf-thqw7"
Nov 02 13:01:29.674233 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:29.674192    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-035dac-openshift-default-6d5cb5658f-8kmlt"
Nov 02 13:01:29.744349 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:29.744318    4559 patch_prober.go:29] interesting pod/e2e-autores-ecec44-openshift-default-6d94dccbcf-thqw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused" start-of-body=
Nov 02 13:01:29.744571 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:29.744552    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-ecec44-openshift-default-6d94dccbcf-thqw7" podUID="abcacc36-3628-44e7-a103-fdf1027634cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused"
Nov 02 13:01:30.406666 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:30.406045    4559 patch_prober.go:29] interesting pod/e2e-extmod-a-035dac-openshift-default-6d5cb5658f-8kmlt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" start-of-body=
Nov 02 13:01:30.406666 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:30.406109    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-035dac-openshift-default-6d5cb5658f-8kmlt" podUID="a47a44e4-6226-4ce7-9760-58ee5eaf65bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused"
Nov 02 13:01:30.751401 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:30.751370    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-ecec44-openshift-default-6d94dccbcf-thqw7"
Nov 02 13:01:30.752559 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:30.752539    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-ecec44-openshift-default-6d94dccbcf-thqw7"
Nov 02 13:01:31.406510 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:31.406477    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-035dac-openshift-default-6d5cb5658f-8kmlt"
Nov 02 13:01:31.407626 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:31.407607    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-035dac-openshift-default-6d5cb5658f-8kmlt"
Nov 02 13:01:33.854911 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:33.854881    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-740f09-776cf8d88b-gtp87"
Nov 02 13:01:38.368662 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:38.362372    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"]
Nov 02 13:01:43.812775 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:43.812730    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-smxns"]
Nov 02 13:01:43.817897 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:43.817870    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-smxns"
Nov 02 13:01:43.864392 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:43.863946    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-25f7a6-openshift-default-dockercfg-z957v\""
Nov 02 13:01:43.867935 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:43.867903    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-smxns"]
Nov 02 13:01:43.948811 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:43.948775    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4eca390-56ae-40c6-a669-de202c2483de-workload-certs\") pod \"e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-smxns\" (UID: \"d4eca390-56ae-40c6-a669-de202c2483de\") " pod="openshift-ingress/e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-smxns"
Nov 02 13:01:43.948968 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:43.948824    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4eca390-56ae-40c6-a669-de202c2483de-istio-data\") pod \"e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-smxns\" (UID: \"d4eca390-56ae-40c6-a669-de202c2483de\") " pod="openshift-ingress/e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-smxns"
Nov 02 13:01:43.948968 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:43.948924    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4eca390-56ae-40c6-a669-de202c2483de-credential-socket\") pod \"e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-smxns\" (UID: \"d4eca390-56ae-40c6-a669-de202c2483de\") " pod="openshift-ingress/e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-smxns"
Nov 02 13:01:43.949059 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:43.949038    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4eca390-56ae-40c6-a669-de202c2483de-istio-token\") pod \"e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-smxns\" (UID: \"d4eca390-56ae-40c6-a669-de202c2483de\") " pod="openshift-ingress/e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-smxns"
Nov 02 13:01:43.949138 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:43.949112    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4eca390-56ae-40c6-a669-de202c2483de-istiod-ca-cert\") pod \"e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-smxns\" (UID: \"d4eca390-56ae-40c6-a669-de202c2483de\") " pod="openshift-ingress/e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-smxns"
Nov 02 13:01:43.949259 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:43.949228    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6q86\" (UniqueName: \"kubernetes.io/projected/d4eca390-56ae-40c6-a669-de202c2483de-kube-api-access-s6q86\") pod \"e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-smxns\" (UID: \"d4eca390-56ae-40c6-a669-de202c2483de\") " pod="openshift-ingress/e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-smxns"
Nov 02 13:01:43.949362 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:43.949320    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4eca390-56ae-40c6-a669-de202c2483de-workload-socket\") pod \"e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-smxns\" (UID: \"d4eca390-56ae-40c6-a669-de202c2483de\") " pod="openshift-ingress/e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-smxns"
Nov 02 13:01:43.949433 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:43.949400    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4eca390-56ae-40c6-a669-de202c2483de-istio-podinfo\") pod \"e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-smxns\" (UID: \"d4eca390-56ae-40c6-a669-de202c2483de\") " pod="openshift-ingress/e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-smxns"
Nov 02 13:01:43.949531 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:43.949496    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4eca390-56ae-40c6-a669-de202c2483de-istio-envoy\") pod \"e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-smxns\" (UID: \"d4eca390-56ae-40c6-a669-de202c2483de\") " pod="openshift-ingress/e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-smxns"
Nov 02 13:01:44.050823 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:44.050780    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4eca390-56ae-40c6-a669-de202c2483de-istiod-ca-cert\") pod \"e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-smxns\" (UID: \"d4eca390-56ae-40c6-a669-de202c2483de\") " pod="openshift-ingress/e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-smxns"
Nov 02 13:01:44.051004 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:44.050839    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6q86\" (UniqueName: \"kubernetes.io/projected/d4eca390-56ae-40c6-a669-de202c2483de-kube-api-access-s6q86\") pod \"e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-smxns\" (UID: \"d4eca390-56ae-40c6-a669-de202c2483de\") " pod="openshift-ingress/e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-smxns"
Nov 02 13:01:44.051004 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:44.050874    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4eca390-56ae-40c6-a669-de202c2483de-workload-socket\") pod \"e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-smxns\" (UID: \"d4eca390-56ae-40c6-a669-de202c2483de\") " pod="openshift-ingress/e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-smxns"
Nov 02 13:01:44.051004 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:44.050969    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4eca390-56ae-40c6-a669-de202c2483de-istio-podinfo\") pod \"e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-smxns\" (UID: \"d4eca390-56ae-40c6-a669-de202c2483de\") " pod="openshift-ingress/e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-smxns"
Nov 02 13:01:44.051210 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:44.051048    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4eca390-56ae-40c6-a669-de202c2483de-istio-envoy\") pod \"e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-smxns\" (UID: \"d4eca390-56ae-40c6-a669-de202c2483de\") " pod="openshift-ingress/e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-smxns"
Nov 02 13:01:44.051210 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:44.051141    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4eca390-56ae-40c6-a669-de202c2483de-workload-certs\") pod \"e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-smxns\" (UID: \"d4eca390-56ae-40c6-a669-de202c2483de\") " pod="openshift-ingress/e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-smxns"
Nov 02 13:01:44.051210 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:44.051202    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4eca390-56ae-40c6-a669-de202c2483de-istio-data\") pod \"e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-smxns\" (UID: \"d4eca390-56ae-40c6-a669-de202c2483de\") " pod="openshift-ingress/e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-smxns"
Nov 02 13:01:44.051382 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:44.051258    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4eca390-56ae-40c6-a669-de202c2483de-credential-socket\") pod \"e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-smxns\" (UID: \"d4eca390-56ae-40c6-a669-de202c2483de\") " pod="openshift-ingress/e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-smxns"
Nov 02 13:01:44.051382 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:44.051331    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4eca390-56ae-40c6-a669-de202c2483de-istio-token\") pod \"e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-smxns\" (UID: \"d4eca390-56ae-40c6-a669-de202c2483de\") " pod="openshift-ingress/e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-smxns"
Nov 02 13:01:44.051612 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:44.051583    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4eca390-56ae-40c6-a669-de202c2483de-istiod-ca-cert\") pod \"e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-smxns\" (UID: \"d4eca390-56ae-40c6-a669-de202c2483de\") " pod="openshift-ingress/e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-smxns"
Nov 02 13:01:44.053624 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:44.052432    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4eca390-56ae-40c6-a669-de202c2483de-istio-data\") pod \"e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-smxns\" (UID: \"d4eca390-56ae-40c6-a669-de202c2483de\") " pod="openshift-ingress/e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-smxns"
Nov 02 13:01:44.055828 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:44.052511    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4eca390-56ae-40c6-a669-de202c2483de-credential-socket\") pod \"e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-smxns\" (UID: \"d4eca390-56ae-40c6-a669-de202c2483de\") " pod="openshift-ingress/e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-smxns"
Nov 02 13:01:44.055971 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:44.052655    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4eca390-56ae-40c6-a669-de202c2483de-workload-certs\") pod \"e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-smxns\" (UID: \"d4eca390-56ae-40c6-a669-de202c2483de\") " pod="openshift-ingress/e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-smxns"
Nov 02 13:01:44.056069 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:44.053339    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4eca390-56ae-40c6-a669-de202c2483de-workload-socket\") pod \"e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-smxns\" (UID: \"d4eca390-56ae-40c6-a669-de202c2483de\") " pod="openshift-ingress/e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-smxns"
Nov 02 13:01:44.070616 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:44.070527    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4eca390-56ae-40c6-a669-de202c2483de-istio-envoy\") pod \"e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-smxns\" (UID: \"d4eca390-56ae-40c6-a669-de202c2483de\") " pod="openshift-ingress/e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-smxns"
Nov 02 13:01:44.072344 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:44.072283    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4eca390-56ae-40c6-a669-de202c2483de-istio-podinfo\") pod \"e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-smxns\" (UID: \"d4eca390-56ae-40c6-a669-de202c2483de\") " pod="openshift-ingress/e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-smxns"
Nov 02 13:01:44.075443 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:44.075418    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6q86\" (UniqueName: \"kubernetes.io/projected/d4eca390-56ae-40c6-a669-de202c2483de-kube-api-access-s6q86\") pod \"e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-smxns\" (UID: \"d4eca390-56ae-40c6-a669-de202c2483de\") " pod="openshift-ingress/e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-smxns"
Nov 02 13:01:44.075917 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:44.075898    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4eca390-56ae-40c6-a669-de202c2483de-istio-token\") pod \"e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-smxns\" (UID: \"d4eca390-56ae-40c6-a669-de202c2483de\") " pod="openshift-ingress/e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-smxns"
Nov 02 13:01:44.152886 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:44.152847    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-25f7a6-openshift-default-dockercfg-z957v\""
Nov 02 13:01:44.166668 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:44.164747    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-smxns"
Nov 02 13:01:44.748726 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:44.741966    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 13:01:44.748726 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:44.742063    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 13:01:44.748726 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:44.742106    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 13:01:44.748726 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:44.746356    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-smxns"]
Nov 02 13:01:45.021036 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:45.020928    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-smxns" event={"ID":"d4eca390-56ae-40c6-a669-de202c2483de","Type":"ContainerStarted","Data":"d8d47f9a5cc855553a22756baceb40d8528085583a6cd8e324c40ece47aec249"}
Nov 02 13:01:45.021036 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:45.020984    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-smxns" event={"ID":"d4eca390-56ae-40c6-a669-de202c2483de","Type":"ContainerStarted","Data":"35f7eee78639776f3e926101807cd6ae214f5bc67eb0a649e436ee0e5996547d"}
Nov 02 13:01:45.076912 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:45.076848    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-smxns" podStartSLOduration=2.076828421 podStartE2EDuration="2.076828421s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:01:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:01:45.068013699 +0000 UTC m=+2101.323412790" watchObservedRunningTime="2026-11-02 13:01:45.076828421 +0000 UTC m=+2101.332227519"
Nov 02 13:01:45.118616 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:01:45.118577    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqcmj" containerName="extractor"
Nov 02 13:01:45.165633 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:45.165595    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-smxns"
Nov 02 13:01:45.166972 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:45.166946    4559 patch_prober.go:29] interesting pod/e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-smxns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused" start-of-body=
Nov 02 13:01:45.167098 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:45.167006    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-smxns" podUID="d4eca390-56ae-40c6-a669-de202c2483de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused"
Nov 02 13:01:46.165418 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:46.165384    4559 patch_prober.go:29] interesting pod/e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-smxns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused" start-of-body=
Nov 02 13:01:46.165837 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:46.165442    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-smxns" podUID="d4eca390-56ae-40c6-a669-de202c2483de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused"
Nov 02 13:01:47.165873 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:47.165838    4559 patch_prober.go:29] interesting pod/e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-smxns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused" start-of-body=
Nov 02 13:01:47.166243 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:47.165891    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-smxns" podUID="d4eca390-56ae-40c6-a669-de202c2483de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused"
Nov 02 13:01:48.174714 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:48.171247    4559 patch_prober.go:29] interesting pod/e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-smxns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused" start-of-body=
Nov 02 13:01:48.174714 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:48.171311    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-smxns" podUID="d4eca390-56ae-40c6-a669-de202c2483de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused"
Nov 02 13:01:49.165933 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:49.165896    4559 patch_prober.go:29] interesting pod/e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-smxns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused" start-of-body=
Nov 02 13:01:49.166148 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:49.165958    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-smxns" podUID="d4eca390-56ae-40c6-a669-de202c2483de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused"
Nov 02 13:01:50.247905 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:50.247865    4559 patch_prober.go:29] interesting pod/e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-smxns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused" start-of-body=
Nov 02 13:01:50.248691 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:50.248574    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-smxns" podUID="d4eca390-56ae-40c6-a669-de202c2483de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused"
Nov 02 13:01:51.166231 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:51.166146    4559 patch_prober.go:29] interesting pod/e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-smxns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused" start-of-body=
Nov 02 13:01:51.166231 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:51.166205    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-smxns" podUID="d4eca390-56ae-40c6-a669-de202c2483de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused"
Nov 02 13:01:51.279042 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:51.279012    4559 generic.go:356] "Generic (PLEG): container finished" podID="8f2bb155-8114-4c86-a74d-479522616e10" containerID="815680021cae23ccb7232840479478c9c9425105ef0b77eee0ee0e05076164ef" exitCode=0
Nov 02 13:01:51.279495 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:51.279088    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-740f09-openshift-default-786896ffcc-lmqlz" event={"ID":"8f2bb155-8114-4c86-a74d-479522616e10","Type":"ContainerDied","Data":"815680021cae23ccb7232840479478c9c9425105ef0b77eee0ee0e05076164ef"}
Nov 02 13:01:51.279545 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:51.279519    4559 scope.go:122] "RemoveContainer" containerID="815680021cae23ccb7232840479478c9c9425105ef0b77eee0ee0e05076164ef"
Nov 02 13:01:51.281216 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:51.281181    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 13:01:51.281330 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:51.281270    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 13:01:51.281330 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:51.281324    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 13:01:52.170825 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:52.170787    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-smxns"
Nov 02 13:01:52.171101 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:52.171080    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-smxns"
Nov 02 13:01:52.171882 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:52.171858    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-smxns"
Nov 02 13:01:52.316666 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:52.316276    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-740f09-openshift-default-786896ffcc-lmqlz" event={"ID":"8f2bb155-8114-4c86-a74d-479522616e10","Type":"ContainerStarted","Data":"168d71daca827a8ae4b13fa24b65d231dfd8557291998d1e831016a70371da98"}
Nov 02 13:01:52.353747 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:52.353696    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-740f09-openshift-default-786896ffcc-lmqlz"
Nov 02 13:01:52.354806 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:52.354778    4559 patch_prober.go:29] interesting pod/e2e-extmod-a-740f09-openshift-default-786896ffcc-lmqlz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body=
Nov 02 13:01:52.354964 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:52.354841    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-740f09-openshift-default-786896ffcc-lmqlz" podUID="8f2bb155-8114-4c86-a74d-479522616e10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused"
Nov 02 13:01:53.353411 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:53.353377    4559 patch_prober.go:29] interesting pod/e2e-extmod-a-740f09-openshift-default-786896ffcc-lmqlz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body=
Nov 02 13:01:53.353831 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:53.353435    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-740f09-openshift-default-786896ffcc-lmqlz" podUID="8f2bb155-8114-4c86-a74d-479522616e10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused"
Nov 02 13:01:54.353592 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:54.353560    4559 patch_prober.go:29] interesting pod/e2e-extmod-a-740f09-openshift-default-786896ffcc-lmqlz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body=
Nov 02 13:01:54.354044 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:54.353672    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-740f09-openshift-default-786896ffcc-lmqlz" podUID="8f2bb155-8114-4c86-a74d-479522616e10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused"
Nov 02 13:01:55.354256 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:55.354175    4559 patch_prober.go:29] interesting pod/e2e-extmod-a-740f09-openshift-default-786896ffcc-lmqlz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body=
Nov 02 13:01:55.354256 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:55.354232    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-740f09-openshift-default-786896ffcc-lmqlz" podUID="8f2bb155-8114-4c86-a74d-479522616e10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused"
Nov 02 13:01:56.249523 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:56.249484    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-740f09-openshift-default-786896ffcc-lmqlz"
Nov 02 13:01:56.353162 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:56.353130    4559 patch_prober.go:29] interesting pod/e2e-extmod-a-740f09-openshift-default-786896ffcc-lmqlz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body=
Nov 02 13:01:56.353363 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:56.353197    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-740f09-openshift-default-786896ffcc-lmqlz" podUID="8f2bb155-8114-4c86-a74d-479522616e10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused"
Nov 02 13:01:57.374355 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:57.374307    4559 patch_prober.go:29] interesting pod/e2e-extmod-a-740f09-openshift-default-786896ffcc-lmqlz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body=
Nov 02 13:01:57.376534 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:57.376487    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-740f09-openshift-default-786896ffcc-lmqlz" podUID="8f2bb155-8114-4c86-a74d-479522616e10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused"
Nov 02 13:01:58.480072 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:58.480038    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-740f09-openshift-default-786896ffcc-lmqlz"
Nov 02 13:01:58.481246 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:01:58.481224    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-740f09-openshift-default-786896ffcc-lmqlz"
Nov 02 13:02:00.629818 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:02:00.629775    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-90cda9-openshift-default-57f8b6f957-b5jzx"]
Nov 02 13:02:00.635188 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:02:00.635161    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-90cda9-openshift-default-57f8b6f957-b5jzx"
Nov 02 13:02:00.660712 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:02:00.660677    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-90cda9-openshift-default-57f8b6f957-b5jzx"]
Nov 02 13:02:00.665829 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:02:00.665802    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-90cda9-openshift-default-dockercfg-xm76t\""
Nov 02 13:02:00.693433 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:02:00.693396    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb3c761d-8641-496c-9d3a-531e8117e7dd-istio-envoy\") pod \"e2e-extmod-b-90cda9-openshift-default-57f8b6f957-b5jzx\" (UID: \"bb3c761d-8641-496c-9d3a-531e8117e7dd\") " pod="openshift-ingress/e2e-extmod-b-90cda9-openshift-default-57f8b6f957-b5jzx"
Nov 02 13:02:00.693629 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:02:00.693447    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb3c761d-8641-496c-9d3a-531e8117e7dd-istio-podinfo\") pod \"e2e-extmod-b-90cda9-openshift-default-57f8b6f957-b5jzx\" (UID: \"bb3c761d-8641-496c-9d3a-531e8117e7dd\") " pod="openshift-ingress/e2e-extmod-b-90cda9-openshift-default-57f8b6f957-b5jzx"
Nov 02 13:02:00.693629 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:02:00.693579    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb3c761d-8641-496c-9d3a-531e8117e7dd-workload-socket\") pod \"e2e-extmod-b-90cda9-openshift-default-57f8b6f957-b5jzx\" (UID: \"bb3c761d-8641-496c-9d3a-531e8117e7dd\") " pod="openshift-ingress/e2e-extmod-b-90cda9-openshift-default-57f8b6f957-b5jzx"
Nov 02 13:02:00.693790 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:02:00.693680    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb3c761d-8641-496c-9d3a-531e8117e7dd-istio-data\") pod \"e2e-extmod-b-90cda9-openshift-default-57f8b6f957-b5jzx\" (UID: \"bb3c761d-8641-496c-9d3a-531e8117e7dd\") " pod="openshift-ingress/e2e-extmod-b-90cda9-openshift-default-57f8b6f957-b5jzx"
Nov 02 13:02:00.693790 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:02:00.693763    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb3c761d-8641-496c-9d3a-531e8117e7dd-istiod-ca-cert\") pod \"e2e-extmod-b-90cda9-openshift-default-57f8b6f957-b5jzx\" (UID: \"bb3c761d-8641-496c-9d3a-531e8117e7dd\") " pod="openshift-ingress/e2e-extmod-b-90cda9-openshift-default-57f8b6f957-b5jzx"
Nov 02 13:02:00.693905 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:02:00.693816    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb3c761d-8641-496c-9d3a-531e8117e7dd-credential-socket\") pod \"e2e-extmod-b-90cda9-openshift-default-57f8b6f957-b5jzx\" (UID: \"bb3c761d-8641-496c-9d3a-531e8117e7dd\") " pod="openshift-ingress/e2e-extmod-b-90cda9-openshift-default-57f8b6f957-b5jzx"
Nov 02 13:02:00.693905 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:02:00.693866    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb3c761d-8641-496c-9d3a-531e8117e7dd-istio-token\") pod \"e2e-extmod-b-90cda9-openshift-default-57f8b6f957-b5jzx\" (UID: \"bb3c761d-8641-496c-9d3a-531e8117e7dd\") " pod="openshift-ingress/e2e-extmod-b-90cda9-openshift-default-57f8b6f957-b5jzx"
Nov 02 13:02:00.694016 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:02:00.693926    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb3c761d-8641-496c-9d3a-531e8117e7dd-workload-certs\") pod \"e2e-extmod-b-90cda9-openshift-default-57f8b6f957-b5jzx\" (UID: \"bb3c761d-8641-496c-9d3a-531e8117e7dd\") " pod="openshift-ingress/e2e-extmod-b-90cda9-openshift-default-57f8b6f957-b5jzx"
Nov 02 13:02:00.694016 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:02:00.693958    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdwxx\" (UniqueName: \"kubernetes.io/projected/bb3c761d-8641-496c-9d3a-531e8117e7dd-kube-api-access-qdwxx\") pod \"e2e-extmod-b-90cda9-openshift-default-57f8b6f957-b5jzx\" (UID: \"bb3c761d-8641-496c-9d3a-531e8117e7dd\") " pod="openshift-ingress/e2e-extmod-b-90cda9-openshift-default-57f8b6f957-b5jzx"
Nov 02 13:02:00.799994 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:02:00.799950    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb3c761d-8641-496c-9d3a-531e8117e7dd-istiod-ca-cert\") pod \"e2e-extmod-b-90cda9-openshift-default-57f8b6f957-b5jzx\" (UID: \"bb3c761d-8641-496c-9d3a-531e8117e7dd\") " pod="openshift-ingress/e2e-extmod-b-90cda9-openshift-default-57f8b6f957-b5jzx"
Nov 02 13:02:00.800338 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:02:00.800314    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb3c761d-8641-496c-9d3a-531e8117e7dd-credential-socket\") pod \"e2e-extmod-b-90cda9-openshift-default-57f8b6f957-b5jzx\" (UID: \"bb3c761d-8641-496c-9d3a-531e8117e7dd\") " pod="openshift-ingress/e2e-extmod-b-90cda9-openshift-default-57f8b6f957-b5jzx"
Nov 02 13:02:00.800505 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:02:00.800486    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb3c761d-8641-496c-9d3a-531e8117e7dd-istio-token\") pod \"e2e-extmod-b-90cda9-openshift-default-57f8b6f957-b5jzx\" (UID: \"bb3c761d-8641-496c-9d3a-531e8117e7dd\") " pod="openshift-ingress/e2e-extmod-b-90cda9-openshift-default-57f8b6f957-b5jzx"
Nov 02 13:02:00.800691 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:02:00.800671    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb3c761d-8641-496c-9d3a-531e8117e7dd-workload-certs\") pod \"e2e-extmod-b-90cda9-openshift-default-57f8b6f957-b5jzx\" (UID: \"bb3c761d-8641-496c-9d3a-531e8117e7dd\") " pod="openshift-ingress/e2e-extmod-b-90cda9-openshift-default-57f8b6f957-b5jzx"
Nov 02 13:02:00.800862 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:02:00.800844    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdwxx\" (UniqueName: \"kubernetes.io/projected/bb3c761d-8641-496c-9d3a-531e8117e7dd-kube-api-access-qdwxx\") pod \"e2e-extmod-b-90cda9-openshift-default-57f8b6f957-b5jzx\" (UID: \"bb3c761d-8641-496c-9d3a-531e8117e7dd\") " pod="openshift-ingress/e2e-extmod-b-90cda9-openshift-default-57f8b6f957-b5jzx"
Nov 02 13:02:00.801063 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:02:00.801046    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb3c761d-8641-496c-9d3a-531e8117e7dd-istio-envoy\") pod \"e2e-extmod-b-90cda9-openshift-default-57f8b6f957-b5jzx\" (UID: \"bb3c761d-8641-496c-9d3a-531e8117e7dd\") " pod="openshift-ingress/e2e-extmod-b-90cda9-openshift-default-57f8b6f957-b5jzx"
Nov 02 13:02:00.801186 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:02:00.801171    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb3c761d-8641-496c-9d3a-531e8117e7dd-istio-podinfo\") pod \"e2e-extmod-b-90cda9-openshift-default-57f8b6f957-b5jzx\" (UID: \"bb3c761d-8641-496c-9d3a-531e8117e7dd\") " pod="openshift-ingress/e2e-extmod-b-90cda9-openshift-default-57f8b6f957-b5jzx"
Nov 02 13:02:00.801353 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:02:00.801330    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb3c761d-8641-496c-9d3a-531e8117e7dd-workload-socket\") pod \"e2e-extmod-b-90cda9-openshift-default-57f8b6f957-b5jzx\" (UID: \"bb3c761d-8641-496c-9d3a-531e8117e7dd\") " pod="openshift-ingress/e2e-extmod-b-90cda9-openshift-default-57f8b6f957-b5jzx"
Nov 02 13:02:00.806124 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:02:00.804070    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb3c761d-8641-496c-9d3a-531e8117e7dd-workload-certs\") pod \"e2e-extmod-b-90cda9-openshift-default-57f8b6f957-b5jzx\" (UID: \"bb3c761d-8641-496c-9d3a-531e8117e7dd\") " pod="openshift-ingress/e2e-extmod-b-90cda9-openshift-default-57f8b6f957-b5jzx"
Nov 02 13:02:00.812666 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:02:00.804631    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb3c761d-8641-496c-9d3a-531e8117e7dd-istiod-ca-cert\") pod \"e2e-extmod-b-90cda9-openshift-default-57f8b6f957-b5jzx\" (UID: \"bb3c761d-8641-496c-9d3a-531e8117e7dd\") " pod="openshift-ingress/e2e-extmod-b-90cda9-openshift-default-57f8b6f957-b5jzx"
Nov 02 13:02:00.812827 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:02:00.804858    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb3c761d-8641-496c-9d3a-531e8117e7dd-credential-socket\") pod \"e2e-extmod-b-90cda9-openshift-default-57f8b6f957-b5jzx\" (UID: \"bb3c761d-8641-496c-9d3a-531e8117e7dd\") " pod="openshift-ingress/e2e-extmod-b-90cda9-openshift-default-57f8b6f957-b5jzx"
Nov 02 13:02:00.812935 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:02:00.806018    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb3c761d-8641-496c-9d3a-531e8117e7dd-workload-socket\") pod \"e2e-extmod-b-90cda9-openshift-default-57f8b6f957-b5jzx\" (UID: \"bb3c761d-8641-496c-9d3a-531e8117e7dd\") " pod="openshift-ingress/e2e-extmod-b-90cda9-openshift-default-57f8b6f957-b5jzx"
Nov 02 13:02:00.827679 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:02:00.814716    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb3c761d-8641-496c-9d3a-531e8117e7dd-istio-data\") pod \"e2e-extmod-b-90cda9-openshift-default-57f8b6f957-b5jzx\" (UID: \"bb3c761d-8641-496c-9d3a-531e8117e7dd\") " pod="openshift-ingress/e2e-extmod-b-90cda9-openshift-default-57f8b6f957-b5jzx"
Nov 02 13:02:00.827679 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:02:00.815342    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb3c761d-8641-496c-9d3a-531e8117e7dd-istio-data\") pod \"e2e-extmod-b-90cda9-openshift-default-57f8b6f957-b5jzx\" (UID: \"bb3c761d-8641-496c-9d3a-531e8117e7dd\") " pod="openshift-ingress/e2e-extmod-b-90cda9-openshift-default-57f8b6f957-b5jzx"
Nov 02 13:02:00.864131 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:02:00.864101    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb3c761d-8641-496c-9d3a-531e8117e7dd-istio-envoy\") pod \"e2e-extmod-b-90cda9-openshift-default-57f8b6f957-b5jzx\" (UID: \"bb3c761d-8641-496c-9d3a-531e8117e7dd\") " pod="openshift-ingress/e2e-extmod-b-90cda9-openshift-default-57f8b6f957-b5jzx"
Nov 02 13:02:00.866632 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:02:00.866568    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb3c761d-8641-496c-9d3a-531e8117e7dd-istio-podinfo\") pod \"e2e-extmod-b-90cda9-openshift-default-57f8b6f957-b5jzx\" (UID: \"bb3c761d-8641-496c-9d3a-531e8117e7dd\") " pod="openshift-ingress/e2e-extmod-b-90cda9-openshift-default-57f8b6f957-b5jzx"
Nov 02 13:02:00.888599 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:02:00.888532    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdwxx\" (UniqueName: \"kubernetes.io/projected/bb3c761d-8641-496c-9d3a-531e8117e7dd-kube-api-access-qdwxx\") pod \"e2e-extmod-b-90cda9-openshift-default-57f8b6f957-b5jzx\" (UID: \"bb3c761d-8641-496c-9d3a-531e8117e7dd\") " pod="openshift-ingress/e2e-extmod-b-90cda9-openshift-default-57f8b6f957-b5jzx"
Nov 02 13:02:00.897170 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:02:00.897138    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb3c761d-8641-496c-9d3a-531e8117e7dd-istio-token\") pod \"e2e-extmod-b-90cda9-openshift-default-57f8b6f957-b5jzx\" (UID: \"bb3c761d-8641-496c-9d3a-531e8117e7dd\") " pod="openshift-ingress/e2e-extmod-b-90cda9-openshift-default-57f8b6f957-b5jzx"
Nov 02 13:02:00.962975 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:02:00.962936    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-90cda9-openshift-default-57f8b6f957-b5jzx"
Nov 02 13:02:01.316601 ip-10-0-128-132 kubenswrapper[4559]: W1102 13:02:01.316565    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb3c761d_8641_496c_9d3a_531e8117e7dd.slice/crio-0c83a6053ae23e43970b631436b163c787562ceb20c18eb8edd097fa1c11a4a3 WatchSource:0}: Error finding container 0c83a6053ae23e43970b631436b163c787562ceb20c18eb8edd097fa1c11a4a3: Status 404 returned error can't find the container with id 0c83a6053ae23e43970b631436b163c787562ceb20c18eb8edd097fa1c11a4a3
Nov 02 13:02:01.318508 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:02:01.318476    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-90cda9-openshift-default-57f8b6f957-b5jzx"]
Nov 02 13:02:01.319995 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:02:01.319960    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 13:02:01.320083 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:02:01.320025    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 13:02:01.320083 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:02:01.320056    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 13:02:01.466237 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:02:01.466194    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-90cda9-openshift-default-57f8b6f957-b5jzx" event={"ID":"bb3c761d-8641-496c-9d3a-531e8117e7dd","Type":"ContainerStarted","Data":"d421f86a3e5b5b24f5e2908085ec94c480c14b86b84e4c22495f03c365aa50fb"}
Nov 02 13:02:01.466444 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:02:01.466265    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-90cda9-openshift-default-57f8b6f957-b5jzx" event={"ID":"bb3c761d-8641-496c-9d3a-531e8117e7dd","Type":"ContainerStarted","Data":"0c83a6053ae23e43970b631436b163c787562ceb20c18eb8edd097fa1c11a4a3"}
Nov 02 13:02:01.513423 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:02:01.513359    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-90cda9-openshift-default-57f8b6f957-b5jzx" podStartSLOduration=1.513338798 podStartE2EDuration="1.513338798s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:02:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:02:01.512056469 +0000 UTC m=+2117.767455577" watchObservedRunningTime="2026-11-02 13:02:01.513338798 +0000 UTC m=+2117.768737894"
Nov 02 13:02:01.963724 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:02:01.963693    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-90cda9-openshift-default-57f8b6f957-b5jzx"
Nov 02 13:02:02.970833 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:02:02.970794    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-90cda9-openshift-default-57f8b6f957-b5jzx"
Nov 02 13:02:03.479719 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:02:03.479679    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-90cda9-openshift-default-57f8b6f957-b5jzx"
Nov 02 13:02:03.480739 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:02:03.480721    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-90cda9-openshift-default-57f8b6f957-b5jzx"
Nov 02 13:02:06.437946 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:02:06.437908    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"]
Nov 02 13:02:15.086689 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:02:15.086618    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqcmj" containerName="extractor"
Nov 02 13:02:18.791802 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:02:18.791767    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-ecec44-openshift-default-6d94dccbcf-thqw7"]
Nov 02 13:02:18.792288 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:02:18.792029    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-ecec44-openshift-default-6d94dccbcf-thqw7" podUID="abcacc36-3628-44e7-a103-fdf1027634cf" containerName="istio-proxy" containerID="cri-o://91b6c8bf5e03dee916a17312abd00deed23b155b5f4959bf629aa3627856e1a9" gracePeriod=30
Nov 02 13:02:21.272862 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:02:21.272820    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-14fcb6-openshift-default-568ddb87b9-tzf8f"]
Nov 02 13:02:21.277735 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:02:21.277707    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-14fcb6-openshift-default-568ddb87b9-tzf8f"
Nov 02 13:02:21.309832 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:02:21.309805    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-14fcb6-openshift-default-dockercfg-q7wgl\""
Nov 02 13:02:21.328910 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:02:21.328876    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-14fcb6-openshift-default-568ddb87b9-tzf8f"]
Nov 02 13:02:21.341387 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:02:21.341351    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0029ba38-71d3-4212-9b50-ded1a42abd1d-credential-socket\") pod \"e2e-extmod-a-14fcb6-openshift-default-568ddb87b9-tzf8f\" (UID: \"0029ba38-71d3-4212-9b50-ded1a42abd1d\") " pod="openshift-ingress/e2e-extmod-a-14fcb6-openshift-default-568ddb87b9-tzf8f"
Nov 02 13:02:21.341569 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:02:21.341397    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0029ba38-71d3-4212-9b50-ded1a42abd1d-workload-socket\") pod \"e2e-extmod-a-14fcb6-openshift-default-568ddb87b9-tzf8f\" (UID: \"0029ba38-71d3-4212-9b50-ded1a42abd1d\") " pod="openshift-ingress/e2e-extmod-a-14fcb6-openshift-default-568ddb87b9-tzf8f"
Nov 02 13:02:21.341569 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:02:21.341501    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6cm7\" (UniqueName: \"kubernetes.io/projected/0029ba38-71d3-4212-9b50-ded1a42abd1d-kube-api-access-r6cm7\") pod \"e2e-extmod-a-14fcb6-openshift-default-568ddb87b9-tzf8f\" (UID: \"0029ba38-71d3-4212-9b50-ded1a42abd1d\") " pod="openshift-ingress/e2e-extmod-a-14fcb6-openshift-default-568ddb87b9-tzf8f"
Nov 02 13:02:21.341569 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:02:21.341553    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0029ba38-71d3-4212-9b50-ded1a42abd1d-istiod-ca-cert\") pod \"e2e-extmod-a-14fcb6-openshift-default-568ddb87b9-tzf8f\" (UID: \"0029ba38-71d3-4212-9b50-ded1a42abd1d\") " pod="openshift-ingress/e2e-extmod-a-14fcb6-openshift-default-568ddb87b9-tzf8f"
Nov 02 13:02:21.341825 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:02:21.341588    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0029ba38-71d3-4212-9b50-ded1a42abd1d-workload-certs\") pod \"e2e-extmod-a-14fcb6-openshift-default-568ddb87b9-tzf8f\" (UID: \"0029ba38-71d3-4212-9b50-ded1a42abd1d\") " pod="openshift-ingress/e2e-extmod-a-14fcb6-openshift-default-568ddb87b9-tzf8f"
Nov 02 13:02:21.341825 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:02:21.341622    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0029ba38-71d3-4212-9b50-ded1a42abd1d-istio-envoy\") pod \"e2e-extmod-a-14fcb6-openshift-default-568ddb87b9-tzf8f\" (UID: \"0029ba38-71d3-4212-9b50-ded1a42abd1d\") " pod="openshift-ingress/e2e-extmod-a-14fcb6-openshift-default-568ddb87b9-tzf8f"
Nov 02 13:02:21.341825 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:02:21.341688    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0029ba38-71d3-4212-9b50-ded1a42abd1d-istio-token\") pod \"e2e-extmod-a-14fcb6-openshift-default-568ddb87b9-tzf8f\" (UID: \"0029ba38-71d3-4212-9b50-ded1a42abd1d\") " pod="openshift-ingress/e2e-extmod-a-14fcb6-openshift-default-568ddb87b9-tzf8f"
Nov 02 13:02:21.341998 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:02:21.341827    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0029ba38-71d3-4212-9b50-ded1a42abd1d-istio-data\") pod \"e2e-extmod-a-14fcb6-openshift-default-568ddb87b9-tzf8f\" (UID: \"0029ba38-71d3-4212-9b50-ded1a42abd1d\") " pod="openshift-ingress/e2e-extmod-a-14fcb6-openshift-default-568ddb87b9-tzf8f"
Nov 02 13:02:21.341998 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:02:21.341850    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0029ba38-71d3-4212-9b50-ded1a42abd1d-istio-podinfo\") pod \"e2e-extmod-a-14fcb6-openshift-default-568ddb87b9-tzf8f\" (UID: \"0029ba38-71d3-4212-9b50-ded1a42abd1d\") " pod="openshift-ingress/e2e-extmod-a-14fcb6-openshift-default-568ddb87b9-tzf8f"
Nov 02 13:02:21.460701 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:02:21.460589    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0029ba38-71d3-4212-9b50-ded1a42abd1d-credential-socket\") pod \"e2e-extmod-a-14fcb6-openshift-default-568ddb87b9-tzf8f\" (UID: \"0029ba38-71d3-4212-9b50-ded1a42abd1d\") " pod="openshift-ingress/e2e-extmod-a-14fcb6-openshift-default-568ddb87b9-tzf8f"
Nov 02 13:02:21.474038 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:02:21.471510    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0029ba38-71d3-4212-9b50-ded1a42abd1d-workload-socket\") pod \"e2e-extmod-a-14fcb6-openshift-default-568ddb87b9-tzf8f\" (UID: \"0029ba38-71d3-4212-9b50-ded1a42abd1d\") " pod="openshift-ingress/e2e-extmod-a-14fcb6-openshift-default-568ddb87b9-tzf8f"
Nov 02 13:02:21.475124 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:02:21.470071    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0029ba38-71d3-4212-9b50-ded1a42abd1d-credential-socket\") pod \"e2e-extmod-a-14fcb6-openshift-default-568ddb87b9-tzf8f\" (UID: \"0029ba38-71d3-4212-9b50-ded1a42abd1d\") " pod="openshift-ingress/e2e-extmod-a-14fcb6-openshift-default-568ddb87b9-tzf8f"
Nov 02 13:02:21.496924 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:02:21.496878    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6cm7\" (UniqueName: \"kubernetes.io/projected/0029ba38-71d3-4212-9b50-ded1a42abd1d-kube-api-access-r6cm7\") pod \"e2e-extmod-a-14fcb6-openshift-default-568ddb87b9-tzf8f\" (UID: \"0029ba38-71d3-4212-9b50-ded1a42abd1d\") " pod="openshift-ingress/e2e-extmod-a-14fcb6-openshift-default-568ddb87b9-tzf8f"
Nov 02 13:02:21.497467 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:02:21.497327    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0029ba38-71d3-4212-9b50-ded1a42abd1d-istiod-ca-cert\") pod \"e2e-extmod-a-14fcb6-openshift-default-568ddb87b9-tzf8f\" (UID: \"0029ba38-71d3-4212-9b50-ded1a42abd1d\") " pod="openshift-ingress/e2e-extmod-a-14fcb6-openshift-default-568ddb87b9-tzf8f"
Nov 02 13:02:21.497775 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:02:21.497757    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0029ba38-71d3-4212-9b50-ded1a42abd1d-workload-certs\") pod \"e2e-extmod-a-14fcb6-openshift-default-568ddb87b9-tzf8f\" (UID: \"0029ba38-71d3-4212-9b50-ded1a42abd1d\") " pod="openshift-ingress/e2e-extmod-a-14fcb6-openshift-default-568ddb87b9-tzf8f"
Nov 02 13:02:21.498021 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:02:21.498005    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0029ba38-71d3-4212-9b50-ded1a42abd1d-istio-envoy\") pod \"e2e-extmod-a-14fcb6-openshift-default-568ddb87b9-tzf8f\" (UID: \"0029ba38-71d3-4212-9b50-ded1a42abd1d\") " pod="openshift-ingress/e2e-extmod-a-14fcb6-openshift-default-568ddb87b9-tzf8f"
Nov 02 13:02:21.498319 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:02:21.498299    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0029ba38-71d3-4212-9b50-ded1a42abd1d-istio-token\") pod \"e2e-extmod-a-14fcb6-openshift-default-568ddb87b9-tzf8f\" (UID: \"0029ba38-71d3-4212-9b50-ded1a42abd1d\") " pod="openshift-ingress/e2e-extmod-a-14fcb6-openshift-default-568ddb87b9-tzf8f"
Nov 02 13:02:21.499036 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:02:21.499005    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0029ba38-71d3-4212-9b50-ded1a42abd1d-istio-data\") pod \"e2e-extmod-a-14fcb6-openshift-default-568ddb87b9-tzf8f\" (UID: \"0029ba38-71d3-4212-9b50-ded1a42abd1d\") " pod="openshift-ingress/e2e-extmod-a-14fcb6-openshift-default-568ddb87b9-tzf8f"
Nov 02 13:02:21.501120 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:02:21.501099    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0029ba38-71d3-4212-9b50-ded1a42abd1d-istio-podinfo\") pod \"e2e-extmod-a-14fcb6-openshift-default-568ddb87b9-tzf8f\" (UID: \"0029ba38-71d3-4212-9b50-ded1a42abd1d\") " pod="openshift-ingress/e2e-extmod-a-14fcb6-openshift-default-568ddb87b9-tzf8f"
Nov 02 13:02:21.501415 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:02:21.499489    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0029ba38-71d3-4212-9b50-ded1a42abd1d-istio-data\") pod \"e2e-extmod-a-14fcb6-openshift-default-568ddb87b9-tzf8f\" (UID: \"0029ba38-71d3-4212-9b50-ded1a42abd1d\") " pod="openshift-ingress/e2e-extmod-a-14fcb6-openshift-default-568ddb87b9-tzf8f"
Nov 02 13:02:21.501610 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:02:21.500502    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0029ba38-71d3-4212-9b50-ded1a42abd1d-istiod-ca-cert\") pod \"e2e-extmod-a-14fcb6-openshift-default-568ddb87b9-tzf8f\" (UID: \"0029ba38-71d3-4212-9b50-ded1a42abd1d\") " pod="openshift-ingress/e2e-extmod-a-14fcb6-openshift-default-568ddb87b9-tzf8f"
Nov 02 13:02:21.501926 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:02:21.501043    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0029ba38-71d3-4212-9b50-ded1a42abd1d-workload-certs\") pod \"e2e-extmod-a-14fcb6-openshift-default-568ddb87b9-tzf8f\" (UID: \"0029ba38-71d3-4212-9b50-ded1a42abd1d\") " pod="openshift-ingress/e2e-extmod-a-14fcb6-openshift-default-568ddb87b9-tzf8f"
Nov 02 13:02:21.502120 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:02:21.498463    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0029ba38-71d3-4212-9b50-ded1a42abd1d-workload-socket\") pod \"e2e-extmod-a-14fcb6-openshift-default-568ddb87b9-tzf8f\" (UID: \"0029ba38-71d3-4212-9b50-ded1a42abd1d\") " pod="openshift-ingress/e2e-extmod-a-14fcb6-openshift-default-568ddb87b9-tzf8f"
Nov 02 13:02:21.506519 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:02:21.506495    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0029ba38-71d3-4212-9b50-ded1a42abd1d-istio-envoy\") pod \"e2e-extmod-a-14fcb6-openshift-default-568ddb87b9-tzf8f\" (UID: \"0029ba38-71d3-4212-9b50-ded1a42abd1d\") " pod="openshift-ingress/e2e-extmod-a-14fcb6-openshift-default-568ddb87b9-tzf8f"
Nov 02 13:02:21.523591 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:02:21.523501    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0029ba38-71d3-4212-9b50-ded1a42abd1d-istio-podinfo\") pod \"e2e-extmod-a-14fcb6-openshift-default-568ddb87b9-tzf8f\" (UID: \"0029ba38-71d3-4212-9b50-ded1a42abd1d\") " pod="openshift-ingress/e2e-extmod-a-14fcb6-openshift-default-568ddb87b9-tzf8f"
Nov 02 13:02:21.564667 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:02:21.562227    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6cm7\" (UniqueName: \"kubernetes.io/projected/0029ba38-71d3-4212-9b50-ded1a42abd1d-kube-api-access-r6cm7\") pod \"e2e-extmod-a-14fcb6-openshift-default-568ddb87b9-tzf8f\" (UID: \"0029ba38-71d3-4212-9b50-ded1a42abd1d\") " pod="openshift-ingress/e2e-extmod-a-14fcb6-openshift-default-568ddb87b9-tzf8f"
Nov 02 13:02:21.564667 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:02:21.563805    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0029ba38-71d3-4212-9b50-ded1a42abd1d-istio-token\") pod \"e2e-extmod-a-14fcb6-openshift-default-568ddb87b9-tzf8f\" (UID: \"0029ba38-71d3-4212-9b50-ded1a42abd1d\") " pod="openshift-ingress/e2e-extmod-a-14fcb6-openshift-default-568ddb87b9-tzf8f"
Nov 02 13:02:21.600670 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:02:21.599681    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-14fcb6-openshift-default-568ddb87b9-tzf8f"
Nov 02 13:02:21.975475 ip-10-0-128-132 kubenswrapper[4559]: W1102 13:02:21.975446    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0029ba38_71d3_4212_9b50_ded1a42abd1d.slice/crio-8a261e86569eb681d412286740390f7f166b3b7619c421e3c6d0144931d7bdb9 WatchSource:0}: Error finding container 8a261e86569eb681d412286740390f7f166b3b7619c421e3c6d0144931d7bdb9: Status 404 returned error can't find the container with id 8a261e86569eb681d412286740390f7f166b3b7619c421e3c6d0144931d7bdb9
Nov 02 13:02:21.976511 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:02:21.976485    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-14fcb6-openshift-default-568ddb87b9-tzf8f"]
Nov 02 13:02:21.978489 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:02:21.978457    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 13:02:21.978552 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:02:21.978526    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 13:02:21.978592 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:02:21.978556    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 13:02:22.668936 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:02:22.668895    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-14fcb6-openshift-default-568ddb87b9-tzf8f" event={"ID":"0029ba38-71d3-4212-9b50-ded1a42abd1d","Type":"ContainerStarted","Data":"3de5c889a2ef40f69cbc4ea9aa9eda016b3925972ef4c9c11a190b59925b248e"}
Nov 02 13:02:22.669350 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:02:22.668960    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-14fcb6-openshift-default-568ddb87b9-tzf8f" event={"ID":"0029ba38-71d3-4212-9b50-ded1a42abd1d","Type":"ContainerStarted","Data":"8a261e86569eb681d412286740390f7f166b3b7619c421e3c6d0144931d7bdb9"}
Nov 02 13:02:22.748169 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:02:22.748113    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-14fcb6-openshift-default-568ddb87b9-tzf8f" podStartSLOduration=1.748099072 podStartE2EDuration="1.748099072s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:02:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:02:22.733861425 +0000 UTC m=+2138.989260520" watchObservedRunningTime="2026-11-02 13:02:22.748099072 +0000 UTC m=+2139.003498164"
Nov 02 13:02:23.601099 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:02:23.601068    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-14fcb6-openshift-default-568ddb87b9-tzf8f"
Nov 02 13:02:23.606910 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:02:23.606883    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-14fcb6-openshift-default-568ddb87b9-tzf8f"
Nov 02 13:02:23.675996 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:02:23.675961    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-14fcb6-openshift-default-568ddb87b9-tzf8f"
Nov 02 13:02:23.677050 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:02:23.677027    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-14fcb6-openshift-default-568ddb87b9-tzf8f"
Nov 02 13:02:24.073362 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:02:24.073337    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-ecec44-openshift-default-6d94dccbcf-thqw7"
Nov 02 13:02:24.149068 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:02:24.149037    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abcacc36-3628-44e7-a103-fdf1027634cf-kube-api-access-zknzt\" (UniqueName: \"kubernetes.io/projected/abcacc36-3628-44e7-a103-fdf1027634cf-kube-api-access-zknzt\") pod \"abcacc36-3628-44e7-a103-fdf1027634cf\" (UID: \"abcacc36-3628-44e7-a103-fdf1027634cf\") "
Nov 02 13:02:24.149265 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:02:24.149080    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abcacc36-3628-44e7-a103-fdf1027634cf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abcacc36-3628-44e7-a103-fdf1027634cf-credential-socket\") pod \"abcacc36-3628-44e7-a103-fdf1027634cf\" (UID: \"abcacc36-3628-44e7-a103-fdf1027634cf\") "
Nov 02 13:02:24.149265 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:02:24.149112    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abcacc36-3628-44e7-a103-fdf1027634cf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abcacc36-3628-44e7-a103-fdf1027634cf-istio-data\") pod \"abcacc36-3628-44e7-a103-fdf1027634cf\" (UID: \"abcacc36-3628-44e7-a103-fdf1027634cf\") "
Nov 02 13:02:24.149265 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:02:24.149160    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abcacc36-3628-44e7-a103-fdf1027634cf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abcacc36-3628-44e7-a103-fdf1027634cf-workload-certs\") pod \"abcacc36-3628-44e7-a103-fdf1027634cf\" (UID: \"abcacc36-3628-44e7-a103-fdf1027634cf\") "
Nov 02 13:02:24.149265 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:02:24.149242    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abcacc36-3628-44e7-a103-fdf1027634cf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abcacc36-3628-44e7-a103-fdf1027634cf-istio-envoy\") pod \"abcacc36-3628-44e7-a103-fdf1027634cf\" (UID: \"abcacc36-3628-44e7-a103-fdf1027634cf\") "
Nov 02 13:02:24.149509 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:02:24.149361    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abcacc36-3628-44e7-a103-fdf1027634cf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abcacc36-3628-44e7-a103-fdf1027634cf-workload-socket\") pod \"abcacc36-3628-44e7-a103-fdf1027634cf\" (UID: \"abcacc36-3628-44e7-a103-fdf1027634cf\") "
Nov 02 13:02:24.149580 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:02:24.149498    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abcacc36-3628-44e7-a103-fdf1027634cf-istio-data" pod "abcacc36-3628-44e7-a103-fdf1027634cf" (UID: "abcacc36-3628-44e7-a103-fdf1027634cf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:02:24.149580 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:02:24.149524    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/abcacc36-3628-44e7-a103-fdf1027634cf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abcacc36-3628-44e7-a103-fdf1027634cf-istiod-ca-cert\") pod \"abcacc36-3628-44e7-a103-fdf1027634cf\" (UID: \"abcacc36-3628-44e7-a103-fdf1027634cf\") "
Nov 02 13:02:24.149724 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:02:24.149580    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abcacc36-3628-44e7-a103-fdf1027634cf-credential-socket" pod "abcacc36-3628-44e7-a103-fdf1027634cf" (UID: "abcacc36-3628-44e7-a103-fdf1027634cf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:02:24.149724 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:02:24.149596    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/abcacc36-3628-44e7-a103-fdf1027634cf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abcacc36-3628-44e7-a103-fdf1027634cf-istio-podinfo\") pod \"abcacc36-3628-44e7-a103-fdf1027634cf\" (UID: \"abcacc36-3628-44e7-a103-fdf1027634cf\") "
Nov 02 13:02:24.149724 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:02:24.149666    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abcacc36-3628-44e7-a103-fdf1027634cf-istio-token\" (UniqueName: \"kubernetes.io/projected/abcacc36-3628-44e7-a103-fdf1027634cf-istio-token\") pod \"abcacc36-3628-44e7-a103-fdf1027634cf\" (UID: \"abcacc36-3628-44e7-a103-fdf1027634cf\") "
Nov 02 13:02:24.149886 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:02:24.149785    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abcacc36-3628-44e7-a103-fdf1027634cf-workload-socket" pod "abcacc36-3628-44e7-a103-fdf1027634cf" (UID: "abcacc36-3628-44e7-a103-fdf1027634cf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:02:24.149950 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:02:24.149893    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abcacc36-3628-44e7-a103-fdf1027634cf-workload-certs" pod "abcacc36-3628-44e7-a103-fdf1027634cf" (UID: "abcacc36-3628-44e7-a103-fdf1027634cf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:02:24.149950 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:02:24.149924    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abcacc36-3628-44e7-a103-fdf1027634cf-istiod-ca-cert" pod "abcacc36-3628-44e7-a103-fdf1027634cf" (UID: "abcacc36-3628-44e7-a103-fdf1027634cf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:02:24.150368 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:02:24.150276    4559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abcacc36-3628-44e7-a103-fdf1027634cf-credential-socket\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:02:24.150368 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:02:24.150301    4559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abcacc36-3628-44e7-a103-fdf1027634cf-istio-data\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:02:24.150368 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:02:24.150316    4559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abcacc36-3628-44e7-a103-fdf1027634cf-workload-certs\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:02:24.150368 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:02:24.150329    4559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abcacc36-3628-44e7-a103-fdf1027634cf-workload-socket\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:02:24.150368 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:02:24.150343    4559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abcacc36-3628-44e7-a103-fdf1027634cf-istiod-ca-cert\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:02:24.152070 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:02:24.152042    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abcacc36-3628-44e7-a103-fdf1027634cf-istio-token" pod "abcacc36-3628-44e7-a103-fdf1027634cf" (UID: "abcacc36-3628-44e7-a103-fdf1027634cf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:02:24.152323 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:02:24.152301    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abcacc36-3628-44e7-a103-fdf1027634cf-istio-envoy" pod "abcacc36-3628-44e7-a103-fdf1027634cf" (UID: "abcacc36-3628-44e7-a103-fdf1027634cf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:02:24.152720 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:02:24.152700    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abcacc36-3628-44e7-a103-fdf1027634cf-kube-api-access-zknzt" pod "abcacc36-3628-44e7-a103-fdf1027634cf" (UID: "abcacc36-3628-44e7-a103-fdf1027634cf"). InnerVolumeSpecName "kube-api-access-zknzt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:02:24.152812 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:02:24.152704    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/abcacc36-3628-44e7-a103-fdf1027634cf-istio-podinfo" pod "abcacc36-3628-44e7-a103-fdf1027634cf" (UID: "abcacc36-3628-44e7-a103-fdf1027634cf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 13:02:24.259824 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:02:24.258233    4559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abcacc36-3628-44e7-a103-fdf1027634cf-istio-podinfo\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:02:24.259824 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:02:24.258273    4559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abcacc36-3628-44e7-a103-fdf1027634cf-istio-token\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:02:24.259824 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:02:24.258290    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zknzt\" (UniqueName: \"kubernetes.io/projected/abcacc36-3628-44e7-a103-fdf1027634cf-kube-api-access-zknzt\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:02:24.259824 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:02:24.258408    4559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abcacc36-3628-44e7-a103-fdf1027634cf-istio-envoy\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:02:24.684985 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:02:24.684948    4559 generic.go:356] "Generic (PLEG): container finished" podID="abcacc36-3628-44e7-a103-fdf1027634cf" containerID="91b6c8bf5e03dee916a17312abd00deed23b155b5f4959bf629aa3627856e1a9" exitCode=0
Nov 02 13:02:24.685472 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:02:24.685004    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-ecec44-openshift-default-6d94dccbcf-thqw7"
Nov 02 13:02:24.685472 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:02:24.685027    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-ecec44-openshift-default-6d94dccbcf-thqw7" event={"ID":"abcacc36-3628-44e7-a103-fdf1027634cf","Type":"ContainerDied","Data":"91b6c8bf5e03dee916a17312abd00deed23b155b5f4959bf629aa3627856e1a9"}
Nov 02 13:02:24.685472 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:02:24.685084    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-ecec44-openshift-default-6d94dccbcf-thqw7" event={"ID":"abcacc36-3628-44e7-a103-fdf1027634cf","Type":"ContainerDied","Data":"f9a7168a19074010c8064fd5f01cda6aab0d4aba9bbe2510e214e745e979a0e3"}
Nov 02 13:02:24.685472 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:02:24.685115    4559 scope.go:122] "RemoveContainer" containerID="91b6c8bf5e03dee916a17312abd00deed23b155b5f4959bf629aa3627856e1a9"
Nov 02 13:02:24.697351 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:02:24.697332    4559 scope.go:122] "RemoveContainer" containerID="ae925b4b0b98226777ab79fddf3d1314563e478a87f64c3170ac1de0d48ab2bf"
Nov 02 13:02:24.710594 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:02:24.709504    4559 scope.go:122] "RemoveContainer" containerID="91b6c8bf5e03dee916a17312abd00deed23b155b5f4959bf629aa3627856e1a9"
Nov 02 13:02:24.710594 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:02:24.710062    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91b6c8bf5e03dee916a17312abd00deed23b155b5f4959bf629aa3627856e1a9\": container with ID starting with 91b6c8bf5e03dee916a17312abd00deed23b155b5f4959bf629aa3627856e1a9 not found: ID does not exist" containerID="91b6c8bf5e03dee916a17312abd00deed23b155b5f4959bf629aa3627856e1a9"
Nov 02 13:02:24.710594 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:02:24.710098    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91b6c8bf5e03dee916a17312abd00deed23b155b5f4959bf629aa3627856e1a9"} err="failed to get container status \"91b6c8bf5e03dee916a17312abd00deed23b155b5f4959bf629aa3627856e1a9\": rpc error: code = NotFound desc = could not find container \"91b6c8bf5e03dee916a17312abd00deed23b155b5f4959bf629aa3627856e1a9\": container with ID starting with 91b6c8bf5e03dee916a17312abd00deed23b155b5f4959bf629aa3627856e1a9 not found: ID does not exist"
Nov 02 13:02:24.710594 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:02:24.710136    4559 scope.go:122] "RemoveContainer" containerID="ae925b4b0b98226777ab79fddf3d1314563e478a87f64c3170ac1de0d48ab2bf"
Nov 02 13:02:24.710594 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:02:24.710568    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae925b4b0b98226777ab79fddf3d1314563e478a87f64c3170ac1de0d48ab2bf\": container with ID starting with ae925b4b0b98226777ab79fddf3d1314563e478a87f64c3170ac1de0d48ab2bf not found: ID does not exist" containerID="ae925b4b0b98226777ab79fddf3d1314563e478a87f64c3170ac1de0d48ab2bf"
Nov 02 13:02:24.710877 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:02:24.710613    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae925b4b0b98226777ab79fddf3d1314563e478a87f64c3170ac1de0d48ab2bf"} err="failed to get container status \"ae925b4b0b98226777ab79fddf3d1314563e478a87f64c3170ac1de0d48ab2bf\": rpc error: code = NotFound desc = could not find container \"ae925b4b0b98226777ab79fddf3d1314563e478a87f64c3170ac1de0d48ab2bf\": container with ID starting with ae925b4b0b98226777ab79fddf3d1314563e478a87f64c3170ac1de0d48ab2bf not found: ID does not exist"
Nov 02 13:02:24.759750 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:02:24.759710    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-ecec44-openshift-default-6d94dccbcf-thqw7"]
Nov 02 13:02:24.770125 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:02:24.770097    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-ecec44-openshift-default-6d94dccbcf-thqw7"]
Nov 02 13:02:25.893510 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:02:25.893473    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="abcacc36-3628-44e7-a103-fdf1027634cf" path="/var/lib/kubelet/pods/abcacc36-3628-44e7-a103-fdf1027634cf/volumes"
Nov 02 13:02:37.517664 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:02:37.517574    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-90cda9-5f86b75b95-bnzqk"]
Nov 02 13:02:37.518248 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:02:37.518233    4559 cpu_manager.go:414] "containerMap: removing container" podUID="abcacc36-3628-44e7-a103-fdf1027634cf" containerName="istio-proxy"
Nov 02 13:02:37.518289 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:02:37.518248    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abcacc36-3628-44e7-a103-fdf1027634cf" containerName="istio-proxy"
Nov 02 13:02:37.518289 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:02:37.518284    4559 cpu_manager.go:414] "containerMap: removing container" podUID="abcacc36-3628-44e7-a103-fdf1027634cf" containerName="istio-proxy"
Nov 02 13:02:37.518376 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:02:37.518290    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abcacc36-3628-44e7-a103-fdf1027634cf" containerName="istio-proxy"
Nov 02 13:02:37.518411 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:02:37.518397    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="abcacc36-3628-44e7-a103-fdf1027634cf" containerName="istio-proxy"
Nov 02 13:02:37.518449 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:02:37.518414    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="abcacc36-3628-44e7-a103-fdf1027634cf" containerName="istio-proxy"
Nov 02 13:02:37.523480 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:02:37.523461    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-90cda9-5f86b75b95-bnzqk"
Nov 02 13:02:37.551394 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:02:37.551342    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pv49h\" (UniqueName: \"kubernetes.io/projected/8acbad7b-5320-4a6c-80b1-b1766c55f340-kube-api-access-pv49h\") pod \"maas-api-e2e-extmod-b-90cda9-5f86b75b95-bnzqk\" (UID: \"8acbad7b-5320-4a6c-80b1-b1766c55f340\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-90cda9-5f86b75b95-bnzqk"
Nov 02 13:02:37.551700 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:02:37.551679    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8acbad7b-5320-4a6c-80b1-b1766c55f340-maas-api-tls\") pod \"maas-api-e2e-extmod-b-90cda9-5f86b75b95-bnzqk\" (UID: \"8acbad7b-5320-4a6c-80b1-b1766c55f340\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-90cda9-5f86b75b95-bnzqk"
Nov 02 13:02:37.552124 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:02:37.552098    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8acbad7b-5320-4a6c-80b1-b1766c55f340-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-90cda9-5f86b75b95-bnzqk\" (UID: \"8acbad7b-5320-4a6c-80b1-b1766c55f340\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-90cda9-5f86b75b95-bnzqk"
Nov 02 13:02:37.552259 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:02:37.552217    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8acbad7b-5320-4a6c-80b1-b1766c55f340-metrics-certs\") pod \"maas-api-e2e-extmod-b-90cda9-5f86b75b95-bnzqk\" (UID: \"8acbad7b-5320-4a6c-80b1-b1766c55f340\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-90cda9-5f86b75b95-bnzqk"
Nov 02 13:02:37.552565 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:02:37.552524    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-90cda9-serving-cert\""
Nov 02 13:02:37.558547 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:02:37.558508    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-90cda9-5f86b75b95-bnzqk"]
Nov 02 13:02:37.653138 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:02:37.653100    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8acbad7b-5320-4a6c-80b1-b1766c55f340-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-90cda9-5f86b75b95-bnzqk\" (UID: \"8acbad7b-5320-4a6c-80b1-b1766c55f340\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-90cda9-5f86b75b95-bnzqk"
Nov 02 13:02:37.653316 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:02:37.653207    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8acbad7b-5320-4a6c-80b1-b1766c55f340-metrics-certs\") pod \"maas-api-e2e-extmod-b-90cda9-5f86b75b95-bnzqk\" (UID: \"8acbad7b-5320-4a6c-80b1-b1766c55f340\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-90cda9-5f86b75b95-bnzqk"
Nov 02 13:02:37.653316 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:02:37.653236    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pv49h\" (UniqueName: \"kubernetes.io/projected/8acbad7b-5320-4a6c-80b1-b1766c55f340-kube-api-access-pv49h\") pod \"maas-api-e2e-extmod-b-90cda9-5f86b75b95-bnzqk\" (UID: \"8acbad7b-5320-4a6c-80b1-b1766c55f340\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-90cda9-5f86b75b95-bnzqk"
Nov 02 13:02:37.653316 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:02:37.653308    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8acbad7b-5320-4a6c-80b1-b1766c55f340-maas-api-tls\") pod \"maas-api-e2e-extmod-b-90cda9-5f86b75b95-bnzqk\" (UID: \"8acbad7b-5320-4a6c-80b1-b1766c55f340\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-90cda9-5f86b75b95-bnzqk"
Nov 02 13:02:37.653844 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:02:37.653813    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8acbad7b-5320-4a6c-80b1-b1766c55f340-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-90cda9-5f86b75b95-bnzqk\" (UID: \"8acbad7b-5320-4a6c-80b1-b1766c55f340\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-90cda9-5f86b75b95-bnzqk"
Nov 02 13:02:37.655811 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:02:37.655790    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8acbad7b-5320-4a6c-80b1-b1766c55f340-maas-api-tls\") pod \"maas-api-e2e-extmod-b-90cda9-5f86b75b95-bnzqk\" (UID: \"8acbad7b-5320-4a6c-80b1-b1766c55f340\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-90cda9-5f86b75b95-bnzqk"
Nov 02 13:02:37.655894 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:02:37.655823    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8acbad7b-5320-4a6c-80b1-b1766c55f340-metrics-certs\") pod \"maas-api-e2e-extmod-b-90cda9-5f86b75b95-bnzqk\" (UID: \"8acbad7b-5320-4a6c-80b1-b1766c55f340\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-90cda9-5f86b75b95-bnzqk"
Nov 02 13:02:37.677260 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:02:37.677229    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pv49h\" (UniqueName: \"kubernetes.io/projected/8acbad7b-5320-4a6c-80b1-b1766c55f340-kube-api-access-pv49h\") pod \"maas-api-e2e-extmod-b-90cda9-5f86b75b95-bnzqk\" (UID: \"8acbad7b-5320-4a6c-80b1-b1766c55f340\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-90cda9-5f86b75b95-bnzqk"
Nov 02 13:02:37.837549 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:02:37.837466    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-90cda9-5f86b75b95-bnzqk"
Nov 02 13:02:38.128595 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:02:38.128508    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"]
Nov 02 13:02:38.152178 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:02:38.152148    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-90cda9-5f86b75b95-bnzqk"]
Nov 02 13:02:38.152328 ip-10-0-128-132 kubenswrapper[4559]: W1102 13:02:38.152214    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8acbad7b_5320_4a6c_80b1_b1766c55f340.slice/crio-39e48fd80e03cafa3375ed50f62f3f99c003a9867230455adcdf352407ab0594 WatchSource:0}: Error finding container 39e48fd80e03cafa3375ed50f62f3f99c003a9867230455adcdf352407ab0594: Status 404 returned error can't find the container with id 39e48fd80e03cafa3375ed50f62f3f99c003a9867230455adcdf352407ab0594
Nov 02 13:02:38.827331 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:02:38.827238    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-90cda9-5f86b75b95-bnzqk" event={"ID":"8acbad7b-5320-4a6c-80b1-b1766c55f340","Type":"ContainerStarted","Data":"9a34e36329226612c7729d6dbca102816c2e41091846d9b166e3718d76347c0a"}
Nov 02 13:02:38.827331 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:02:38.827315    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-90cda9-5f86b75b95-bnzqk" event={"ID":"8acbad7b-5320-4a6c-80b1-b1766c55f340","Type":"ContainerStarted","Data":"39e48fd80e03cafa3375ed50f62f3f99c003a9867230455adcdf352407ab0594"}
Nov 02 13:02:38.827802 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:02:38.827441    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-90cda9-5f86b75b95-bnzqk"
Nov 02 13:02:38.872427 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:02:38.872374    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-90cda9-5f86b75b95-bnzqk" podStartSLOduration=1.5516053859999999 podStartE2EDuration="1.872358227s" totalImagesPullingTime="320.752841ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:02:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:02:38.871149791 +0000 UTC m=+2155.126548902" watchObservedRunningTime="2026-11-02 13:02:38.872358227 +0000 UTC m=+2155.127757321"
Nov 02 13:02:44.850833 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:02:44.850800    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-90cda9-5f86b75b95-bnzqk"
Nov 02 13:02:45.153222 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:02:45.153132    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqcmj" containerName="extractor"
Nov 02 13:02:58.923614 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:02:58.923576    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-14fcb6-7cd8b6b866-tdw6v"]
Nov 02 13:02:58.929559 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:02:58.929534    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-14fcb6-7cd8b6b866-tdw6v"
Nov 02 13:02:58.953680 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:02:58.953649    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-14fcb6-7cd8b6b866-tdw6v"]
Nov 02 13:02:58.956499 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:02:58.956468    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-14fcb6-serving-cert\""
Nov 02 13:02:59.099737 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:02:59.099706    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b49dc55f-dea9-4a2d-be97-0bc8219e8fe1-metrics-certs\") pod \"maas-api-e2e-extmod-a-14fcb6-7cd8b6b866-tdw6v\" (UID: \"b49dc55f-dea9-4a2d-be97-0bc8219e8fe1\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-14fcb6-7cd8b6b866-tdw6v"
Nov 02 13:02:59.099924 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:02:59.099842    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b49dc55f-dea9-4a2d-be97-0bc8219e8fe1-maas-api-tls\") pod \"maas-api-e2e-extmod-a-14fcb6-7cd8b6b866-tdw6v\" (UID: \"b49dc55f-dea9-4a2d-be97-0bc8219e8fe1\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-14fcb6-7cd8b6b866-tdw6v"
Nov 02 13:02:59.099924 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:02:59.099897    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b49dc55f-dea9-4a2d-be97-0bc8219e8fe1-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-14fcb6-7cd8b6b866-tdw6v\" (UID: \"b49dc55f-dea9-4a2d-be97-0bc8219e8fe1\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-14fcb6-7cd8b6b866-tdw6v"
Nov 02 13:02:59.100008 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:02:59.099943    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5pfj\" (UniqueName: \"kubernetes.io/projected/b49dc55f-dea9-4a2d-be97-0bc8219e8fe1-kube-api-access-j5pfj\") pod \"maas-api-e2e-extmod-a-14fcb6-7cd8b6b866-tdw6v\" (UID: \"b49dc55f-dea9-4a2d-be97-0bc8219e8fe1\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-14fcb6-7cd8b6b866-tdw6v"
Nov 02 13:02:59.212773 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:02:59.206266    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b49dc55f-dea9-4a2d-be97-0bc8219e8fe1-metrics-certs\") pod \"maas-api-e2e-extmod-a-14fcb6-7cd8b6b866-tdw6v\" (UID: \"b49dc55f-dea9-4a2d-be97-0bc8219e8fe1\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-14fcb6-7cd8b6b866-tdw6v"
Nov 02 13:02:59.212773 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:02:59.206963    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b49dc55f-dea9-4a2d-be97-0bc8219e8fe1-maas-api-tls\") pod \"maas-api-e2e-extmod-a-14fcb6-7cd8b6b866-tdw6v\" (UID: \"b49dc55f-dea9-4a2d-be97-0bc8219e8fe1\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-14fcb6-7cd8b6b866-tdw6v"
Nov 02 13:02:59.212773 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:02:59.207012    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b49dc55f-dea9-4a2d-be97-0bc8219e8fe1-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-14fcb6-7cd8b6b866-tdw6v\" (UID: \"b49dc55f-dea9-4a2d-be97-0bc8219e8fe1\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-14fcb6-7cd8b6b866-tdw6v"
Nov 02 13:02:59.212773 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:02:59.207073    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5pfj\" (UniqueName: \"kubernetes.io/projected/b49dc55f-dea9-4a2d-be97-0bc8219e8fe1-kube-api-access-j5pfj\") pod \"maas-api-e2e-extmod-a-14fcb6-7cd8b6b866-tdw6v\" (UID: \"b49dc55f-dea9-4a2d-be97-0bc8219e8fe1\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-14fcb6-7cd8b6b866-tdw6v"
Nov 02 13:02:59.212773 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:02:59.208557    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b49dc55f-dea9-4a2d-be97-0bc8219e8fe1-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-14fcb6-7cd8b6b866-tdw6v\" (UID: \"b49dc55f-dea9-4a2d-be97-0bc8219e8fe1\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-14fcb6-7cd8b6b866-tdw6v"
Nov 02 13:02:59.245382 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:02:59.245313    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b49dc55f-dea9-4a2d-be97-0bc8219e8fe1-metrics-certs\") pod \"maas-api-e2e-extmod-a-14fcb6-7cd8b6b866-tdw6v\" (UID: \"b49dc55f-dea9-4a2d-be97-0bc8219e8fe1\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-14fcb6-7cd8b6b866-tdw6v"
Nov 02 13:02:59.253233 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:02:59.253199    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b49dc55f-dea9-4a2d-be97-0bc8219e8fe1-maas-api-tls\") pod \"maas-api-e2e-extmod-a-14fcb6-7cd8b6b866-tdw6v\" (UID: \"b49dc55f-dea9-4a2d-be97-0bc8219e8fe1\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-14fcb6-7cd8b6b866-tdw6v"
Nov 02 13:02:59.253709 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:02:59.253682    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5pfj\" (UniqueName: \"kubernetes.io/projected/b49dc55f-dea9-4a2d-be97-0bc8219e8fe1-kube-api-access-j5pfj\") pod \"maas-api-e2e-extmod-a-14fcb6-7cd8b6b866-tdw6v\" (UID: \"b49dc55f-dea9-4a2d-be97-0bc8219e8fe1\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-14fcb6-7cd8b6b866-tdw6v"
Nov 02 13:02:59.543447 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:02:59.543369    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-14fcb6-7cd8b6b866-tdw6v"
Nov 02 13:02:59.847552 ip-10-0-128-132 kubenswrapper[4559]: W1102 13:02:59.847519    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb49dc55f_dea9_4a2d_be97_0bc8219e8fe1.slice/crio-f3a37fdc25f755542f3959b01684dd875da9d652ac7169a06798dec546bf36ca WatchSource:0}: Error finding container f3a37fdc25f755542f3959b01684dd875da9d652ac7169a06798dec546bf36ca: Status 404 returned error can't find the container with id f3a37fdc25f755542f3959b01684dd875da9d652ac7169a06798dec546bf36ca
Nov 02 13:02:59.849717 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:02:59.849688    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-14fcb6-7cd8b6b866-tdw6v"]
Nov 02 13:03:00.006250 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:03:00.006208    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-14fcb6-7cd8b6b866-tdw6v" event={"ID":"b49dc55f-dea9-4a2d-be97-0bc8219e8fe1","Type":"ContainerStarted","Data":"f3a37fdc25f755542f3959b01684dd875da9d652ac7169a06798dec546bf36ca"}
Nov 02 13:03:01.014877 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:03:01.014845    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-14fcb6-7cd8b6b866-tdw6v" event={"ID":"b49dc55f-dea9-4a2d-be97-0bc8219e8fe1","Type":"ContainerStarted","Data":"ebaae3d5bfa136ba1ab72cdfe04326deebb3eb91aa991dab18cc972d97af0b6c"}
Nov 02 13:03:01.015311 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:03:01.014974    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-14fcb6-7cd8b6b866-tdw6v"
Nov 02 13:03:01.059806 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:03:01.059751    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-14fcb6-7cd8b6b866-tdw6v" podStartSLOduration=2.71553688 podStartE2EDuration="3.059736515s" totalImagesPullingTime="344.199635ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:02:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:03:01.058079187 +0000 UTC m=+2177.313478287" watchObservedRunningTime="2026-11-02 13:03:01.059736515 +0000 UTC m=+2177.315135639"
Nov 02 13:03:07.026983 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:03:07.026952    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-14fcb6-7cd8b6b866-tdw6v"
Nov 02 13:03:12.125330 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:03:12.125290    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"]
Nov 02 13:03:15.134663 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:03:15.127824    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqcmj" containerName="extractor"
Nov 02 13:03:24.729716 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:03:24.728122    4559 patch_prober.go:29] interesting pod/e2e-extmod-a-14fcb6-openshift-default-568ddb87b9-tzf8f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.217:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 13:03:24.729716 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:03:24.728208    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-14fcb6-openshift-default-568ddb87b9-tzf8f" podUID="0029ba38-71d3-4212-9b50-ded1a42abd1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.217:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:03:27.230819 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:03:27.230778    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-936529-openshift-default-5578bccc97-9cw4k"]
Nov 02 13:03:27.264845 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:03:27.264808    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-936529-openshift-default-5578bccc97-9cw4k"
Nov 02 13:03:27.265153 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:03:27.265114    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-936529-openshift-default-5578bccc97-9cw4k"]
Nov 02 13:03:27.287137 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:03:27.287108    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-936529-openshift-default-dockercfg-7z5jv\""
Nov 02 13:03:27.361114 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:03:27.351776    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ae3594b-0d9b-431e-b165-bb290002858a-istio-data\") pod \"e2e-extmod-b-936529-openshift-default-5578bccc97-9cw4k\" (UID: \"6ae3594b-0d9b-431e-b165-bb290002858a\") " pod="openshift-ingress/e2e-extmod-b-936529-openshift-default-5578bccc97-9cw4k"
Nov 02 13:03:27.361114 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:03:27.351884    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kck68\" (UniqueName: \"kubernetes.io/projected/6ae3594b-0d9b-431e-b165-bb290002858a-kube-api-access-kck68\") pod \"e2e-extmod-b-936529-openshift-default-5578bccc97-9cw4k\" (UID: \"6ae3594b-0d9b-431e-b165-bb290002858a\") " pod="openshift-ingress/e2e-extmod-b-936529-openshift-default-5578bccc97-9cw4k"
Nov 02 13:03:27.361114 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:03:27.351928    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ae3594b-0d9b-431e-b165-bb290002858a-istio-podinfo\") pod \"e2e-extmod-b-936529-openshift-default-5578bccc97-9cw4k\" (UID: \"6ae3594b-0d9b-431e-b165-bb290002858a\") " pod="openshift-ingress/e2e-extmod-b-936529-openshift-default-5578bccc97-9cw4k"
Nov 02 13:03:27.361114 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:03:27.351969    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ae3594b-0d9b-431e-b165-bb290002858a-istiod-ca-cert\") pod \"e2e-extmod-b-936529-openshift-default-5578bccc97-9cw4k\" (UID: \"6ae3594b-0d9b-431e-b165-bb290002858a\") " pod="openshift-ingress/e2e-extmod-b-936529-openshift-default-5578bccc97-9cw4k"
Nov 02 13:03:27.361114 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:03:27.352033    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ae3594b-0d9b-431e-b165-bb290002858a-istio-token\") pod \"e2e-extmod-b-936529-openshift-default-5578bccc97-9cw4k\" (UID: \"6ae3594b-0d9b-431e-b165-bb290002858a\") " pod="openshift-ingress/e2e-extmod-b-936529-openshift-default-5578bccc97-9cw4k"
Nov 02 13:03:27.361114 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:03:27.352075    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ae3594b-0d9b-431e-b165-bb290002858a-workload-certs\") pod \"e2e-extmod-b-936529-openshift-default-5578bccc97-9cw4k\" (UID: \"6ae3594b-0d9b-431e-b165-bb290002858a\") " pod="openshift-ingress/e2e-extmod-b-936529-openshift-default-5578bccc97-9cw4k"
Nov 02 13:03:27.361114 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:03:27.352180    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ae3594b-0d9b-431e-b165-bb290002858a-workload-socket\") pod \"e2e-extmod-b-936529-openshift-default-5578bccc97-9cw4k\" (UID: \"6ae3594b-0d9b-431e-b165-bb290002858a\") " pod="openshift-ingress/e2e-extmod-b-936529-openshift-default-5578bccc97-9cw4k"
Nov 02 13:03:27.361114 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:03:27.352207    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ae3594b-0d9b-431e-b165-bb290002858a-credential-socket\") pod \"e2e-extmod-b-936529-openshift-default-5578bccc97-9cw4k\" (UID: \"6ae3594b-0d9b-431e-b165-bb290002858a\") " pod="openshift-ingress/e2e-extmod-b-936529-openshift-default-5578bccc97-9cw4k"
Nov 02 13:03:27.361114 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:03:27.352325    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ae3594b-0d9b-431e-b165-bb290002858a-istio-envoy\") pod \"e2e-extmod-b-936529-openshift-default-5578bccc97-9cw4k\" (UID: \"6ae3594b-0d9b-431e-b165-bb290002858a\") " pod="openshift-ingress/e2e-extmod-b-936529-openshift-default-5578bccc97-9cw4k"
Nov 02 13:03:27.465767 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:03:27.465729    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ae3594b-0d9b-431e-b165-bb290002858a-workload-socket\") pod \"e2e-extmod-b-936529-openshift-default-5578bccc97-9cw4k\" (UID: \"6ae3594b-0d9b-431e-b165-bb290002858a\") " pod="openshift-ingress/e2e-extmod-b-936529-openshift-default-5578bccc97-9cw4k"
Nov 02 13:03:27.466077 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:03:27.466054    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ae3594b-0d9b-431e-b165-bb290002858a-credential-socket\") pod \"e2e-extmod-b-936529-openshift-default-5578bccc97-9cw4k\" (UID: \"6ae3594b-0d9b-431e-b165-bb290002858a\") " pod="openshift-ingress/e2e-extmod-b-936529-openshift-default-5578bccc97-9cw4k"
Nov 02 13:03:27.466316 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:03:27.466300    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ae3594b-0d9b-431e-b165-bb290002858a-istio-envoy\") pod \"e2e-extmod-b-936529-openshift-default-5578bccc97-9cw4k\" (UID: \"6ae3594b-0d9b-431e-b165-bb290002858a\") " pod="openshift-ingress/e2e-extmod-b-936529-openshift-default-5578bccc97-9cw4k"
Nov 02 13:03:27.466492 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:03:27.466474    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ae3594b-0d9b-431e-b165-bb290002858a-istio-data\") pod \"e2e-extmod-b-936529-openshift-default-5578bccc97-9cw4k\" (UID: \"6ae3594b-0d9b-431e-b165-bb290002858a\") " pod="openshift-ingress/e2e-extmod-b-936529-openshift-default-5578bccc97-9cw4k"
Nov 02 13:03:27.467003 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:03:27.466980    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kck68\" (UniqueName: \"kubernetes.io/projected/6ae3594b-0d9b-431e-b165-bb290002858a-kube-api-access-kck68\") pod \"e2e-extmod-b-936529-openshift-default-5578bccc97-9cw4k\" (UID: \"6ae3594b-0d9b-431e-b165-bb290002858a\") " pod="openshift-ingress/e2e-extmod-b-936529-openshift-default-5578bccc97-9cw4k"
Nov 02 13:03:27.467160 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:03:27.467146    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ae3594b-0d9b-431e-b165-bb290002858a-istio-podinfo\") pod \"e2e-extmod-b-936529-openshift-default-5578bccc97-9cw4k\" (UID: \"6ae3594b-0d9b-431e-b165-bb290002858a\") " pod="openshift-ingress/e2e-extmod-b-936529-openshift-default-5578bccc97-9cw4k"
Nov 02 13:03:27.467298 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:03:27.467281    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ae3594b-0d9b-431e-b165-bb290002858a-istiod-ca-cert\") pod \"e2e-extmod-b-936529-openshift-default-5578bccc97-9cw4k\" (UID: \"6ae3594b-0d9b-431e-b165-bb290002858a\") " pod="openshift-ingress/e2e-extmod-b-936529-openshift-default-5578bccc97-9cw4k"
Nov 02 13:03:27.467574 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:03:27.467552    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ae3594b-0d9b-431e-b165-bb290002858a-istio-token\") pod \"e2e-extmod-b-936529-openshift-default-5578bccc97-9cw4k\" (UID: \"6ae3594b-0d9b-431e-b165-bb290002858a\") " pod="openshift-ingress/e2e-extmod-b-936529-openshift-default-5578bccc97-9cw4k"
Nov 02 13:03:27.472066 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:03:27.467758    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ae3594b-0d9b-431e-b165-bb290002858a-workload-certs\") pod \"e2e-extmod-b-936529-openshift-default-5578bccc97-9cw4k\" (UID: \"6ae3594b-0d9b-431e-b165-bb290002858a\") " pod="openshift-ingress/e2e-extmod-b-936529-openshift-default-5578bccc97-9cw4k"
Nov 02 13:03:27.472066 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:03:27.468282    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ae3594b-0d9b-431e-b165-bb290002858a-workload-certs\") pod \"e2e-extmod-b-936529-openshift-default-5578bccc97-9cw4k\" (UID: \"6ae3594b-0d9b-431e-b165-bb290002858a\") " pod="openshift-ingress/e2e-extmod-b-936529-openshift-default-5578bccc97-9cw4k"
Nov 02 13:03:27.472066 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:03:27.468538    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ae3594b-0d9b-431e-b165-bb290002858a-workload-socket\") pod \"e2e-extmod-b-936529-openshift-default-5578bccc97-9cw4k\" (UID: \"6ae3594b-0d9b-431e-b165-bb290002858a\") " pod="openshift-ingress/e2e-extmod-b-936529-openshift-default-5578bccc97-9cw4k"
Nov 02 13:03:27.472066 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:03:27.468764    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ae3594b-0d9b-431e-b165-bb290002858a-credential-socket\") pod \"e2e-extmod-b-936529-openshift-default-5578bccc97-9cw4k\" (UID: \"6ae3594b-0d9b-431e-b165-bb290002858a\") " pod="openshift-ingress/e2e-extmod-b-936529-openshift-default-5578bccc97-9cw4k"
Nov 02 13:03:27.478208 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:03:27.474819    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ae3594b-0d9b-431e-b165-bb290002858a-istio-envoy\") pod \"e2e-extmod-b-936529-openshift-default-5578bccc97-9cw4k\" (UID: \"6ae3594b-0d9b-431e-b165-bb290002858a\") " pod="openshift-ingress/e2e-extmod-b-936529-openshift-default-5578bccc97-9cw4k"
Nov 02 13:03:27.478208 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:03:27.475498    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ae3594b-0d9b-431e-b165-bb290002858a-istio-data\") pod \"e2e-extmod-b-936529-openshift-default-5578bccc97-9cw4k\" (UID: \"6ae3594b-0d9b-431e-b165-bb290002858a\") " pod="openshift-ingress/e2e-extmod-b-936529-openshift-default-5578bccc97-9cw4k"
Nov 02 13:03:27.484013 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:03:27.483946    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ae3594b-0d9b-431e-b165-bb290002858a-istiod-ca-cert\") pod \"e2e-extmod-b-936529-openshift-default-5578bccc97-9cw4k\" (UID: \"6ae3594b-0d9b-431e-b165-bb290002858a\") " pod="openshift-ingress/e2e-extmod-b-936529-openshift-default-5578bccc97-9cw4k"
Nov 02 13:03:27.505663 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:03:27.505165    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ae3594b-0d9b-431e-b165-bb290002858a-istio-podinfo\") pod \"e2e-extmod-b-936529-openshift-default-5578bccc97-9cw4k\" (UID: \"6ae3594b-0d9b-431e-b165-bb290002858a\") " pod="openshift-ingress/e2e-extmod-b-936529-openshift-default-5578bccc97-9cw4k"
Nov 02 13:03:27.512769 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:03:27.509511    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ae3594b-0d9b-431e-b165-bb290002858a-istio-token\") pod \"e2e-extmod-b-936529-openshift-default-5578bccc97-9cw4k\" (UID: \"6ae3594b-0d9b-431e-b165-bb290002858a\") " pod="openshift-ingress/e2e-extmod-b-936529-openshift-default-5578bccc97-9cw4k"
Nov 02 13:03:27.524124 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:03:27.524072    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kck68\" (UniqueName: \"kubernetes.io/projected/6ae3594b-0d9b-431e-b165-bb290002858a-kube-api-access-kck68\") pod \"e2e-extmod-b-936529-openshift-default-5578bccc97-9cw4k\" (UID: \"6ae3594b-0d9b-431e-b165-bb290002858a\") " pod="openshift-ingress/e2e-extmod-b-936529-openshift-default-5578bccc97-9cw4k"
Nov 02 13:03:27.629667 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:03:27.613010    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-936529-openshift-default-5578bccc97-9cw4k"
Nov 02 13:03:28.131916 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:03:28.003833    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-936529-openshift-default-5578bccc97-9cw4k"]
Nov 02 13:03:28.131916 ip-10-0-128-132 kubenswrapper[4559]: W1102 13:03:28.031065    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ae3594b_0d9b_431e_b165_bb290002858a.slice/crio-35975e05c621db548df1a1b13766762be48d4beb4e97d764c2ba617843304e94 WatchSource:0}: Error finding container 35975e05c621db548df1a1b13766762be48d4beb4e97d764c2ba617843304e94: Status 404 returned error can't find the container with id 35975e05c621db548df1a1b13766762be48d4beb4e97d764c2ba617843304e94
Nov 02 13:03:28.131916 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:03:28.033938    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 13:03:28.131916 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:03:28.034008    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 13:03:28.131916 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:03:28.034038    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 13:03:28.435505 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:03:28.435470    4559 generic.go:356] "Generic (PLEG): container finished" podID="0029ba38-71d3-4212-9b50-ded1a42abd1d" containerID="3de5c889a2ef40f69cbc4ea9aa9eda016b3925972ef4c9c11a190b59925b248e" exitCode=0
Nov 02 13:03:28.435997 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:03:28.435540    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-14fcb6-openshift-default-568ddb87b9-tzf8f" event={"ID":"0029ba38-71d3-4212-9b50-ded1a42abd1d","Type":"ContainerDied","Data":"3de5c889a2ef40f69cbc4ea9aa9eda016b3925972ef4c9c11a190b59925b248e"}
Nov 02 13:03:28.436075 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:03:28.436053    4559 scope.go:122] "RemoveContainer" containerID="3de5c889a2ef40f69cbc4ea9aa9eda016b3925972ef4c9c11a190b59925b248e"
Nov 02 13:03:28.437587 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:03:28.437563    4559 generic.go:356] "Generic (PLEG): container finished" podID="bb3c761d-8641-496c-9d3a-531e8117e7dd" containerID="d421f86a3e5b5b24f5e2908085ec94c480c14b86b84e4c22495f03c365aa50fb" exitCode=0
Nov 02 13:03:28.437750 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:03:28.437672    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-90cda9-openshift-default-57f8b6f957-b5jzx" event={"ID":"bb3c761d-8641-496c-9d3a-531e8117e7dd","Type":"ContainerDied","Data":"d421f86a3e5b5b24f5e2908085ec94c480c14b86b84e4c22495f03c365aa50fb"}
Nov 02 13:03:28.438064 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:03:28.438021    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 13:03:28.438064 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:03:28.438058    4559 scope.go:122] "RemoveContainer" containerID="d421f86a3e5b5b24f5e2908085ec94c480c14b86b84e4c22495f03c365aa50fb"
Nov 02 13:03:28.438241 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:03:28.438118    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 13:03:28.438241 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:03:28.438159    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 13:03:28.439766 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:03:28.439742    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 13:03:28.439856 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:03:28.439803    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 13:03:28.439856 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:03:28.439833    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 13:03:28.440214 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:03:28.440193    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-936529-openshift-default-5578bccc97-9cw4k" event={"ID":"6ae3594b-0d9b-431e-b165-bb290002858a","Type":"ContainerStarted","Data":"9558dea98056df34d056ae3419f34b5c7a53a5c076914de1f956a03cf8c76cf4"}
Nov 02 13:03:28.440359 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:03:28.440240    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-936529-openshift-default-5578bccc97-9cw4k" event={"ID":"6ae3594b-0d9b-431e-b165-bb290002858a","Type":"ContainerStarted","Data":"35975e05c621db548df1a1b13766762be48d4beb4e97d764c2ba617843304e94"}
Nov 02 13:03:28.593258 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:03:28.593144    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-936529-openshift-default-5578bccc97-9cw4k" podStartSLOduration=1.5931279969999999 podStartE2EDuration="1.593127997s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:03:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:03:28.54363162 +0000 UTC m=+2204.799030707" watchObservedRunningTime="2026-11-02 13:03:28.593127997 +0000 UTC m=+2204.848527092"
Nov 02 13:03:28.640715 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:03:28.638815    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-936529-openshift-default-5578bccc97-9cw4k"
Nov 02 13:03:29.457355 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:03:29.457317    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-90cda9-openshift-default-57f8b6f957-b5jzx" event={"ID":"bb3c761d-8641-496c-9d3a-531e8117e7dd","Type":"ContainerStarted","Data":"e59de548036ec32daed50a5119084a5473e6be9275af17787db33e25b4b53425"}
Nov 02 13:03:29.460502 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:03:29.460464    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-14fcb6-openshift-default-568ddb87b9-tzf8f" event={"ID":"0029ba38-71d3-4212-9b50-ded1a42abd1d","Type":"ContainerStarted","Data":"c8634cb2c9ca821baf4648fb98052f709ae9905476a7296de6da8b14e581e117"}
Nov 02 13:03:29.605669 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:03:29.605597    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-14fcb6-openshift-default-568ddb87b9-tzf8f"
Nov 02 13:03:29.605891 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:03:29.605789    4559 patch_prober.go:29] interesting pod/e2e-extmod-a-14fcb6-openshift-default-568ddb87b9-tzf8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused" start-of-body=
Nov 02 13:03:29.605891 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:03:29.605855    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-14fcb6-openshift-default-568ddb87b9-tzf8f" podUID="0029ba38-71d3-4212-9b50-ded1a42abd1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused"
Nov 02 13:03:29.639692 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:03:29.638948    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-936529-openshift-default-5578bccc97-9cw4k"
Nov 02 13:03:29.963859 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:03:29.963829    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-90cda9-openshift-default-57f8b6f957-b5jzx"
Nov 02 13:03:29.965486 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:03:29.965460    4559 patch_prober.go:29] interesting pod/e2e-extmod-b-90cda9-openshift-default-57f8b6f957-b5jzx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused" start-of-body=
Nov 02 13:03:29.965606 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:03:29.965514    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-90cda9-openshift-default-57f8b6f957-b5jzx" podUID="bb3c761d-8641-496c-9d3a-531e8117e7dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused"
Nov 02 13:03:30.467376 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:03:30.467335    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-936529-openshift-default-5578bccc97-9cw4k"
Nov 02 13:03:30.468208 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:03:30.468189    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-936529-openshift-default-5578bccc97-9cw4k"
Nov 02 13:03:30.624668 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:03:30.624066    4559 patch_prober.go:29] interesting pod/e2e-extmod-a-14fcb6-openshift-default-568ddb87b9-tzf8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused" start-of-body=
Nov 02 13:03:30.624668 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:03:30.624127    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-14fcb6-openshift-default-568ddb87b9-tzf8f" podUID="0029ba38-71d3-4212-9b50-ded1a42abd1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused"
Nov 02 13:03:30.964234 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:03:30.964191    4559 patch_prober.go:29] interesting pod/e2e-extmod-b-90cda9-openshift-default-57f8b6f957-b5jzx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused" start-of-body=
Nov 02 13:03:30.964458 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:03:30.964256    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-90cda9-openshift-default-57f8b6f957-b5jzx" podUID="bb3c761d-8641-496c-9d3a-531e8117e7dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused"
Nov 02 13:03:31.470841 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:03:31.470808    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-v9zqm"]
Nov 02 13:03:31.471352 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:03:31.470990    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-v9zqm" podUID="ef76ad04-45eb-4b3b-b314-e93122bb18f5" containerName="istio-proxy" containerID="cri-o://70e5ba11c272127890ba4f3d4cba2d3574441162cb27abe1498ce4870f5b0a4c" gracePeriod=30
Nov 02 13:03:31.643718 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:03:31.639264    4559 patch_prober.go:29] interesting pod/e2e-extmod-a-14fcb6-openshift-default-568ddb87b9-tzf8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused" start-of-body=
Nov 02 13:03:31.643718 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:03:31.639330    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-14fcb6-openshift-default-568ddb87b9-tzf8f" podUID="0029ba38-71d3-4212-9b50-ded1a42abd1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused"
Nov 02 13:03:31.964239 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:03:31.964205    4559 patch_prober.go:29] interesting pod/e2e-extmod-b-90cda9-openshift-default-57f8b6f957-b5jzx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused" start-of-body=
Nov 02 13:03:31.964433 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:03:31.964260    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-90cda9-openshift-default-57f8b6f957-b5jzx" podUID="bb3c761d-8641-496c-9d3a-531e8117e7dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused"
Nov 02 13:03:32.604666 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:03:32.603330    4559 patch_prober.go:29] interesting pod/e2e-extmod-a-14fcb6-openshift-default-568ddb87b9-tzf8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused" start-of-body=
Nov 02 13:03:32.604666 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:03:32.603391    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-14fcb6-openshift-default-568ddb87b9-tzf8f" podUID="0029ba38-71d3-4212-9b50-ded1a42abd1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused"
Nov 02 13:03:32.964157 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:03:32.964120    4559 patch_prober.go:29] interesting pod/e2e-extmod-b-90cda9-openshift-default-57f8b6f957-b5jzx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused" start-of-body=
Nov 02 13:03:32.964416 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:03:32.964378    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-90cda9-openshift-default-57f8b6f957-b5jzx" podUID="bb3c761d-8641-496c-9d3a-531e8117e7dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused"
Nov 02 13:03:33.480464 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:03:33.480433    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-90cda9-openshift-default-57f8b6f957-b5jzx"
Nov 02 13:03:33.617696 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:03:33.615778    4559 patch_prober.go:29] interesting pod/e2e-extmod-a-14fcb6-openshift-default-568ddb87b9-tzf8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused" start-of-body=
Nov 02 13:03:33.617696 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:03:33.615862    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-14fcb6-openshift-default-568ddb87b9-tzf8f" podUID="0029ba38-71d3-4212-9b50-ded1a42abd1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused"
Nov 02 13:03:33.964366 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:03:33.964333    4559 patch_prober.go:29] interesting pod/e2e-extmod-b-90cda9-openshift-default-57f8b6f957-b5jzx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused" start-of-body=
Nov 02 13:03:33.964541 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:03:33.964386    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-90cda9-openshift-default-57f8b6f957-b5jzx" podUID="bb3c761d-8641-496c-9d3a-531e8117e7dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused"
Nov 02 13:03:34.615666 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:03:34.615337    4559 patch_prober.go:29] interesting pod/e2e-extmod-a-14fcb6-openshift-default-568ddb87b9-tzf8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused" start-of-body=
Nov 02 13:03:34.615666 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:03:34.615404    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-14fcb6-openshift-default-568ddb87b9-tzf8f" podUID="0029ba38-71d3-4212-9b50-ded1a42abd1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused"
Nov 02 13:03:34.963605 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:03:34.963565    4559 patch_prober.go:29] interesting pod/e2e-extmod-b-90cda9-openshift-default-57f8b6f957-b5jzx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused" start-of-body=
Nov 02 13:03:34.964064 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:03:34.963634    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-90cda9-openshift-default-57f8b6f957-b5jzx" podUID="bb3c761d-8641-496c-9d3a-531e8117e7dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused"
Nov 02 13:03:35.605703 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:03:35.605673    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-14fcb6-openshift-default-568ddb87b9-tzf8f"
Nov 02 13:03:35.606163 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:03:35.606138    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-14fcb6-openshift-default-568ddb87b9-tzf8f"
Nov 02 13:03:35.606942 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:03:35.606923    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-14fcb6-openshift-default-568ddb87b9-tzf8f"
Nov 02 13:03:35.980672 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:03:35.974056    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-90cda9-openshift-default-57f8b6f957-b5jzx"
Nov 02 13:03:35.999662 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:03:35.996926    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-90cda9-openshift-default-57f8b6f957-b5jzx"
Nov 02 13:03:36.750569 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:03:36.750510    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-v9zqm"
Nov 02 13:03:36.872262 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:03:36.872228    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef76ad04-45eb-4b3b-b314-e93122bb18f5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef76ad04-45eb-4b3b-b314-e93122bb18f5-istio-envoy\") pod \"ef76ad04-45eb-4b3b-b314-e93122bb18f5\" (UID: \"ef76ad04-45eb-4b3b-b314-e93122bb18f5\") "
Nov 02 13:03:36.872449 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:03:36.872291    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ef76ad04-45eb-4b3b-b314-e93122bb18f5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef76ad04-45eb-4b3b-b314-e93122bb18f5-istio-podinfo\") pod \"ef76ad04-45eb-4b3b-b314-e93122bb18f5\" (UID: \"ef76ad04-45eb-4b3b-b314-e93122bb18f5\") "
Nov 02 13:03:36.872449 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:03:36.872345    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef76ad04-45eb-4b3b-b314-e93122bb18f5-kube-api-access-mlszr\" (UniqueName: \"kubernetes.io/projected/ef76ad04-45eb-4b3b-b314-e93122bb18f5-kube-api-access-mlszr\") pod \"ef76ad04-45eb-4b3b-b314-e93122bb18f5\" (UID: \"ef76ad04-45eb-4b3b-b314-e93122bb18f5\") "
Nov 02 13:03:36.872449 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:03:36.872372    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef76ad04-45eb-4b3b-b314-e93122bb18f5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef76ad04-45eb-4b3b-b314-e93122bb18f5-workload-certs\") pod \"ef76ad04-45eb-4b3b-b314-e93122bb18f5\" (UID: \"ef76ad04-45eb-4b3b-b314-e93122bb18f5\") "
Nov 02 13:03:36.872449 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:03:36.872413    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef76ad04-45eb-4b3b-b314-e93122bb18f5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef76ad04-45eb-4b3b-b314-e93122bb18f5-istio-data\") pod \"ef76ad04-45eb-4b3b-b314-e93122bb18f5\" (UID: \"ef76ad04-45eb-4b3b-b314-e93122bb18f5\") "
Nov 02 13:03:36.872747 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:03:36.872520    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef76ad04-45eb-4b3b-b314-e93122bb18f5-istio-token\" (UniqueName: \"kubernetes.io/projected/ef76ad04-45eb-4b3b-b314-e93122bb18f5-istio-token\") pod \"ef76ad04-45eb-4b3b-b314-e93122bb18f5\" (UID: \"ef76ad04-45eb-4b3b-b314-e93122bb18f5\") "
Nov 02 13:03:36.872747 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:03:36.872588    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef76ad04-45eb-4b3b-b314-e93122bb18f5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef76ad04-45eb-4b3b-b314-e93122bb18f5-credential-socket\") pod \"ef76ad04-45eb-4b3b-b314-e93122bb18f5\" (UID: \"ef76ad04-45eb-4b3b-b314-e93122bb18f5\") "
Nov 02 13:03:36.872884 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:03:36.872727    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef76ad04-45eb-4b3b-b314-e93122bb18f5-workload-certs" pod "ef76ad04-45eb-4b3b-b314-e93122bb18f5" (UID: "ef76ad04-45eb-4b3b-b314-e93122bb18f5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:03:36.872884 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:03:36.872782    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef76ad04-45eb-4b3b-b314-e93122bb18f5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef76ad04-45eb-4b3b-b314-e93122bb18f5-istiod-ca-cert\") pod \"ef76ad04-45eb-4b3b-b314-e93122bb18f5\" (UID: \"ef76ad04-45eb-4b3b-b314-e93122bb18f5\") "
Nov 02 13:03:36.872884 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:03:36.872803    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef76ad04-45eb-4b3b-b314-e93122bb18f5-istio-data" pod "ef76ad04-45eb-4b3b-b314-e93122bb18f5" (UID: "ef76ad04-45eb-4b3b-b314-e93122bb18f5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:03:36.873063 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:03:36.872922    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef76ad04-45eb-4b3b-b314-e93122bb18f5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef76ad04-45eb-4b3b-b314-e93122bb18f5-workload-socket\") pod \"ef76ad04-45eb-4b3b-b314-e93122bb18f5\" (UID: \"ef76ad04-45eb-4b3b-b314-e93122bb18f5\") "
Nov 02 13:03:36.873063 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:03:36.872975    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef76ad04-45eb-4b3b-b314-e93122bb18f5-credential-socket" pod "ef76ad04-45eb-4b3b-b314-e93122bb18f5" (UID: "ef76ad04-45eb-4b3b-b314-e93122bb18f5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:03:36.873310 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:03:36.873287    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef76ad04-45eb-4b3b-b314-e93122bb18f5-workload-socket" pod "ef76ad04-45eb-4b3b-b314-e93122bb18f5" (UID: "ef76ad04-45eb-4b3b-b314-e93122bb18f5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:03:36.873407 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:03:36.873225    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef76ad04-45eb-4b3b-b314-e93122bb18f5-istiod-ca-cert" pod "ef76ad04-45eb-4b3b-b314-e93122bb18f5" (UID: "ef76ad04-45eb-4b3b-b314-e93122bb18f5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:03:36.873634 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:03:36.873602    4559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef76ad04-45eb-4b3b-b314-e93122bb18f5-workload-socket\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:03:36.873758 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:03:36.873673    4559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef76ad04-45eb-4b3b-b314-e93122bb18f5-workload-certs\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:03:36.873758 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:03:36.873691    4559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef76ad04-45eb-4b3b-b314-e93122bb18f5-istio-data\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:03:36.873758 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:03:36.873705    4559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef76ad04-45eb-4b3b-b314-e93122bb18f5-credential-socket\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:03:36.873758 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:03:36.873719    4559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef76ad04-45eb-4b3b-b314-e93122bb18f5-istiod-ca-cert\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:03:36.874575 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:03:36.874553    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ef76ad04-45eb-4b3b-b314-e93122bb18f5-istio-podinfo" pod "ef76ad04-45eb-4b3b-b314-e93122bb18f5" (UID: "ef76ad04-45eb-4b3b-b314-e93122bb18f5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 13:03:36.874838 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:03:36.874817    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef76ad04-45eb-4b3b-b314-e93122bb18f5-istio-envoy" pod "ef76ad04-45eb-4b3b-b314-e93122bb18f5" (UID: "ef76ad04-45eb-4b3b-b314-e93122bb18f5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:03:36.874936 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:03:36.874854    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef76ad04-45eb-4b3b-b314-e93122bb18f5-istio-token" pod "ef76ad04-45eb-4b3b-b314-e93122bb18f5" (UID: "ef76ad04-45eb-4b3b-b314-e93122bb18f5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:03:36.874936 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:03:36.874879    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef76ad04-45eb-4b3b-b314-e93122bb18f5-kube-api-access-mlszr" pod "ef76ad04-45eb-4b3b-b314-e93122bb18f5" (UID: "ef76ad04-45eb-4b3b-b314-e93122bb18f5"). InnerVolumeSpecName "kube-api-access-mlszr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:03:36.975305 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:03:36.975274    4559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef76ad04-45eb-4b3b-b314-e93122bb18f5-istio-envoy\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:03:36.975305 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:03:36.975303    4559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef76ad04-45eb-4b3b-b314-e93122bb18f5-istio-podinfo\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:03:36.975305 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:03:36.975312    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mlszr\" (UniqueName: \"kubernetes.io/projected/ef76ad04-45eb-4b3b-b314-e93122bb18f5-kube-api-access-mlszr\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:03:36.975692 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:03:36.975325    4559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef76ad04-45eb-4b3b-b314-e93122bb18f5-istio-token\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:03:37.581239 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:03:37.581202    4559 generic.go:356] "Generic (PLEG): container finished" podID="ef76ad04-45eb-4b3b-b314-e93122bb18f5" containerID="70e5ba11c272127890ba4f3d4cba2d3574441162cb27abe1498ce4870f5b0a4c" exitCode=0
Nov 02 13:03:37.581796 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:03:37.581771    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-v9zqm"
Nov 02 13:03:37.582273 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:03:37.582249    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-v9zqm" event={"ID":"ef76ad04-45eb-4b3b-b314-e93122bb18f5","Type":"ContainerDied","Data":"70e5ba11c272127890ba4f3d4cba2d3574441162cb27abe1498ce4870f5b0a4c"}
Nov 02 13:03:37.582360 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:03:37.582305    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-v9zqm" event={"ID":"ef76ad04-45eb-4b3b-b314-e93122bb18f5","Type":"ContainerDied","Data":"ade342c7d53f0fa54032a2fc71a2d3f0234e84568b94fe8f5177bacd0dee65eb"}
Nov 02 13:03:37.582360 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:03:37.582337    4559 scope.go:122] "RemoveContainer" containerID="70e5ba11c272127890ba4f3d4cba2d3574441162cb27abe1498ce4870f5b0a4c"
Nov 02 13:03:37.594028 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:03:37.594008    4559 scope.go:122] "RemoveContainer" containerID="70e5ba11c272127890ba4f3d4cba2d3574441162cb27abe1498ce4870f5b0a4c"
Nov 02 13:03:37.594348 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:03:37.594319    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70e5ba11c272127890ba4f3d4cba2d3574441162cb27abe1498ce4870f5b0a4c\": container with ID starting with 70e5ba11c272127890ba4f3d4cba2d3574441162cb27abe1498ce4870f5b0a4c not found: ID does not exist" containerID="70e5ba11c272127890ba4f3d4cba2d3574441162cb27abe1498ce4870f5b0a4c"
Nov 02 13:03:37.594443 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:03:37.594353    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70e5ba11c272127890ba4f3d4cba2d3574441162cb27abe1498ce4870f5b0a4c"} err="failed to get container status \"70e5ba11c272127890ba4f3d4cba2d3574441162cb27abe1498ce4870f5b0a4c\": rpc error: code = NotFound desc = could not find container \"70e5ba11c272127890ba4f3d4cba2d3574441162cb27abe1498ce4870f5b0a4c\": container with ID starting with 70e5ba11c272127890ba4f3d4cba2d3574441162cb27abe1498ce4870f5b0a4c not found: ID does not exist"
Nov 02 13:03:37.672691 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:03:37.667023    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-v9zqm"]
Nov 02 13:03:37.672691 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:03:37.670734    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-v9zqm"]
Nov 02 13:03:37.894279 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:03:37.894196    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef76ad04-45eb-4b3b-b314-e93122bb18f5" path="/var/lib/kubelet/pods/ef76ad04-45eb-4b3b-b314-e93122bb18f5/volumes"
Nov 02 13:03:45.088833 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:03:45.088681    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqcmj" containerName="extractor"
Nov 02 13:03:49.332908 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:03:49.332869    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"]
Nov 02 13:04:07.895514 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:04:07.895478    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-936529-d8d95fdc9-62p9f"]
Nov 02 13:04:07.896255 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:04:07.896233    4559 cpu_manager.go:414] "containerMap: removing container" podUID="ef76ad04-45eb-4b3b-b314-e93122bb18f5" containerName="istio-proxy"
Nov 02 13:04:07.896304 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:04:07.896257    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef76ad04-45eb-4b3b-b314-e93122bb18f5" containerName="istio-proxy"
Nov 02 13:04:07.896360 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:04:07.896351    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef76ad04-45eb-4b3b-b314-e93122bb18f5" containerName="istio-proxy"
Nov 02 13:04:07.901255 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:04:07.901236    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-936529-d8d95fdc9-62p9f"
Nov 02 13:04:07.915247 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:04:07.915219    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-936529-d8d95fdc9-62p9f"]
Nov 02 13:04:07.926691 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:04:07.922626    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-936529-serving-cert\""
Nov 02 13:04:08.030092 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:04:08.030052    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70323c42-ec34-40c6-a906-d646b3025523-metrics-certs\") pod \"maas-api-e2e-extmod-b-936529-d8d95fdc9-62p9f\" (UID: \"70323c42-ec34-40c6-a906-d646b3025523\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-936529-d8d95fdc9-62p9f"
Nov 02 13:04:08.030276 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:04:08.030122    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70323c42-ec34-40c6-a906-d646b3025523-maas-api-tls\") pod \"maas-api-e2e-extmod-b-936529-d8d95fdc9-62p9f\" (UID: \"70323c42-ec34-40c6-a906-d646b3025523\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-936529-d8d95fdc9-62p9f"
Nov 02 13:04:08.030276 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:04:08.030252    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/70323c42-ec34-40c6-a906-d646b3025523-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-936529-d8d95fdc9-62p9f\" (UID: \"70323c42-ec34-40c6-a906-d646b3025523\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-936529-d8d95fdc9-62p9f"
Nov 02 13:04:08.030353 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:04:08.030293    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fg9f8\" (UniqueName: \"kubernetes.io/projected/70323c42-ec34-40c6-a906-d646b3025523-kube-api-access-fg9f8\") pod \"maas-api-e2e-extmod-b-936529-d8d95fdc9-62p9f\" (UID: \"70323c42-ec34-40c6-a906-d646b3025523\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-936529-d8d95fdc9-62p9f"
Nov 02 13:04:08.131294 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:04:08.131262    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/70323c42-ec34-40c6-a906-d646b3025523-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-936529-d8d95fdc9-62p9f\" (UID: \"70323c42-ec34-40c6-a906-d646b3025523\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-936529-d8d95fdc9-62p9f"
Nov 02 13:04:08.131487 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:04:08.131306    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fg9f8\" (UniqueName: \"kubernetes.io/projected/70323c42-ec34-40c6-a906-d646b3025523-kube-api-access-fg9f8\") pod \"maas-api-e2e-extmod-b-936529-d8d95fdc9-62p9f\" (UID: \"70323c42-ec34-40c6-a906-d646b3025523\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-936529-d8d95fdc9-62p9f"
Nov 02 13:04:08.131487 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:04:08.131473    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70323c42-ec34-40c6-a906-d646b3025523-metrics-certs\") pod \"maas-api-e2e-extmod-b-936529-d8d95fdc9-62p9f\" (UID: \"70323c42-ec34-40c6-a906-d646b3025523\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-936529-d8d95fdc9-62p9f"
Nov 02 13:04:08.131600 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:04:08.131505    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70323c42-ec34-40c6-a906-d646b3025523-maas-api-tls\") pod \"maas-api-e2e-extmod-b-936529-d8d95fdc9-62p9f\" (UID: \"70323c42-ec34-40c6-a906-d646b3025523\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-936529-d8d95fdc9-62p9f"
Nov 02 13:04:08.132262 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:04:08.132228    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/70323c42-ec34-40c6-a906-d646b3025523-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-936529-d8d95fdc9-62p9f\" (UID: \"70323c42-ec34-40c6-a906-d646b3025523\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-936529-d8d95fdc9-62p9f"
Nov 02 13:04:08.134179 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:04:08.134155    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70323c42-ec34-40c6-a906-d646b3025523-maas-api-tls\") pod \"maas-api-e2e-extmod-b-936529-d8d95fdc9-62p9f\" (UID: \"70323c42-ec34-40c6-a906-d646b3025523\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-936529-d8d95fdc9-62p9f"
Nov 02 13:04:08.134266 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:04:08.134176    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70323c42-ec34-40c6-a906-d646b3025523-metrics-certs\") pod \"maas-api-e2e-extmod-b-936529-d8d95fdc9-62p9f\" (UID: \"70323c42-ec34-40c6-a906-d646b3025523\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-936529-d8d95fdc9-62p9f"
Nov 02 13:04:08.181744 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:04:08.178569    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fg9f8\" (UniqueName: \"kubernetes.io/projected/70323c42-ec34-40c6-a906-d646b3025523-kube-api-access-fg9f8\") pod \"maas-api-e2e-extmod-b-936529-d8d95fdc9-62p9f\" (UID: \"70323c42-ec34-40c6-a906-d646b3025523\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-936529-d8d95fdc9-62p9f"
Nov 02 13:04:08.224663 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:04:08.223022    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-936529-d8d95fdc9-62p9f"
Nov 02 13:04:08.531992 ip-10-0-128-132 kubenswrapper[4559]: W1102 13:04:08.531965    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70323c42_ec34_40c6_a906_d646b3025523.slice/crio-e7b67b328b1771abbe7bf3a745ad4cebdb057923bb87e99b68137b7ba8dff7d9 WatchSource:0}: Error finding container e7b67b328b1771abbe7bf3a745ad4cebdb057923bb87e99b68137b7ba8dff7d9: Status 404 returned error can't find the container with id e7b67b328b1771abbe7bf3a745ad4cebdb057923bb87e99b68137b7ba8dff7d9
Nov 02 13:04:08.533454 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:04:08.533431    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-936529-d8d95fdc9-62p9f"]
Nov 02 13:04:08.898085 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:04:08.898000    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-936529-d8d95fdc9-62p9f" event={"ID":"70323c42-ec34-40c6-a906-d646b3025523","Type":"ContainerStarted","Data":"e7b67b328b1771abbe7bf3a745ad4cebdb057923bb87e99b68137b7ba8dff7d9"}
Nov 02 13:04:09.925261 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:04:09.925226    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-936529-d8d95fdc9-62p9f"
Nov 02 13:04:09.925725 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:04:09.925278    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-936529-d8d95fdc9-62p9f" event={"ID":"70323c42-ec34-40c6-a906-d646b3025523","Type":"ContainerStarted","Data":"ecab3a9a4db6830de1e1e398be6871fde6add381f0857429ee57626c40979cfc"}
Nov 02 13:04:09.965598 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:04:09.965549    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-936529-d8d95fdc9-62p9f" podStartSLOduration=2.3996122570000002 podStartE2EDuration="2.965530523s" totalImagesPullingTime="565.918266ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:04:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:04:09.964544309 +0000 UTC m=+2246.219943404" watchObservedRunningTime="2026-11-02 13:04:09.965530523 +0000 UTC m=+2246.220929617"
Nov 02 13:04:15.089672 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:04:15.089613    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqcmj" containerName="extractor"
Nov 02 13:04:15.934750 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:04:15.934719    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-936529-d8d95fdc9-62p9f"
Nov 02 13:04:21.047666 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:04:21.032437    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"]
Nov 02 13:04:45.088382 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:04:45.088338    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqcmj" containerName="extractor"
Nov 02 13:04:51.978667 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:04:51.978035    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"]
Nov 02 13:05:01.507788 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:05:01.507746    4559 patch_prober.go:29] interesting pod/e2e-extmod-b-936529-openshift-default-5578bccc97-9cw4k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.220:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 13:05:01.508343 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:05:01.507856    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-936529-openshift-default-5578bccc97-9cw4k" podUID="6ae3594b-0d9b-431e-b165-bb290002858a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.220:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:05:02.463854 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:05:02.463821    4559 generic.go:356] "Generic (PLEG): container finished" podID="6ae3594b-0d9b-431e-b165-bb290002858a" containerID="9558dea98056df34d056ae3419f34b5c7a53a5c076914de1f956a03cf8c76cf4" exitCode=0
Nov 02 13:05:02.464058 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:05:02.463896    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-936529-openshift-default-5578bccc97-9cw4k" event={"ID":"6ae3594b-0d9b-431e-b165-bb290002858a","Type":"ContainerDied","Data":"9558dea98056df34d056ae3419f34b5c7a53a5c076914de1f956a03cf8c76cf4"}
Nov 02 13:05:02.464285 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:05:02.464270    4559 scope.go:122] "RemoveContainer" containerID="9558dea98056df34d056ae3419f34b5c7a53a5c076914de1f956a03cf8c76cf4"
Nov 02 13:05:02.466081 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:05:02.466051    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 13:05:02.466149 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:05:02.466126    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 13:05:02.466186 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:05:02.466158    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 13:05:03.473953 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:05:03.473921    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-936529-openshift-default-5578bccc97-9cw4k" event={"ID":"6ae3594b-0d9b-431e-b165-bb290002858a","Type":"ContainerStarted","Data":"ce6a7a7afb91e946a884853ec8f96119491db4ef21f11aeb621a8d0189d32f66"}
Nov 02 13:05:03.614210 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:05:03.614172    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-936529-openshift-default-5578bccc97-9cw4k"
Nov 02 13:05:03.614456 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:05:03.614429    4559 patch_prober.go:29] interesting pod/e2e-extmod-b-936529-openshift-default-5578bccc97-9cw4k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.220:15021/healthz/ready\": dial tcp 10.133.0.220:15021: connect: connection refused" start-of-body=
Nov 02 13:05:03.614547 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:05:03.614491    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-936529-openshift-default-5578bccc97-9cw4k" podUID="6ae3594b-0d9b-431e-b165-bb290002858a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.220:15021/healthz/ready\": dial tcp 10.133.0.220:15021: connect: connection refused"
Nov 02 13:05:04.614094 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:05:04.614061    4559 patch_prober.go:29] interesting pod/e2e-extmod-b-936529-openshift-default-5578bccc97-9cw4k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.220:15021/healthz/ready\": dial tcp 10.133.0.220:15021: connect: connection refused" start-of-body=
Nov 02 13:05:04.614512 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:05:04.614113    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-936529-openshift-default-5578bccc97-9cw4k" podUID="6ae3594b-0d9b-431e-b165-bb290002858a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.220:15021/healthz/ready\": dial tcp 10.133.0.220:15021: connect: connection refused"
Nov 02 13:05:05.614170 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:05:05.614130    4559 patch_prober.go:29] interesting pod/e2e-extmod-b-936529-openshift-default-5578bccc97-9cw4k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.220:15021/healthz/ready\": dial tcp 10.133.0.220:15021: connect: connection refused" start-of-body=
Nov 02 13:05:05.614587 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:05:05.614198    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-936529-openshift-default-5578bccc97-9cw4k" podUID="6ae3594b-0d9b-431e-b165-bb290002858a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.220:15021/healthz/ready\": dial tcp 10.133.0.220:15021: connect: connection refused"
Nov 02 13:05:06.614035 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:05:06.614001    4559 patch_prober.go:29] interesting pod/e2e-extmod-b-936529-openshift-default-5578bccc97-9cw4k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.220:15021/healthz/ready\": dial tcp 10.133.0.220:15021: connect: connection refused" start-of-body=
Nov 02 13:05:06.614215 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:05:06.614054    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-936529-openshift-default-5578bccc97-9cw4k" podUID="6ae3594b-0d9b-431e-b165-bb290002858a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.220:15021/healthz/ready\": dial tcp 10.133.0.220:15021: connect: connection refused"
Nov 02 13:05:07.614144 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:05:07.614106    4559 patch_prober.go:29] interesting pod/e2e-extmod-b-936529-openshift-default-5578bccc97-9cw4k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.220:15021/healthz/ready\": dial tcp 10.133.0.220:15021: connect: connection refused" start-of-body=
Nov 02 13:05:07.614334 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:05:07.614164    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-936529-openshift-default-5578bccc97-9cw4k" podUID="6ae3594b-0d9b-431e-b165-bb290002858a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.220:15021/healthz/ready\": dial tcp 10.133.0.220:15021: connect: connection refused"
Nov 02 13:05:08.614089 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:05:08.614045    4559 patch_prober.go:29] interesting pod/e2e-extmod-b-936529-openshift-default-5578bccc97-9cw4k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.220:15021/healthz/ready\": dial tcp 10.133.0.220:15021: connect: connection refused" start-of-body=
Nov 02 13:05:08.614390 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:05:08.614360    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-936529-openshift-default-5578bccc97-9cw4k" podUID="6ae3594b-0d9b-431e-b165-bb290002858a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.220:15021/healthz/ready\": dial tcp 10.133.0.220:15021: connect: connection refused"
Nov 02 13:05:09.618403 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:05:09.618371    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-936529-openshift-default-5578bccc97-9cw4k"
Nov 02 13:05:09.618888 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:05:09.618866    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-936529-openshift-default-5578bccc97-9cw4k"
Nov 02 13:05:09.619588 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:05:09.619569    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-936529-openshift-default-5578bccc97-9cw4k"
Nov 02 13:05:15.087702 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:05:15.087629    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqcmj" containerName="extractor"
Nov 02 13:05:39.803342 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:05:39.803305    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"]
Nov 02 13:05:45.243664 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:05:45.225687    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqcmj" containerName="extractor"
Nov 02 13:06:15.087676 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:06:15.087624    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqcmj" containerName="extractor"
Nov 02 13:06:35.265594 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:06:35.265498    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"]
Nov 02 13:06:45.089655 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:06:45.089599    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqcmj" containerName="extractor"
Nov 02 13:07:15.087606 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:07:15.087564    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqcmj" containerName="extractor"
Nov 02 13:07:37.253345 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:07:37.253313    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"]
Nov 02 13:07:45.089769 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:07:45.089723    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqcmj" containerName="extractor"
Nov 02 13:07:46.560432 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:07:46.560400    4559 scope.go:122] "RemoveContainer" containerID="53b372e8a46893c77498b89edcd4add5a74c5700156e921a0359889920aad8e1"
Nov 02 13:08:15.086724 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:08:15.086630    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqcmj" containerName="extractor"
Nov 02 13:08:23.509746 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:08:23.509704    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"]
Nov 02 13:08:45.087197 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:08:45.087158    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqcmj" containerName="extractor"
Nov 02 13:09:15.091089 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:09:15.091048    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqcmj" containerName="extractor"
Nov 02 13:09:27.163283 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:09:27.163086    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-xj4xk"]
Nov 02 13:09:27.185999 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:09:27.185949    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-xj4xk"
Nov 02 13:09:27.195657 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:09:27.194447    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-xj4xk"]
Nov 02 13:09:27.315539 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:09:27.315496    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2c5f43a-3a0c-444b-afc1-e2bf090dc05c-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-xj4xk\" (UID: \"d2c5f43a-3a0c-444b-afc1-e2bf090dc05c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-xj4xk"
Nov 02 13:09:27.315769 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:09:27.315555    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2c5f43a-3a0c-444b-afc1-e2bf090dc05c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-xj4xk\" (UID: \"d2c5f43a-3a0c-444b-afc1-e2bf090dc05c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-xj4xk"
Nov 02 13:09:27.315769 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:09:27.315607    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2c5f43a-3a0c-444b-afc1-e2bf090dc05c-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-xj4xk\" (UID: \"d2c5f43a-3a0c-444b-afc1-e2bf090dc05c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-xj4xk"
Nov 02 13:09:27.315769 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:09:27.315624    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2c5f43a-3a0c-444b-afc1-e2bf090dc05c-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-xj4xk\" (UID: \"d2c5f43a-3a0c-444b-afc1-e2bf090dc05c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-xj4xk"
Nov 02 13:09:27.316023 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:09:27.315898    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znb9l\" (UniqueName: \"kubernetes.io/projected/d2c5f43a-3a0c-444b-afc1-e2bf090dc05c-kube-api-access-znb9l\") pod \"maas-default-gateway-openshift-default-5bc884dd87-xj4xk\" (UID: \"d2c5f43a-3a0c-444b-afc1-e2bf090dc05c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-xj4xk"
Nov 02 13:09:27.316023 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:09:27.316007    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2c5f43a-3a0c-444b-afc1-e2bf090dc05c-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-xj4xk\" (UID: \"d2c5f43a-3a0c-444b-afc1-e2bf090dc05c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-xj4xk"
Nov 02 13:09:27.316139 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:09:27.316059    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2c5f43a-3a0c-444b-afc1-e2bf090dc05c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-xj4xk\" (UID: \"d2c5f43a-3a0c-444b-afc1-e2bf090dc05c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-xj4xk"
Nov 02 13:09:27.316199 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:09:27.316167    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2c5f43a-3a0c-444b-afc1-e2bf090dc05c-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-xj4xk\" (UID: \"d2c5f43a-3a0c-444b-afc1-e2bf090dc05c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-xj4xk"
Nov 02 13:09:27.316253 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:09:27.316215    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2c5f43a-3a0c-444b-afc1-e2bf090dc05c-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-xj4xk\" (UID: \"d2c5f43a-3a0c-444b-afc1-e2bf090dc05c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-xj4xk"
Nov 02 13:09:27.417286 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:09:27.417260    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2c5f43a-3a0c-444b-afc1-e2bf090dc05c-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-xj4xk\" (UID: \"d2c5f43a-3a0c-444b-afc1-e2bf090dc05c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-xj4xk"
Nov 02 13:09:27.417450 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:09:27.417302    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2c5f43a-3a0c-444b-afc1-e2bf090dc05c-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-xj4xk\" (UID: \"d2c5f43a-3a0c-444b-afc1-e2bf090dc05c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-xj4xk"
Nov 02 13:09:27.417450 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:09:27.417339    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2c5f43a-3a0c-444b-afc1-e2bf090dc05c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-xj4xk\" (UID: \"d2c5f43a-3a0c-444b-afc1-e2bf090dc05c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-xj4xk"
Nov 02 13:09:27.417450 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:09:27.417360    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2c5f43a-3a0c-444b-afc1-e2bf090dc05c-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-xj4xk\" (UID: \"d2c5f43a-3a0c-444b-afc1-e2bf090dc05c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-xj4xk"
Nov 02 13:09:27.417450 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:09:27.417377    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2c5f43a-3a0c-444b-afc1-e2bf090dc05c-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-xj4xk\" (UID: \"d2c5f43a-3a0c-444b-afc1-e2bf090dc05c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-xj4xk"
Nov 02 13:09:27.417751 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:09:27.417507    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znb9l\" (UniqueName: \"kubernetes.io/projected/d2c5f43a-3a0c-444b-afc1-e2bf090dc05c-kube-api-access-znb9l\") pod \"maas-default-gateway-openshift-default-5bc884dd87-xj4xk\" (UID: \"d2c5f43a-3a0c-444b-afc1-e2bf090dc05c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-xj4xk"
Nov 02 13:09:27.417751 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:09:27.417573    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2c5f43a-3a0c-444b-afc1-e2bf090dc05c-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-xj4xk\" (UID: \"d2c5f43a-3a0c-444b-afc1-e2bf090dc05c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-xj4xk"
Nov 02 13:09:27.417751 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:09:27.417615    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2c5f43a-3a0c-444b-afc1-e2bf090dc05c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-xj4xk\" (UID: \"d2c5f43a-3a0c-444b-afc1-e2bf090dc05c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-xj4xk"
Nov 02 13:09:27.417751 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:09:27.417708    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2c5f43a-3a0c-444b-afc1-e2bf090dc05c-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-xj4xk\" (UID: \"d2c5f43a-3a0c-444b-afc1-e2bf090dc05c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-xj4xk"
Nov 02 13:09:27.417963 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:09:27.417785    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2c5f43a-3a0c-444b-afc1-e2bf090dc05c-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-xj4xk\" (UID: \"d2c5f43a-3a0c-444b-afc1-e2bf090dc05c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-xj4xk"
Nov 02 13:09:27.417963 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:09:27.417811    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2c5f43a-3a0c-444b-afc1-e2bf090dc05c-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-xj4xk\" (UID: \"d2c5f43a-3a0c-444b-afc1-e2bf090dc05c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-xj4xk"
Nov 02 13:09:27.417963 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:09:27.417867    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2c5f43a-3a0c-444b-afc1-e2bf090dc05c-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-xj4xk\" (UID: \"d2c5f43a-3a0c-444b-afc1-e2bf090dc05c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-xj4xk"
Nov 02 13:09:27.418145 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:09:27.417938    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2c5f43a-3a0c-444b-afc1-e2bf090dc05c-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-xj4xk\" (UID: \"d2c5f43a-3a0c-444b-afc1-e2bf090dc05c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-xj4xk"
Nov 02 13:09:27.418329 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:09:27.418306    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2c5f43a-3a0c-444b-afc1-e2bf090dc05c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-xj4xk\" (UID: \"d2c5f43a-3a0c-444b-afc1-e2bf090dc05c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-xj4xk"
Nov 02 13:09:27.419891 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:09:27.419869    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2c5f43a-3a0c-444b-afc1-e2bf090dc05c-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-xj4xk\" (UID: \"d2c5f43a-3a0c-444b-afc1-e2bf090dc05c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-xj4xk"
Nov 02 13:09:27.420127 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:09:27.420109    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2c5f43a-3a0c-444b-afc1-e2bf090dc05c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-xj4xk\" (UID: \"d2c5f43a-3a0c-444b-afc1-e2bf090dc05c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-xj4xk"
Nov 02 13:09:27.451621 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:09:27.451592    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2c5f43a-3a0c-444b-afc1-e2bf090dc05c-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-xj4xk\" (UID: \"d2c5f43a-3a0c-444b-afc1-e2bf090dc05c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-xj4xk"
Nov 02 13:09:27.451805 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:09:27.451787    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-znb9l\" (UniqueName: \"kubernetes.io/projected/d2c5f43a-3a0c-444b-afc1-e2bf090dc05c-kube-api-access-znb9l\") pod \"maas-default-gateway-openshift-default-5bc884dd87-xj4xk\" (UID: \"d2c5f43a-3a0c-444b-afc1-e2bf090dc05c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-xj4xk"
Nov 02 13:09:27.508305 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:09:27.508273    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-xj4xk"
Nov 02 13:09:27.828725 ip-10-0-128-132 kubenswrapper[4559]: W1102 13:09:27.828692    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2c5f43a_3a0c_444b_afc1_e2bf090dc05c.slice/crio-534dda2fb3c1a0513a827ead60da78c811c71e9ebb60f1ede628359096d82bb2 WatchSource:0}: Error finding container 534dda2fb3c1a0513a827ead60da78c811c71e9ebb60f1ede628359096d82bb2: Status 404 returned error can't find the container with id 534dda2fb3c1a0513a827ead60da78c811c71e9ebb60f1ede628359096d82bb2
Nov 02 13:09:27.828970 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:09:27.828947    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-xj4xk"]
Nov 02 13:09:27.831449 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:09:27.831419    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 13:09:27.831529 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:09:27.831486    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 13:09:27.831529 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:09:27.831518    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 13:09:28.230687 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:09:28.230650    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-xj4xk" event={"ID":"d2c5f43a-3a0c-444b-afc1-e2bf090dc05c","Type":"ContainerStarted","Data":"96ed21c4883f0d9a981d09f6cefccbd271ccb36cf3d4a1146353b01d1680067a"}
Nov 02 13:09:28.231075 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:09:28.230704    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-xj4xk" event={"ID":"d2c5f43a-3a0c-444b-afc1-e2bf090dc05c","Type":"ContainerStarted","Data":"534dda2fb3c1a0513a827ead60da78c811c71e9ebb60f1ede628359096d82bb2"}
Nov 02 13:09:28.508606 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:09:28.508524    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-xj4xk"
Nov 02 13:09:28.510207 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:09:28.510178    4559 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-xj4xk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" start-of-body=
Nov 02 13:09:28.510334 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:09:28.510224    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-xj4xk" podUID="d2c5f43a-3a0c-444b-afc1-e2bf090dc05c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused"
Nov 02 13:09:29.509774 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:09:29.509696    4559 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-xj4xk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" start-of-body=
Nov 02 13:09:29.510472 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:09:29.510438    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-xj4xk" podUID="d2c5f43a-3a0c-444b-afc1-e2bf090dc05c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused"
Nov 02 13:09:30.509822 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:09:30.509782    4559 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-xj4xk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" start-of-body=
Nov 02 13:09:30.510448 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:09:30.510413    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-xj4xk" podUID="d2c5f43a-3a0c-444b-afc1-e2bf090dc05c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused"
Nov 02 13:09:31.509152 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:09:31.509112    4559 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-xj4xk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" start-of-body=
Nov 02 13:09:31.509342 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:09:31.509184    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-xj4xk" podUID="d2c5f43a-3a0c-444b-afc1-e2bf090dc05c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused"
Nov 02 13:09:32.515663 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:09:32.512771    4559 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-xj4xk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" start-of-body=
Nov 02 13:09:32.515663 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:09:32.512843    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-xj4xk" podUID="d2c5f43a-3a0c-444b-afc1-e2bf090dc05c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused"
Nov 02 13:09:33.113057 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:09:33.110994    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-xj4xk" podStartSLOduration=6.110969925 podStartE2EDuration="6.110969925s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:09:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:09:28.298768845 +0000 UTC m=+2564.554167940" watchObservedRunningTime="2026-11-02 13:09:33.110969925 +0000 UTC m=+2569.366369021"
Nov 02 13:09:33.113057 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:09:33.112895    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"]
Nov 02 13:09:33.509168 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:09:33.509123    4559 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-xj4xk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" start-of-body=
Nov 02 13:09:33.509367 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:09:33.509211    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-xj4xk" podUID="d2c5f43a-3a0c-444b-afc1-e2bf090dc05c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused"
Nov 02 13:09:34.510664 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:09:34.509899    4559 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-xj4xk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" start-of-body=
Nov 02 13:09:34.510664 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:09:34.510024    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-xj4xk" podUID="d2c5f43a-3a0c-444b-afc1-e2bf090dc05c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused"
Nov 02 13:09:35.509594 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:09:35.509541    4559 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-xj4xk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" start-of-body=
Nov 02 13:09:35.509846 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:09:35.509604    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-xj4xk" podUID="d2c5f43a-3a0c-444b-afc1-e2bf090dc05c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused"
Nov 02 13:09:36.509020 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:09:36.508976    4559 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-xj4xk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" start-of-body=
Nov 02 13:09:36.509540 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:09:36.509060    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-xj4xk" podUID="d2c5f43a-3a0c-444b-afc1-e2bf090dc05c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused"
Nov 02 13:09:38.304483 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:09:38.304447    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-xj4xk"
Nov 02 13:09:38.305363 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:09:38.305326    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-xj4xk"
Nov 02 13:09:38.308718 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:09:38.308689    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-xj4xk"
Nov 02 13:09:42.228122 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:09:42.228087    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ckslt"]
Nov 02 13:09:42.232465 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:09:42.232443    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ckslt"
Nov 02 13:09:42.377658 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:09:42.340624    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/826b92f5-161c-4f6a-9055-d0c838c9ce09-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ckslt\" (UID: \"826b92f5-161c-4f6a-9055-d0c838c9ce09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ckslt"
Nov 02 13:09:42.378046 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:09:42.378001    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/826b92f5-161c-4f6a-9055-d0c838c9ce09-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ckslt\" (UID: \"826b92f5-161c-4f6a-9055-d0c838c9ce09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ckslt"
Nov 02 13:09:42.378368 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:09:42.378347    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/826b92f5-161c-4f6a-9055-d0c838c9ce09-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ckslt\" (UID: \"826b92f5-161c-4f6a-9055-d0c838c9ce09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ckslt"
Nov 02 13:09:42.402664 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:09:42.378528    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/826b92f5-161c-4f6a-9055-d0c838c9ce09-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ckslt\" (UID: \"826b92f5-161c-4f6a-9055-d0c838c9ce09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ckslt"
Nov 02 13:09:42.403059 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:09:42.403033    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/826b92f5-161c-4f6a-9055-d0c838c9ce09-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ckslt\" (UID: \"826b92f5-161c-4f6a-9055-d0c838c9ce09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ckslt"
Nov 02 13:09:42.403292 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:09:42.403257    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/826b92f5-161c-4f6a-9055-d0c838c9ce09-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ckslt\" (UID: \"826b92f5-161c-4f6a-9055-d0c838c9ce09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ckslt"
Nov 02 13:09:42.403564 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:09:42.403548    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/826b92f5-161c-4f6a-9055-d0c838c9ce09-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ckslt\" (UID: \"826b92f5-161c-4f6a-9055-d0c838c9ce09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ckslt"
Nov 02 13:09:42.404036 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:09:42.404015    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/826b92f5-161c-4f6a-9055-d0c838c9ce09-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ckslt\" (UID: \"826b92f5-161c-4f6a-9055-d0c838c9ce09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ckslt"
Nov 02 13:09:42.410793 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:09:42.410713    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dq79r\" (UniqueName: \"kubernetes.io/projected/826b92f5-161c-4f6a-9055-d0c838c9ce09-kube-api-access-dq79r\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ckslt\" (UID: \"826b92f5-161c-4f6a-9055-d0c838c9ce09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ckslt"
Nov 02 13:09:42.461740 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:09:42.461695    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ckslt"]
Nov 02 13:09:42.523729 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:09:42.519735    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/826b92f5-161c-4f6a-9055-d0c838c9ce09-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ckslt\" (UID: \"826b92f5-161c-4f6a-9055-d0c838c9ce09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ckslt"
Nov 02 13:09:42.523729 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:09:42.519789    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dq79r\" (UniqueName: \"kubernetes.io/projected/826b92f5-161c-4f6a-9055-d0c838c9ce09-kube-api-access-dq79r\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ckslt\" (UID: \"826b92f5-161c-4f6a-9055-d0c838c9ce09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ckslt"
Nov 02 13:09:42.523729 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:09:42.519844    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/826b92f5-161c-4f6a-9055-d0c838c9ce09-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ckslt\" (UID: \"826b92f5-161c-4f6a-9055-d0c838c9ce09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ckslt"
Nov 02 13:09:42.523729 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:09:42.519885    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/826b92f5-161c-4f6a-9055-d0c838c9ce09-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ckslt\" (UID: \"826b92f5-161c-4f6a-9055-d0c838c9ce09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ckslt"
Nov 02 13:09:42.523729 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:09:42.520000    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/826b92f5-161c-4f6a-9055-d0c838c9ce09-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ckslt\" (UID: \"826b92f5-161c-4f6a-9055-d0c838c9ce09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ckslt"
Nov 02 13:09:42.523729 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:09:42.520039    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/826b92f5-161c-4f6a-9055-d0c838c9ce09-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ckslt\" (UID: \"826b92f5-161c-4f6a-9055-d0c838c9ce09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ckslt"
Nov 02 13:09:42.523729 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:09:42.520064    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/826b92f5-161c-4f6a-9055-d0c838c9ce09-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ckslt\" (UID: \"826b92f5-161c-4f6a-9055-d0c838c9ce09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ckslt"
Nov 02 13:09:42.523729 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:09:42.520112    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/826b92f5-161c-4f6a-9055-d0c838c9ce09-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ckslt\" (UID: \"826b92f5-161c-4f6a-9055-d0c838c9ce09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ckslt"
Nov 02 13:09:42.523729 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:09:42.520204    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/826b92f5-161c-4f6a-9055-d0c838c9ce09-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ckslt\" (UID: \"826b92f5-161c-4f6a-9055-d0c838c9ce09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ckslt"
Nov 02 13:09:42.523729 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:09:42.520733    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/826b92f5-161c-4f6a-9055-d0c838c9ce09-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ckslt\" (UID: \"826b92f5-161c-4f6a-9055-d0c838c9ce09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ckslt"
Nov 02 13:09:42.523729 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:09:42.521421    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/826b92f5-161c-4f6a-9055-d0c838c9ce09-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ckslt\" (UID: \"826b92f5-161c-4f6a-9055-d0c838c9ce09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ckslt"
Nov 02 13:09:42.523729 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:09:42.522030    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/826b92f5-161c-4f6a-9055-d0c838c9ce09-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ckslt\" (UID: \"826b92f5-161c-4f6a-9055-d0c838c9ce09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ckslt"
Nov 02 13:09:42.523729 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:09:42.522562    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/826b92f5-161c-4f6a-9055-d0c838c9ce09-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ckslt\" (UID: \"826b92f5-161c-4f6a-9055-d0c838c9ce09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ckslt"
Nov 02 13:09:42.529970 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:09:42.529944    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/826b92f5-161c-4f6a-9055-d0c838c9ce09-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ckslt\" (UID: \"826b92f5-161c-4f6a-9055-d0c838c9ce09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ckslt"
Nov 02 13:09:42.538788 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:09:42.538095    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/826b92f5-161c-4f6a-9055-d0c838c9ce09-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ckslt\" (UID: \"826b92f5-161c-4f6a-9055-d0c838c9ce09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ckslt"
Nov 02 13:09:42.539004 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:09:42.538447    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/826b92f5-161c-4f6a-9055-d0c838c9ce09-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ckslt\" (UID: \"826b92f5-161c-4f6a-9055-d0c838c9ce09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ckslt"
Nov 02 13:09:42.548200 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:09:42.548177    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dq79r\" (UniqueName: \"kubernetes.io/projected/826b92f5-161c-4f6a-9055-d0c838c9ce09-kube-api-access-dq79r\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ckslt\" (UID: \"826b92f5-161c-4f6a-9055-d0c838c9ce09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ckslt"
Nov 02 13:09:42.550150 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:09:42.550126    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/826b92f5-161c-4f6a-9055-d0c838c9ce09-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ckslt\" (UID: \"826b92f5-161c-4f6a-9055-d0c838c9ce09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ckslt"
Nov 02 13:09:42.623580 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:09:42.623549    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ckslt"
Nov 02 13:09:42.984014 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:09:42.983982    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ckslt"]
Nov 02 13:09:42.993857 ip-10-0-128-132 kubenswrapper[4559]: W1102 13:09:42.993831    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod826b92f5_161c_4f6a_9055_d0c838c9ce09.slice/crio-63d729291d2408508b974a1dfb81e28eb6361ef4cf776ba39a410e68f9d30fd3 WatchSource:0}: Error finding container 63d729291d2408508b974a1dfb81e28eb6361ef4cf776ba39a410e68f9d30fd3: Status 404 returned error can't find the container with id 63d729291d2408508b974a1dfb81e28eb6361ef4cf776ba39a410e68f9d30fd3
Nov 02 13:09:42.996489 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:09:42.996456    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 13:09:42.996560 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:09:42.996526    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 13:09:42.996599 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:09:42.996556    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 13:09:43.745666 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:09:43.742460    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ckslt" event={"ID":"826b92f5-161c-4f6a-9055-d0c838c9ce09","Type":"ContainerStarted","Data":"bd06e4fbfb52ae78f4d1d88b88740481e7fb5e29e35825a9512b1bec633649b4"}
Nov 02 13:09:43.745666 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:09:43.742589    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ckslt" event={"ID":"826b92f5-161c-4f6a-9055-d0c838c9ce09","Type":"ContainerStarted","Data":"63d729291d2408508b974a1dfb81e28eb6361ef4cf776ba39a410e68f9d30fd3"}
Nov 02 13:09:44.624214 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:09:44.624181    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ckslt"
Nov 02 13:09:44.625979 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:09:44.625948    4559 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-ckslt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused" start-of-body=
Nov 02 13:09:44.626118 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:09:44.625999    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ckslt" podUID="826b92f5-161c-4f6a-9055-d0c838c9ce09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused"
Nov 02 13:09:45.108438 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:09:45.108390    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqcmj" containerName="extractor"
Nov 02 13:09:45.624793 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:09:45.624763    4559 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-ckslt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused" start-of-body=
Nov 02 13:09:45.624985 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:09:45.624811    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ckslt" podUID="826b92f5-161c-4f6a-9055-d0c838c9ce09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused"
Nov 02 13:09:46.631850 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:09:46.631808    4559 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-ckslt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused" start-of-body=
Nov 02 13:09:46.632687 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:09:46.632593    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ckslt" podUID="826b92f5-161c-4f6a-9055-d0c838c9ce09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused"
Nov 02 13:09:47.625150 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:09:47.625114    4559 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-ckslt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused" start-of-body=
Nov 02 13:09:47.625339 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:09:47.625172    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ckslt" podUID="826b92f5-161c-4f6a-9055-d0c838c9ce09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused"
Nov 02 13:09:48.624210 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:09:48.624170    4559 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-ckslt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused" start-of-body=
Nov 02 13:09:48.624621 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:09:48.624246    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ckslt" podUID="826b92f5-161c-4f6a-9055-d0c838c9ce09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused"
Nov 02 13:09:49.623925 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:09:49.623892    4559 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-ckslt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused" start-of-body=
Nov 02 13:09:49.624120 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:09:49.623955    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ckslt" podUID="826b92f5-161c-4f6a-9055-d0c838c9ce09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused"
Nov 02 13:09:50.624926 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:09:50.624885    4559 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-ckslt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused" start-of-body=
Nov 02 13:09:50.625341 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:09:50.624986    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ckslt" podUID="826b92f5-161c-4f6a-9055-d0c838c9ce09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused"
Nov 02 13:09:51.624822 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:09:51.624785    4559 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-ckslt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused" start-of-body=
Nov 02 13:09:51.624999 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:09:51.624838    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ckslt" podUID="826b92f5-161c-4f6a-9055-d0c838c9ce09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused"
Nov 02 13:09:53.279865 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:09:53.279828    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ckslt"
Nov 02 13:09:53.280713 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:09:53.280687    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ckslt"
Nov 02 13:09:53.281560 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:09:53.281537    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ckslt"
Nov 02 13:09:53.342277 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:09:53.342197    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ckslt" podStartSLOduration=11.342182079 podStartE2EDuration="11.342182079s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:09:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:09:43.850362589 +0000 UTC m=+2580.105761681" watchObservedRunningTime="2026-11-02 13:09:53.342182079 +0000 UTC m=+2589.597581173"
Nov 02 13:10:06.253402 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:10:06.253370    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-ef5efe-openshift-default-679746c7d9-x5qtc"]
Nov 02 13:10:06.257538 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:10:06.257523    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-ef5efe-openshift-default-679746c7d9-x5qtc"
Nov 02 13:10:06.282659 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:10:06.282624    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-ef5efe-openshift-default-dockercfg-4cfq2\""
Nov 02 13:10:06.301575 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:10:06.301548    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-ef5efe-openshift-default-679746c7d9-x5qtc"]
Nov 02 13:10:06.322901 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:10:06.322861    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2fd94eba-ab06-4d75-a63b-e31bca0326d6-istio-podinfo\") pod \"e2e-extmod-a-ef5efe-openshift-default-679746c7d9-x5qtc\" (UID: \"2fd94eba-ab06-4d75-a63b-e31bca0326d6\") " pod="openshift-ingress/e2e-extmod-a-ef5efe-openshift-default-679746c7d9-x5qtc"
Nov 02 13:10:06.323048 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:10:06.322936    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5gtj\" (UniqueName: \"kubernetes.io/projected/2fd94eba-ab06-4d75-a63b-e31bca0326d6-kube-api-access-n5gtj\") pod \"e2e-extmod-a-ef5efe-openshift-default-679746c7d9-x5qtc\" (UID: \"2fd94eba-ab06-4d75-a63b-e31bca0326d6\") " pod="openshift-ingress/e2e-extmod-a-ef5efe-openshift-default-679746c7d9-x5qtc"
Nov 02 13:10:06.323048 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:10:06.323035    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2fd94eba-ab06-4d75-a63b-e31bca0326d6-workload-certs\") pod \"e2e-extmod-a-ef5efe-openshift-default-679746c7d9-x5qtc\" (UID: \"2fd94eba-ab06-4d75-a63b-e31bca0326d6\") " pod="openshift-ingress/e2e-extmod-a-ef5efe-openshift-default-679746c7d9-x5qtc"
Nov 02 13:10:06.323179 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:10:06.323083    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2fd94eba-ab06-4d75-a63b-e31bca0326d6-istio-data\") pod \"e2e-extmod-a-ef5efe-openshift-default-679746c7d9-x5qtc\" (UID: \"2fd94eba-ab06-4d75-a63b-e31bca0326d6\") " pod="openshift-ingress/e2e-extmod-a-ef5efe-openshift-default-679746c7d9-x5qtc"
Nov 02 13:10:06.323179 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:10:06.323143    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2fd94eba-ab06-4d75-a63b-e31bca0326d6-istiod-ca-cert\") pod \"e2e-extmod-a-ef5efe-openshift-default-679746c7d9-x5qtc\" (UID: \"2fd94eba-ab06-4d75-a63b-e31bca0326d6\") " pod="openshift-ingress/e2e-extmod-a-ef5efe-openshift-default-679746c7d9-x5qtc"
Nov 02 13:10:06.323338 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:10:06.323309    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2fd94eba-ab06-4d75-a63b-e31bca0326d6-istio-envoy\") pod \"e2e-extmod-a-ef5efe-openshift-default-679746c7d9-x5qtc\" (UID: \"2fd94eba-ab06-4d75-a63b-e31bca0326d6\") " pod="openshift-ingress/e2e-extmod-a-ef5efe-openshift-default-679746c7d9-x5qtc"
Nov 02 13:10:06.323507 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:10:06.323486    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fd94eba-ab06-4d75-a63b-e31bca0326d6-workload-socket\") pod \"e2e-extmod-a-ef5efe-openshift-default-679746c7d9-x5qtc\" (UID: \"2fd94eba-ab06-4d75-a63b-e31bca0326d6\") " pod="openshift-ingress/e2e-extmod-a-ef5efe-openshift-default-679746c7d9-x5qtc"
Nov 02 13:10:06.323560 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:10:06.323545    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2fd94eba-ab06-4d75-a63b-e31bca0326d6-istio-token\") pod \"e2e-extmod-a-ef5efe-openshift-default-679746c7d9-x5qtc\" (UID: \"2fd94eba-ab06-4d75-a63b-e31bca0326d6\") " pod="openshift-ingress/e2e-extmod-a-ef5efe-openshift-default-679746c7d9-x5qtc"
Nov 02 13:10:06.323598 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:10:06.323576    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fd94eba-ab06-4d75-a63b-e31bca0326d6-credential-socket\") pod \"e2e-extmod-a-ef5efe-openshift-default-679746c7d9-x5qtc\" (UID: \"2fd94eba-ab06-4d75-a63b-e31bca0326d6\") " pod="openshift-ingress/e2e-extmod-a-ef5efe-openshift-default-679746c7d9-x5qtc"
Nov 02 13:10:06.425192 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:10:06.425105    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2fd94eba-ab06-4d75-a63b-e31bca0326d6-istio-envoy\") pod \"e2e-extmod-a-ef5efe-openshift-default-679746c7d9-x5qtc\" (UID: \"2fd94eba-ab06-4d75-a63b-e31bca0326d6\") " pod="openshift-ingress/e2e-extmod-a-ef5efe-openshift-default-679746c7d9-x5qtc"
Nov 02 13:10:06.425350 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:10:06.425281    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fd94eba-ab06-4d75-a63b-e31bca0326d6-workload-socket\") pod \"e2e-extmod-a-ef5efe-openshift-default-679746c7d9-x5qtc\" (UID: \"2fd94eba-ab06-4d75-a63b-e31bca0326d6\") " pod="openshift-ingress/e2e-extmod-a-ef5efe-openshift-default-679746c7d9-x5qtc"
Nov 02 13:10:06.425350 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:10:06.425317    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2fd94eba-ab06-4d75-a63b-e31bca0326d6-istio-token\") pod \"e2e-extmod-a-ef5efe-openshift-default-679746c7d9-x5qtc\" (UID: \"2fd94eba-ab06-4d75-a63b-e31bca0326d6\") " pod="openshift-ingress/e2e-extmod-a-ef5efe-openshift-default-679746c7d9-x5qtc"
Nov 02 13:10:06.425350 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:10:06.425335    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fd94eba-ab06-4d75-a63b-e31bca0326d6-credential-socket\") pod \"e2e-extmod-a-ef5efe-openshift-default-679746c7d9-x5qtc\" (UID: \"2fd94eba-ab06-4d75-a63b-e31bca0326d6\") " pod="openshift-ingress/e2e-extmod-a-ef5efe-openshift-default-679746c7d9-x5qtc"
Nov 02 13:10:06.425537 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:10:06.425392    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2fd94eba-ab06-4d75-a63b-e31bca0326d6-istio-podinfo\") pod \"e2e-extmod-a-ef5efe-openshift-default-679746c7d9-x5qtc\" (UID: \"2fd94eba-ab06-4d75-a63b-e31bca0326d6\") " pod="openshift-ingress/e2e-extmod-a-ef5efe-openshift-default-679746c7d9-x5qtc"
Nov 02 13:10:06.425537 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:10:06.425423    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5gtj\" (UniqueName: \"kubernetes.io/projected/2fd94eba-ab06-4d75-a63b-e31bca0326d6-kube-api-access-n5gtj\") pod \"e2e-extmod-a-ef5efe-openshift-default-679746c7d9-x5qtc\" (UID: \"2fd94eba-ab06-4d75-a63b-e31bca0326d6\") " pod="openshift-ingress/e2e-extmod-a-ef5efe-openshift-default-679746c7d9-x5qtc"
Nov 02 13:10:06.425537 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:10:06.425476    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2fd94eba-ab06-4d75-a63b-e31bca0326d6-workload-certs\") pod \"e2e-extmod-a-ef5efe-openshift-default-679746c7d9-x5qtc\" (UID: \"2fd94eba-ab06-4d75-a63b-e31bca0326d6\") " pod="openshift-ingress/e2e-extmod-a-ef5efe-openshift-default-679746c7d9-x5qtc"
Nov 02 13:10:06.425537 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:10:06.425523    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2fd94eba-ab06-4d75-a63b-e31bca0326d6-istio-data\") pod \"e2e-extmod-a-ef5efe-openshift-default-679746c7d9-x5qtc\" (UID: \"2fd94eba-ab06-4d75-a63b-e31bca0326d6\") " pod="openshift-ingress/e2e-extmod-a-ef5efe-openshift-default-679746c7d9-x5qtc"
Nov 02 13:10:06.425823 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:10:06.425580    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2fd94eba-ab06-4d75-a63b-e31bca0326d6-istiod-ca-cert\") pod \"e2e-extmod-a-ef5efe-openshift-default-679746c7d9-x5qtc\" (UID: \"2fd94eba-ab06-4d75-a63b-e31bca0326d6\") " pod="openshift-ingress/e2e-extmod-a-ef5efe-openshift-default-679746c7d9-x5qtc"
Nov 02 13:10:06.425823 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:10:06.425699    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fd94eba-ab06-4d75-a63b-e31bca0326d6-workload-socket\") pod \"e2e-extmod-a-ef5efe-openshift-default-679746c7d9-x5qtc\" (UID: \"2fd94eba-ab06-4d75-a63b-e31bca0326d6\") " pod="openshift-ingress/e2e-extmod-a-ef5efe-openshift-default-679746c7d9-x5qtc"
Nov 02 13:10:06.425823 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:10:06.425780    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fd94eba-ab06-4d75-a63b-e31bca0326d6-credential-socket\") pod \"e2e-extmod-a-ef5efe-openshift-default-679746c7d9-x5qtc\" (UID: \"2fd94eba-ab06-4d75-a63b-e31bca0326d6\") " pod="openshift-ingress/e2e-extmod-a-ef5efe-openshift-default-679746c7d9-x5qtc"
Nov 02 13:10:06.430598 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:10:06.425929    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2fd94eba-ab06-4d75-a63b-e31bca0326d6-workload-certs\") pod \"e2e-extmod-a-ef5efe-openshift-default-679746c7d9-x5qtc\" (UID: \"2fd94eba-ab06-4d75-a63b-e31bca0326d6\") " pod="openshift-ingress/e2e-extmod-a-ef5efe-openshift-default-679746c7d9-x5qtc"
Nov 02 13:10:06.430598 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:10:06.426367    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2fd94eba-ab06-4d75-a63b-e31bca0326d6-istio-data\") pod \"e2e-extmod-a-ef5efe-openshift-default-679746c7d9-x5qtc\" (UID: \"2fd94eba-ab06-4d75-a63b-e31bca0326d6\") " pod="openshift-ingress/e2e-extmod-a-ef5efe-openshift-default-679746c7d9-x5qtc"
Nov 02 13:10:06.430598 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:10:06.426724    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2fd94eba-ab06-4d75-a63b-e31bca0326d6-istiod-ca-cert\") pod \"e2e-extmod-a-ef5efe-openshift-default-679746c7d9-x5qtc\" (UID: \"2fd94eba-ab06-4d75-a63b-e31bca0326d6\") " pod="openshift-ingress/e2e-extmod-a-ef5efe-openshift-default-679746c7d9-x5qtc"
Nov 02 13:10:06.430598 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:10:06.427901    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2fd94eba-ab06-4d75-a63b-e31bca0326d6-istio-envoy\") pod \"e2e-extmod-a-ef5efe-openshift-default-679746c7d9-x5qtc\" (UID: \"2fd94eba-ab06-4d75-a63b-e31bca0326d6\") " pod="openshift-ingress/e2e-extmod-a-ef5efe-openshift-default-679746c7d9-x5qtc"
Nov 02 13:10:06.430598 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:10:06.428967    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2fd94eba-ab06-4d75-a63b-e31bca0326d6-istio-podinfo\") pod \"e2e-extmod-a-ef5efe-openshift-default-679746c7d9-x5qtc\" (UID: \"2fd94eba-ab06-4d75-a63b-e31bca0326d6\") " pod="openshift-ingress/e2e-extmod-a-ef5efe-openshift-default-679746c7d9-x5qtc"
Nov 02 13:10:06.449285 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:10:06.449259    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2fd94eba-ab06-4d75-a63b-e31bca0326d6-istio-token\") pod \"e2e-extmod-a-ef5efe-openshift-default-679746c7d9-x5qtc\" (UID: \"2fd94eba-ab06-4d75-a63b-e31bca0326d6\") " pod="openshift-ingress/e2e-extmod-a-ef5efe-openshift-default-679746c7d9-x5qtc"
Nov 02 13:10:06.450059 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:10:06.450032    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5gtj\" (UniqueName: \"kubernetes.io/projected/2fd94eba-ab06-4d75-a63b-e31bca0326d6-kube-api-access-n5gtj\") pod \"e2e-extmod-a-ef5efe-openshift-default-679746c7d9-x5qtc\" (UID: \"2fd94eba-ab06-4d75-a63b-e31bca0326d6\") " pod="openshift-ingress/e2e-extmod-a-ef5efe-openshift-default-679746c7d9-x5qtc"
Nov 02 13:10:06.571988 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:10:06.571912    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-ef5efe-openshift-default-679746c7d9-x5qtc"
Nov 02 13:10:06.893695 ip-10-0-128-132 kubenswrapper[4559]: W1102 13:10:06.893663    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fd94eba_ab06_4d75_a63b_e31bca0326d6.slice/crio-4968e28cecd962a397484e1e22fc9607aa9437e56f9ac96e46e1e0587e378ba1 WatchSource:0}: Error finding container 4968e28cecd962a397484e1e22fc9607aa9437e56f9ac96e46e1e0587e378ba1: Status 404 returned error can't find the container with id 4968e28cecd962a397484e1e22fc9607aa9437e56f9ac96e46e1e0587e378ba1
Nov 02 13:10:06.894505 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:10:06.894480    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-ef5efe-openshift-default-679746c7d9-x5qtc"]
Nov 02 13:10:06.896445 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:10:06.896407    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 13:10:06.896555 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:10:06.896476    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 13:10:06.896555 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:10:06.896505    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 13:10:07.168934 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:10:07.168898    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-ef5efe-openshift-default-679746c7d9-x5qtc" event={"ID":"2fd94eba-ab06-4d75-a63b-e31bca0326d6","Type":"ContainerStarted","Data":"712549f074112340d65eb3ef1b661beb6e934332b2dbc804bc4c71d14cd447ca"}
Nov 02 13:10:07.169175 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:10:07.168965    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-ef5efe-openshift-default-679746c7d9-x5qtc" event={"ID":"2fd94eba-ab06-4d75-a63b-e31bca0326d6","Type":"ContainerStarted","Data":"4968e28cecd962a397484e1e22fc9607aa9437e56f9ac96e46e1e0587e378ba1"}
Nov 02 13:10:07.231660 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:10:07.229073    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-ef5efe-openshift-default-679746c7d9-x5qtc" podStartSLOduration=1.229053495 podStartE2EDuration="1.229053495s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:10:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:10:07.222827664 +0000 UTC m=+2603.478226760" watchObservedRunningTime="2026-11-02 13:10:07.229053495 +0000 UTC m=+2603.484452590"
Nov 02 13:10:07.572942 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:10:07.572860    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-ef5efe-openshift-default-679746c7d9-x5qtc"
Nov 02 13:10:07.658630 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:10:07.658600    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-740f09/payload-processing-external-model-e2e-extmod-a-740f09-645cjzv9l"]
Nov 02 13:10:07.663001 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:10:07.662982    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-740f09/payload-processing-external-model-e2e-extmod-a-740f09-645cjzv9l"
Nov 02 13:10:07.683264 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:10:07.683137    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-740f09\"/\"agc-cred-0c5fa1a8b1b9\""
Nov 02 13:10:07.683264 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:10:07.683239    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-740f09\"/\"routing-overlay\""
Nov 02 13:10:07.705824 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:10:07.704948    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-740f09\"/\"payload-processing-external-model-plugins-e2e-extmod-a-740f09\""
Nov 02 13:10:07.705824 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:10:07.705023    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-740f09\"/\"openshift-service-ca.crt\""
Nov 02 13:10:07.706212 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:10:07.706186    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-740f09\"/\"kube-root-ca.crt\""
Nov 02 13:10:07.706424 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:10:07.706406    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-740f09\"/\"agc-cred-6990cad4d83b\""
Nov 02 13:10:07.706656 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:10:07.706624    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-740f09\"/\"payload-processing-external-model-e2e--49869bd7-dockercfg-j49vg\""
Nov 02 13:10:07.737659 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:10:07.732011    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-740f09/payload-processing-external-model-e2e-extmod-a-740f09-645cjzv9l"]
Nov 02 13:10:07.751666 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:10:07.751603    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/dc513c22-60e4-4789-90db-4a23974df75c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-740f09-645cjzv9l\" (UID: \"dc513c22-60e4-4789-90db-4a23974df75c\") " pod="ai-tenant-e2e-extmod-a-740f09/payload-processing-external-model-e2e-extmod-a-740f09-645cjzv9l"
Nov 02 13:10:07.751804 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:10:07.751684    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc513c22-60e4-4789-90db-4a23974df75c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-740f09-645cjzv9l\" (UID: \"dc513c22-60e4-4789-90db-4a23974df75c\") " pod="ai-tenant-e2e-extmod-a-740f09/payload-processing-external-model-e2e-extmod-a-740f09-645cjzv9l"
Nov 02 13:10:07.751804 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:10:07.751725    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/dc513c22-60e4-4789-90db-4a23974df75c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-740f09-645cjzv9l\" (UID: \"dc513c22-60e4-4789-90db-4a23974df75c\") " pod="ai-tenant-e2e-extmod-a-740f09/payload-processing-external-model-e2e-extmod-a-740f09-645cjzv9l"
Nov 02 13:10:07.751928 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:10:07.751858    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2cfp\" (UniqueName: \"kubernetes.io/projected/dc513c22-60e4-4789-90db-4a23974df75c-kube-api-access-d2cfp\") pod \"payload-processing-external-model-e2e-extmod-a-740f09-645cjzv9l\" (UID: \"dc513c22-60e4-4789-90db-4a23974df75c\") " pod="ai-tenant-e2e-extmod-a-740f09/payload-processing-external-model-e2e-extmod-a-740f09-645cjzv9l"
Nov 02 13:10:07.853081 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:10:07.853000    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2cfp\" (UniqueName: \"kubernetes.io/projected/dc513c22-60e4-4789-90db-4a23974df75c-kube-api-access-d2cfp\") pod \"payload-processing-external-model-e2e-extmod-a-740f09-645cjzv9l\" (UID: \"dc513c22-60e4-4789-90db-4a23974df75c\") " pod="ai-tenant-e2e-extmod-a-740f09/payload-processing-external-model-e2e-extmod-a-740f09-645cjzv9l"
Nov 02 13:10:07.853273 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:10:07.853113    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/dc513c22-60e4-4789-90db-4a23974df75c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-740f09-645cjzv9l\" (UID: \"dc513c22-60e4-4789-90db-4a23974df75c\") " pod="ai-tenant-e2e-extmod-a-740f09/payload-processing-external-model-e2e-extmod-a-740f09-645cjzv9l"
Nov 02 13:10:07.853273 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:10:07.853142    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc513c22-60e4-4789-90db-4a23974df75c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-740f09-645cjzv9l\" (UID: \"dc513c22-60e4-4789-90db-4a23974df75c\") " pod="ai-tenant-e2e-extmod-a-740f09/payload-processing-external-model-e2e-extmod-a-740f09-645cjzv9l"
Nov 02 13:10:07.853273 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:10:07.853183    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/dc513c22-60e4-4789-90db-4a23974df75c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-740f09-645cjzv9l\" (UID: \"dc513c22-60e4-4789-90db-4a23974df75c\") " pod="ai-tenant-e2e-extmod-a-740f09/payload-processing-external-model-e2e-extmod-a-740f09-645cjzv9l"
Nov 02 13:10:07.853996 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:10:07.853959    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc513c22-60e4-4789-90db-4a23974df75c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-740f09-645cjzv9l\" (UID: \"dc513c22-60e4-4789-90db-4a23974df75c\") " pod="ai-tenant-e2e-extmod-a-740f09/payload-processing-external-model-e2e-extmod-a-740f09-645cjzv9l"
Nov 02 13:10:07.854128 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:10:07.853960    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/dc513c22-60e4-4789-90db-4a23974df75c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-740f09-645cjzv9l\" (UID: \"dc513c22-60e4-4789-90db-4a23974df75c\") " pod="ai-tenant-e2e-extmod-a-740f09/payload-processing-external-model-e2e-extmod-a-740f09-645cjzv9l"
Nov 02 13:10:07.856158 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:10:07.856132    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/dc513c22-60e4-4789-90db-4a23974df75c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-740f09-645cjzv9l\" (UID: \"dc513c22-60e4-4789-90db-4a23974df75c\") " pod="ai-tenant-e2e-extmod-a-740f09/payload-processing-external-model-e2e-extmod-a-740f09-645cjzv9l"
Nov 02 13:10:07.877489 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:10:07.877457    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2cfp\" (UniqueName: \"kubernetes.io/projected/dc513c22-60e4-4789-90db-4a23974df75c-kube-api-access-d2cfp\") pod \"payload-processing-external-model-e2e-extmod-a-740f09-645cjzv9l\" (UID: \"dc513c22-60e4-4789-90db-4a23974df75c\") " pod="ai-tenant-e2e-extmod-a-740f09/payload-processing-external-model-e2e-extmod-a-740f09-645cjzv9l"
Nov 02 13:10:07.979540 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:10:07.979503    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-740f09/payload-processing-external-model-e2e-extmod-a-740f09-645cjzv9l"
Nov 02 13:10:08.317783 ip-10-0-128-132 kubenswrapper[4559]: W1102 13:10:08.317752    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc513c22_60e4_4789_90db_4a23974df75c.slice/crio-12eb07ab4b877a303e3e9c612f12a2ce90213ace8ada8b6ce1574d7f97b7191e WatchSource:0}: Error finding container 12eb07ab4b877a303e3e9c612f12a2ce90213ace8ada8b6ce1574d7f97b7191e: Status 404 returned error can't find the container with id 12eb07ab4b877a303e3e9c612f12a2ce90213ace8ada8b6ce1574d7f97b7191e
Nov 02 13:10:08.319977 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:10:08.319952    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-740f09/payload-processing-external-model-e2e-extmod-a-740f09-645cjzv9l"]
Nov 02 13:10:08.573834 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:10:08.573754    4559 patch_prober.go:29] interesting pod/e2e-extmod-a-ef5efe-openshift-default-679746c7d9-x5qtc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.224:15021/healthz/ready\": dial tcp 10.133.0.224:15021: connect: connection refused" start-of-body=
Nov 02 13:10:08.573834 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:10:08.573811    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-ef5efe-openshift-default-679746c7d9-x5qtc" podUID="2fd94eba-ab06-4d75-a63b-e31bca0326d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.224:15021/healthz/ready\": dial tcp 10.133.0.224:15021: connect: connection refused"
Nov 02 13:10:09.215428 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:10:09.215385    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-740f09/payload-processing-external-model-e2e-extmod-a-740f09-645cjzv9l" event={"ID":"dc513c22-60e4-4789-90db-4a23974df75c","Type":"ContainerStarted","Data":"3652276d6d67bf6a6efceb4619e8ac12b511ac8ae34e205a8f4b41a22abe235c"}
Nov 02 13:10:09.215656 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:10:09.215498    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-740f09/payload-processing-external-model-e2e-extmod-a-740f09-645cjzv9l" event={"ID":"dc513c22-60e4-4789-90db-4a23974df75c","Type":"ContainerStarted","Data":"12eb07ab4b877a303e3e9c612f12a2ce90213ace8ada8b6ce1574d7f97b7191e"}
Nov 02 13:10:09.215856 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:10:09.215828    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-740f09/payload-processing-external-model-e2e-extmod-a-740f09-645cjzv9l"
Nov 02 13:10:09.259107 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:10:09.259053    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-740f09/payload-processing-external-model-e2e-extmod-a-740f09-645cjzv9l" podStartSLOduration=2.259041515 podStartE2EDuration="2.259041515s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:10:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:10:09.257073779 +0000 UTC m=+2605.512472874" watchObservedRunningTime="2026-11-02 13:10:09.259041515 +0000 UTC m=+2605.514440609"
Nov 02 13:10:09.572306 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:10:09.572206    4559 patch_prober.go:29] interesting pod/e2e-extmod-a-ef5efe-openshift-default-679746c7d9-x5qtc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.224:15021/healthz/ready\": dial tcp 10.133.0.224:15021: connect: connection refused" start-of-body=
Nov 02 13:10:09.572306 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:10:09.572267    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-ef5efe-openshift-default-679746c7d9-x5qtc" podUID="2fd94eba-ab06-4d75-a63b-e31bca0326d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.224:15021/healthz/ready\": dial tcp 10.133.0.224:15021: connect: connection refused"
Nov 02 13:10:10.572189 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:10:10.572152    4559 patch_prober.go:29] interesting pod/e2e-extmod-a-ef5efe-openshift-default-679746c7d9-x5qtc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.224:15021/healthz/ready\": dial tcp 10.133.0.224:15021: connect: connection refused" start-of-body=
Nov 02 13:10:10.572683 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:10:10.572208    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-ef5efe-openshift-default-679746c7d9-x5qtc" podUID="2fd94eba-ab06-4d75-a63b-e31bca0326d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.224:15021/healthz/ready\": dial tcp 10.133.0.224:15021: connect: connection refused"
Nov 02 13:10:11.572606 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:10:11.572573    4559 patch_prober.go:29] interesting pod/e2e-extmod-a-ef5efe-openshift-default-679746c7d9-x5qtc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.224:15021/healthz/ready\": dial tcp 10.133.0.224:15021: connect: connection refused" start-of-body=
Nov 02 13:10:11.573005 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:10:11.572628    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-ef5efe-openshift-default-679746c7d9-x5qtc" podUID="2fd94eba-ab06-4d75-a63b-e31bca0326d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.224:15021/healthz/ready\": dial tcp 10.133.0.224:15021: connect: connection refused"
Nov 02 13:10:12.572247 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:10:12.572213    4559 patch_prober.go:29] interesting pod/e2e-extmod-a-ef5efe-openshift-default-679746c7d9-x5qtc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.224:15021/healthz/ready\": dial tcp 10.133.0.224:15021: connect: connection refused" start-of-body=
Nov 02 13:10:12.572416 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:10:12.572268    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-ef5efe-openshift-default-679746c7d9-x5qtc" podUID="2fd94eba-ab06-4d75-a63b-e31bca0326d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.224:15021/healthz/ready\": dial tcp 10.133.0.224:15021: connect: connection refused"
Nov 02 13:10:13.577925 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:10:13.577893    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-ef5efe-openshift-default-679746c7d9-x5qtc"
Nov 02 13:10:13.578330 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:10:13.578267    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-ef5efe-openshift-default-679746c7d9-x5qtc"
Nov 02 13:10:13.578993 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:10:13.578971    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-ef5efe-openshift-default-679746c7d9-x5qtc"
Nov 02 13:10:15.089363 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:10:15.089326    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqcmj" containerName="extractor"
Nov 02 13:10:16.886055 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:10:16.886019    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-ef5efe/payload-processing-external-model-e2e-extmod-a-ef5efe-59bb6rs2g"]
Nov 02 13:10:16.890541 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:10:16.890524    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-ef5efe/payload-processing-external-model-e2e-extmod-a-ef5efe-59bb6rs2g"
Nov 02 13:10:16.909989 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:10:16.909966    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-ef5efe/payload-processing-external-model-e2e-extmod-a-ef5efe-59bb6rs2g"]
Nov 02 13:10:16.919013 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:10:16.918946    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-ef5efe\"/\"payload-processing-external-model-plugins-e2e-extmod-a-ef5efe\""
Nov 02 13:10:16.919013 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:10:16.918979    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-ef5efe\"/\"routing-overlay\""
Nov 02 13:10:16.919202 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:10:16.919121    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-ef5efe\"/\"agc-cred-7577f881eb5f\""
Nov 02 13:10:16.919202 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:10:16.919148    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-ef5efe\"/\"kube-root-ca.crt\""
Nov 02 13:10:16.923606 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:10:16.923583    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-ef5efe\"/\"openshift-service-ca.crt\""
Nov 02 13:10:16.923885 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:10:16.923866    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-ef5efe\"/\"payload-processing-external-model-e2e--00beae95-dockercfg-t2lnb\""
Nov 02 13:10:16.977203 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:10:16.977157    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f3f421b-784d-4112-b1fc-2a1425a502b4-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ef5efe-59bb6rs2g\" (UID: \"5f3f421b-784d-4112-b1fc-2a1425a502b4\") " pod="ai-tenant-e2e-extmod-a-ef5efe/payload-processing-external-model-e2e-extmod-a-ef5efe-59bb6rs2g"
Nov 02 13:10:16.977459 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:10:16.977438    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvz64\" (UniqueName: \"kubernetes.io/projected/5f3f421b-784d-4112-b1fc-2a1425a502b4-kube-api-access-zvz64\") pod \"payload-processing-external-model-e2e-extmod-a-ef5efe-59bb6rs2g\" (UID: \"5f3f421b-784d-4112-b1fc-2a1425a502b4\") " pod="ai-tenant-e2e-extmod-a-ef5efe/payload-processing-external-model-e2e-extmod-a-ef5efe-59bb6rs2g"
Nov 02 13:10:16.977555 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:10:16.977510    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5f3f421b-784d-4112-b1fc-2a1425a502b4-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-ef5efe-59bb6rs2g\" (UID: \"5f3f421b-784d-4112-b1fc-2a1425a502b4\") " pod="ai-tenant-e2e-extmod-a-ef5efe/payload-processing-external-model-e2e-extmod-a-ef5efe-59bb6rs2g"
Nov 02 13:10:16.977739 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:10:16.977710    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5f3f421b-784d-4112-b1fc-2a1425a502b4-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ef5efe-59bb6rs2g\" (UID: \"5f3f421b-784d-4112-b1fc-2a1425a502b4\") " pod="ai-tenant-e2e-extmod-a-ef5efe/payload-processing-external-model-e2e-extmod-a-ef5efe-59bb6rs2g"
Nov 02 13:10:17.079466 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:10:17.079429    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvz64\" (UniqueName: \"kubernetes.io/projected/5f3f421b-784d-4112-b1fc-2a1425a502b4-kube-api-access-zvz64\") pod \"payload-processing-external-model-e2e-extmod-a-ef5efe-59bb6rs2g\" (UID: \"5f3f421b-784d-4112-b1fc-2a1425a502b4\") " pod="ai-tenant-e2e-extmod-a-ef5efe/payload-processing-external-model-e2e-extmod-a-ef5efe-59bb6rs2g"
Nov 02 13:10:17.079609 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:10:17.079481    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5f3f421b-784d-4112-b1fc-2a1425a502b4-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-ef5efe-59bb6rs2g\" (UID: \"5f3f421b-784d-4112-b1fc-2a1425a502b4\") " pod="ai-tenant-e2e-extmod-a-ef5efe/payload-processing-external-model-e2e-extmod-a-ef5efe-59bb6rs2g"
Nov 02 13:10:17.079701 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:10:17.079617    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5f3f421b-784d-4112-b1fc-2a1425a502b4-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ef5efe-59bb6rs2g\" (UID: \"5f3f421b-784d-4112-b1fc-2a1425a502b4\") " pod="ai-tenant-e2e-extmod-a-ef5efe/payload-processing-external-model-e2e-extmod-a-ef5efe-59bb6rs2g"
Nov 02 13:10:17.079701 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:10:17.079685    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f3f421b-784d-4112-b1fc-2a1425a502b4-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ef5efe-59bb6rs2g\" (UID: \"5f3f421b-784d-4112-b1fc-2a1425a502b4\") " pod="ai-tenant-e2e-extmod-a-ef5efe/payload-processing-external-model-e2e-extmod-a-ef5efe-59bb6rs2g"
Nov 02 13:10:17.080295 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:10:17.080266    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5f3f421b-784d-4112-b1fc-2a1425a502b4-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ef5efe-59bb6rs2g\" (UID: \"5f3f421b-784d-4112-b1fc-2a1425a502b4\") " pod="ai-tenant-e2e-extmod-a-ef5efe/payload-processing-external-model-e2e-extmod-a-ef5efe-59bb6rs2g"
Nov 02 13:10:17.080428 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:10:17.080410    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f3f421b-784d-4112-b1fc-2a1425a502b4-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ef5efe-59bb6rs2g\" (UID: \"5f3f421b-784d-4112-b1fc-2a1425a502b4\") " pod="ai-tenant-e2e-extmod-a-ef5efe/payload-processing-external-model-e2e-extmod-a-ef5efe-59bb6rs2g"
Nov 02 13:10:17.081996 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:10:17.081977    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5f3f421b-784d-4112-b1fc-2a1425a502b4-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-ef5efe-59bb6rs2g\" (UID: \"5f3f421b-784d-4112-b1fc-2a1425a502b4\") " pod="ai-tenant-e2e-extmod-a-ef5efe/payload-processing-external-model-e2e-extmod-a-ef5efe-59bb6rs2g"
Nov 02 13:10:17.108358 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:10:17.108326    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvz64\" (UniqueName: \"kubernetes.io/projected/5f3f421b-784d-4112-b1fc-2a1425a502b4-kube-api-access-zvz64\") pod \"payload-processing-external-model-e2e-extmod-a-ef5efe-59bb6rs2g\" (UID: \"5f3f421b-784d-4112-b1fc-2a1425a502b4\") " pod="ai-tenant-e2e-extmod-a-ef5efe/payload-processing-external-model-e2e-extmod-a-ef5efe-59bb6rs2g"
Nov 02 13:10:17.204785 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:10:17.204752    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-ef5efe/payload-processing-external-model-e2e-extmod-a-ef5efe-59bb6rs2g"
Nov 02 13:10:17.481759 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:10:17.481684    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-ef5efe/payload-processing-external-model-e2e-extmod-a-ef5efe-59bb6rs2g"]
Nov 02 13:10:17.523897 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:10:17.523858    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-ef5efe/payload-processing-external-model-e2e-extmod-a-ef5efe-59bb6rs2g"]
Nov 02 13:10:17.530612 ip-10-0-128-132 kubenswrapper[4559]: W1102 13:10:17.530581    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f3f421b_784d_4112_b1fc_2a1425a502b4.slice/crio-cc63ab77bbe65c5e2173807c2fb57b6cbce35718e601e53010ea6c6f9650613a WatchSource:0}: Error finding container cc63ab77bbe65c5e2173807c2fb57b6cbce35718e601e53010ea6c6f9650613a: Status 404 returned error can't find the container with id cc63ab77bbe65c5e2173807c2fb57b6cbce35718e601e53010ea6c6f9650613a
Nov 02 13:10:18.285486 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:10:18.285443    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-ef5efe/payload-processing-external-model-e2e-extmod-a-ef5efe-59bb6rs2g" event={"ID":"5f3f421b-784d-4112-b1fc-2a1425a502b4","Type":"ContainerStarted","Data":"23dded2c9df90e3cd9363e2fc4f28ebcf3423f4096318d795c858161be8cb899"}
Nov 02 13:10:18.286059 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:10:18.285544    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-ef5efe/payload-processing-external-model-e2e-extmod-a-ef5efe-59bb6rs2g" event={"ID":"5f3f421b-784d-4112-b1fc-2a1425a502b4","Type":"ContainerStarted","Data":"cc63ab77bbe65c5e2173807c2fb57b6cbce35718e601e53010ea6c6f9650613a"}
Nov 02 13:10:18.286059 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:10:18.285573    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-ef5efe/payload-processing-external-model-e2e-extmod-a-ef5efe-59bb6rs2g"
Nov 02 13:10:18.328940 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:10:18.328887    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-ef5efe/payload-processing-external-model-e2e-extmod-a-ef5efe-59bb6rs2g" podStartSLOduration=2.328873095 podStartE2EDuration="2.328873095s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:10:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:10:18.327347321 +0000 UTC m=+2614.582746416" watchObservedRunningTime="2026-11-02 13:10:18.328873095 +0000 UTC m=+2614.584272190"
Nov 02 13:10:20.226133 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:10:20.226099    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-740f09/payload-processing-external-model-e2e-extmod-a-740f09-645cjzv9l"
Nov 02 13:10:23.311663 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:10:23.306174    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-ef5efe/payload-processing-external-model-e2e-extmod-a-ef5efe-59bb6rs2g" podUID="5f3f421b-784d-4112-b1fc-2a1425a502b4" containerName="payload-processing" containerID="cri-o://23dded2c9df90e3cd9363e2fc4f28ebcf3423f4096318d795c858161be8cb899" gracePeriod=25
Nov 02 13:10:23.478509 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:10:23.478376    4559 generic.go:356] "Generic (PLEG): container finished" podID="5f3f421b-784d-4112-b1fc-2a1425a502b4" containerID="23dded2c9df90e3cd9363e2fc4f28ebcf3423f4096318d795c858161be8cb899" exitCode=0
Nov 02 13:10:23.478509 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:10:23.478455    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-ef5efe/payload-processing-external-model-e2e-extmod-a-ef5efe-59bb6rs2g" event={"ID":"5f3f421b-784d-4112-b1fc-2a1425a502b4","Type":"ContainerDied","Data":"23dded2c9df90e3cd9363e2fc4f28ebcf3423f4096318d795c858161be8cb899"}
Nov 02 13:10:24.082048 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:10:24.082021    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-ef5efe/payload-processing-external-model-e2e-extmod-a-ef5efe-59bb6rs2g"
Nov 02 13:10:24.109080 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:10:24.109036    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f3f421b-784d-4112-b1fc-2a1425a502b4-routing-overlay-volume" pod "5f3f421b-784d-4112-b1fc-2a1425a502b4" (UID: "5f3f421b-784d-4112-b1fc-2a1425a502b4"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:10:24.109424 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:10:24.109403    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f3f421b-784d-4112-b1fc-2a1425a502b4-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5f3f421b-784d-4112-b1fc-2a1425a502b4-routing-overlay-volume\") pod \"5f3f421b-784d-4112-b1fc-2a1425a502b4\" (UID: \"5f3f421b-784d-4112-b1fc-2a1425a502b4\") "
Nov 02 13:10:24.110841 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:10:24.110818    4559 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5f3f421b-784d-4112-b1fc-2a1425a502b4-routing-overlay-volume\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:10:24.253718 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:10:24.253140    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f3f421b-784d-4112-b1fc-2a1425a502b4-provider-credentials\" (UniqueName: \"kubernetes.io/projected/5f3f421b-784d-4112-b1fc-2a1425a502b4-provider-credentials\") pod \"5f3f421b-784d-4112-b1fc-2a1425a502b4\" (UID: \"5f3f421b-784d-4112-b1fc-2a1425a502b4\") "
Nov 02 13:10:24.253718 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:10:24.253235    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f3f421b-784d-4112-b1fc-2a1425a502b4-kube-api-access-zvz64\" (UniqueName: \"kubernetes.io/projected/5f3f421b-784d-4112-b1fc-2a1425a502b4-kube-api-access-zvz64\") pod \"5f3f421b-784d-4112-b1fc-2a1425a502b4\" (UID: \"5f3f421b-784d-4112-b1fc-2a1425a502b4\") "
Nov 02 13:10:24.253718 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:10:24.253553    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f3f421b-784d-4112-b1fc-2a1425a502b4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f3f421b-784d-4112-b1fc-2a1425a502b4-plugins-config-volume\") pod \"5f3f421b-784d-4112-b1fc-2a1425a502b4\" (UID: \"5f3f421b-784d-4112-b1fc-2a1425a502b4\") "
Nov 02 13:10:24.276710 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:10:24.273888    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f3f421b-784d-4112-b1fc-2a1425a502b4-plugins-config-volume" pod "5f3f421b-784d-4112-b1fc-2a1425a502b4" (UID: "5f3f421b-784d-4112-b1fc-2a1425a502b4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:10:24.279588 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:10:24.279550    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f3f421b-784d-4112-b1fc-2a1425a502b4-kube-api-access-zvz64" pod "5f3f421b-784d-4112-b1fc-2a1425a502b4" (UID: "5f3f421b-784d-4112-b1fc-2a1425a502b4"). InnerVolumeSpecName "kube-api-access-zvz64". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:10:24.279758 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:10:24.279670    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f3f421b-784d-4112-b1fc-2a1425a502b4-provider-credentials" pod "5f3f421b-784d-4112-b1fc-2a1425a502b4" (UID: "5f3f421b-784d-4112-b1fc-2a1425a502b4"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:10:24.329674 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:10:24.322739    4559 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-ckslt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.223:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 13:10:24.329674 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:10:24.322852    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ckslt" podUID="826b92f5-161c-4f6a-9055-d0c838c9ce09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.223:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:10:24.364333 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:10:24.364292    4559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f3f421b-784d-4112-b1fc-2a1425a502b4-plugins-config-volume\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:10:24.364555 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:10:24.364536    4559 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5f3f421b-784d-4112-b1fc-2a1425a502b4-provider-credentials\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:10:24.364711 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:10:24.364694    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvz64\" (UniqueName: \"kubernetes.io/projected/5f3f421b-784d-4112-b1fc-2a1425a502b4-kube-api-access-zvz64\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:10:24.406675 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:10:24.404760    4559 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-xj4xk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.222:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 13:10:24.406675 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:10:24.404831    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-xj4xk" podUID="d2c5f43a-3a0c-444b-afc1-e2bf090dc05c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.222:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:10:24.493666 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:10:24.491202    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-ef5efe/payload-processing-external-model-e2e-extmod-a-ef5efe-59bb6rs2g" event={"ID":"5f3f421b-784d-4112-b1fc-2a1425a502b4","Type":"ContainerDied","Data":"cc63ab77bbe65c5e2173807c2fb57b6cbce35718e601e53010ea6c6f9650613a"}
Nov 02 13:10:24.493666 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:10:24.491277    4559 scope.go:122] "RemoveContainer" containerID="23dded2c9df90e3cd9363e2fc4f28ebcf3423f4096318d795c858161be8cb899"
Nov 02 13:10:24.493666 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:10:24.491433    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-ef5efe/payload-processing-external-model-e2e-extmod-a-ef5efe-59bb6rs2g"
Nov 02 13:10:24.591800 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:10:24.591728    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-ef5efe/payload-processing-external-model-e2e-extmod-a-ef5efe-59bb6rs2g"]
Nov 02 13:10:24.648663 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:10:24.645949    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-ef5efe/payload-processing-external-model-e2e-extmod-a-ef5efe-59bb6rs2g"]
Nov 02 13:10:25.897910 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:10:25.897877    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f3f421b-784d-4112-b1fc-2a1425a502b4" path="/var/lib/kubelet/pods/5f3f421b-784d-4112-b1fc-2a1425a502b4/volumes"
Nov 02 13:10:45.088189 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:10:45.088155    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqcmj" containerName="extractor"
Nov 02 13:10:54.595663 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:10:54.595550    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"]
Nov 02 13:11:13.124708 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:11:13.124676    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-579d767848-hbb8t"]
Nov 02 13:11:13.125542 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:11:13.125523    4559 cpu_manager.go:414] "containerMap: removing container" podUID="5f3f421b-784d-4112-b1fc-2a1425a502b4" containerName="payload-processing"
Nov 02 13:11:13.125542 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:11:13.125543    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f3f421b-784d-4112-b1fc-2a1425a502b4" containerName="payload-processing"
Nov 02 13:11:13.125719 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:11:13.125705    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f3f421b-784d-4112-b1fc-2a1425a502b4" containerName="payload-processing"
Nov 02 13:11:13.129364 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:11:13.129347    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-579d767848-hbb8t"
Nov 02 13:11:13.151769 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:11:13.151740    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-579d767848-hbb8t"]
Nov 02 13:11:13.161682 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:11:13.158917    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\""
Nov 02 13:11:13.161682 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:11:13.158926    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\""
Nov 02 13:11:13.161682 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:11:13.158964    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-90aa2c2f03a4\""
Nov 02 13:11:13.161682 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:11:13.159047    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\""
Nov 02 13:11:13.161682 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:11:13.159054    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\""
Nov 02 13:11:13.161682 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:11:13.159179    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-vjcqq\""
Nov 02 13:11:13.163342 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:11:13.163312    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8dhf\" (UniqueName: \"kubernetes.io/projected/e54fd919-3dc6-4b33-8ad6-67d31a2bf664-kube-api-access-m8dhf\") pod \"payload-processing-external-model-579d767848-hbb8t\" (UID: \"e54fd919-3dc6-4b33-8ad6-67d31a2bf664\") " pod="models-as-a-service/payload-processing-external-model-579d767848-hbb8t"
Nov 02 13:11:13.163460 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:11:13.163425    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e54fd919-3dc6-4b33-8ad6-67d31a2bf664-routing-overlay-volume\") pod \"payload-processing-external-model-579d767848-hbb8t\" (UID: \"e54fd919-3dc6-4b33-8ad6-67d31a2bf664\") " pod="models-as-a-service/payload-processing-external-model-579d767848-hbb8t"
Nov 02 13:11:13.163538 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:11:13.163471    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e54fd919-3dc6-4b33-8ad6-67d31a2bf664-plugins-config-volume\") pod \"payload-processing-external-model-579d767848-hbb8t\" (UID: \"e54fd919-3dc6-4b33-8ad6-67d31a2bf664\") " pod="models-as-a-service/payload-processing-external-model-579d767848-hbb8t"
Nov 02 13:11:13.163715 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:11:13.163696    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e54fd919-3dc6-4b33-8ad6-67d31a2bf664-provider-credentials\") pod \"payload-processing-external-model-579d767848-hbb8t\" (UID: \"e54fd919-3dc6-4b33-8ad6-67d31a2bf664\") " pod="models-as-a-service/payload-processing-external-model-579d767848-hbb8t"
Nov 02 13:11:13.279971 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:11:13.279923    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e54fd919-3dc6-4b33-8ad6-67d31a2bf664-provider-credentials\") pod \"payload-processing-external-model-579d767848-hbb8t\" (UID: \"e54fd919-3dc6-4b33-8ad6-67d31a2bf664\") " pod="models-as-a-service/payload-processing-external-model-579d767848-hbb8t"
Nov 02 13:11:13.283881 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:11:13.283847    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8dhf\" (UniqueName: \"kubernetes.io/projected/e54fd919-3dc6-4b33-8ad6-67d31a2bf664-kube-api-access-m8dhf\") pod \"payload-processing-external-model-579d767848-hbb8t\" (UID: \"e54fd919-3dc6-4b33-8ad6-67d31a2bf664\") " pod="models-as-a-service/payload-processing-external-model-579d767848-hbb8t"
Nov 02 13:11:13.284246 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:11:13.284214    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e54fd919-3dc6-4b33-8ad6-67d31a2bf664-routing-overlay-volume\") pod \"payload-processing-external-model-579d767848-hbb8t\" (UID: \"e54fd919-3dc6-4b33-8ad6-67d31a2bf664\") " pod="models-as-a-service/payload-processing-external-model-579d767848-hbb8t"
Nov 02 13:11:13.284681 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:11:13.284442    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e54fd919-3dc6-4b33-8ad6-67d31a2bf664-plugins-config-volume\") pod \"payload-processing-external-model-579d767848-hbb8t\" (UID: \"e54fd919-3dc6-4b33-8ad6-67d31a2bf664\") " pod="models-as-a-service/payload-processing-external-model-579d767848-hbb8t"
Nov 02 13:11:13.288778 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:11:13.285548    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e54fd919-3dc6-4b33-8ad6-67d31a2bf664-plugins-config-volume\") pod \"payload-processing-external-model-579d767848-hbb8t\" (UID: \"e54fd919-3dc6-4b33-8ad6-67d31a2bf664\") " pod="models-as-a-service/payload-processing-external-model-579d767848-hbb8t"
Nov 02 13:11:13.297909 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:11:13.297877    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e54fd919-3dc6-4b33-8ad6-67d31a2bf664-routing-overlay-volume\") pod \"payload-processing-external-model-579d767848-hbb8t\" (UID: \"e54fd919-3dc6-4b33-8ad6-67d31a2bf664\") " pod="models-as-a-service/payload-processing-external-model-579d767848-hbb8t"
Nov 02 13:11:13.302654 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:11:13.302592    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e54fd919-3dc6-4b33-8ad6-67d31a2bf664-provider-credentials\") pod \"payload-processing-external-model-579d767848-hbb8t\" (UID: \"e54fd919-3dc6-4b33-8ad6-67d31a2bf664\") " pod="models-as-a-service/payload-processing-external-model-579d767848-hbb8t"
Nov 02 13:11:13.321509 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:11:13.321482    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8dhf\" (UniqueName: \"kubernetes.io/projected/e54fd919-3dc6-4b33-8ad6-67d31a2bf664-kube-api-access-m8dhf\") pod \"payload-processing-external-model-579d767848-hbb8t\" (UID: \"e54fd919-3dc6-4b33-8ad6-67d31a2bf664\") " pod="models-as-a-service/payload-processing-external-model-579d767848-hbb8t"
Nov 02 13:11:13.446583 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:11:13.446456    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-579d767848-hbb8t"
Nov 02 13:11:13.949747 ip-10-0-128-132 kubenswrapper[4559]: W1102 13:11:13.949713    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode54fd919_3dc6_4b33_8ad6_67d31a2bf664.slice/crio-84549395add54355cf5e8ec9d8980a079fc015013f390f060466cdd554458b29 WatchSource:0}: Error finding container 84549395add54355cf5e8ec9d8980a079fc015013f390f060466cdd554458b29: Status 404 returned error can't find the container with id 84549395add54355cf5e8ec9d8980a079fc015013f390f060466cdd554458b29
Nov 02 13:11:13.950840 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:11:13.950813    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-579d767848-hbb8t"]
Nov 02 13:11:14.036458 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:11:14.036421    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-579d767848-hbb8t" event={"ID":"e54fd919-3dc6-4b33-8ad6-67d31a2bf664","Type":"ContainerStarted","Data":"832ead65a1c6e6f99d604db96504b8332789d4356ec8401e898659dad978cd37"}
Nov 02 13:11:14.036689 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:11:14.036477    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-579d767848-hbb8t" event={"ID":"e54fd919-3dc6-4b33-8ad6-67d31a2bf664","Type":"ContainerStarted","Data":"84549395add54355cf5e8ec9d8980a079fc015013f390f060466cdd554458b29"}
Nov 02 13:11:14.036689 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:11:14.036562    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-579d767848-hbb8t"
Nov 02 13:11:14.082969 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:11:14.082924    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-579d767848-hbb8t" podStartSLOduration=1.082910555 podStartE2EDuration="1.082910555s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:11:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:11:14.080535258 +0000 UTC m=+2670.335934352" watchObservedRunningTime="2026-11-02 13:11:14.082910555 +0000 UTC m=+2670.338309649"
Nov 02 13:11:15.413452 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:11:15.413416    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqcmj" containerName="extractor"
Nov 02 13:11:21.475618 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:11:21.475584    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-bbb7d0-7cddbf6b47-z2q7w"]
Nov 02 13:11:21.481762 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:11:21.481740    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bbb7d0-7cddbf6b47-z2q7w"
Nov 02 13:11:21.500434 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:11:21.500412    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-bbb7d0-7cddbf6b47-z2q7w"]
Nov 02 13:11:21.509038 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:11:21.509010    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-bbb7d0\""
Nov 02 13:11:21.509177 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:11:21.509026    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-bbb7d0-dockercfg-98s9r\""
Nov 02 13:11:21.526737 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:11:21.526714    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-bbb7d0-6bcb48d58c-h8gsp"]
Nov 02 13:11:21.531081 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:11:21.531060    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-bbb7d0-6bcb48d58c-h8gsp"
Nov 02 13:11:21.553895 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:11:21.553868    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-bbb7d0-6bcb48d58c-h8gsp"]
Nov 02 13:11:21.628665 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:11:21.626332    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvscc\" (UniqueName: \"kubernetes.io/projected/affbe914-1335-4829-b1d5-c31bf0bd44ea-kube-api-access-rvscc\") pod \"payload-pre-processing-e2e-inf-a-bbb7d0-7cddbf6b47-z2q7w\" (UID: \"affbe914-1335-4829-b1d5-c31bf0bd44ea\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bbb7d0-7cddbf6b47-z2q7w"
Nov 02 13:11:21.628665 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:11:21.626399    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b3239ce-770b-4f22-9461-89f8c62f850a-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-bbb7d0-6bcb48d58c-h8gsp\" (UID: \"1b3239ce-770b-4f22-9461-89f8c62f850a\") " pod="openshift-ingress/payload-processing-e2e-inf-a-bbb7d0-6bcb48d58c-h8gsp"
Nov 02 13:11:21.628665 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:11:21.626619    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/affbe914-1335-4829-b1d5-c31bf0bd44ea-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-bbb7d0-7cddbf6b47-z2q7w\" (UID: \"affbe914-1335-4829-b1d5-c31bf0bd44ea\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bbb7d0-7cddbf6b47-z2q7w"
Nov 02 13:11:21.628665 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:11:21.626686    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmdr5\" (UniqueName: \"kubernetes.io/projected/1b3239ce-770b-4f22-9461-89f8c62f850a-kube-api-access-rmdr5\") pod \"payload-processing-e2e-inf-a-bbb7d0-6bcb48d58c-h8gsp\" (UID: \"1b3239ce-770b-4f22-9461-89f8c62f850a\") " pod="openshift-ingress/payload-processing-e2e-inf-a-bbb7d0-6bcb48d58c-h8gsp"
Nov 02 13:11:21.774722 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:11:21.768146    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/affbe914-1335-4829-b1d5-c31bf0bd44ea-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-bbb7d0-7cddbf6b47-z2q7w\" (UID: \"affbe914-1335-4829-b1d5-c31bf0bd44ea\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bbb7d0-7cddbf6b47-z2q7w"
Nov 02 13:11:21.774722 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:11:21.768262    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmdr5\" (UniqueName: \"kubernetes.io/projected/1b3239ce-770b-4f22-9461-89f8c62f850a-kube-api-access-rmdr5\") pod \"payload-processing-e2e-inf-a-bbb7d0-6bcb48d58c-h8gsp\" (UID: \"1b3239ce-770b-4f22-9461-89f8c62f850a\") " pod="openshift-ingress/payload-processing-e2e-inf-a-bbb7d0-6bcb48d58c-h8gsp"
Nov 02 13:11:21.774722 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:11:21.768349    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvscc\" (UniqueName: \"kubernetes.io/projected/affbe914-1335-4829-b1d5-c31bf0bd44ea-kube-api-access-rvscc\") pod \"payload-pre-processing-e2e-inf-a-bbb7d0-7cddbf6b47-z2q7w\" (UID: \"affbe914-1335-4829-b1d5-c31bf0bd44ea\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bbb7d0-7cddbf6b47-z2q7w"
Nov 02 13:11:21.774722 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:11:21.768396    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b3239ce-770b-4f22-9461-89f8c62f850a-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-bbb7d0-6bcb48d58c-h8gsp\" (UID: \"1b3239ce-770b-4f22-9461-89f8c62f850a\") " pod="openshift-ingress/payload-processing-e2e-inf-a-bbb7d0-6bcb48d58c-h8gsp"
Nov 02 13:11:21.812707 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:11:21.777409    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/affbe914-1335-4829-b1d5-c31bf0bd44ea-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-bbb7d0-7cddbf6b47-z2q7w\" (UID: \"affbe914-1335-4829-b1d5-c31bf0bd44ea\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bbb7d0-7cddbf6b47-z2q7w"
Nov 02 13:11:21.820817 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:11:21.819182    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b3239ce-770b-4f22-9461-89f8c62f850a-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-bbb7d0-6bcb48d58c-h8gsp\" (UID: \"1b3239ce-770b-4f22-9461-89f8c62f850a\") " pod="openshift-ingress/payload-processing-e2e-inf-a-bbb7d0-6bcb48d58c-h8gsp"
Nov 02 13:11:21.830585 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:11:21.830521    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmdr5\" (UniqueName: \"kubernetes.io/projected/1b3239ce-770b-4f22-9461-89f8c62f850a-kube-api-access-rmdr5\") pod \"payload-processing-e2e-inf-a-bbb7d0-6bcb48d58c-h8gsp\" (UID: \"1b3239ce-770b-4f22-9461-89f8c62f850a\") " pod="openshift-ingress/payload-processing-e2e-inf-a-bbb7d0-6bcb48d58c-h8gsp"
Nov 02 13:11:21.833297 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:11:21.833226    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvscc\" (UniqueName: \"kubernetes.io/projected/affbe914-1335-4829-b1d5-c31bf0bd44ea-kube-api-access-rvscc\") pod \"payload-pre-processing-e2e-inf-a-bbb7d0-7cddbf6b47-z2q7w\" (UID: \"affbe914-1335-4829-b1d5-c31bf0bd44ea\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bbb7d0-7cddbf6b47-z2q7w"
Nov 02 13:11:21.869591 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:11:21.869566    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-bbb7d0-6bcb48d58c-h8gsp"
Nov 02 13:11:22.096717 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:11:22.096620    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bbb7d0-7cddbf6b47-z2q7w"
Nov 02 13:11:22.177429 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:11:22.177396    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-bbb7d0-6bcb48d58c-h8gsp"]
Nov 02 13:11:22.186141 ip-10-0-128-132 kubenswrapper[4559]: W1102 13:11:22.186107    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b3239ce_770b_4f22_9461_89f8c62f850a.slice/crio-84e01fcc7e6fad6a13d28aa29e46ef9d2609500039e021277e14d711395675f9 WatchSource:0}: Error finding container 84e01fcc7e6fad6a13d28aa29e46ef9d2609500039e021277e14d711395675f9: Status 404 returned error can't find the container with id 84e01fcc7e6fad6a13d28aa29e46ef9d2609500039e021277e14d711395675f9
Nov 02 13:11:22.400953 ip-10-0-128-132 kubenswrapper[4559]: W1102 13:11:22.400913    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaffbe914_1335_4829_b1d5_c31bf0bd44ea.slice/crio-7bb85d058ffcd7f77ac7c6735699b541ed47551fc4807d6ebd0ada9c6bf34315 WatchSource:0}: Error finding container 7bb85d058ffcd7f77ac7c6735699b541ed47551fc4807d6ebd0ada9c6bf34315: Status 404 returned error can't find the container with id 7bb85d058ffcd7f77ac7c6735699b541ed47551fc4807d6ebd0ada9c6bf34315
Nov 02 13:11:22.401748 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:11:22.401717    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-bbb7d0-7cddbf6b47-z2q7w"]
Nov 02 13:11:23.221747 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:11:23.221714    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bbb7d0-7cddbf6b47-z2q7w" event={"ID":"affbe914-1335-4829-b1d5-c31bf0bd44ea","Type":"ContainerStarted","Data":"051e6af572a8d3d0d97733c531e6f5ca206ee4c0db728d65036300d25b90f83a"}
Nov 02 13:11:23.222253 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:11:23.221775    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bbb7d0-7cddbf6b47-z2q7w" event={"ID":"affbe914-1335-4829-b1d5-c31bf0bd44ea","Type":"ContainerStarted","Data":"7bb85d058ffcd7f77ac7c6735699b541ed47551fc4807d6ebd0ada9c6bf34315"}
Nov 02 13:11:23.222253 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:11:23.221879    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bbb7d0-7cddbf6b47-z2q7w"
Nov 02 13:11:23.235701 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:11:23.235555    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-bbb7d0-6bcb48d58c-h8gsp" event={"ID":"1b3239ce-770b-4f22-9461-89f8c62f850a","Type":"ContainerStarted","Data":"b6dceb868d01a25c8ae20abe82d66e591fbb1e19f352dbdcab4bc1ad6ca97c69"}
Nov 02 13:11:23.235846 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:11:23.235760    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-bbb7d0-6bcb48d58c-h8gsp" event={"ID":"1b3239ce-770b-4f22-9461-89f8c62f850a","Type":"ContainerStarted","Data":"84e01fcc7e6fad6a13d28aa29e46ef9d2609500039e021277e14d711395675f9"}
Nov 02 13:11:23.235846 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:11:23.235790    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-bbb7d0-6bcb48d58c-h8gsp"
Nov 02 13:11:23.267290 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:11:23.267244    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bbb7d0-7cddbf6b47-z2q7w" podStartSLOduration=2.267231588 podStartE2EDuration="2.267231588s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:11:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:11:23.263622111 +0000 UTC m=+2679.519021206" watchObservedRunningTime="2026-11-02 13:11:23.267231588 +0000 UTC m=+2679.522630682"
Nov 02 13:11:23.306093 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:11:23.306038    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-bbb7d0-6bcb48d58c-h8gsp" podStartSLOduration=2.306024859 podStartE2EDuration="2.306024859s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:11:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:11:23.304822348 +0000 UTC m=+2679.560221446" watchObservedRunningTime="2026-11-02 13:11:23.306024859 +0000 UTC m=+2679.561423958"
Nov 02 13:11:25.120549 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:11:25.120514    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-579d767848-hbb8t" podUID="e54fd919-3dc6-4b33-8ad6-67d31a2bf664" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 13:11:34.246878 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:11:34.246848    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-bbb7d0-6bcb48d58c-h8gsp"
Nov 02 13:11:34.247373 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:11:34.246932    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bbb7d0-7cddbf6b47-z2q7w"
Nov 02 13:11:35.117899 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:11:35.117861    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-579d767848-hbb8t" podUID="e54fd919-3dc6-4b33-8ad6-67d31a2bf664" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 13:11:45.086606 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:11:45.086567    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqcmj" containerName="extractor"
Nov 02 13:11:45.118464 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:11:45.118430    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-579d767848-hbb8t" podUID="e54fd919-3dc6-4b33-8ad6-67d31a2bf664" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 13:11:50.450809 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:11:50.450771    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"]
Nov 02 13:11:54.323664 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:11:54.322723    4559 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-ckslt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.223:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 13:11:54.323664 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:11:54.322801    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ckslt" podUID="826b92f5-161c-4f6a-9055-d0c838c9ce09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.223:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:11:54.361805 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:11:54.361765    4559 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-xj4xk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.222:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 13:11:54.361999 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:11:54.361840    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-xj4xk" podUID="d2c5f43a-3a0c-444b-afc1-e2bf090dc05c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.222:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:11:55.118512 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:11:55.118477    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-579d767848-hbb8t" podUID="e54fd919-3dc6-4b33-8ad6-67d31a2bf664" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 13:12:02.838840 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:12:02.838805    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-bbb7d0-84c974d75-jlqqq"]
Nov 02 13:12:02.844686 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:12:02.844666    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bbb7d0-84c974d75-jlqqq"
Nov 02 13:12:02.865373 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:12:02.865343    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-bbb7d0-84c974d75-jlqqq"]
Nov 02 13:12:02.873811 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:12:02.873782    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-bbb7d0-serving-cert\""
Nov 02 13:12:03.022655 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:12:03.022611    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4436e381-1572-416e-ad03-2805362dde64-maas-api-tls\") pod \"maas-api-e2e-inf-a-bbb7d0-84c974d75-jlqqq\" (UID: \"4436e381-1572-416e-ad03-2805362dde64\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bbb7d0-84c974d75-jlqqq"
Nov 02 13:12:03.022886 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:12:03.022864    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4436e381-1572-416e-ad03-2805362dde64-metrics-certs\") pod \"maas-api-e2e-inf-a-bbb7d0-84c974d75-jlqqq\" (UID: \"4436e381-1572-416e-ad03-2805362dde64\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bbb7d0-84c974d75-jlqqq"
Nov 02 13:12:03.022983 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:12:03.022960    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zj8fz\" (UniqueName: \"kubernetes.io/projected/4436e381-1572-416e-ad03-2805362dde64-kube-api-access-zj8fz\") pod \"maas-api-e2e-inf-a-bbb7d0-84c974d75-jlqqq\" (UID: \"4436e381-1572-416e-ad03-2805362dde64\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bbb7d0-84c974d75-jlqqq"
Nov 02 13:12:03.023053 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:12:03.023035    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4436e381-1572-416e-ad03-2805362dde64-openshift-service-ca\") pod \"maas-api-e2e-inf-a-bbb7d0-84c974d75-jlqqq\" (UID: \"4436e381-1572-416e-ad03-2805362dde64\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bbb7d0-84c974d75-jlqqq"
Nov 02 13:12:03.124925 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:12:03.124835    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4436e381-1572-416e-ad03-2805362dde64-metrics-certs\") pod \"maas-api-e2e-inf-a-bbb7d0-84c974d75-jlqqq\" (UID: \"4436e381-1572-416e-ad03-2805362dde64\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bbb7d0-84c974d75-jlqqq"
Nov 02 13:12:03.124925 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:12:03.124911    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zj8fz\" (UniqueName: \"kubernetes.io/projected/4436e381-1572-416e-ad03-2805362dde64-kube-api-access-zj8fz\") pod \"maas-api-e2e-inf-a-bbb7d0-84c974d75-jlqqq\" (UID: \"4436e381-1572-416e-ad03-2805362dde64\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bbb7d0-84c974d75-jlqqq"
Nov 02 13:12:03.125187 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:12:03.124972    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4436e381-1572-416e-ad03-2805362dde64-openshift-service-ca\") pod \"maas-api-e2e-inf-a-bbb7d0-84c974d75-jlqqq\" (UID: \"4436e381-1572-416e-ad03-2805362dde64\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bbb7d0-84c974d75-jlqqq"
Nov 02 13:12:03.125187 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:12:03.125145    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4436e381-1572-416e-ad03-2805362dde64-maas-api-tls\") pod \"maas-api-e2e-inf-a-bbb7d0-84c974d75-jlqqq\" (UID: \"4436e381-1572-416e-ad03-2805362dde64\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bbb7d0-84c974d75-jlqqq"
Nov 02 13:12:03.125710 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:12:03.125688    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4436e381-1572-416e-ad03-2805362dde64-openshift-service-ca\") pod \"maas-api-e2e-inf-a-bbb7d0-84c974d75-jlqqq\" (UID: \"4436e381-1572-416e-ad03-2805362dde64\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bbb7d0-84c974d75-jlqqq"
Nov 02 13:12:03.127419 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:12:03.127399    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4436e381-1572-416e-ad03-2805362dde64-metrics-certs\") pod \"maas-api-e2e-inf-a-bbb7d0-84c974d75-jlqqq\" (UID: \"4436e381-1572-416e-ad03-2805362dde64\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bbb7d0-84c974d75-jlqqq"
Nov 02 13:12:03.127548 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:12:03.127527    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4436e381-1572-416e-ad03-2805362dde64-maas-api-tls\") pod \"maas-api-e2e-inf-a-bbb7d0-84c974d75-jlqqq\" (UID: \"4436e381-1572-416e-ad03-2805362dde64\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bbb7d0-84c974d75-jlqqq"
Nov 02 13:12:03.179967 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:12:03.179913    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zj8fz\" (UniqueName: \"kubernetes.io/projected/4436e381-1572-416e-ad03-2805362dde64-kube-api-access-zj8fz\") pod \"maas-api-e2e-inf-a-bbb7d0-84c974d75-jlqqq\" (UID: \"4436e381-1572-416e-ad03-2805362dde64\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bbb7d0-84c974d75-jlqqq"
Nov 02 13:12:03.458768 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:12:03.458736    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bbb7d0-84c974d75-jlqqq"
Nov 02 13:12:03.764806 ip-10-0-128-132 kubenswrapper[4559]: W1102 13:12:03.764778    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4436e381_1572_416e_ad03_2805362dde64.slice/crio-256ffb7640772b7d6a503acf12150fd1fbb24bd77f963bd6787538916e18d9c9 WatchSource:0}: Error finding container 256ffb7640772b7d6a503acf12150fd1fbb24bd77f963bd6787538916e18d9c9: Status 404 returned error can't find the container with id 256ffb7640772b7d6a503acf12150fd1fbb24bd77f963bd6787538916e18d9c9
Nov 02 13:12:03.765128 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:12:03.765104    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-bbb7d0-84c974d75-jlqqq"]
Nov 02 13:12:03.766289 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:12:03.766267    4559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 13:12:04.650919 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:12:04.650835    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bbb7d0-84c974d75-jlqqq" event={"ID":"4436e381-1572-416e-ad03-2805362dde64","Type":"ContainerStarted","Data":"6ebaad0b6c4564b1bb0480b10003ff6f0898fbd29c966e61317a5ec30282d8bb"}
Nov 02 13:12:04.650919 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:12:04.650896    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bbb7d0-84c974d75-jlqqq" event={"ID":"4436e381-1572-416e-ad03-2805362dde64","Type":"ContainerStarted","Data":"256ffb7640772b7d6a503acf12150fd1fbb24bd77f963bd6787538916e18d9c9"}
Nov 02 13:12:04.651410 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:12:04.650954    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bbb7d0-84c974d75-jlqqq"
Nov 02 13:12:04.700967 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:12:04.700908    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bbb7d0-84c974d75-jlqqq" podStartSLOduration=2.248878702 podStartE2EDuration="2.700890045s" totalImagesPullingTime="452.011343ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:12:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:12:04.699479846 +0000 UTC m=+2720.954878943" watchObservedRunningTime="2026-11-02 13:12:04.700890045 +0000 UTC m=+2720.956289140"
Nov 02 13:12:05.118210 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:12:05.118174    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-579d767848-hbb8t" podUID="e54fd919-3dc6-4b33-8ad6-67d31a2bf664" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 13:12:10.663772 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:12:10.663727    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bbb7d0-84c974d75-jlqqq"
Nov 02 13:12:15.087354 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:12:15.087316    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqcmj" containerName="extractor"
Nov 02 13:12:15.118714 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:12:15.118682    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-579d767848-hbb8t" podUID="e54fd919-3dc6-4b33-8ad6-67d31a2bf664" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 13:12:25.119005 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:12:25.118917    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-579d767848-hbb8t" podUID="e54fd919-3dc6-4b33-8ad6-67d31a2bf664" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 13:12:35.118785 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:12:35.118746    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-579d767848-hbb8t" podUID="e54fd919-3dc6-4b33-8ad6-67d31a2bf664" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 13:12:36.024665 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:12:36.020718    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"]
Nov 02 13:12:43.892168 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:12:43.892127    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-579d767848-hbb8t" podUID="e54fd919-3dc6-4b33-8ad6-67d31a2bf664" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 13:12:45.094165 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:12:45.094129    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqcmj" containerName="extractor"
Nov 02 13:12:53.892343 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:12:53.892308    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-579d767848-hbb8t" podUID="e54fd919-3dc6-4b33-8ad6-67d31a2bf664" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 13:12:57.160338 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:12:57.160301    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-579d767848-hbb8t"]
Nov 02 13:12:57.161907 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:12:57.161870    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-579d767848-hbb8t" podUID="e54fd919-3dc6-4b33-8ad6-67d31a2bf664" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 13:12:59.459853 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:12:59.459819    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-j7rnm"]
Nov 02 13:12:59.464444 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:12:59.464424    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-j7rnm"
Nov 02 13:12:59.485126 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:12:59.485105    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-6563ed-openshift-default-dockercfg-dh4qv\""
Nov 02 13:12:59.496784 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:12:59.496755    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85ec8b4f-746f-4d0c-9560-7a79f0c64ef6-istio-envoy\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-j7rnm\" (UID: \"85ec8b4f-746f-4d0c-9560-7a79f0c64ef6\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-j7rnm"
Nov 02 13:12:59.496926 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:12:59.496818    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85ec8b4f-746f-4d0c-9560-7a79f0c64ef6-istio-data\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-j7rnm\" (UID: \"85ec8b4f-746f-4d0c-9560-7a79f0c64ef6\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-j7rnm"
Nov 02 13:12:59.496991 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:12:59.496926    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpwfm\" (UniqueName: \"kubernetes.io/projected/85ec8b4f-746f-4d0c-9560-7a79f0c64ef6-kube-api-access-fpwfm\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-j7rnm\" (UID: \"85ec8b4f-746f-4d0c-9560-7a79f0c64ef6\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-j7rnm"
Nov 02 13:12:59.497068 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:12:59.497046    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85ec8b4f-746f-4d0c-9560-7a79f0c64ef6-istio-podinfo\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-j7rnm\" (UID: \"85ec8b4f-746f-4d0c-9560-7a79f0c64ef6\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-j7rnm"
Nov 02 13:12:59.497134 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:12:59.497099    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85ec8b4f-746f-4d0c-9560-7a79f0c64ef6-istio-token\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-j7rnm\" (UID: \"85ec8b4f-746f-4d0c-9560-7a79f0c64ef6\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-j7rnm"
Nov 02 13:12:59.497201 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:12:59.497163    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85ec8b4f-746f-4d0c-9560-7a79f0c64ef6-workload-socket\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-j7rnm\" (UID: \"85ec8b4f-746f-4d0c-9560-7a79f0c64ef6\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-j7rnm"
Nov 02 13:12:59.497274 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:12:59.497257    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85ec8b4f-746f-4d0c-9560-7a79f0c64ef6-credential-socket\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-j7rnm\" (UID: \"85ec8b4f-746f-4d0c-9560-7a79f0c64ef6\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-j7rnm"
Nov 02 13:12:59.497527 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:12:59.497502    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85ec8b4f-746f-4d0c-9560-7a79f0c64ef6-istiod-ca-cert\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-j7rnm\" (UID: \"85ec8b4f-746f-4d0c-9560-7a79f0c64ef6\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-j7rnm"
Nov 02 13:12:59.497700 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:12:59.497553    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85ec8b4f-746f-4d0c-9560-7a79f0c64ef6-workload-certs\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-j7rnm\" (UID: \"85ec8b4f-746f-4d0c-9560-7a79f0c64ef6\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-j7rnm"
Nov 02 13:12:59.503287 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:12:59.503263    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-j7rnm"]
Nov 02 13:12:59.702687 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:12:59.702626    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85ec8b4f-746f-4d0c-9560-7a79f0c64ef6-istiod-ca-cert\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-j7rnm\" (UID: \"85ec8b4f-746f-4d0c-9560-7a79f0c64ef6\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-j7rnm"
Nov 02 13:12:59.702969 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:12:59.702944    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85ec8b4f-746f-4d0c-9560-7a79f0c64ef6-workload-certs\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-j7rnm\" (UID: \"85ec8b4f-746f-4d0c-9560-7a79f0c64ef6\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-j7rnm"
Nov 02 13:12:59.703173 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:12:59.703153    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85ec8b4f-746f-4d0c-9560-7a79f0c64ef6-istio-envoy\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-j7rnm\" (UID: \"85ec8b4f-746f-4d0c-9560-7a79f0c64ef6\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-j7rnm"
Nov 02 13:12:59.703317 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:12:59.703280    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85ec8b4f-746f-4d0c-9560-7a79f0c64ef6-istio-data\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-j7rnm\" (UID: \"85ec8b4f-746f-4d0c-9560-7a79f0c64ef6\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-j7rnm"
Nov 02 13:12:59.703459 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:12:59.703439    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpwfm\" (UniqueName: \"kubernetes.io/projected/85ec8b4f-746f-4d0c-9560-7a79f0c64ef6-kube-api-access-fpwfm\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-j7rnm\" (UID: \"85ec8b4f-746f-4d0c-9560-7a79f0c64ef6\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-j7rnm"
Nov 02 13:12:59.703701 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:12:59.703680    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85ec8b4f-746f-4d0c-9560-7a79f0c64ef6-istio-podinfo\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-j7rnm\" (UID: \"85ec8b4f-746f-4d0c-9560-7a79f0c64ef6\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-j7rnm"
Nov 02 13:12:59.703850 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:12:59.703833    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85ec8b4f-746f-4d0c-9560-7a79f0c64ef6-istio-token\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-j7rnm\" (UID: \"85ec8b4f-746f-4d0c-9560-7a79f0c64ef6\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-j7rnm"
Nov 02 13:12:59.704020 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:12:59.704004    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85ec8b4f-746f-4d0c-9560-7a79f0c64ef6-workload-socket\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-j7rnm\" (UID: \"85ec8b4f-746f-4d0c-9560-7a79f0c64ef6\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-j7rnm"
Nov 02 13:12:59.704199 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:12:59.704184    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85ec8b4f-746f-4d0c-9560-7a79f0c64ef6-credential-socket\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-j7rnm\" (UID: \"85ec8b4f-746f-4d0c-9560-7a79f0c64ef6\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-j7rnm"
Nov 02 13:12:59.718147 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:12:59.717457    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85ec8b4f-746f-4d0c-9560-7a79f0c64ef6-workload-socket\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-j7rnm\" (UID: \"85ec8b4f-746f-4d0c-9560-7a79f0c64ef6\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-j7rnm"
Nov 02 13:12:59.824665 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:12:59.766220    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85ec8b4f-746f-4d0c-9560-7a79f0c64ef6-istiod-ca-cert\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-j7rnm\" (UID: \"85ec8b4f-746f-4d0c-9560-7a79f0c64ef6\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-j7rnm"
Nov 02 13:12:59.824665 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:12:59.780097    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85ec8b4f-746f-4d0c-9560-7a79f0c64ef6-istio-envoy\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-j7rnm\" (UID: \"85ec8b4f-746f-4d0c-9560-7a79f0c64ef6\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-j7rnm"
Nov 02 13:12:59.824665 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:12:59.780754    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85ec8b4f-746f-4d0c-9560-7a79f0c64ef6-credential-socket\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-j7rnm\" (UID: \"85ec8b4f-746f-4d0c-9560-7a79f0c64ef6\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-j7rnm"
Nov 02 13:12:59.824665 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:12:59.781205    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85ec8b4f-746f-4d0c-9560-7a79f0c64ef6-workload-certs\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-j7rnm\" (UID: \"85ec8b4f-746f-4d0c-9560-7a79f0c64ef6\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-j7rnm"
Nov 02 13:12:59.824665 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:12:59.781785    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85ec8b4f-746f-4d0c-9560-7a79f0c64ef6-istio-data\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-j7rnm\" (UID: \"85ec8b4f-746f-4d0c-9560-7a79f0c64ef6\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-j7rnm"
Nov 02 13:12:59.846703 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:12:59.846553    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85ec8b4f-746f-4d0c-9560-7a79f0c64ef6-istio-token\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-j7rnm\" (UID: \"85ec8b4f-746f-4d0c-9560-7a79f0c64ef6\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-j7rnm"
Nov 02 13:12:59.868072 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:12:59.868027    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85ec8b4f-746f-4d0c-9560-7a79f0c64ef6-istio-podinfo\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-j7rnm\" (UID: \"85ec8b4f-746f-4d0c-9560-7a79f0c64ef6\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-j7rnm"
Nov 02 13:12:59.883799 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:12:59.883767    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpwfm\" (UniqueName: \"kubernetes.io/projected/85ec8b4f-746f-4d0c-9560-7a79f0c64ef6-kube-api-access-fpwfm\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-j7rnm\" (UID: \"85ec8b4f-746f-4d0c-9560-7a79f0c64ef6\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-j7rnm"
Nov 02 13:13:00.078490 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:13:00.078410    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-j7rnm"
Nov 02 13:13:00.486989 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:13:00.476543    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 13:13:00.486989 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:13:00.480920    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-j7rnm"]
Nov 02 13:13:00.486989 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:13:00.484885    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 13:13:00.486989 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:13:00.484964    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 13:13:01.464557 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:13:01.464511    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-j7rnm" event={"ID":"85ec8b4f-746f-4d0c-9560-7a79f0c64ef6","Type":"ContainerStarted","Data":"07fc6ecc53d115dd90560bf90a6b96e92920642b83d1fb307440c71e204217de"}
Nov 02 13:13:01.466009 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:13:01.465980    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-j7rnm" event={"ID":"85ec8b4f-746f-4d0c-9560-7a79f0c64ef6","Type":"ContainerStarted","Data":"161f787b1c7a84e4c93d97b63575af60d73efbf09faf5293b33884c10646b849"}
Nov 02 13:13:01.530957 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:13:01.530902    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-j7rnm" podStartSLOduration=2.5308871269999997 podStartE2EDuration="2.530887127s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:12:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:13:01.526948948 +0000 UTC m=+2777.782348066" watchObservedRunningTime="2026-11-02 13:13:01.530887127 +0000 UTC m=+2777.786286221"
Nov 02 13:13:02.078979 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:13:02.078948    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-j7rnm"
Nov 02 13:13:02.086015 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:13:02.085985    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-j7rnm"
Nov 02 13:13:02.160668 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:13:02.160617    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-579d767848-hbb8t" podUID="e54fd919-3dc6-4b33-8ad6-67d31a2bf664" containerName="payload-processing" containerID="cri-o://832ead65a1c6e6f99d604db96504b8332789d4356ec8401e898659dad978cd37" gracePeriod=25
Nov 02 13:13:02.442687 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:13:02.442661    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-579d767848-hbb8t"
Nov 02 13:13:02.473422 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:13:02.473395    4559 generic.go:356] "Generic (PLEG): container finished" podID="e54fd919-3dc6-4b33-8ad6-67d31a2bf664" containerID="832ead65a1c6e6f99d604db96504b8332789d4356ec8401e898659dad978cd37" exitCode=0
Nov 02 13:13:02.473549 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:13:02.473442    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-579d767848-hbb8t"
Nov 02 13:13:02.473549 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:13:02.473475    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-579d767848-hbb8t" event={"ID":"e54fd919-3dc6-4b33-8ad6-67d31a2bf664","Type":"ContainerDied","Data":"832ead65a1c6e6f99d604db96504b8332789d4356ec8401e898659dad978cd37"}
Nov 02 13:13:02.473549 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:13:02.473523    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-579d767848-hbb8t" event={"ID":"e54fd919-3dc6-4b33-8ad6-67d31a2bf664","Type":"ContainerDied","Data":"84549395add54355cf5e8ec9d8980a079fc015013f390f060466cdd554458b29"}
Nov 02 13:13:02.473549 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:13:02.473548    4559 scope.go:122] "RemoveContainer" containerID="832ead65a1c6e6f99d604db96504b8332789d4356ec8401e898659dad978cd37"
Nov 02 13:13:02.473913 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:13:02.473890    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-j7rnm"
Nov 02 13:13:02.474985 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:13:02.474966    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-j7rnm"
Nov 02 13:13:02.483351 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:13:02.483335    4559 scope.go:122] "RemoveContainer" containerID="832ead65a1c6e6f99d604db96504b8332789d4356ec8401e898659dad978cd37"
Nov 02 13:13:02.483584 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:13:02.483565    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"832ead65a1c6e6f99d604db96504b8332789d4356ec8401e898659dad978cd37\": container with ID starting with 832ead65a1c6e6f99d604db96504b8332789d4356ec8401e898659dad978cd37 not found: ID does not exist" containerID="832ead65a1c6e6f99d604db96504b8332789d4356ec8401e898659dad978cd37"
Nov 02 13:13:02.483683 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:13:02.483591    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"832ead65a1c6e6f99d604db96504b8332789d4356ec8401e898659dad978cd37"} err="failed to get container status \"832ead65a1c6e6f99d604db96504b8332789d4356ec8401e898659dad978cd37\": rpc error: code = NotFound desc = could not find container \"832ead65a1c6e6f99d604db96504b8332789d4356ec8401e898659dad978cd37\": container with ID starting with 832ead65a1c6e6f99d604db96504b8332789d4356ec8401e898659dad978cd37 not found: ID does not exist"
Nov 02 13:13:02.546071 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:13:02.546038    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e54fd919-3dc6-4b33-8ad6-67d31a2bf664-kube-api-access-m8dhf\" (UniqueName: \"kubernetes.io/projected/e54fd919-3dc6-4b33-8ad6-67d31a2bf664-kube-api-access-m8dhf\") pod \"e54fd919-3dc6-4b33-8ad6-67d31a2bf664\" (UID: \"e54fd919-3dc6-4b33-8ad6-67d31a2bf664\") "
Nov 02 13:13:02.546450 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:13:02.546167    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e54fd919-3dc6-4b33-8ad6-67d31a2bf664-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e54fd919-3dc6-4b33-8ad6-67d31a2bf664-routing-overlay-volume\") pod \"e54fd919-3dc6-4b33-8ad6-67d31a2bf664\" (UID: \"e54fd919-3dc6-4b33-8ad6-67d31a2bf664\") "
Nov 02 13:13:02.546450 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:13:02.546238    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e54fd919-3dc6-4b33-8ad6-67d31a2bf664-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e54fd919-3dc6-4b33-8ad6-67d31a2bf664-plugins-config-volume\") pod \"e54fd919-3dc6-4b33-8ad6-67d31a2bf664\" (UID: \"e54fd919-3dc6-4b33-8ad6-67d31a2bf664\") "
Nov 02 13:13:02.546563 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:13:02.546535    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e54fd919-3dc6-4b33-8ad6-67d31a2bf664-provider-credentials\" (UniqueName: \"kubernetes.io/projected/e54fd919-3dc6-4b33-8ad6-67d31a2bf664-provider-credentials\") pod \"e54fd919-3dc6-4b33-8ad6-67d31a2bf664\" (UID: \"e54fd919-3dc6-4b33-8ad6-67d31a2bf664\") "
Nov 02 13:13:02.546626 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:13:02.546581    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e54fd919-3dc6-4b33-8ad6-67d31a2bf664-routing-overlay-volume" pod "e54fd919-3dc6-4b33-8ad6-67d31a2bf664" (UID: "e54fd919-3dc6-4b33-8ad6-67d31a2bf664"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:13:02.546626 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:13:02.546585    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e54fd919-3dc6-4b33-8ad6-67d31a2bf664-plugins-config-volume" pod "e54fd919-3dc6-4b33-8ad6-67d31a2bf664" (UID: "e54fd919-3dc6-4b33-8ad6-67d31a2bf664"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:13:02.547354 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:13:02.547331    4559 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e54fd919-3dc6-4b33-8ad6-67d31a2bf664-routing-overlay-volume\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:13:02.547462 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:13:02.547359    4559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e54fd919-3dc6-4b33-8ad6-67d31a2bf664-plugins-config-volume\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:13:02.548723 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:13:02.548698    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e54fd919-3dc6-4b33-8ad6-67d31a2bf664-kube-api-access-m8dhf" pod "e54fd919-3dc6-4b33-8ad6-67d31a2bf664" (UID: "e54fd919-3dc6-4b33-8ad6-67d31a2bf664"). InnerVolumeSpecName "kube-api-access-m8dhf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:13:02.549091 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:13:02.549068    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e54fd919-3dc6-4b33-8ad6-67d31a2bf664-provider-credentials" pod "e54fd919-3dc6-4b33-8ad6-67d31a2bf664" (UID: "e54fd919-3dc6-4b33-8ad6-67d31a2bf664"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:13:02.648726 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:13:02.648650    4559 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e54fd919-3dc6-4b33-8ad6-67d31a2bf664-provider-credentials\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:13:02.648726 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:13:02.648678    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8dhf\" (UniqueName: \"kubernetes.io/projected/e54fd919-3dc6-4b33-8ad6-67d31a2bf664-kube-api-access-m8dhf\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:13:02.824998 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:13:02.824965    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-579d767848-hbb8t"]
Nov 02 13:13:02.828705 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:13:02.828678    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-579d767848-hbb8t"]
Nov 02 13:13:03.895283 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:13:03.895253    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e54fd919-3dc6-4b33-8ad6-67d31a2bf664" path="/var/lib/kubelet/pods/e54fd919-3dc6-4b33-8ad6-67d31a2bf664/volumes"
Nov 02 13:13:15.086686 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:13:15.086620    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqcmj" containerName="extractor"
Nov 02 13:13:25.083387 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:13:25.083350    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-5f8899-59548dd96c-5p585"]
Nov 02 13:13:25.084522 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:13:25.084496    4559 cpu_manager.go:414] "containerMap: removing container" podUID="e54fd919-3dc6-4b33-8ad6-67d31a2bf664" containerName="payload-processing"
Nov 02 13:13:25.084522 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:13:25.084519    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e54fd919-3dc6-4b33-8ad6-67d31a2bf664" containerName="payload-processing"
Nov 02 13:13:25.084755 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:13:25.084674    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="e54fd919-3dc6-4b33-8ad6-67d31a2bf664" containerName="payload-processing"
Nov 02 13:13:25.089682 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:13:25.089661    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5f8899-59548dd96c-5p585"
Nov 02 13:13:25.118900 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:13:25.118866    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-5f8899-59548dd96c-5p585"]
Nov 02 13:13:25.125115 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:13:25.125091    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-5f8899\""
Nov 02 13:13:25.125398 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:13:25.125381    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-5f8899-dockercfg-72pwx\""
Nov 02 13:13:25.129387 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:13:25.129358    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8qkp\" (UniqueName: \"kubernetes.io/projected/da6dd2b6-1e1c-4128-a988-cf86451095e1-kube-api-access-s8qkp\") pod \"payload-pre-processing-e2e-inf-a-5f8899-59548dd96c-5p585\" (UID: \"da6dd2b6-1e1c-4128-a988-cf86451095e1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5f8899-59548dd96c-5p585"
Nov 02 13:13:25.129788 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:13:25.129765    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da6dd2b6-1e1c-4128-a988-cf86451095e1-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-5f8899-59548dd96c-5p585\" (UID: \"da6dd2b6-1e1c-4128-a988-cf86451095e1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5f8899-59548dd96c-5p585"
Nov 02 13:13:25.325667 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:13:25.291791    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8qkp\" (UniqueName: \"kubernetes.io/projected/da6dd2b6-1e1c-4128-a988-cf86451095e1-kube-api-access-s8qkp\") pod \"payload-pre-processing-e2e-inf-a-5f8899-59548dd96c-5p585\" (UID: \"da6dd2b6-1e1c-4128-a988-cf86451095e1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5f8899-59548dd96c-5p585"
Nov 02 13:13:25.338713 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:13:25.326021    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da6dd2b6-1e1c-4128-a988-cf86451095e1-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-5f8899-59548dd96c-5p585\" (UID: \"da6dd2b6-1e1c-4128-a988-cf86451095e1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5f8899-59548dd96c-5p585"
Nov 02 13:13:25.375894 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:13:25.375852    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da6dd2b6-1e1c-4128-a988-cf86451095e1-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-5f8899-59548dd96c-5p585\" (UID: \"da6dd2b6-1e1c-4128-a988-cf86451095e1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5f8899-59548dd96c-5p585"
Nov 02 13:13:25.542623 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:13:25.542047    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8qkp\" (UniqueName: \"kubernetes.io/projected/da6dd2b6-1e1c-4128-a988-cf86451095e1-kube-api-access-s8qkp\") pod \"payload-pre-processing-e2e-inf-a-5f8899-59548dd96c-5p585\" (UID: \"da6dd2b6-1e1c-4128-a988-cf86451095e1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5f8899-59548dd96c-5p585"
Nov 02 13:13:25.728846 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:13:25.728815    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5f8899-59548dd96c-5p585"
Nov 02 13:13:26.054822 ip-10-0-128-132 kubenswrapper[4559]: W1102 13:13:26.054782    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda6dd2b6_1e1c_4128_a988_cf86451095e1.slice/crio-085fffe8b6ded69c4b42668a1f0862be37cb6c2efbc60b24eacdec6b261b5864 WatchSource:0}: Error finding container 085fffe8b6ded69c4b42668a1f0862be37cb6c2efbc60b24eacdec6b261b5864: Status 404 returned error can't find the container with id 085fffe8b6ded69c4b42668a1f0862be37cb6c2efbc60b24eacdec6b261b5864
Nov 02 13:13:26.057277 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:13:26.057246    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-5f8899-59548dd96c-5p585"]
Nov 02 13:13:27.021129 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:13:27.021081    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5f8899-59548dd96c-5p585" event={"ID":"da6dd2b6-1e1c-4128-a988-cf86451095e1","Type":"ContainerStarted","Data":"31e8306d86f94bf7ad0f16e6dca4a5bc07d2df8c306d0ec8455f4152832b2d61"}
Nov 02 13:13:27.021595 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:13:27.021148    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5f8899-59548dd96c-5p585" event={"ID":"da6dd2b6-1e1c-4128-a988-cf86451095e1","Type":"ContainerStarted","Data":"085fffe8b6ded69c4b42668a1f0862be37cb6c2efbc60b24eacdec6b261b5864"}
Nov 02 13:13:27.021595 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:13:27.021175    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5f8899-59548dd96c-5p585"
Nov 02 13:13:27.066681 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:13:27.066617    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5f8899-59548dd96c-5p585" podStartSLOduration=2.066603859 podStartE2EDuration="2.066603859s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:13:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:13:27.062621783 +0000 UTC m=+2803.318020878" watchObservedRunningTime="2026-11-02 13:13:27.066603859 +0000 UTC m=+2803.322002953"
Nov 02 13:13:38.031996 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:13:38.031966    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5f8899-59548dd96c-5p585"
Nov 02 13:13:41.610089 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:13:41.610054    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-9139ec-74b46d9789-dtps2"]
Nov 02 13:13:41.615930 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:13:41.615907    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9139ec-74b46d9789-dtps2"
Nov 02 13:13:41.635842 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:13:41.635815    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-9139ec-74b46d9789-dtps2"]
Nov 02 13:13:41.637398 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:13:41.637371    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-9139ec-serving-cert\""
Nov 02 13:13:41.714761 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:13:41.714728    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cptfx\" (UniqueName: \"kubernetes.io/projected/004721a0-56e6-42d6-bca7-c9cb11cf0e5f-kube-api-access-cptfx\") pod \"maas-api-e2e-inf-b-9139ec-74b46d9789-dtps2\" (UID: \"004721a0-56e6-42d6-bca7-c9cb11cf0e5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9139ec-74b46d9789-dtps2"
Nov 02 13:13:41.714926 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:13:41.714878    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/004721a0-56e6-42d6-bca7-c9cb11cf0e5f-openshift-service-ca\") pod \"maas-api-e2e-inf-b-9139ec-74b46d9789-dtps2\" (UID: \"004721a0-56e6-42d6-bca7-c9cb11cf0e5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9139ec-74b46d9789-dtps2"
Nov 02 13:13:41.714926 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:13:41.714920    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/004721a0-56e6-42d6-bca7-c9cb11cf0e5f-maas-api-tls\") pod \"maas-api-e2e-inf-b-9139ec-74b46d9789-dtps2\" (UID: \"004721a0-56e6-42d6-bca7-c9cb11cf0e5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9139ec-74b46d9789-dtps2"
Nov 02 13:13:41.715027 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:13:41.715002    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/004721a0-56e6-42d6-bca7-c9cb11cf0e5f-metrics-certs\") pod \"maas-api-e2e-inf-b-9139ec-74b46d9789-dtps2\" (UID: \"004721a0-56e6-42d6-bca7-c9cb11cf0e5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9139ec-74b46d9789-dtps2"
Nov 02 13:13:41.824715 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:13:41.817872    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/004721a0-56e6-42d6-bca7-c9cb11cf0e5f-openshift-service-ca\") pod \"maas-api-e2e-inf-b-9139ec-74b46d9789-dtps2\" (UID: \"004721a0-56e6-42d6-bca7-c9cb11cf0e5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9139ec-74b46d9789-dtps2"
Nov 02 13:13:41.824715 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:13:41.817949    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/004721a0-56e6-42d6-bca7-c9cb11cf0e5f-maas-api-tls\") pod \"maas-api-e2e-inf-b-9139ec-74b46d9789-dtps2\" (UID: \"004721a0-56e6-42d6-bca7-c9cb11cf0e5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9139ec-74b46d9789-dtps2"
Nov 02 13:13:41.824715 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:13:41.817985    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/004721a0-56e6-42d6-bca7-c9cb11cf0e5f-metrics-certs\") pod \"maas-api-e2e-inf-b-9139ec-74b46d9789-dtps2\" (UID: \"004721a0-56e6-42d6-bca7-c9cb11cf0e5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9139ec-74b46d9789-dtps2"
Nov 02 13:13:41.824715 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:13:41.818271    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cptfx\" (UniqueName: \"kubernetes.io/projected/004721a0-56e6-42d6-bca7-c9cb11cf0e5f-kube-api-access-cptfx\") pod \"maas-api-e2e-inf-b-9139ec-74b46d9789-dtps2\" (UID: \"004721a0-56e6-42d6-bca7-c9cb11cf0e5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9139ec-74b46d9789-dtps2"
Nov 02 13:13:41.836040 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:13:41.835462    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/004721a0-56e6-42d6-bca7-c9cb11cf0e5f-openshift-service-ca\") pod \"maas-api-e2e-inf-b-9139ec-74b46d9789-dtps2\" (UID: \"004721a0-56e6-42d6-bca7-c9cb11cf0e5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9139ec-74b46d9789-dtps2"
Nov 02 13:13:41.839489 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:13:41.839454    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/004721a0-56e6-42d6-bca7-c9cb11cf0e5f-maas-api-tls\") pod \"maas-api-e2e-inf-b-9139ec-74b46d9789-dtps2\" (UID: \"004721a0-56e6-42d6-bca7-c9cb11cf0e5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9139ec-74b46d9789-dtps2"
Nov 02 13:13:41.843687 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:13:41.843627    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/004721a0-56e6-42d6-bca7-c9cb11cf0e5f-metrics-certs\") pod \"maas-api-e2e-inf-b-9139ec-74b46d9789-dtps2\" (UID: \"004721a0-56e6-42d6-bca7-c9cb11cf0e5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9139ec-74b46d9789-dtps2"
Nov 02 13:13:41.849079 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:13:41.849051    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cptfx\" (UniqueName: \"kubernetes.io/projected/004721a0-56e6-42d6-bca7-c9cb11cf0e5f-kube-api-access-cptfx\") pod \"maas-api-e2e-inf-b-9139ec-74b46d9789-dtps2\" (UID: \"004721a0-56e6-42d6-bca7-c9cb11cf0e5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9139ec-74b46d9789-dtps2"
Nov 02 13:13:41.938144 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:13:41.938112    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9139ec-74b46d9789-dtps2"
Nov 02 13:13:42.243164 ip-10-0-128-132 kubenswrapper[4559]: W1102 13:13:42.243137    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod004721a0_56e6_42d6_bca7_c9cb11cf0e5f.slice/crio-fd3f1a557070281ad707d6253fd4a9dcf4d418515688e63c228234bb98e9776d WatchSource:0}: Error finding container fd3f1a557070281ad707d6253fd4a9dcf4d418515688e63c228234bb98e9776d: Status 404 returned error can't find the container with id fd3f1a557070281ad707d6253fd4a9dcf4d418515688e63c228234bb98e9776d
Nov 02 13:13:42.243804 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:13:42.243702    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-9139ec-74b46d9789-dtps2"]
Nov 02 13:13:43.140277 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:13:43.140177    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9139ec-74b46d9789-dtps2" event={"ID":"004721a0-56e6-42d6-bca7-c9cb11cf0e5f","Type":"ContainerStarted","Data":"3097121f84d7c13654d14342b17b4b3beca591f7d7dad6e84bcf0045fe2c033e"}
Nov 02 13:13:43.140277 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:13:43.140243    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9139ec-74b46d9789-dtps2" event={"ID":"004721a0-56e6-42d6-bca7-c9cb11cf0e5f","Type":"ContainerStarted","Data":"fd3f1a557070281ad707d6253fd4a9dcf4d418515688e63c228234bb98e9776d"}
Nov 02 13:13:43.140760 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:13:43.140289    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9139ec-74b46d9789-dtps2"
Nov 02 13:13:43.188124 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:13:43.188066    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9139ec-74b46d9789-dtps2" podStartSLOduration=1.824787643 podStartE2EDuration="2.188052439s" totalImagesPullingTime="363.264796ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:13:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:13:43.18753732 +0000 UTC m=+2819.442936415" watchObservedRunningTime="2026-11-02 13:13:43.188052439 +0000 UTC m=+2819.443451533"
Nov 02 13:13:45.088207 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:13:45.088152    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqcmj" containerName="extractor"
Nov 02 13:13:49.156107 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:13:49.156075    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9139ec-74b46d9789-dtps2"
Nov 02 13:13:58.217748 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:13:58.217632    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"]
Nov 02 13:14:08.012710 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:14:08.012661    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-5f8899-69b5fb7989-zz4xv"]
Nov 02 13:14:08.018688 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:14:08.018667    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5f8899-69b5fb7989-zz4xv"
Nov 02 13:14:08.039903 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:14:08.039876    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-5f8899-69b5fb7989-zz4xv"]
Nov 02 13:14:08.040937 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:14:08.040912    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-5f8899-serving-cert\""
Nov 02 13:14:08.099543 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:14:08.099510    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqvcm\" (UniqueName: \"kubernetes.io/projected/ada43a30-3953-47ab-9bb7-d623b8025e13-kube-api-access-kqvcm\") pod \"maas-api-e2e-inf-a-5f8899-69b5fb7989-zz4xv\" (UID: \"ada43a30-3953-47ab-9bb7-d623b8025e13\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5f8899-69b5fb7989-zz4xv"
Nov 02 13:14:08.099745 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:14:08.099592    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ada43a30-3953-47ab-9bb7-d623b8025e13-metrics-certs\") pod \"maas-api-e2e-inf-a-5f8899-69b5fb7989-zz4xv\" (UID: \"ada43a30-3953-47ab-9bb7-d623b8025e13\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5f8899-69b5fb7989-zz4xv"
Nov 02 13:14:08.099745 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:14:08.099687    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ada43a30-3953-47ab-9bb7-d623b8025e13-openshift-service-ca\") pod \"maas-api-e2e-inf-a-5f8899-69b5fb7989-zz4xv\" (UID: \"ada43a30-3953-47ab-9bb7-d623b8025e13\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5f8899-69b5fb7989-zz4xv"
Nov 02 13:14:08.099916 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:14:08.099881    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ada43a30-3953-47ab-9bb7-d623b8025e13-maas-api-tls\") pod \"maas-api-e2e-inf-a-5f8899-69b5fb7989-zz4xv\" (UID: \"ada43a30-3953-47ab-9bb7-d623b8025e13\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5f8899-69b5fb7989-zz4xv"
Nov 02 13:14:08.201760 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:14:08.201733    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ada43a30-3953-47ab-9bb7-d623b8025e13-maas-api-tls\") pod \"maas-api-e2e-inf-a-5f8899-69b5fb7989-zz4xv\" (UID: \"ada43a30-3953-47ab-9bb7-d623b8025e13\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5f8899-69b5fb7989-zz4xv"
Nov 02 13:14:08.201913 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:14:08.201854    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqvcm\" (UniqueName: \"kubernetes.io/projected/ada43a30-3953-47ab-9bb7-d623b8025e13-kube-api-access-kqvcm\") pod \"maas-api-e2e-inf-a-5f8899-69b5fb7989-zz4xv\" (UID: \"ada43a30-3953-47ab-9bb7-d623b8025e13\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5f8899-69b5fb7989-zz4xv"
Nov 02 13:14:08.201913 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:14:08.201902    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ada43a30-3953-47ab-9bb7-d623b8025e13-metrics-certs\") pod \"maas-api-e2e-inf-a-5f8899-69b5fb7989-zz4xv\" (UID: \"ada43a30-3953-47ab-9bb7-d623b8025e13\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5f8899-69b5fb7989-zz4xv"
Nov 02 13:14:08.202009 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:14:08.201954    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ada43a30-3953-47ab-9bb7-d623b8025e13-openshift-service-ca\") pod \"maas-api-e2e-inf-a-5f8899-69b5fb7989-zz4xv\" (UID: \"ada43a30-3953-47ab-9bb7-d623b8025e13\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5f8899-69b5fb7989-zz4xv"
Nov 02 13:14:08.202588 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:14:08.202568    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ada43a30-3953-47ab-9bb7-d623b8025e13-openshift-service-ca\") pod \"maas-api-e2e-inf-a-5f8899-69b5fb7989-zz4xv\" (UID: \"ada43a30-3953-47ab-9bb7-d623b8025e13\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5f8899-69b5fb7989-zz4xv"
Nov 02 13:14:08.204160 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:14:08.204137    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ada43a30-3953-47ab-9bb7-d623b8025e13-maas-api-tls\") pod \"maas-api-e2e-inf-a-5f8899-69b5fb7989-zz4xv\" (UID: \"ada43a30-3953-47ab-9bb7-d623b8025e13\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5f8899-69b5fb7989-zz4xv"
Nov 02 13:14:08.204346 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:14:08.204327    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ada43a30-3953-47ab-9bb7-d623b8025e13-metrics-certs\") pod \"maas-api-e2e-inf-a-5f8899-69b5fb7989-zz4xv\" (UID: \"ada43a30-3953-47ab-9bb7-d623b8025e13\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5f8899-69b5fb7989-zz4xv"
Nov 02 13:14:08.225993 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:14:08.225967    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqvcm\" (UniqueName: \"kubernetes.io/projected/ada43a30-3953-47ab-9bb7-d623b8025e13-kube-api-access-kqvcm\") pod \"maas-api-e2e-inf-a-5f8899-69b5fb7989-zz4xv\" (UID: \"ada43a30-3953-47ab-9bb7-d623b8025e13\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5f8899-69b5fb7989-zz4xv"
Nov 02 13:14:08.338715 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:14:08.338573    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5f8899-69b5fb7989-zz4xv"
Nov 02 13:14:08.655721 ip-10-0-128-132 kubenswrapper[4559]: W1102 13:14:08.655695    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podada43a30_3953_47ab_9bb7_d623b8025e13.slice/crio-1c30bdc437bb1b01e6a2059cb710f9ae781e3bba96bfaf6698fe06bd8e24df69 WatchSource:0}: Error finding container 1c30bdc437bb1b01e6a2059cb710f9ae781e3bba96bfaf6698fe06bd8e24df69: Status 404 returned error can't find the container with id 1c30bdc437bb1b01e6a2059cb710f9ae781e3bba96bfaf6698fe06bd8e24df69
Nov 02 13:14:08.656927 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:14:08.656905    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-5f8899-69b5fb7989-zz4xv"]
Nov 02 13:14:09.444116 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:14:09.444035    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5f8899-69b5fb7989-zz4xv" event={"ID":"ada43a30-3953-47ab-9bb7-d623b8025e13","Type":"ContainerStarted","Data":"7e29594fb9ab173d1c431e732f04f7efa4ae94ebdb8ee39d4d7576be8b982318"}
Nov 02 13:14:09.444116 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:14:09.444086    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5f8899-69b5fb7989-zz4xv" event={"ID":"ada43a30-3953-47ab-9bb7-d623b8025e13","Type":"ContainerStarted","Data":"1c30bdc437bb1b01e6a2059cb710f9ae781e3bba96bfaf6698fe06bd8e24df69"}
Nov 02 13:14:09.444524 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:14:09.444168    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5f8899-69b5fb7989-zz4xv"
Nov 02 13:14:09.490666 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:14:09.490585    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5f8899-69b5fb7989-zz4xv" podStartSLOduration=1.982846525 podStartE2EDuration="2.490571153s" totalImagesPullingTime="507.724628ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:14:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:14:09.486972433 +0000 UTC m=+2845.742371528" watchObservedRunningTime="2026-11-02 13:14:09.490571153 +0000 UTC m=+2845.745970247"
Nov 02 13:14:15.089795 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:14:15.089757    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqcmj" containerName="extractor"
Nov 02 13:14:15.456766 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:14:15.456739    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5f8899-69b5fb7989-zz4xv"
Nov 02 13:14:35.179099 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:14:35.179055    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"]
Nov 02 13:14:45.087580 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:14:45.087503    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqcmj" containerName="extractor"
Nov 02 13:14:48.223733 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:14:48.223697    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-bmcxg"]
Nov 02 13:14:48.228679 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:14:48.228650    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-bmcxg"
Nov 02 13:14:48.256341 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:14:48.256313    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-5f8899-openshift-default-dockercfg-rrzwf\""
Nov 02 13:14:48.261368 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:14:48.261348    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-bmcxg"]
Nov 02 13:14:48.372665 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:14:48.367304    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/185fcba5-401f-4a43-bad1-f82f845221b8-istiod-ca-cert\") pod \"e2e-inf-a-5f8899-openshift-default-67cc5d8789-bmcxg\" (UID: \"185fcba5-401f-4a43-bad1-f82f845221b8\") " pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-bmcxg"
Nov 02 13:14:48.372665 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:14:48.367357    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/185fcba5-401f-4a43-bad1-f82f845221b8-credential-socket\") pod \"e2e-inf-a-5f8899-openshift-default-67cc5d8789-bmcxg\" (UID: \"185fcba5-401f-4a43-bad1-f82f845221b8\") " pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-bmcxg"
Nov 02 13:14:48.372665 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:14:48.367431    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/185fcba5-401f-4a43-bad1-f82f845221b8-istio-podinfo\") pod \"e2e-inf-a-5f8899-openshift-default-67cc5d8789-bmcxg\" (UID: \"185fcba5-401f-4a43-bad1-f82f845221b8\") " pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-bmcxg"
Nov 02 13:14:48.372665 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:14:48.367472    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/185fcba5-401f-4a43-bad1-f82f845221b8-istio-token\") pod \"e2e-inf-a-5f8899-openshift-default-67cc5d8789-bmcxg\" (UID: \"185fcba5-401f-4a43-bad1-f82f845221b8\") " pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-bmcxg"
Nov 02 13:14:48.372665 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:14:48.367516    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/185fcba5-401f-4a43-bad1-f82f845221b8-workload-certs\") pod \"e2e-inf-a-5f8899-openshift-default-67cc5d8789-bmcxg\" (UID: \"185fcba5-401f-4a43-bad1-f82f845221b8\") " pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-bmcxg"
Nov 02 13:14:48.372665 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:14:48.367609    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbzgw\" (UniqueName: \"kubernetes.io/projected/185fcba5-401f-4a43-bad1-f82f845221b8-kube-api-access-lbzgw\") pod \"e2e-inf-a-5f8899-openshift-default-67cc5d8789-bmcxg\" (UID: \"185fcba5-401f-4a43-bad1-f82f845221b8\") " pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-bmcxg"
Nov 02 13:14:48.372665 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:14:48.367658    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/185fcba5-401f-4a43-bad1-f82f845221b8-istio-data\") pod \"e2e-inf-a-5f8899-openshift-default-67cc5d8789-bmcxg\" (UID: \"185fcba5-401f-4a43-bad1-f82f845221b8\") " pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-bmcxg"
Nov 02 13:14:48.372665 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:14:48.367724    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/185fcba5-401f-4a43-bad1-f82f845221b8-workload-socket\") pod \"e2e-inf-a-5f8899-openshift-default-67cc5d8789-bmcxg\" (UID: \"185fcba5-401f-4a43-bad1-f82f845221b8\") " pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-bmcxg"
Nov 02 13:14:48.372665 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:14:48.367801    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/185fcba5-401f-4a43-bad1-f82f845221b8-istio-envoy\") pod \"e2e-inf-a-5f8899-openshift-default-67cc5d8789-bmcxg\" (UID: \"185fcba5-401f-4a43-bad1-f82f845221b8\") " pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-bmcxg"
Nov 02 13:14:48.478710 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:14:48.477828    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/185fcba5-401f-4a43-bad1-f82f845221b8-istiod-ca-cert\") pod \"e2e-inf-a-5f8899-openshift-default-67cc5d8789-bmcxg\" (UID: \"185fcba5-401f-4a43-bad1-f82f845221b8\") " pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-bmcxg"
Nov 02 13:14:48.478710 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:14:48.477880    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/185fcba5-401f-4a43-bad1-f82f845221b8-credential-socket\") pod \"e2e-inf-a-5f8899-openshift-default-67cc5d8789-bmcxg\" (UID: \"185fcba5-401f-4a43-bad1-f82f845221b8\") " pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-bmcxg"
Nov 02 13:14:48.478710 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:14:48.477945    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/185fcba5-401f-4a43-bad1-f82f845221b8-istio-podinfo\") pod \"e2e-inf-a-5f8899-openshift-default-67cc5d8789-bmcxg\" (UID: \"185fcba5-401f-4a43-bad1-f82f845221b8\") " pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-bmcxg"
Nov 02 13:14:48.478710 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:14:48.477990    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/185fcba5-401f-4a43-bad1-f82f845221b8-istio-token\") pod \"e2e-inf-a-5f8899-openshift-default-67cc5d8789-bmcxg\" (UID: \"185fcba5-401f-4a43-bad1-f82f845221b8\") " pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-bmcxg"
Nov 02 13:14:48.478710 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:14:48.478029    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/185fcba5-401f-4a43-bad1-f82f845221b8-workload-certs\") pod \"e2e-inf-a-5f8899-openshift-default-67cc5d8789-bmcxg\" (UID: \"185fcba5-401f-4a43-bad1-f82f845221b8\") " pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-bmcxg"
Nov 02 13:14:48.478710 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:14:48.478114    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbzgw\" (UniqueName: \"kubernetes.io/projected/185fcba5-401f-4a43-bad1-f82f845221b8-kube-api-access-lbzgw\") pod \"e2e-inf-a-5f8899-openshift-default-67cc5d8789-bmcxg\" (UID: \"185fcba5-401f-4a43-bad1-f82f845221b8\") " pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-bmcxg"
Nov 02 13:14:48.478710 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:14:48.478145    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/185fcba5-401f-4a43-bad1-f82f845221b8-istio-data\") pod \"e2e-inf-a-5f8899-openshift-default-67cc5d8789-bmcxg\" (UID: \"185fcba5-401f-4a43-bad1-f82f845221b8\") " pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-bmcxg"
Nov 02 13:14:48.478710 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:14:48.478192    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/185fcba5-401f-4a43-bad1-f82f845221b8-workload-socket\") pod \"e2e-inf-a-5f8899-openshift-default-67cc5d8789-bmcxg\" (UID: \"185fcba5-401f-4a43-bad1-f82f845221b8\") " pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-bmcxg"
Nov 02 13:14:48.478710 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:14:48.478253    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/185fcba5-401f-4a43-bad1-f82f845221b8-istio-envoy\") pod \"e2e-inf-a-5f8899-openshift-default-67cc5d8789-bmcxg\" (UID: \"185fcba5-401f-4a43-bad1-f82f845221b8\") " pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-bmcxg"
Nov 02 13:14:48.507675 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:14:48.506485    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/185fcba5-401f-4a43-bad1-f82f845221b8-credential-socket\") pod \"e2e-inf-a-5f8899-openshift-default-67cc5d8789-bmcxg\" (UID: \"185fcba5-401f-4a43-bad1-f82f845221b8\") " pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-bmcxg"
Nov 02 13:14:48.580430 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:14:48.523701    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/185fcba5-401f-4a43-bad1-f82f845221b8-workload-certs\") pod \"e2e-inf-a-5f8899-openshift-default-67cc5d8789-bmcxg\" (UID: \"185fcba5-401f-4a43-bad1-f82f845221b8\") " pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-bmcxg"
Nov 02 13:14:48.580430 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:14:48.524523    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/185fcba5-401f-4a43-bad1-f82f845221b8-istio-data\") pod \"e2e-inf-a-5f8899-openshift-default-67cc5d8789-bmcxg\" (UID: \"185fcba5-401f-4a43-bad1-f82f845221b8\") " pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-bmcxg"
Nov 02 13:14:48.600624 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:14:48.600562    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/185fcba5-401f-4a43-bad1-f82f845221b8-workload-socket\") pod \"e2e-inf-a-5f8899-openshift-default-67cc5d8789-bmcxg\" (UID: \"185fcba5-401f-4a43-bad1-f82f845221b8\") " pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-bmcxg"
Nov 02 13:14:48.620724 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:14:48.620525    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/185fcba5-401f-4a43-bad1-f82f845221b8-istio-envoy\") pod \"e2e-inf-a-5f8899-openshift-default-67cc5d8789-bmcxg\" (UID: \"185fcba5-401f-4a43-bad1-f82f845221b8\") " pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-bmcxg"
Nov 02 13:14:48.620941 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:14:48.620748    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/185fcba5-401f-4a43-bad1-f82f845221b8-istiod-ca-cert\") pod \"e2e-inf-a-5f8899-openshift-default-67cc5d8789-bmcxg\" (UID: \"185fcba5-401f-4a43-bad1-f82f845221b8\") " pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-bmcxg"
Nov 02 13:14:48.639079 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:14:48.638705    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/185fcba5-401f-4a43-bad1-f82f845221b8-istio-podinfo\") pod \"e2e-inf-a-5f8899-openshift-default-67cc5d8789-bmcxg\" (UID: \"185fcba5-401f-4a43-bad1-f82f845221b8\") " pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-bmcxg"
Nov 02 13:14:48.682568 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:14:48.681915    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/185fcba5-401f-4a43-bad1-f82f845221b8-istio-token\") pod \"e2e-inf-a-5f8899-openshift-default-67cc5d8789-bmcxg\" (UID: \"185fcba5-401f-4a43-bad1-f82f845221b8\") " pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-bmcxg"
Nov 02 13:14:48.692776 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:14:48.692750    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbzgw\" (UniqueName: \"kubernetes.io/projected/185fcba5-401f-4a43-bad1-f82f845221b8-kube-api-access-lbzgw\") pod \"e2e-inf-a-5f8899-openshift-default-67cc5d8789-bmcxg\" (UID: \"185fcba5-401f-4a43-bad1-f82f845221b8\") " pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-bmcxg"
Nov 02 13:14:48.842920 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:14:48.842843    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-bmcxg"
Nov 02 13:14:49.178997 ip-10-0-128-132 kubenswrapper[4559]: W1102 13:14:49.178967    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod185fcba5_401f_4a43_bad1_f82f845221b8.slice/crio-f00875f9f6cdf35bf678f81f0754b90a53b9caf8c5c95855266cd52e64f71f45 WatchSource:0}: Error finding container f00875f9f6cdf35bf678f81f0754b90a53b9caf8c5c95855266cd52e64f71f45: Status 404 returned error can't find the container with id f00875f9f6cdf35bf678f81f0754b90a53b9caf8c5c95855266cd52e64f71f45
Nov 02 13:14:49.179134 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:14:49.179073    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-bmcxg"]
Nov 02 13:14:49.182237 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:14:49.182198    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 13:14:49.182350 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:14:49.182288    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 13:14:49.182350 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:14:49.182332    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 13:14:49.828614 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:14:49.828457    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-bmcxg" event={"ID":"185fcba5-401f-4a43-bad1-f82f845221b8","Type":"ContainerStarted","Data":"190aeadec88fade680508bb06145e8268d599a5ce8bcfb0c305de74114cb52ff"}
Nov 02 13:14:49.828614 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:14:49.828524    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-bmcxg" event={"ID":"185fcba5-401f-4a43-bad1-f82f845221b8","Type":"ContainerStarted","Data":"f00875f9f6cdf35bf678f81f0754b90a53b9caf8c5c95855266cd52e64f71f45"}
Nov 02 13:14:49.843534 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:14:49.843509    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-bmcxg"
Nov 02 13:14:49.913664 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:14:49.913455    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-bmcxg" podStartSLOduration=1.9134354770000002 podStartE2EDuration="1.913435477s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:14:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:14:49.88807389 +0000 UTC m=+2886.143472986" watchObservedRunningTime="2026-11-02 13:14:49.913435477 +0000 UTC m=+2886.168834573"
Nov 02 13:14:50.844363 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:14:50.844325    4559 patch_prober.go:29] interesting pod/e2e-inf-a-5f8899-openshift-default-67cc5d8789-bmcxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused" start-of-body=
Nov 02 13:14:50.844823 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:14:50.844371    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-bmcxg" podUID="185fcba5-401f-4a43-bad1-f82f845221b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused"
Nov 02 13:14:51.843430 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:14:51.843396    4559 patch_prober.go:29] interesting pod/e2e-inf-a-5f8899-openshift-default-67cc5d8789-bmcxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused" start-of-body=
Nov 02 13:14:51.843608 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:14:51.843456    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-bmcxg" podUID="185fcba5-401f-4a43-bad1-f82f845221b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused"
Nov 02 13:14:52.846662 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:14:52.846281    4559 patch_prober.go:29] interesting pod/e2e-inf-a-5f8899-openshift-default-67cc5d8789-bmcxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused" start-of-body=
Nov 02 13:14:52.846662 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:14:52.846345    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-bmcxg" podUID="185fcba5-401f-4a43-bad1-f82f845221b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused"
Nov 02 13:14:53.844133 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:14:53.844096    4559 patch_prober.go:29] interesting pod/e2e-inf-a-5f8899-openshift-default-67cc5d8789-bmcxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused" start-of-body=
Nov 02 13:14:53.844309 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:14:53.844163    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-bmcxg" podUID="185fcba5-401f-4a43-bad1-f82f845221b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused"
Nov 02 13:14:54.843961 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:14:54.843923    4559 patch_prober.go:29] interesting pod/e2e-inf-a-5f8899-openshift-default-67cc5d8789-bmcxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused" start-of-body=
Nov 02 13:14:54.844385 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:14:54.843986    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-bmcxg" podUID="185fcba5-401f-4a43-bad1-f82f845221b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused"
Nov 02 13:14:55.848902 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:14:55.848863    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-bmcxg"
Nov 02 13:14:55.849417 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:14:55.849326    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-bmcxg"
Nov 02 13:14:55.850254 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:14:55.850233    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-bmcxg"
Nov 02 13:15:00.308343 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:00.308295    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9139ec-29893755-ghrkl"]
Nov 02 13:15:00.313897 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:00.313872    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9139ec-29893755-ghrkl"
Nov 02 13:15:00.326715 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:00.326681    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893755-nfv7m"]
Nov 02 13:15:00.347866 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:00.347838    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6563ed-29893755-49cdm"]
Nov 02 13:15:00.349947 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:00.349438    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893755-nfv7m"
Nov 02 13:15:00.354903 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:00.354889    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9139ec-29893755-ghrkl"]
Nov 02 13:15:00.355055 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:00.355043    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893755-nfv7m"]
Nov 02 13:15:00.355161 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:00.355152    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6563ed-29893755-49cdm"]
Nov 02 13:15:00.355264 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:00.355005    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6563ed-29893755-49cdm"
Nov 02 13:15:00.401601 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:00.401569    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w9cwp\""
Nov 02 13:15:00.445763 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:00.445731    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/74991db3-76e1-4a11-8fa8-e2dff64107e1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-6563ed-29893755-49cdm\" (UID: \"74991db3-76e1-4a11-8fa8-e2dff64107e1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6563ed-29893755-49cdm"
Nov 02 13:15:00.445954 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:00.445795    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a35985fb-30d2-49a6-a34d-3b6c2d4ce2b6-maas-api-service-ca\") pod \"maas-api-key-cleanup-29893755-nfv7m\" (UID: \"a35985fb-30d2-49a6-a34d-3b6c2d4ce2b6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893755-nfv7m"
Nov 02 13:15:00.446045 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:00.446019    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3dc1fcaf-e4d8-4912-af05-a63c7fba2506-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-9139ec-29893755-ghrkl\" (UID: \"3dc1fcaf-e4d8-4912-af05-a63c7fba2506\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9139ec-29893755-ghrkl"
Nov 02 13:15:00.446249 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:00.446224    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gsblx\" (UniqueName: \"kubernetes.io/projected/74991db3-76e1-4a11-8fa8-e2dff64107e1-kube-api-access-gsblx\") pod \"maas-api-key-cleanup-e2e-inf-a-6563ed-29893755-49cdm\" (UID: \"74991db3-76e1-4a11-8fa8-e2dff64107e1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6563ed-29893755-49cdm"
Nov 02 13:15:00.446302 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:00.446271    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gb5ct\" (UniqueName: \"kubernetes.io/projected/3dc1fcaf-e4d8-4912-af05-a63c7fba2506-kube-api-access-gb5ct\") pod \"maas-api-key-cleanup-e2e-inf-b-9139ec-29893755-ghrkl\" (UID: \"3dc1fcaf-e4d8-4912-af05-a63c7fba2506\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9139ec-29893755-ghrkl"
Nov 02 13:15:00.446391 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:00.446376    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87vcl\" (UniqueName: \"kubernetes.io/projected/a35985fb-30d2-49a6-a34d-3b6c2d4ce2b6-kube-api-access-87vcl\") pod \"maas-api-key-cleanup-29893755-nfv7m\" (UID: \"a35985fb-30d2-49a6-a34d-3b6c2d4ce2b6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893755-nfv7m"
Nov 02 13:15:00.547961 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:00.547922    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3dc1fcaf-e4d8-4912-af05-a63c7fba2506-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-9139ec-29893755-ghrkl\" (UID: \"3dc1fcaf-e4d8-4912-af05-a63c7fba2506\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9139ec-29893755-ghrkl"
Nov 02 13:15:00.548175 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:00.548031    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsblx\" (UniqueName: \"kubernetes.io/projected/74991db3-76e1-4a11-8fa8-e2dff64107e1-kube-api-access-gsblx\") pod \"maas-api-key-cleanup-e2e-inf-a-6563ed-29893755-49cdm\" (UID: \"74991db3-76e1-4a11-8fa8-e2dff64107e1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6563ed-29893755-49cdm"
Nov 02 13:15:00.548175 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:00.548055    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gb5ct\" (UniqueName: \"kubernetes.io/projected/3dc1fcaf-e4d8-4912-af05-a63c7fba2506-kube-api-access-gb5ct\") pod \"maas-api-key-cleanup-e2e-inf-b-9139ec-29893755-ghrkl\" (UID: \"3dc1fcaf-e4d8-4912-af05-a63c7fba2506\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9139ec-29893755-ghrkl"
Nov 02 13:15:00.548175 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:00.548091    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87vcl\" (UniqueName: \"kubernetes.io/projected/a35985fb-30d2-49a6-a34d-3b6c2d4ce2b6-kube-api-access-87vcl\") pod \"maas-api-key-cleanup-29893755-nfv7m\" (UID: \"a35985fb-30d2-49a6-a34d-3b6c2d4ce2b6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893755-nfv7m"
Nov 02 13:15:00.548175 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:00.548161    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/74991db3-76e1-4a11-8fa8-e2dff64107e1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-6563ed-29893755-49cdm\" (UID: \"74991db3-76e1-4a11-8fa8-e2dff64107e1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6563ed-29893755-49cdm"
Nov 02 13:15:00.548435 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:00.548203    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a35985fb-30d2-49a6-a34d-3b6c2d4ce2b6-maas-api-service-ca\") pod \"maas-api-key-cleanup-29893755-nfv7m\" (UID: \"a35985fb-30d2-49a6-a34d-3b6c2d4ce2b6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893755-nfv7m"
Nov 02 13:15:00.548734 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:00.548709    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3dc1fcaf-e4d8-4912-af05-a63c7fba2506-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-9139ec-29893755-ghrkl\" (UID: \"3dc1fcaf-e4d8-4912-af05-a63c7fba2506\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9139ec-29893755-ghrkl"
Nov 02 13:15:00.548891 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:00.548873    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a35985fb-30d2-49a6-a34d-3b6c2d4ce2b6-maas-api-service-ca\") pod \"maas-api-key-cleanup-29893755-nfv7m\" (UID: \"a35985fb-30d2-49a6-a34d-3b6c2d4ce2b6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893755-nfv7m"
Nov 02 13:15:00.548947 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:00.548897    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/74991db3-76e1-4a11-8fa8-e2dff64107e1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-6563ed-29893755-49cdm\" (UID: \"74991db3-76e1-4a11-8fa8-e2dff64107e1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6563ed-29893755-49cdm"
Nov 02 13:15:00.575331 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:00.575264    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gb5ct\" (UniqueName: \"kubernetes.io/projected/3dc1fcaf-e4d8-4912-af05-a63c7fba2506-kube-api-access-gb5ct\") pod \"maas-api-key-cleanup-e2e-inf-b-9139ec-29893755-ghrkl\" (UID: \"3dc1fcaf-e4d8-4912-af05-a63c7fba2506\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9139ec-29893755-ghrkl"
Nov 02 13:15:00.575331 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:00.575299    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-87vcl\" (UniqueName: \"kubernetes.io/projected/a35985fb-30d2-49a6-a34d-3b6c2d4ce2b6-kube-api-access-87vcl\") pod \"maas-api-key-cleanup-29893755-nfv7m\" (UID: \"a35985fb-30d2-49a6-a34d-3b6c2d4ce2b6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893755-nfv7m"
Nov 02 13:15:00.575521 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:00.575449    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsblx\" (UniqueName: \"kubernetes.io/projected/74991db3-76e1-4a11-8fa8-e2dff64107e1-kube-api-access-gsblx\") pod \"maas-api-key-cleanup-e2e-inf-a-6563ed-29893755-49cdm\" (UID: \"74991db3-76e1-4a11-8fa8-e2dff64107e1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6563ed-29893755-49cdm"
Nov 02 13:15:00.649683 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:00.649625    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9139ec-29893755-ghrkl"
Nov 02 13:15:00.678968 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:00.678940    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893755-nfv7m"
Nov 02 13:15:00.694239 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:00.694188    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6563ed-29893755-49cdm"
Nov 02 13:15:00.990116 ip-10-0-128-132 kubenswrapper[4559]: W1102 13:15:00.990075    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3dc1fcaf_e4d8_4912_af05_a63c7fba2506.slice/crio-0906b63924f9e5828f10aff9eed38b7786e8a3434f228cde599e6f0ce7a92c9b WatchSource:0}: Error finding container 0906b63924f9e5828f10aff9eed38b7786e8a3434f228cde599e6f0ce7a92c9b: Status 404 returned error can't find the container with id 0906b63924f9e5828f10aff9eed38b7786e8a3434f228cde599e6f0ce7a92c9b
Nov 02 13:15:00.992054 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:00.992008    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9139ec-29893755-ghrkl"]
Nov 02 13:15:01.030708 ip-10-0-128-132 kubenswrapper[4559]: W1102 13:15:01.030680    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda35985fb_30d2_49a6_a34d_3b6c2d4ce2b6.slice/crio-df347d48907f0e6815535c7eefc3acb31a9e37329f53f5506183cb38dc90850a WatchSource:0}: Error finding container df347d48907f0e6815535c7eefc3acb31a9e37329f53f5506183cb38dc90850a: Status 404 returned error can't find the container with id df347d48907f0e6815535c7eefc3acb31a9e37329f53f5506183cb38dc90850a
Nov 02 13:15:01.032923 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:01.032897    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893755-nfv7m"]
Nov 02 13:15:01.063518 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:01.063489    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6563ed-29893755-49cdm"]
Nov 02 13:15:01.976680 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:01.976620    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893755-nfv7m" event={"ID":"a35985fb-30d2-49a6-a34d-3b6c2d4ce2b6","Type":"ContainerStarted","Data":"ef84699c2c89bd9c9b07892470ab9a55d0ae6d8e0bdd9dbedde4bc899a4fb35c"}
Nov 02 13:15:01.977139 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:01.976696    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893755-nfv7m" event={"ID":"a35985fb-30d2-49a6-a34d-3b6c2d4ce2b6","Type":"ContainerStarted","Data":"df347d48907f0e6815535c7eefc3acb31a9e37329f53f5506183cb38dc90850a"}
Nov 02 13:15:01.978626 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:01.978594    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6563ed-29893755-49cdm" event={"ID":"74991db3-76e1-4a11-8fa8-e2dff64107e1","Type":"ContainerStarted","Data":"8db10451cfc35940fb2b9d6db5b4226b4ff81d250a6d404404a0117409b99d09"}
Nov 02 13:15:01.978772 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:01.978667    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6563ed-29893755-49cdm" event={"ID":"74991db3-76e1-4a11-8fa8-e2dff64107e1","Type":"ContainerStarted","Data":"d6dc4af36ea2f35406d7416d225e7b0109849953ca440d642760681ced9051b0"}
Nov 02 13:15:01.980330 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:01.980303    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9139ec-29893755-ghrkl" event={"ID":"3dc1fcaf-e4d8-4912-af05-a63c7fba2506","Type":"ContainerStarted","Data":"d6b43a7bbea748ddb40fd23f602e749d9ae87db042ae5fb2e446590786c057a1"}
Nov 02 13:15:01.980427 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:01.980354    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9139ec-29893755-ghrkl" event={"ID":"3dc1fcaf-e4d8-4912-af05-a63c7fba2506","Type":"ContainerStarted","Data":"0906b63924f9e5828f10aff9eed38b7786e8a3434f228cde599e6f0ce7a92c9b"}
Nov 02 13:15:02.018885 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:02.018838    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893755-nfv7m" podStartSLOduration=2.018823048 podStartE2EDuration="2.018823048s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:15:02.016904552 +0000 UTC m=+2898.272303647" watchObservedRunningTime="2026-11-02 13:15:02.018823048 +0000 UTC m=+2898.274222145"
Nov 02 13:15:02.060032 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:02.059984    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9139ec-29893755-ghrkl" podStartSLOduration=2.059971217 podStartE2EDuration="2.059971217s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:15:02.056530832 +0000 UTC m=+2898.311929938" watchObservedRunningTime="2026-11-02 13:15:02.059971217 +0000 UTC m=+2898.315370312"
Nov 02 13:15:02.101281 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:02.101239    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6563ed-29893755-49cdm" podStartSLOduration=2.101227155 podStartE2EDuration="2.101227155s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:15:02.097280919 +0000 UTC m=+2898.352680031" watchObservedRunningTime="2026-11-02 13:15:02.101227155 +0000 UTC m=+2898.356626250"
Nov 02 13:15:02.990569 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:02.990526    4559 generic.go:356] "Generic (PLEG): container finished" podID="a35985fb-30d2-49a6-a34d-3b6c2d4ce2b6" containerID="ef84699c2c89bd9c9b07892470ab9a55d0ae6d8e0bdd9dbedde4bc899a4fb35c" exitCode=60
Nov 02 13:15:02.991107 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:02.990626    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893755-nfv7m" event={"ID":"a35985fb-30d2-49a6-a34d-3b6c2d4ce2b6","Type":"ContainerDied","Data":"ef84699c2c89bd9c9b07892470ab9a55d0ae6d8e0bdd9dbedde4bc899a4fb35c"}
Nov 02 13:15:02.991107 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:02.990907    4559 scope.go:122] "RemoveContainer" containerID="ef84699c2c89bd9c9b07892470ab9a55d0ae6d8e0bdd9dbedde4bc899a4fb35c"
Nov 02 13:15:02.992559 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:02.992531    4559 generic.go:356] "Generic (PLEG): container finished" podID="74991db3-76e1-4a11-8fa8-e2dff64107e1" containerID="8db10451cfc35940fb2b9d6db5b4226b4ff81d250a6d404404a0117409b99d09" exitCode=60
Nov 02 13:15:02.992744 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:02.992616    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6563ed-29893755-49cdm" event={"ID":"74991db3-76e1-4a11-8fa8-e2dff64107e1","Type":"ContainerDied","Data":"8db10451cfc35940fb2b9d6db5b4226b4ff81d250a6d404404a0117409b99d09"}
Nov 02 13:15:02.992888 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:02.992878    4559 scope.go:122] "RemoveContainer" containerID="8db10451cfc35940fb2b9d6db5b4226b4ff81d250a6d404404a0117409b99d09"
Nov 02 13:15:02.994751 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:02.994732    4559 generic.go:356] "Generic (PLEG): container finished" podID="3dc1fcaf-e4d8-4912-af05-a63c7fba2506" containerID="d6b43a7bbea748ddb40fd23f602e749d9ae87db042ae5fb2e446590786c057a1" exitCode=60
Nov 02 13:15:02.994975 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:02.994870    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9139ec-29893755-ghrkl" event={"ID":"3dc1fcaf-e4d8-4912-af05-a63c7fba2506","Type":"ContainerDied","Data":"d6b43a7bbea748ddb40fd23f602e749d9ae87db042ae5fb2e446590786c057a1"}
Nov 02 13:15:02.995042 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:02.995028    4559 scope.go:122] "RemoveContainer" containerID="d6b43a7bbea748ddb40fd23f602e749d9ae87db042ae5fb2e446590786c057a1"
Nov 02 13:15:04.004878 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:04.004848    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6563ed-29893755-49cdm" event={"ID":"74991db3-76e1-4a11-8fa8-e2dff64107e1","Type":"ContainerStarted","Data":"497b0f9814dab067108cffc3664cfd621154c52a4d1871b432eece5998eb4c11"}
Nov 02 13:15:04.006828 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:04.006806    4559 generic.go:356] "Generic (PLEG): container finished" podID="3dc1fcaf-e4d8-4912-af05-a63c7fba2506" containerID="ded28c257830ee4bd7b71538d490eff7816b613a6e5707d6a38ed217f8b9bac5" exitCode=60
Nov 02 13:15:04.006929 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:04.006888    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9139ec-29893755-ghrkl" event={"ID":"3dc1fcaf-e4d8-4912-af05-a63c7fba2506","Type":"ContainerDied","Data":"ded28c257830ee4bd7b71538d490eff7816b613a6e5707d6a38ed217f8b9bac5"}
Nov 02 13:15:04.006976 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:04.006946    4559 scope.go:122] "RemoveContainer" containerID="d6b43a7bbea748ddb40fd23f602e749d9ae87db042ae5fb2e446590786c057a1"
Nov 02 13:15:04.007202 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:04.007189    4559 scope.go:122] "RemoveContainer" containerID="ded28c257830ee4bd7b71538d490eff7816b613a6e5707d6a38ed217f8b9bac5"
Nov 02 13:15:04.007502 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:15:04.007478    4559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-9139ec-29893755-ghrkl_odh-ai-gateway-infra(3dc1fcaf-e4d8-4912-af05-a63c7fba2506)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9139ec-29893755-ghrkl" podUID="3dc1fcaf-e4d8-4912-af05-a63c7fba2506"
Nov 02 13:15:04.009131 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:04.009112    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893755-nfv7m" event={"ID":"a35985fb-30d2-49a6-a34d-3b6c2d4ce2b6","Type":"ContainerStarted","Data":"4bf93eae922a6431d329e2f4e2155eee2b21bc422d3a4747fd42d88cdfeb874c"}
Nov 02 13:15:05.022603 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:05.022557    4559 generic.go:356] "Generic (PLEG): container finished" podID="a35985fb-30d2-49a6-a34d-3b6c2d4ce2b6" containerID="4bf93eae922a6431d329e2f4e2155eee2b21bc422d3a4747fd42d88cdfeb874c" exitCode=60
Nov 02 13:15:05.023235 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:05.022654    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893755-nfv7m" event={"ID":"a35985fb-30d2-49a6-a34d-3b6c2d4ce2b6","Type":"ContainerDied","Data":"4bf93eae922a6431d329e2f4e2155eee2b21bc422d3a4747fd42d88cdfeb874c"}
Nov 02 13:15:05.023235 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:05.022754    4559 scope.go:122] "RemoveContainer" containerID="ef84699c2c89bd9c9b07892470ab9a55d0ae6d8e0bdd9dbedde4bc899a4fb35c"
Nov 02 13:15:05.023235 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:05.023111    4559 scope.go:122] "RemoveContainer" containerID="4bf93eae922a6431d329e2f4e2155eee2b21bc422d3a4747fd42d88cdfeb874c"
Nov 02 13:15:05.023482 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:15:05.023461    4559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29893755-nfv7m_odh-ai-gateway-infra(a35985fb-30d2-49a6-a34d-3b6c2d4ce2b6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893755-nfv7m" podUID="a35985fb-30d2-49a6-a34d-3b6c2d4ce2b6"
Nov 02 13:15:05.026140 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:05.026114    4559 generic.go:356] "Generic (PLEG): container finished" podID="74991db3-76e1-4a11-8fa8-e2dff64107e1" containerID="497b0f9814dab067108cffc3664cfd621154c52a4d1871b432eece5998eb4c11" exitCode=60
Nov 02 13:15:05.026279 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:05.026194    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6563ed-29893755-49cdm" event={"ID":"74991db3-76e1-4a11-8fa8-e2dff64107e1","Type":"ContainerDied","Data":"497b0f9814dab067108cffc3664cfd621154c52a4d1871b432eece5998eb4c11"}
Nov 02 13:15:05.026475 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:05.026460    4559 scope.go:122] "RemoveContainer" containerID="497b0f9814dab067108cffc3664cfd621154c52a4d1871b432eece5998eb4c11"
Nov 02 13:15:05.026902 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:15:05.026873    4559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-6563ed-29893755-49cdm_odh-ai-gateway-infra(74991db3-76e1-4a11-8fa8-e2dff64107e1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6563ed-29893755-49cdm" podUID="74991db3-76e1-4a11-8fa8-e2dff64107e1"
Nov 02 13:15:05.028076 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:05.028059    4559 scope.go:122] "RemoveContainer" containerID="ded28c257830ee4bd7b71538d490eff7816b613a6e5707d6a38ed217f8b9bac5"
Nov 02 13:15:05.028343 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:15:05.028325    4559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-9139ec-29893755-ghrkl_odh-ai-gateway-infra(3dc1fcaf-e4d8-4912-af05-a63c7fba2506)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9139ec-29893755-ghrkl" podUID="3dc1fcaf-e4d8-4912-af05-a63c7fba2506"
Nov 02 13:15:05.037476 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:05.037455    4559 scope.go:122] "RemoveContainer" containerID="8db10451cfc35940fb2b9d6db5b4226b4ff81d250a6d404404a0117409b99d09"
Nov 02 13:15:06.078666 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:06.070486    4559 scope.go:122] "RemoveContainer" containerID="4bf93eae922a6431d329e2f4e2155eee2b21bc422d3a4747fd42d88cdfeb874c"
Nov 02 13:15:06.078666 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:15:06.071059    4559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29893755-nfv7m_odh-ai-gateway-infra(a35985fb-30d2-49a6-a34d-3b6c2d4ce2b6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893755-nfv7m" podUID="a35985fb-30d2-49a6-a34d-3b6c2d4ce2b6"
Nov 02 13:15:06.078666 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:06.074264    4559 scope.go:122] "RemoveContainer" containerID="497b0f9814dab067108cffc3664cfd621154c52a4d1871b432eece5998eb4c11"
Nov 02 13:15:06.089823 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:15:06.089752    4559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-6563ed-29893755-49cdm_odh-ai-gateway-infra(74991db3-76e1-4a11-8fa8-e2dff64107e1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6563ed-29893755-49cdm" podUID="74991db3-76e1-4a11-8fa8-e2dff64107e1"
Nov 02 13:15:07.539317 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:07.539279    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-48d74e-77f759984c-j6jsk"]
Nov 02 13:15:07.544089 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:07.544067    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-48d74e-77f759984c-j6jsk"
Nov 02 13:15:07.566182 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:07.566159    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-48d74e-77f759984c-j6jsk"]
Nov 02 13:15:07.573702 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:07.573680    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-48d74e\""
Nov 02 13:15:07.573865 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:07.573715    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-48d74e-dockercfg-nj5gm\""
Nov 02 13:15:07.772715 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:07.711656    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e74c7087-2b2c-44c5-b480-8d692b91d134-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-48d74e-77f759984c-j6jsk\" (UID: \"e74c7087-2b2c-44c5-b480-8d692b91d134\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-48d74e-77f759984c-j6jsk"
Nov 02 13:15:07.778785 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:07.778743    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bw6tf\" (UniqueName: \"kubernetes.io/projected/e74c7087-2b2c-44c5-b480-8d692b91d134-kube-api-access-bw6tf\") pod \"payload-pre-processing-e2e-inf-b-48d74e-77f759984c-j6jsk\" (UID: \"e74c7087-2b2c-44c5-b480-8d692b91d134\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-48d74e-77f759984c-j6jsk"
Nov 02 13:15:07.900712 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:07.853871    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-48d74e-57f8487987-np4k8"]
Nov 02 13:15:07.900712 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:07.873951    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-48d74e-57f8487987-np4k8"
Nov 02 13:15:07.931768 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:07.925718    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-48d74e-57f8487987-np4k8"]
Nov 02 13:15:08.003560 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:08.003515    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zcnsv\" (UniqueName: \"kubernetes.io/projected/4f71b7db-e5b6-4fb6-b489-ea87e96450a7-kube-api-access-zcnsv\") pod \"payload-processing-e2e-inf-b-48d74e-57f8487987-np4k8\" (UID: \"4f71b7db-e5b6-4fb6-b489-ea87e96450a7\") " pod="openshift-ingress/payload-processing-e2e-inf-b-48d74e-57f8487987-np4k8"
Nov 02 13:15:08.003749 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:08.003707    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f71b7db-e5b6-4fb6-b489-ea87e96450a7-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-48d74e-57f8487987-np4k8\" (UID: \"4f71b7db-e5b6-4fb6-b489-ea87e96450a7\") " pod="openshift-ingress/payload-processing-e2e-inf-b-48d74e-57f8487987-np4k8"
Nov 02 13:15:08.003812 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:08.003751    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e74c7087-2b2c-44c5-b480-8d692b91d134-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-48d74e-77f759984c-j6jsk\" (UID: \"e74c7087-2b2c-44c5-b480-8d692b91d134\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-48d74e-77f759984c-j6jsk"
Nov 02 13:15:08.003812 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:08.003773    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bw6tf\" (UniqueName: \"kubernetes.io/projected/e74c7087-2b2c-44c5-b480-8d692b91d134-kube-api-access-bw6tf\") pod \"payload-pre-processing-e2e-inf-b-48d74e-77f759984c-j6jsk\" (UID: \"e74c7087-2b2c-44c5-b480-8d692b91d134\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-48d74e-77f759984c-j6jsk"
Nov 02 13:15:08.004566 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:08.004542    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e74c7087-2b2c-44c5-b480-8d692b91d134-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-48d74e-77f759984c-j6jsk\" (UID: \"e74c7087-2b2c-44c5-b480-8d692b91d134\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-48d74e-77f759984c-j6jsk"
Nov 02 13:15:08.042272 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:08.042242    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bw6tf\" (UniqueName: \"kubernetes.io/projected/e74c7087-2b2c-44c5-b480-8d692b91d134-kube-api-access-bw6tf\") pod \"payload-pre-processing-e2e-inf-b-48d74e-77f759984c-j6jsk\" (UID: \"e74c7087-2b2c-44c5-b480-8d692b91d134\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-48d74e-77f759984c-j6jsk"
Nov 02 13:15:08.104787 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:08.104756    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcnsv\" (UniqueName: \"kubernetes.io/projected/4f71b7db-e5b6-4fb6-b489-ea87e96450a7-kube-api-access-zcnsv\") pod \"payload-processing-e2e-inf-b-48d74e-57f8487987-np4k8\" (UID: \"4f71b7db-e5b6-4fb6-b489-ea87e96450a7\") " pod="openshift-ingress/payload-processing-e2e-inf-b-48d74e-57f8487987-np4k8"
Nov 02 13:15:08.104966 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:08.104866    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f71b7db-e5b6-4fb6-b489-ea87e96450a7-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-48d74e-57f8487987-np4k8\" (UID: \"4f71b7db-e5b6-4fb6-b489-ea87e96450a7\") " pod="openshift-ingress/payload-processing-e2e-inf-b-48d74e-57f8487987-np4k8"
Nov 02 13:15:08.105535 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:08.105516    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f71b7db-e5b6-4fb6-b489-ea87e96450a7-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-48d74e-57f8487987-np4k8\" (UID: \"4f71b7db-e5b6-4fb6-b489-ea87e96450a7\") " pod="openshift-ingress/payload-processing-e2e-inf-b-48d74e-57f8487987-np4k8"
Nov 02 13:15:08.128973 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:08.128948    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcnsv\" (UniqueName: \"kubernetes.io/projected/4f71b7db-e5b6-4fb6-b489-ea87e96450a7-kube-api-access-zcnsv\") pod \"payload-processing-e2e-inf-b-48d74e-57f8487987-np4k8\" (UID: \"4f71b7db-e5b6-4fb6-b489-ea87e96450a7\") " pod="openshift-ingress/payload-processing-e2e-inf-b-48d74e-57f8487987-np4k8"
Nov 02 13:15:08.158375 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:08.158311    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-48d74e-77f759984c-j6jsk"
Nov 02 13:15:08.250972 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:08.244533    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-48d74e-57f8487987-np4k8"
Nov 02 13:15:08.562054 ip-10-0-128-132 kubenswrapper[4559]: W1102 13:15:08.562022    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode74c7087_2b2c_44c5_b480_8d692b91d134.slice/crio-c150f9b8b237e6e9d2655944b584363a8d5f52c6ed94759ef2c16da1bb2dce3a WatchSource:0}: Error finding container c150f9b8b237e6e9d2655944b584363a8d5f52c6ed94759ef2c16da1bb2dce3a: Status 404 returned error can't find the container with id c150f9b8b237e6e9d2655944b584363a8d5f52c6ed94759ef2c16da1bb2dce3a
Nov 02 13:15:08.562594 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:08.562572    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-48d74e-77f759984c-j6jsk"]
Nov 02 13:15:08.584784 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:08.584754    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-48d74e-57f8487987-np4k8"]
Nov 02 13:15:08.585131 ip-10-0-128-132 kubenswrapper[4559]: W1102 13:15:08.585110    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f71b7db_e5b6_4fb6_b489_ea87e96450a7.slice/crio-da3d6702df157d50b23ef9e29dc1e5c9e20f40e64ee3d279168a631fded75447 WatchSource:0}: Error finding container da3d6702df157d50b23ef9e29dc1e5c9e20f40e64ee3d279168a631fded75447: Status 404 returned error can't find the container with id da3d6702df157d50b23ef9e29dc1e5c9e20f40e64ee3d279168a631fded75447
Nov 02 13:15:09.099263 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:09.099217    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-48d74e-57f8487987-np4k8" event={"ID":"4f71b7db-e5b6-4fb6-b489-ea87e96450a7","Type":"ContainerStarted","Data":"59932a3d203e08813bfa9712eb627a678d24f71a13a0a4db874e73375caf8b24"}
Nov 02 13:15:09.099481 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:09.099279    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-48d74e-57f8487987-np4k8" event={"ID":"4f71b7db-e5b6-4fb6-b489-ea87e96450a7","Type":"ContainerStarted","Data":"da3d6702df157d50b23ef9e29dc1e5c9e20f40e64ee3d279168a631fded75447"}
Nov 02 13:15:09.099481 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:09.099304    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-48d74e-57f8487987-np4k8"
Nov 02 13:15:09.101165 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:09.101135    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-48d74e-77f759984c-j6jsk" event={"ID":"e74c7087-2b2c-44c5-b480-8d692b91d134","Type":"ContainerStarted","Data":"2303c6e317d118f062da5202a64703c41c14837340f2f762686ee91c2bda8580"}
Nov 02 13:15:09.101296 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:09.101182    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-48d74e-77f759984c-j6jsk" event={"ID":"e74c7087-2b2c-44c5-b480-8d692b91d134","Type":"ContainerStarted","Data":"c150f9b8b237e6e9d2655944b584363a8d5f52c6ed94759ef2c16da1bb2dce3a"}
Nov 02 13:15:09.101296 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:09.101203    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-48d74e-77f759984c-j6jsk"
Nov 02 13:15:09.143817 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:09.143765    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-48d74e-57f8487987-np4k8" podStartSLOduration=2.143751255 podStartE2EDuration="2.143751255s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:15:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:15:09.142029215 +0000 UTC m=+2905.397428310" watchObservedRunningTime="2026-11-02 13:15:09.143751255 +0000 UTC m=+2905.399150409"
Nov 02 13:15:09.186268 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:09.186219    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-48d74e-77f759984c-j6jsk" podStartSLOduration=2.186206896 podStartE2EDuration="2.186206896s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:15:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:15:09.182721829 +0000 UTC m=+2905.438120925" watchObservedRunningTime="2026-11-02 13:15:09.186206896 +0000 UTC m=+2905.441606022"
Nov 02 13:15:15.088679 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:15:15.088625    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqcmj" containerName="extractor"
Nov 02 13:15:15.895840 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:15.895809    4559 scope.go:122] "RemoveContainer" containerID="497b0f9814dab067108cffc3664cfd621154c52a4d1871b432eece5998eb4c11"
Nov 02 13:15:15.896053 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:15.896015    4559 scope.go:122] "RemoveContainer" containerID="4bf93eae922a6431d329e2f4e2155eee2b21bc422d3a4747fd42d88cdfeb874c"
Nov 02 13:15:17.158445 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:17.158364    4559 generic.go:356] "Generic (PLEG): container finished" podID="a35985fb-30d2-49a6-a34d-3b6c2d4ce2b6" containerID="011361b066ccf78ecb1ac626391323338c1b6c7f87c586ab4c2f418c8a4ede10" exitCode=60
Nov 02 13:15:17.158951 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:17.158444    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893755-nfv7m" event={"ID":"a35985fb-30d2-49a6-a34d-3b6c2d4ce2b6","Type":"ContainerDied","Data":"011361b066ccf78ecb1ac626391323338c1b6c7f87c586ab4c2f418c8a4ede10"}
Nov 02 13:15:17.158951 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:17.158511    4559 scope.go:122] "RemoveContainer" containerID="4bf93eae922a6431d329e2f4e2155eee2b21bc422d3a4747fd42d88cdfeb874c"
Nov 02 13:15:17.158951 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:17.158910    4559 scope.go:122] "RemoveContainer" containerID="011361b066ccf78ecb1ac626391323338c1b6c7f87c586ab4c2f418c8a4ede10"
Nov 02 13:15:17.159256 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:15:17.159232    4559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29893755-nfv7m_odh-ai-gateway-infra(a35985fb-30d2-49a6-a34d-3b6c2d4ce2b6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893755-nfv7m" podUID="a35985fb-30d2-49a6-a34d-3b6c2d4ce2b6"
Nov 02 13:15:17.160901 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:17.160881    4559 generic.go:356] "Generic (PLEG): container finished" podID="74991db3-76e1-4a11-8fa8-e2dff64107e1" containerID="a7c8c8c018d47b64773c2969a06653531f9c303fd7373b2a5a0e67ac1cd2f5be" exitCode=60
Nov 02 13:15:17.161006 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:17.160945    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6563ed-29893755-49cdm" event={"ID":"74991db3-76e1-4a11-8fa8-e2dff64107e1","Type":"ContainerDied","Data":"a7c8c8c018d47b64773c2969a06653531f9c303fd7373b2a5a0e67ac1cd2f5be"}
Nov 02 13:15:17.161291 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:17.161277    4559 scope.go:122] "RemoveContainer" containerID="a7c8c8c018d47b64773c2969a06653531f9c303fd7373b2a5a0e67ac1cd2f5be"
Nov 02 13:15:17.161593 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:15:17.161576    4559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-6563ed-29893755-49cdm_odh-ai-gateway-infra(74991db3-76e1-4a11-8fa8-e2dff64107e1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6563ed-29893755-49cdm" podUID="74991db3-76e1-4a11-8fa8-e2dff64107e1"
Nov 02 13:15:17.172441 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:17.172424    4559 scope.go:122] "RemoveContainer" containerID="497b0f9814dab067108cffc3664cfd621154c52a4d1871b432eece5998eb4c11"
Nov 02 13:15:17.896844 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:17.896816    4559 scope.go:122] "RemoveContainer" containerID="ded28c257830ee4bd7b71538d490eff7816b613a6e5707d6a38ed217f8b9bac5"
Nov 02 13:15:18.243945 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:18.243910    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893755-nfv7m"]
Nov 02 13:15:18.257061 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:18.257037    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6563ed-29893755-49cdm"]
Nov 02 13:15:18.467414 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:18.467388    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893755-nfv7m"
Nov 02 13:15:18.526069 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:18.526044    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6563ed-29893755-49cdm"
Nov 02 13:15:18.588355 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:18.588323    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74991db3-76e1-4a11-8fa8-e2dff64107e1-kube-api-access-gsblx\" (UniqueName: \"kubernetes.io/projected/74991db3-76e1-4a11-8fa8-e2dff64107e1-kube-api-access-gsblx\") pod \"74991db3-76e1-4a11-8fa8-e2dff64107e1\" (UID: \"74991db3-76e1-4a11-8fa8-e2dff64107e1\") "
Nov 02 13:15:18.588516 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:18.588455    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a35985fb-30d2-49a6-a34d-3b6c2d4ce2b6-kube-api-access-87vcl\" (UniqueName: \"kubernetes.io/projected/a35985fb-30d2-49a6-a34d-3b6c2d4ce2b6-kube-api-access-87vcl\") pod \"a35985fb-30d2-49a6-a34d-3b6c2d4ce2b6\" (UID: \"a35985fb-30d2-49a6-a34d-3b6c2d4ce2b6\") "
Nov 02 13:15:18.588738 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:18.588713    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a35985fb-30d2-49a6-a34d-3b6c2d4ce2b6-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a35985fb-30d2-49a6-a34d-3b6c2d4ce2b6-maas-api-service-ca\") pod \"a35985fb-30d2-49a6-a34d-3b6c2d4ce2b6\" (UID: \"a35985fb-30d2-49a6-a34d-3b6c2d4ce2b6\") "
Nov 02 13:15:18.588914 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:18.588889    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74991db3-76e1-4a11-8fa8-e2dff64107e1-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/74991db3-76e1-4a11-8fa8-e2dff64107e1-maas-api-service-ca\") pod \"74991db3-76e1-4a11-8fa8-e2dff64107e1\" (UID: \"74991db3-76e1-4a11-8fa8-e2dff64107e1\") "
Nov 02 13:15:18.589772 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:18.589742    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a35985fb-30d2-49a6-a34d-3b6c2d4ce2b6-maas-api-service-ca" pod "a35985fb-30d2-49a6-a34d-3b6c2d4ce2b6" (UID: "a35985fb-30d2-49a6-a34d-3b6c2d4ce2b6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:15:18.589980 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:18.589944    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74991db3-76e1-4a11-8fa8-e2dff64107e1-maas-api-service-ca" pod "74991db3-76e1-4a11-8fa8-e2dff64107e1" (UID: "74991db3-76e1-4a11-8fa8-e2dff64107e1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:15:18.591122 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:18.591100    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a35985fb-30d2-49a6-a34d-3b6c2d4ce2b6-kube-api-access-87vcl" pod "a35985fb-30d2-49a6-a34d-3b6c2d4ce2b6" (UID: "a35985fb-30d2-49a6-a34d-3b6c2d4ce2b6"). InnerVolumeSpecName "kube-api-access-87vcl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:15:18.591244 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:18.591221    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74991db3-76e1-4a11-8fa8-e2dff64107e1-kube-api-access-gsblx" pod "74991db3-76e1-4a11-8fa8-e2dff64107e1" (UID: "74991db3-76e1-4a11-8fa8-e2dff64107e1"). InnerVolumeSpecName "kube-api-access-gsblx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:15:18.691160 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:18.691074    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/74991db3-76e1-4a11-8fa8-e2dff64107e1-maas-api-service-ca\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:15:18.691160 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:18.691119    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gsblx\" (UniqueName: \"kubernetes.io/projected/74991db3-76e1-4a11-8fa8-e2dff64107e1-kube-api-access-gsblx\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:15:18.691160 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:18.691134    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-87vcl\" (UniqueName: \"kubernetes.io/projected/a35985fb-30d2-49a6-a34d-3b6c2d4ce2b6-kube-api-access-87vcl\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:15:18.691160 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:18.691152    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a35985fb-30d2-49a6-a34d-3b6c2d4ce2b6-maas-api-service-ca\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:15:19.255632 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:19.255419    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893755-nfv7m"
Nov 02 13:15:19.255632 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:19.255419    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893755-nfv7m" event={"ID":"a35985fb-30d2-49a6-a34d-3b6c2d4ce2b6","Type":"ContainerDied","Data":"df347d48907f0e6815535c7eefc3acb31a9e37329f53f5506183cb38dc90850a"}
Nov 02 13:15:19.255632 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:19.255569    4559 scope.go:122] "RemoveContainer" containerID="011361b066ccf78ecb1ac626391323338c1b6c7f87c586ab4c2f418c8a4ede10"
Nov 02 13:15:19.257015 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:19.256996    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6563ed-29893755-49cdm" event={"ID":"74991db3-76e1-4a11-8fa8-e2dff64107e1","Type":"ContainerDied","Data":"d6dc4af36ea2f35406d7416d225e7b0109849953ca440d642760681ced9051b0"}
Nov 02 13:15:19.257072 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:19.257066    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6563ed-29893755-49cdm"
Nov 02 13:15:19.260007 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:19.259980    4559 generic.go:356] "Generic (PLEG): container finished" podID="3dc1fcaf-e4d8-4912-af05-a63c7fba2506" containerID="599f3006d59a900708c4741f59084b0e0b30def41796a2bddb5f1012da517988" exitCode=60
Nov 02 13:15:19.260126 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:19.260008    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9139ec-29893755-ghrkl" event={"ID":"3dc1fcaf-e4d8-4912-af05-a63c7fba2506","Type":"ContainerDied","Data":"599f3006d59a900708c4741f59084b0e0b30def41796a2bddb5f1012da517988"}
Nov 02 13:15:19.260431 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:19.260413    4559 scope.go:122] "RemoveContainer" containerID="599f3006d59a900708c4741f59084b0e0b30def41796a2bddb5f1012da517988"
Nov 02 13:15:19.260820 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:15:19.260793    4559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-9139ec-29893755-ghrkl_odh-ai-gateway-infra(3dc1fcaf-e4d8-4912-af05-a63c7fba2506)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9139ec-29893755-ghrkl" podUID="3dc1fcaf-e4d8-4912-af05-a63c7fba2506"
Nov 02 13:15:19.269314 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:19.269294    4559 scope.go:122] "RemoveContainer" containerID="a7c8c8c018d47b64773c2969a06653531f9c303fd7373b2a5a0e67ac1cd2f5be"
Nov 02 13:15:19.279303 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:19.279279    4559 scope.go:122] "RemoveContainer" containerID="ded28c257830ee4bd7b71538d490eff7816b613a6e5707d6a38ed217f8b9bac5"
Nov 02 13:15:19.312706 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:19.312662    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893755-nfv7m"]
Nov 02 13:15:19.318765 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:19.318743    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893755-nfv7m"]
Nov 02 13:15:19.354243 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:19.354219    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6563ed-29893755-49cdm"]
Nov 02 13:15:19.366647 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:19.366613    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6563ed-29893755-49cdm"]
Nov 02 13:15:19.895343 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:19.895308    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74991db3-76e1-4a11-8fa8-e2dff64107e1" path="/var/lib/kubelet/pods/74991db3-76e1-4a11-8fa8-e2dff64107e1/volumes"
Nov 02 13:15:19.895831 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:19.895815    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a35985fb-30d2-49a6-a34d-3b6c2d4ce2b6" path="/var/lib/kubelet/pods/a35985fb-30d2-49a6-a34d-3b6c2d4ce2b6/volumes"
Nov 02 13:15:20.112004 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:20.111973    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-48d74e-77f759984c-j6jsk"
Nov 02 13:15:20.112155 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:20.112043    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-48d74e-57f8487987-np4k8"
Nov 02 13:15:20.424408 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:20.424374    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9139ec-29893755-ghrkl"]
Nov 02 13:15:20.596238 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:20.596214    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9139ec-29893755-ghrkl"
Nov 02 13:15:20.729547 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:20.729452    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3dc1fcaf-e4d8-4912-af05-a63c7fba2506-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3dc1fcaf-e4d8-4912-af05-a63c7fba2506-maas-api-service-ca\") pod \"3dc1fcaf-e4d8-4912-af05-a63c7fba2506\" (UID: \"3dc1fcaf-e4d8-4912-af05-a63c7fba2506\") "
Nov 02 13:15:20.729547 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:20.729529    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3dc1fcaf-e4d8-4912-af05-a63c7fba2506-kube-api-access-gb5ct\" (UniqueName: \"kubernetes.io/projected/3dc1fcaf-e4d8-4912-af05-a63c7fba2506-kube-api-access-gb5ct\") pod \"3dc1fcaf-e4d8-4912-af05-a63c7fba2506\" (UID: \"3dc1fcaf-e4d8-4912-af05-a63c7fba2506\") "
Nov 02 13:15:20.729887 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:20.729857    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3dc1fcaf-e4d8-4912-af05-a63c7fba2506-maas-api-service-ca" pod "3dc1fcaf-e4d8-4912-af05-a63c7fba2506" (UID: "3dc1fcaf-e4d8-4912-af05-a63c7fba2506"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:15:20.730315 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:20.730292    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3dc1fcaf-e4d8-4912-af05-a63c7fba2506-maas-api-service-ca\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:15:20.731655 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:20.731622    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3dc1fcaf-e4d8-4912-af05-a63c7fba2506-kube-api-access-gb5ct" pod "3dc1fcaf-e4d8-4912-af05-a63c7fba2506" (UID: "3dc1fcaf-e4d8-4912-af05-a63c7fba2506"). InnerVolumeSpecName "kube-api-access-gb5ct". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:15:20.832272 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:20.832237    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gb5ct\" (UniqueName: \"kubernetes.io/projected/3dc1fcaf-e4d8-4912-af05-a63c7fba2506-kube-api-access-gb5ct\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:15:21.280050 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:21.280025    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9139ec-29893755-ghrkl"
Nov 02 13:15:21.280209 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:21.280020    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9139ec-29893755-ghrkl" event={"ID":"3dc1fcaf-e4d8-4912-af05-a63c7fba2506","Type":"ContainerDied","Data":"0906b63924f9e5828f10aff9eed38b7786e8a3434f228cde599e6f0ce7a92c9b"}
Nov 02 13:15:21.280209 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:21.280192    4559 scope.go:122] "RemoveContainer" containerID="599f3006d59a900708c4741f59084b0e0b30def41796a2bddb5f1012da517988"
Nov 02 13:15:21.337203 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:21.337176    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9139ec-29893755-ghrkl"]
Nov 02 13:15:21.348888 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:21.348863    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9139ec-29893755-ghrkl"]
Nov 02 13:15:21.969595 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:21.969559    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3dc1fcaf-e4d8-4912-af05-a63c7fba2506" path="/var/lib/kubelet/pods/3dc1fcaf-e4d8-4912-af05-a63c7fba2506/volumes"
Nov 02 13:15:33.362097 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:33.362017    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-q77g2"]
Nov 02 13:15:33.362821 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:33.362797    4559 cpu_manager.go:414] "containerMap: removing container" podUID="74991db3-76e1-4a11-8fa8-e2dff64107e1" containerName="cleanup"
Nov 02 13:15:33.362821 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:33.362813    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74991db3-76e1-4a11-8fa8-e2dff64107e1" containerName="cleanup"
Nov 02 13:15:33.362914 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:33.362828    4559 cpu_manager.go:414] "containerMap: removing container" podUID="3dc1fcaf-e4d8-4912-af05-a63c7fba2506" containerName="cleanup"
Nov 02 13:15:33.362914 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:33.362834    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3dc1fcaf-e4d8-4912-af05-a63c7fba2506" containerName="cleanup"
Nov 02 13:15:33.362914 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:33.362840    4559 cpu_manager.go:414] "containerMap: removing container" podUID="74991db3-76e1-4a11-8fa8-e2dff64107e1" containerName="cleanup"
Nov 02 13:15:33.362914 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:33.362847    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74991db3-76e1-4a11-8fa8-e2dff64107e1" containerName="cleanup"
Nov 02 13:15:33.362914 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:33.362869    4559 cpu_manager.go:414] "containerMap: removing container" podUID="a35985fb-30d2-49a6-a34d-3b6c2d4ce2b6" containerName="cleanup"
Nov 02 13:15:33.362914 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:33.362876    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a35985fb-30d2-49a6-a34d-3b6c2d4ce2b6" containerName="cleanup"
Nov 02 13:15:33.362914 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:33.362904    4559 cpu_manager.go:414] "containerMap: removing container" podUID="74991db3-76e1-4a11-8fa8-e2dff64107e1" containerName="cleanup"
Nov 02 13:15:33.362914 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:33.362911    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74991db3-76e1-4a11-8fa8-e2dff64107e1" containerName="cleanup"
Nov 02 13:15:33.363159 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:33.362929    4559 cpu_manager.go:414] "containerMap: removing container" podUID="a35985fb-30d2-49a6-a34d-3b6c2d4ce2b6" containerName="cleanup"
Nov 02 13:15:33.363159 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:33.362935    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a35985fb-30d2-49a6-a34d-3b6c2d4ce2b6" containerName="cleanup"
Nov 02 13:15:33.363159 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:33.362943    4559 cpu_manager.go:414] "containerMap: removing container" podUID="a35985fb-30d2-49a6-a34d-3b6c2d4ce2b6" containerName="cleanup"
Nov 02 13:15:33.363159 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:33.362947    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a35985fb-30d2-49a6-a34d-3b6c2d4ce2b6" containerName="cleanup"
Nov 02 13:15:33.363159 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:33.362952    4559 cpu_manager.go:414] "containerMap: removing container" podUID="3dc1fcaf-e4d8-4912-af05-a63c7fba2506" containerName="cleanup"
Nov 02 13:15:33.363159 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:33.362957    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3dc1fcaf-e4d8-4912-af05-a63c7fba2506" containerName="cleanup"
Nov 02 13:15:33.363159 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:33.362966    4559 cpu_manager.go:414] "containerMap: removing container" podUID="3dc1fcaf-e4d8-4912-af05-a63c7fba2506" containerName="cleanup"
Nov 02 13:15:33.363159 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:33.362969    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3dc1fcaf-e4d8-4912-af05-a63c7fba2506" containerName="cleanup"
Nov 02 13:15:33.363159 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:33.363084    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="74991db3-76e1-4a11-8fa8-e2dff64107e1" containerName="cleanup"
Nov 02 13:15:33.363159 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:33.363093    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="3dc1fcaf-e4d8-4912-af05-a63c7fba2506" containerName="cleanup"
Nov 02 13:15:33.363159 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:33.363099    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="74991db3-76e1-4a11-8fa8-e2dff64107e1" containerName="cleanup"
Nov 02 13:15:33.363159 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:33.363110    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="a35985fb-30d2-49a6-a34d-3b6c2d4ce2b6" containerName="cleanup"
Nov 02 13:15:33.363159 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:33.363120    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="74991db3-76e1-4a11-8fa8-e2dff64107e1" containerName="cleanup"
Nov 02 13:15:33.363159 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:33.363126    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="a35985fb-30d2-49a6-a34d-3b6c2d4ce2b6" containerName="cleanup"
Nov 02 13:15:33.363159 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:33.363132    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="a35985fb-30d2-49a6-a34d-3b6c2d4ce2b6" containerName="cleanup"
Nov 02 13:15:33.363159 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:33.363140    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="3dc1fcaf-e4d8-4912-af05-a63c7fba2506" containerName="cleanup"
Nov 02 13:15:33.368181 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:33.368163    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-q77g2"
Nov 02 13:15:33.387547 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:33.387521    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-q77g2"]
Nov 02 13:15:33.465862 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:33.465829    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/279e09ef-b780-457c-a0b0-66948ee0f699-istio-envoy\") pod \"e2e-inf-a-5f8899-openshift-default-67cc5d8789-q77g2\" (UID: \"279e09ef-b780-457c-a0b0-66948ee0f699\") " pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-q77g2"
Nov 02 13:15:33.465862 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:33.465866    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7tqn\" (UniqueName: \"kubernetes.io/projected/279e09ef-b780-457c-a0b0-66948ee0f699-kube-api-access-x7tqn\") pod \"e2e-inf-a-5f8899-openshift-default-67cc5d8789-q77g2\" (UID: \"279e09ef-b780-457c-a0b0-66948ee0f699\") " pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-q77g2"
Nov 02 13:15:33.466065 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:33.465980    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/279e09ef-b780-457c-a0b0-66948ee0f699-istio-token\") pod \"e2e-inf-a-5f8899-openshift-default-67cc5d8789-q77g2\" (UID: \"279e09ef-b780-457c-a0b0-66948ee0f699\") " pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-q77g2"
Nov 02 13:15:33.466065 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:33.466008    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/279e09ef-b780-457c-a0b0-66948ee0f699-istio-data\") pod \"e2e-inf-a-5f8899-openshift-default-67cc5d8789-q77g2\" (UID: \"279e09ef-b780-457c-a0b0-66948ee0f699\") " pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-q77g2"
Nov 02 13:15:33.466065 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:33.466025    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/279e09ef-b780-457c-a0b0-66948ee0f699-istio-podinfo\") pod \"e2e-inf-a-5f8899-openshift-default-67cc5d8789-q77g2\" (UID: \"279e09ef-b780-457c-a0b0-66948ee0f699\") " pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-q77g2"
Nov 02 13:15:33.466175 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:33.466144    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/279e09ef-b780-457c-a0b0-66948ee0f699-workload-certs\") pod \"e2e-inf-a-5f8899-openshift-default-67cc5d8789-q77g2\" (UID: \"279e09ef-b780-457c-a0b0-66948ee0f699\") " pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-q77g2"
Nov 02 13:15:33.466250 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:33.466237    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/279e09ef-b780-457c-a0b0-66948ee0f699-credential-socket\") pod \"e2e-inf-a-5f8899-openshift-default-67cc5d8789-q77g2\" (UID: \"279e09ef-b780-457c-a0b0-66948ee0f699\") " pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-q77g2"
Nov 02 13:15:33.466312 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:33.466302    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/279e09ef-b780-457c-a0b0-66948ee0f699-workload-socket\") pod \"e2e-inf-a-5f8899-openshift-default-67cc5d8789-q77g2\" (UID: \"279e09ef-b780-457c-a0b0-66948ee0f699\") " pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-q77g2"
Nov 02 13:15:33.466386 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:33.466372    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/279e09ef-b780-457c-a0b0-66948ee0f699-istiod-ca-cert\") pod \"e2e-inf-a-5f8899-openshift-default-67cc5d8789-q77g2\" (UID: \"279e09ef-b780-457c-a0b0-66948ee0f699\") " pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-q77g2"
Nov 02 13:15:33.568125 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:33.568082    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/279e09ef-b780-457c-a0b0-66948ee0f699-istio-envoy\") pod \"e2e-inf-a-5f8899-openshift-default-67cc5d8789-q77g2\" (UID: \"279e09ef-b780-457c-a0b0-66948ee0f699\") " pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-q77g2"
Nov 02 13:15:33.568333 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:33.568142    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7tqn\" (UniqueName: \"kubernetes.io/projected/279e09ef-b780-457c-a0b0-66948ee0f699-kube-api-access-x7tqn\") pod \"e2e-inf-a-5f8899-openshift-default-67cc5d8789-q77g2\" (UID: \"279e09ef-b780-457c-a0b0-66948ee0f699\") " pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-q77g2"
Nov 02 13:15:33.568408 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:33.568336    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/279e09ef-b780-457c-a0b0-66948ee0f699-istio-token\") pod \"e2e-inf-a-5f8899-openshift-default-67cc5d8789-q77g2\" (UID: \"279e09ef-b780-457c-a0b0-66948ee0f699\") " pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-q77g2"
Nov 02 13:15:33.568408 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:33.568394    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/279e09ef-b780-457c-a0b0-66948ee0f699-istio-data\") pod \"e2e-inf-a-5f8899-openshift-default-67cc5d8789-q77g2\" (UID: \"279e09ef-b780-457c-a0b0-66948ee0f699\") " pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-q77g2"
Nov 02 13:15:33.568529 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:33.568512    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/279e09ef-b780-457c-a0b0-66948ee0f699-istio-podinfo\") pod \"e2e-inf-a-5f8899-openshift-default-67cc5d8789-q77g2\" (UID: \"279e09ef-b780-457c-a0b0-66948ee0f699\") " pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-q77g2"
Nov 02 13:15:33.568673 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:33.568629    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/279e09ef-b780-457c-a0b0-66948ee0f699-workload-certs\") pod \"e2e-inf-a-5f8899-openshift-default-67cc5d8789-q77g2\" (UID: \"279e09ef-b780-457c-a0b0-66948ee0f699\") " pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-q77g2"
Nov 02 13:15:33.568744 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:33.568727    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/279e09ef-b780-457c-a0b0-66948ee0f699-istio-data\") pod \"e2e-inf-a-5f8899-openshift-default-67cc5d8789-q77g2\" (UID: \"279e09ef-b780-457c-a0b0-66948ee0f699\") " pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-q77g2"
Nov 02 13:15:33.568804 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:33.568794    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/279e09ef-b780-457c-a0b0-66948ee0f699-credential-socket\") pod \"e2e-inf-a-5f8899-openshift-default-67cc5d8789-q77g2\" (UID: \"279e09ef-b780-457c-a0b0-66948ee0f699\") " pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-q77g2"
Nov 02 13:15:33.568872 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:33.568863    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/279e09ef-b780-457c-a0b0-66948ee0f699-workload-socket\") pod \"e2e-inf-a-5f8899-openshift-default-67cc5d8789-q77g2\" (UID: \"279e09ef-b780-457c-a0b0-66948ee0f699\") " pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-q77g2"
Nov 02 13:15:33.568960 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:33.568944    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/279e09ef-b780-457c-a0b0-66948ee0f699-istiod-ca-cert\") pod \"e2e-inf-a-5f8899-openshift-default-67cc5d8789-q77g2\" (UID: \"279e09ef-b780-457c-a0b0-66948ee0f699\") " pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-q77g2"
Nov 02 13:15:33.569018 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:33.568984    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/279e09ef-b780-457c-a0b0-66948ee0f699-workload-certs\") pod \"e2e-inf-a-5f8899-openshift-default-67cc5d8789-q77g2\" (UID: \"279e09ef-b780-457c-a0b0-66948ee0f699\") " pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-q77g2"
Nov 02 13:15:33.569260 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:33.569234    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/279e09ef-b780-457c-a0b0-66948ee0f699-credential-socket\") pod \"e2e-inf-a-5f8899-openshift-default-67cc5d8789-q77g2\" (UID: \"279e09ef-b780-457c-a0b0-66948ee0f699\") " pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-q77g2"
Nov 02 13:15:33.569299 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:33.569255    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/279e09ef-b780-457c-a0b0-66948ee0f699-workload-socket\") pod \"e2e-inf-a-5f8899-openshift-default-67cc5d8789-q77g2\" (UID: \"279e09ef-b780-457c-a0b0-66948ee0f699\") " pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-q77g2"
Nov 02 13:15:33.569570 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:33.569554    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/279e09ef-b780-457c-a0b0-66948ee0f699-istiod-ca-cert\") pod \"e2e-inf-a-5f8899-openshift-default-67cc5d8789-q77g2\" (UID: \"279e09ef-b780-457c-a0b0-66948ee0f699\") " pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-q77g2"
Nov 02 13:15:33.570570 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:33.570545    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/279e09ef-b780-457c-a0b0-66948ee0f699-istio-envoy\") pod \"e2e-inf-a-5f8899-openshift-default-67cc5d8789-q77g2\" (UID: \"279e09ef-b780-457c-a0b0-66948ee0f699\") " pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-q77g2"
Nov 02 13:15:33.571046 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:33.571022    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/279e09ef-b780-457c-a0b0-66948ee0f699-istio-podinfo\") pod \"e2e-inf-a-5f8899-openshift-default-67cc5d8789-q77g2\" (UID: \"279e09ef-b780-457c-a0b0-66948ee0f699\") " pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-q77g2"
Nov 02 13:15:33.592467 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:33.592437    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/279e09ef-b780-457c-a0b0-66948ee0f699-istio-token\") pod \"e2e-inf-a-5f8899-openshift-default-67cc5d8789-q77g2\" (UID: \"279e09ef-b780-457c-a0b0-66948ee0f699\") " pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-q77g2"
Nov 02 13:15:33.592595 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:33.592580    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7tqn\" (UniqueName: \"kubernetes.io/projected/279e09ef-b780-457c-a0b0-66948ee0f699-kube-api-access-x7tqn\") pod \"e2e-inf-a-5f8899-openshift-default-67cc5d8789-q77g2\" (UID: \"279e09ef-b780-457c-a0b0-66948ee0f699\") " pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-q77g2"
Nov 02 13:15:33.682933 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:33.682904    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-q77g2"
Nov 02 13:15:34.014265 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:34.014240    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-q77g2"]
Nov 02 13:15:34.016031 ip-10-0-128-132 kubenswrapper[4559]: W1102 13:15:34.016005    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod279e09ef_b780_457c_a0b0_66948ee0f699.slice/crio-33bf23a44a39e9a239aea3f710edbc6478151d4d5d04aa88cb03a231d3651467 WatchSource:0}: Error finding container 33bf23a44a39e9a239aea3f710edbc6478151d4d5d04aa88cb03a231d3651467: Status 404 returned error can't find the container with id 33bf23a44a39e9a239aea3f710edbc6478151d4d5d04aa88cb03a231d3651467
Nov 02 13:15:34.019016 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:34.018984    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 13:15:34.019099 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:34.019057    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 13:15:34.019099 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:34.019086    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 13:15:34.398561 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:34.398447    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-q77g2" event={"ID":"279e09ef-b780-457c-a0b0-66948ee0f699","Type":"ContainerStarted","Data":"6611f45b056f82ddcaaac585cbe7560a0cd621b022bbf602a9c07b50152dcad5"}
Nov 02 13:15:34.398561 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:34.398554    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-q77g2" event={"ID":"279e09ef-b780-457c-a0b0-66948ee0f699","Type":"ContainerStarted","Data":"33bf23a44a39e9a239aea3f710edbc6478151d4d5d04aa88cb03a231d3651467"}
Nov 02 13:15:34.451215 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:34.451155    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-q77g2" podStartSLOduration=1.451138214 podStartE2EDuration="1.451138214s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:15:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:15:34.447200553 +0000 UTC m=+2930.702599661" watchObservedRunningTime="2026-11-02 13:15:34.451138214 +0000 UTC m=+2930.706537308"
Nov 02 13:15:34.683200 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:34.683163    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-q77g2"
Nov 02 13:15:35.727660 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:35.726831    4559 patch_prober.go:29] interesting pod/e2e-inf-a-5f8899-openshift-default-67cc5d8789-q77g2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused" start-of-body=
Nov 02 13:15:35.727660 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:35.726897    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-q77g2" podUID="279e09ef-b780-457c-a0b0-66948ee0f699" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused"
Nov 02 13:15:36.707662 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:36.706830    4559 patch_prober.go:29] interesting pod/e2e-inf-a-5f8899-openshift-default-67cc5d8789-q77g2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused" start-of-body=
Nov 02 13:15:36.707662 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:36.706904    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-q77g2" podUID="279e09ef-b780-457c-a0b0-66948ee0f699" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused"
Nov 02 13:15:37.688130 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:37.688089    4559 patch_prober.go:29] interesting pod/e2e-inf-a-5f8899-openshift-default-67cc5d8789-q77g2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused" start-of-body=
Nov 02 13:15:37.688816 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:37.688781    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-q77g2" podUID="279e09ef-b780-457c-a0b0-66948ee0f699" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused"
Nov 02 13:15:38.707669 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:38.701723    4559 patch_prober.go:29] interesting pod/e2e-inf-a-5f8899-openshift-default-67cc5d8789-q77g2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused" start-of-body=
Nov 02 13:15:38.707669 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:38.701803    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-q77g2" podUID="279e09ef-b780-457c-a0b0-66948ee0f699" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused"
Nov 02 13:15:39.642036 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:39.641953    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"]
Nov 02 13:15:39.705664 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:39.696652    4559 patch_prober.go:29] interesting pod/e2e-inf-a-5f8899-openshift-default-67cc5d8789-q77g2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused" start-of-body=
Nov 02 13:15:39.705664 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:39.696722    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-q77g2" podUID="279e09ef-b780-457c-a0b0-66948ee0f699" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused"
Nov 02 13:15:40.689661 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:40.689413    4559 patch_prober.go:29] interesting pod/e2e-inf-a-5f8899-openshift-default-67cc5d8789-q77g2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused" start-of-body=
Nov 02 13:15:40.689661 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:40.689550    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-q77g2" podUID="279e09ef-b780-457c-a0b0-66948ee0f699" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused"
Nov 02 13:15:41.689429 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:41.689393    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-q77g2"
Nov 02 13:15:41.689846 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:41.689821    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-q77g2"
Nov 02 13:15:41.690683 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:41.690633    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-q77g2"
Nov 02 13:15:45.087468 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:15:45.087425    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqcmj" containerName="extractor"
Nov 02 13:15:55.027027 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:55.026987    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-48d74e-55dbcdf789-hznjw"]
Nov 02 13:15:55.028159 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:55.028133    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="3dc1fcaf-e4d8-4912-af05-a63c7fba2506" containerName="cleanup"
Nov 02 13:15:55.033415 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:55.033393    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-48d74e-55dbcdf789-hznjw"
Nov 02 13:15:55.052414 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:55.052381    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-48d74e-55dbcdf789-hznjw"]
Nov 02 13:15:55.053622 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:55.053587    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-48d74e-serving-cert\""
Nov 02 13:15:55.115184 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:55.115145    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5c39b0a-8b09-4c62-87e4-223aad82d163-metrics-certs\") pod \"maas-api-e2e-inf-b-48d74e-55dbcdf789-hznjw\" (UID: \"d5c39b0a-8b09-4c62-87e4-223aad82d163\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-48d74e-55dbcdf789-hznjw"
Nov 02 13:15:55.115356 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:55.115223    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5c39b0a-8b09-4c62-87e4-223aad82d163-maas-api-tls\") pod \"maas-api-e2e-inf-b-48d74e-55dbcdf789-hznjw\" (UID: \"d5c39b0a-8b09-4c62-87e4-223aad82d163\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-48d74e-55dbcdf789-hznjw"
Nov 02 13:15:55.115434 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:55.115416    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfm7w\" (UniqueName: \"kubernetes.io/projected/d5c39b0a-8b09-4c62-87e4-223aad82d163-kube-api-access-jfm7w\") pod \"maas-api-e2e-inf-b-48d74e-55dbcdf789-hznjw\" (UID: \"d5c39b0a-8b09-4c62-87e4-223aad82d163\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-48d74e-55dbcdf789-hznjw"
Nov 02 13:15:55.115473 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:55.115464    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5c39b0a-8b09-4c62-87e4-223aad82d163-openshift-service-ca\") pod \"maas-api-e2e-inf-b-48d74e-55dbcdf789-hznjw\" (UID: \"d5c39b0a-8b09-4c62-87e4-223aad82d163\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-48d74e-55dbcdf789-hznjw"
Nov 02 13:15:55.260661 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:55.248624    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5c39b0a-8b09-4c62-87e4-223aad82d163-openshift-service-ca\") pod \"maas-api-e2e-inf-b-48d74e-55dbcdf789-hznjw\" (UID: \"d5c39b0a-8b09-4c62-87e4-223aad82d163\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-48d74e-55dbcdf789-hznjw"
Nov 02 13:15:55.279716 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:55.272990    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5c39b0a-8b09-4c62-87e4-223aad82d163-metrics-certs\") pod \"maas-api-e2e-inf-b-48d74e-55dbcdf789-hznjw\" (UID: \"d5c39b0a-8b09-4c62-87e4-223aad82d163\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-48d74e-55dbcdf789-hznjw"
Nov 02 13:15:55.279716 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:55.273076    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5c39b0a-8b09-4c62-87e4-223aad82d163-maas-api-tls\") pod \"maas-api-e2e-inf-b-48d74e-55dbcdf789-hznjw\" (UID: \"d5c39b0a-8b09-4c62-87e4-223aad82d163\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-48d74e-55dbcdf789-hznjw"
Nov 02 13:15:55.279716 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:55.273288    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfm7w\" (UniqueName: \"kubernetes.io/projected/d5c39b0a-8b09-4c62-87e4-223aad82d163-kube-api-access-jfm7w\") pod \"maas-api-e2e-inf-b-48d74e-55dbcdf789-hznjw\" (UID: \"d5c39b0a-8b09-4c62-87e4-223aad82d163\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-48d74e-55dbcdf789-hznjw"
Nov 02 13:15:55.279716 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:55.260097    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5c39b0a-8b09-4c62-87e4-223aad82d163-openshift-service-ca\") pod \"maas-api-e2e-inf-b-48d74e-55dbcdf789-hznjw\" (UID: \"d5c39b0a-8b09-4c62-87e4-223aad82d163\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-48d74e-55dbcdf789-hznjw"
Nov 02 13:15:55.298667 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:55.289007    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5c39b0a-8b09-4c62-87e4-223aad82d163-metrics-certs\") pod \"maas-api-e2e-inf-b-48d74e-55dbcdf789-hznjw\" (UID: \"d5c39b0a-8b09-4c62-87e4-223aad82d163\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-48d74e-55dbcdf789-hznjw"
Nov 02 13:15:55.307274 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:55.307240    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5c39b0a-8b09-4c62-87e4-223aad82d163-maas-api-tls\") pod \"maas-api-e2e-inf-b-48d74e-55dbcdf789-hznjw\" (UID: \"d5c39b0a-8b09-4c62-87e4-223aad82d163\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-48d74e-55dbcdf789-hznjw"
Nov 02 13:15:55.334705 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:55.332578    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfm7w\" (UniqueName: \"kubernetes.io/projected/d5c39b0a-8b09-4c62-87e4-223aad82d163-kube-api-access-jfm7w\") pod \"maas-api-e2e-inf-b-48d74e-55dbcdf789-hznjw\" (UID: \"d5c39b0a-8b09-4c62-87e4-223aad82d163\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-48d74e-55dbcdf789-hznjw"
Nov 02 13:15:55.349508 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:55.349481    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-48d74e-55dbcdf789-hznjw"
Nov 02 13:15:55.665693 ip-10-0-128-132 kubenswrapper[4559]: W1102 13:15:55.665665    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5c39b0a_8b09_4c62_87e4_223aad82d163.slice/crio-6be2bbac28b21c4b0e8ec91fe1b10be29e3ca0a7f8eef4aaeaea16d4169e699f WatchSource:0}: Error finding container 6be2bbac28b21c4b0e8ec91fe1b10be29e3ca0a7f8eef4aaeaea16d4169e699f: Status 404 returned error can't find the container with id 6be2bbac28b21c4b0e8ec91fe1b10be29e3ca0a7f8eef4aaeaea16d4169e699f
Nov 02 13:15:55.666891 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:55.666863    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-48d74e-55dbcdf789-hznjw"]
Nov 02 13:15:56.678219 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:56.678184    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-48d74e-55dbcdf789-hznjw" event={"ID":"d5c39b0a-8b09-4c62-87e4-223aad82d163","Type":"ContainerStarted","Data":"1895262d20402f67d415982b8f84a9be237de44013def41310b4f67b2bef063e"}
Nov 02 13:15:56.678717 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:56.678247    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-48d74e-55dbcdf789-hznjw" event={"ID":"d5c39b0a-8b09-4c62-87e4-223aad82d163","Type":"ContainerStarted","Data":"6be2bbac28b21c4b0e8ec91fe1b10be29e3ca0a7f8eef4aaeaea16d4169e699f"}
Nov 02 13:15:56.678717 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:56.678278    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-48d74e-55dbcdf789-hznjw"
Nov 02 13:15:56.724178 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:15:56.724121    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-48d74e-55dbcdf789-hznjw" podStartSLOduration=2.362784978 podStartE2EDuration="2.724106138s" totalImagesPullingTime="361.32116ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:15:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:15:56.721138582 +0000 UTC m=+2952.976537696" watchObservedRunningTime="2026-11-02 13:15:56.724106138 +0000 UTC m=+2952.979505234"
Nov 02 13:16:02.766808 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:16:02.765785    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-48d74e-55dbcdf789-hznjw"
Nov 02 13:16:15.087179 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:16:15.087137    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqcmj" containerName="extractor"
Nov 02 13:16:31.320680 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:16:31.312664    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"]
Nov 02 13:16:45.087154 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:16:45.087107    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqcmj" containerName="extractor"
Nov 02 13:16:52.276715 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:16:52.276681    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-361c3a/test-model-361c3a-kserve-5fdb8ffcc-jjfbm"]
Nov 02 13:16:52.281616 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:16:52.281589    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-361c3a/test-model-361c3a-kserve-5fdb8ffcc-jjfbm"
Nov 02 13:16:52.300558 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:16:52.300530    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-361c3a/test-model-361c3a-kserve-5fdb8ffcc-jjfbm"]
Nov 02 13:16:52.306774 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:16:52.306748    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-361c3a\"/\"openshift-service-ca.crt\""
Nov 02 13:16:52.306774 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:16:52.306757    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-361c3a\"/\"kube-root-ca.crt\""
Nov 02 13:16:52.310481 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:16:52.310460    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-361c3a\"/\"test-model-361c3a-kserve-self-signed-certs\""
Nov 02 13:16:52.310481 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:16:52.310464    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-361c3a\"/\"default-dockercfg-kl9rc\""
Nov 02 13:16:52.372449 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:16:52.372419    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f43f207-7f92-4aab-a554-903c5d18c8f3-dshm\") pod \"test-model-361c3a-kserve-5fdb8ffcc-jjfbm\" (UID: \"2f43f207-7f92-4aab-a554-903c5d18c8f3\") " pod="ai-tenant-e2e-inf-b-361c3a/test-model-361c3a-kserve-5fdb8ffcc-jjfbm"
Nov 02 13:16:52.372617 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:16:52.372528    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f43f207-7f92-4aab-a554-903c5d18c8f3-tls-certs\") pod \"test-model-361c3a-kserve-5fdb8ffcc-jjfbm\" (UID: \"2f43f207-7f92-4aab-a554-903c5d18c8f3\") " pod="ai-tenant-e2e-inf-b-361c3a/test-model-361c3a-kserve-5fdb8ffcc-jjfbm"
Nov 02 13:16:52.372713 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:16:52.372624    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f43f207-7f92-4aab-a554-903c5d18c8f3-model-cache\") pod \"test-model-361c3a-kserve-5fdb8ffcc-jjfbm\" (UID: \"2f43f207-7f92-4aab-a554-903c5d18c8f3\") " pod="ai-tenant-e2e-inf-b-361c3a/test-model-361c3a-kserve-5fdb8ffcc-jjfbm"
Nov 02 13:16:52.372775 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:16:52.372763    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjc2x\" (UniqueName: \"kubernetes.io/projected/2f43f207-7f92-4aab-a554-903c5d18c8f3-kube-api-access-bjc2x\") pod \"test-model-361c3a-kserve-5fdb8ffcc-jjfbm\" (UID: \"2f43f207-7f92-4aab-a554-903c5d18c8f3\") " pod="ai-tenant-e2e-inf-b-361c3a/test-model-361c3a-kserve-5fdb8ffcc-jjfbm"
Nov 02 13:16:52.372830 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:16:52.372789    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f43f207-7f92-4aab-a554-903c5d18c8f3-home\") pod \"test-model-361c3a-kserve-5fdb8ffcc-jjfbm\" (UID: \"2f43f207-7f92-4aab-a554-903c5d18c8f3\") " pod="ai-tenant-e2e-inf-b-361c3a/test-model-361c3a-kserve-5fdb8ffcc-jjfbm"
Nov 02 13:16:52.372884 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:16:52.372824    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f43f207-7f92-4aab-a554-903c5d18c8f3-tmp-dir\") pod \"test-model-361c3a-kserve-5fdb8ffcc-jjfbm\" (UID: \"2f43f207-7f92-4aab-a554-903c5d18c8f3\") " pod="ai-tenant-e2e-inf-b-361c3a/test-model-361c3a-kserve-5fdb8ffcc-jjfbm"
Nov 02 13:16:52.473946 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:16:52.473912    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f43f207-7f92-4aab-a554-903c5d18c8f3-model-cache\") pod \"test-model-361c3a-kserve-5fdb8ffcc-jjfbm\" (UID: \"2f43f207-7f92-4aab-a554-903c5d18c8f3\") " pod="ai-tenant-e2e-inf-b-361c3a/test-model-361c3a-kserve-5fdb8ffcc-jjfbm"
Nov 02 13:16:52.474087 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:16:52.474032    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjc2x\" (UniqueName: \"kubernetes.io/projected/2f43f207-7f92-4aab-a554-903c5d18c8f3-kube-api-access-bjc2x\") pod \"test-model-361c3a-kserve-5fdb8ffcc-jjfbm\" (UID: \"2f43f207-7f92-4aab-a554-903c5d18c8f3\") " pod="ai-tenant-e2e-inf-b-361c3a/test-model-361c3a-kserve-5fdb8ffcc-jjfbm"
Nov 02 13:16:52.474132 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:16:52.474083    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f43f207-7f92-4aab-a554-903c5d18c8f3-home\") pod \"test-model-361c3a-kserve-5fdb8ffcc-jjfbm\" (UID: \"2f43f207-7f92-4aab-a554-903c5d18c8f3\") " pod="ai-tenant-e2e-inf-b-361c3a/test-model-361c3a-kserve-5fdb8ffcc-jjfbm"
Nov 02 13:16:52.474179 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:16:52.474137    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f43f207-7f92-4aab-a554-903c5d18c8f3-tmp-dir\") pod \"test-model-361c3a-kserve-5fdb8ffcc-jjfbm\" (UID: \"2f43f207-7f92-4aab-a554-903c5d18c8f3\") " pod="ai-tenant-e2e-inf-b-361c3a/test-model-361c3a-kserve-5fdb8ffcc-jjfbm"
Nov 02 13:16:52.474313 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:16:52.474296    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f43f207-7f92-4aab-a554-903c5d18c8f3-dshm\") pod \"test-model-361c3a-kserve-5fdb8ffcc-jjfbm\" (UID: \"2f43f207-7f92-4aab-a554-903c5d18c8f3\") " pod="ai-tenant-e2e-inf-b-361c3a/test-model-361c3a-kserve-5fdb8ffcc-jjfbm"
Nov 02 13:16:52.474409 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:16:52.474359    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f43f207-7f92-4aab-a554-903c5d18c8f3-model-cache\") pod \"test-model-361c3a-kserve-5fdb8ffcc-jjfbm\" (UID: \"2f43f207-7f92-4aab-a554-903c5d18c8f3\") " pod="ai-tenant-e2e-inf-b-361c3a/test-model-361c3a-kserve-5fdb8ffcc-jjfbm"
Nov 02 13:16:52.474474 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:16:52.474420    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f43f207-7f92-4aab-a554-903c5d18c8f3-tls-certs\") pod \"test-model-361c3a-kserve-5fdb8ffcc-jjfbm\" (UID: \"2f43f207-7f92-4aab-a554-903c5d18c8f3\") " pod="ai-tenant-e2e-inf-b-361c3a/test-model-361c3a-kserve-5fdb8ffcc-jjfbm"
Nov 02 13:16:52.474474 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:16:52.474425    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f43f207-7f92-4aab-a554-903c5d18c8f3-home\") pod \"test-model-361c3a-kserve-5fdb8ffcc-jjfbm\" (UID: \"2f43f207-7f92-4aab-a554-903c5d18c8f3\") " pod="ai-tenant-e2e-inf-b-361c3a/test-model-361c3a-kserve-5fdb8ffcc-jjfbm"
Nov 02 13:16:52.474594 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:16:52.474549    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f43f207-7f92-4aab-a554-903c5d18c8f3-tmp-dir\") pod \"test-model-361c3a-kserve-5fdb8ffcc-jjfbm\" (UID: \"2f43f207-7f92-4aab-a554-903c5d18c8f3\") " pod="ai-tenant-e2e-inf-b-361c3a/test-model-361c3a-kserve-5fdb8ffcc-jjfbm"
Nov 02 13:16:52.476578 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:16:52.476554    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f43f207-7f92-4aab-a554-903c5d18c8f3-dshm\") pod \"test-model-361c3a-kserve-5fdb8ffcc-jjfbm\" (UID: \"2f43f207-7f92-4aab-a554-903c5d18c8f3\") " pod="ai-tenant-e2e-inf-b-361c3a/test-model-361c3a-kserve-5fdb8ffcc-jjfbm"
Nov 02 13:16:52.476862 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:16:52.476843    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f43f207-7f92-4aab-a554-903c5d18c8f3-tls-certs\") pod \"test-model-361c3a-kserve-5fdb8ffcc-jjfbm\" (UID: \"2f43f207-7f92-4aab-a554-903c5d18c8f3\") " pod="ai-tenant-e2e-inf-b-361c3a/test-model-361c3a-kserve-5fdb8ffcc-jjfbm"
Nov 02 13:16:52.575752 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:16:52.574364    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjc2x\" (UniqueName: \"kubernetes.io/projected/2f43f207-7f92-4aab-a554-903c5d18c8f3-kube-api-access-bjc2x\") pod \"test-model-361c3a-kserve-5fdb8ffcc-jjfbm\" (UID: \"2f43f207-7f92-4aab-a554-903c5d18c8f3\") " pod="ai-tenant-e2e-inf-b-361c3a/test-model-361c3a-kserve-5fdb8ffcc-jjfbm"
Nov 02 13:16:52.603685 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:16:52.601179    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-361c3a/test-model-361c3a-kserve-5fdb8ffcc-jjfbm"
Nov 02 13:16:53.031935 ip-10-0-128-132 kubenswrapper[4559]: W1102 13:16:53.031906    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2f43f207_7f92_4aab_a554_903c5d18c8f3.slice/crio-01950ec6304c475264cf9172ca4b9a14fd1da0088b2b75f7de6ec9efa860202d WatchSource:0}: Error finding container 01950ec6304c475264cf9172ca4b9a14fd1da0088b2b75f7de6ec9efa860202d: Status 404 returned error can't find the container with id 01950ec6304c475264cf9172ca4b9a14fd1da0088b2b75f7de6ec9efa860202d
Nov 02 13:16:53.034498 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:16:53.034465    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-361c3a/test-model-361c3a-kserve-5fdb8ffcc-jjfbm"]
Nov 02 13:16:53.300082 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:16:53.299993    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-361c3a/test-model-361c3a-kserve-5fdb8ffcc-jjfbm" event={"ID":"2f43f207-7f92-4aab-a554-903c5d18c8f3","Type":"ContainerStarted","Data":"b32179fe53f896ba73cfa06d9a9d8814520e60af1f1c6c944b3b8a2b3b01ec7d"}
Nov 02 13:16:53.300082 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:16:53.300047    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-361c3a/test-model-361c3a-kserve-5fdb8ffcc-jjfbm" event={"ID":"2f43f207-7f92-4aab-a554-903c5d18c8f3","Type":"ContainerStarted","Data":"01950ec6304c475264cf9172ca4b9a14fd1da0088b2b75f7de6ec9efa860202d"}
Nov 02 13:16:53.347468 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:16:53.347417    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-361c3a/test-model-361c3a-kserve-5fdb8ffcc-jjfbm" podStartSLOduration=1.347404684 podStartE2EDuration="1.347404684s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:16:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:16:53.343690367 +0000 UTC m=+3009.599089462" watchObservedRunningTime="2026-11-02 13:16:53.347404684 +0000 UTC m=+3009.602803810"
Nov 02 13:16:53.601864 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:16:53.601779    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-361c3a/test-model-361c3a-kserve-5fdb8ffcc-jjfbm"
Nov 02 13:16:56.354109 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:16:56.354015    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-5f8899/test-model-5f8899-kserve-585d469589-78dz5"]
Nov 02 13:16:56.363212 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:16:56.363186    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-5f8899/test-model-5f8899-kserve-585d469589-78dz5"
Nov 02 13:16:56.377972 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:16:56.377942    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-5f8899/test-model-5f8899-kserve-585d469589-78dz5"]
Nov 02 13:16:56.383958 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:16:56.383935    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-5f8899\"/\"openshift-service-ca.crt\""
Nov 02 13:16:56.384073 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:16:56.383988    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-5f8899\"/\"kube-root-ca.crt\""
Nov 02 13:16:56.392221 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:16:56.392199    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-5f8899\"/\"default-dockercfg-h9k79\""
Nov 02 13:16:56.392221 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:16:56.392210    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-5f8899\"/\"test-model-5f8899-kserve-self-signed-certs\""
Nov 02 13:16:56.465360 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:16:56.465328    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3799b3ab-6927-4a3c-a610-2d7eb7fe507c-tmp-dir\") pod \"test-model-5f8899-kserve-585d469589-78dz5\" (UID: \"3799b3ab-6927-4a3c-a610-2d7eb7fe507c\") " pod="ai-tenant-e2e-inf-a-5f8899/test-model-5f8899-kserve-585d469589-78dz5"
Nov 02 13:16:56.465505 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:16:56.465474    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3799b3ab-6927-4a3c-a610-2d7eb7fe507c-dshm\") pod \"test-model-5f8899-kserve-585d469589-78dz5\" (UID: \"3799b3ab-6927-4a3c-a610-2d7eb7fe507c\") " pod="ai-tenant-e2e-inf-a-5f8899/test-model-5f8899-kserve-585d469589-78dz5"
Nov 02 13:16:56.465565 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:16:56.465550    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5w9ng\" (UniqueName: \"kubernetes.io/projected/3799b3ab-6927-4a3c-a610-2d7eb7fe507c-kube-api-access-5w9ng\") pod \"test-model-5f8899-kserve-585d469589-78dz5\" (UID: \"3799b3ab-6927-4a3c-a610-2d7eb7fe507c\") " pod="ai-tenant-e2e-inf-a-5f8899/test-model-5f8899-kserve-585d469589-78dz5"
Nov 02 13:16:56.465611 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:16:56.465588    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3799b3ab-6927-4a3c-a610-2d7eb7fe507c-tls-certs\") pod \"test-model-5f8899-kserve-585d469589-78dz5\" (UID: \"3799b3ab-6927-4a3c-a610-2d7eb7fe507c\") " pod="ai-tenant-e2e-inf-a-5f8899/test-model-5f8899-kserve-585d469589-78dz5"
Nov 02 13:16:56.465694 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:16:56.465681    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3799b3ab-6927-4a3c-a610-2d7eb7fe507c-model-cache\") pod \"test-model-5f8899-kserve-585d469589-78dz5\" (UID: \"3799b3ab-6927-4a3c-a610-2d7eb7fe507c\") " pod="ai-tenant-e2e-inf-a-5f8899/test-model-5f8899-kserve-585d469589-78dz5"
Nov 02 13:16:56.465737 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:16:56.465706    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3799b3ab-6927-4a3c-a610-2d7eb7fe507c-home\") pod \"test-model-5f8899-kserve-585d469589-78dz5\" (UID: \"3799b3ab-6927-4a3c-a610-2d7eb7fe507c\") " pod="ai-tenant-e2e-inf-a-5f8899/test-model-5f8899-kserve-585d469589-78dz5"
Nov 02 13:16:56.567178 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:16:56.567137    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3799b3ab-6927-4a3c-a610-2d7eb7fe507c-tmp-dir\") pod \"test-model-5f8899-kserve-585d469589-78dz5\" (UID: \"3799b3ab-6927-4a3c-a610-2d7eb7fe507c\") " pod="ai-tenant-e2e-inf-a-5f8899/test-model-5f8899-kserve-585d469589-78dz5"
Nov 02 13:16:56.567412 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:16:56.567278    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3799b3ab-6927-4a3c-a610-2d7eb7fe507c-dshm\") pod \"test-model-5f8899-kserve-585d469589-78dz5\" (UID: \"3799b3ab-6927-4a3c-a610-2d7eb7fe507c\") " pod="ai-tenant-e2e-inf-a-5f8899/test-model-5f8899-kserve-585d469589-78dz5"
Nov 02 13:16:56.567412 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:16:56.567386    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5w9ng\" (UniqueName: \"kubernetes.io/projected/3799b3ab-6927-4a3c-a610-2d7eb7fe507c-kube-api-access-5w9ng\") pod \"test-model-5f8899-kserve-585d469589-78dz5\" (UID: \"3799b3ab-6927-4a3c-a610-2d7eb7fe507c\") " pod="ai-tenant-e2e-inf-a-5f8899/test-model-5f8899-kserve-585d469589-78dz5"
Nov 02 13:16:56.567551 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:16:56.567452    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3799b3ab-6927-4a3c-a610-2d7eb7fe507c-tls-certs\") pod \"test-model-5f8899-kserve-585d469589-78dz5\" (UID: \"3799b3ab-6927-4a3c-a610-2d7eb7fe507c\") " pod="ai-tenant-e2e-inf-a-5f8899/test-model-5f8899-kserve-585d469589-78dz5"
Nov 02 13:16:56.567551 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:16:56.567542    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3799b3ab-6927-4a3c-a610-2d7eb7fe507c-model-cache\") pod \"test-model-5f8899-kserve-585d469589-78dz5\" (UID: \"3799b3ab-6927-4a3c-a610-2d7eb7fe507c\") " pod="ai-tenant-e2e-inf-a-5f8899/test-model-5f8899-kserve-585d469589-78dz5"
Nov 02 13:16:56.567725 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:16:56.567680    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3799b3ab-6927-4a3c-a610-2d7eb7fe507c-home\") pod \"test-model-5f8899-kserve-585d469589-78dz5\" (UID: \"3799b3ab-6927-4a3c-a610-2d7eb7fe507c\") " pod="ai-tenant-e2e-inf-a-5f8899/test-model-5f8899-kserve-585d469589-78dz5"
Nov 02 13:16:56.567795 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:16:56.567773    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3799b3ab-6927-4a3c-a610-2d7eb7fe507c-tmp-dir\") pod \"test-model-5f8899-kserve-585d469589-78dz5\" (UID: \"3799b3ab-6927-4a3c-a610-2d7eb7fe507c\") " pod="ai-tenant-e2e-inf-a-5f8899/test-model-5f8899-kserve-585d469589-78dz5"
Nov 02 13:16:56.567965 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:16:56.567945    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3799b3ab-6927-4a3c-a610-2d7eb7fe507c-model-cache\") pod \"test-model-5f8899-kserve-585d469589-78dz5\" (UID: \"3799b3ab-6927-4a3c-a610-2d7eb7fe507c\") " pod="ai-tenant-e2e-inf-a-5f8899/test-model-5f8899-kserve-585d469589-78dz5"
Nov 02 13:16:56.568464 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:16:56.567973    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3799b3ab-6927-4a3c-a610-2d7eb7fe507c-home\") pod \"test-model-5f8899-kserve-585d469589-78dz5\" (UID: \"3799b3ab-6927-4a3c-a610-2d7eb7fe507c\") " pod="ai-tenant-e2e-inf-a-5f8899/test-model-5f8899-kserve-585d469589-78dz5"
Nov 02 13:16:56.570034 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:16:56.570010    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3799b3ab-6927-4a3c-a610-2d7eb7fe507c-dshm\") pod \"test-model-5f8899-kserve-585d469589-78dz5\" (UID: \"3799b3ab-6927-4a3c-a610-2d7eb7fe507c\") " pod="ai-tenant-e2e-inf-a-5f8899/test-model-5f8899-kserve-585d469589-78dz5"
Nov 02 13:16:56.570359 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:16:56.570338    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3799b3ab-6927-4a3c-a610-2d7eb7fe507c-tls-certs\") pod \"test-model-5f8899-kserve-585d469589-78dz5\" (UID: \"3799b3ab-6927-4a3c-a610-2d7eb7fe507c\") " pod="ai-tenant-e2e-inf-a-5f8899/test-model-5f8899-kserve-585d469589-78dz5"
Nov 02 13:16:56.612399 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:16:56.612324    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5w9ng\" (UniqueName: \"kubernetes.io/projected/3799b3ab-6927-4a3c-a610-2d7eb7fe507c-kube-api-access-5w9ng\") pod \"test-model-5f8899-kserve-585d469589-78dz5\" (UID: \"3799b3ab-6927-4a3c-a610-2d7eb7fe507c\") " pod="ai-tenant-e2e-inf-a-5f8899/test-model-5f8899-kserve-585d469589-78dz5"
Nov 02 13:16:56.684023 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:16:56.683982    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-5f8899/test-model-5f8899-kserve-585d469589-78dz5"
Nov 02 13:16:57.098190 ip-10-0-128-132 kubenswrapper[4559]: W1102 13:16:57.098161    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3799b3ab_6927_4a3c_a610_2d7eb7fe507c.slice/crio-b902b6e1348a2b4181060a980e4c7f9cb2803d38d5f58555ca2c70be52cb74af WatchSource:0}: Error finding container b902b6e1348a2b4181060a980e4c7f9cb2803d38d5f58555ca2c70be52cb74af: Status 404 returned error can't find the container with id b902b6e1348a2b4181060a980e4c7f9cb2803d38d5f58555ca2c70be52cb74af
Nov 02 13:16:57.100260 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:16:57.100237    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-5f8899/test-model-5f8899-kserve-585d469589-78dz5"]
Nov 02 13:16:57.331541 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:16:57.331505    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-5f8899/test-model-5f8899-kserve-585d469589-78dz5" event={"ID":"3799b3ab-6927-4a3c-a610-2d7eb7fe507c","Type":"ContainerStarted","Data":"9eb7ef16d099bb488fd4418cf51e91a4be81a35ca555dde1dbcfd0cfd7f370ee"}
Nov 02 13:16:57.331541 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:16:57.331551    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-5f8899/test-model-5f8899-kserve-585d469589-78dz5" event={"ID":"3799b3ab-6927-4a3c-a610-2d7eb7fe507c","Type":"ContainerStarted","Data":"b902b6e1348a2b4181060a980e4c7f9cb2803d38d5f58555ca2c70be52cb74af"}
Nov 02 13:16:57.374389 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:16:57.374299    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-5f8899/test-model-5f8899-kserve-585d469589-78dz5" podStartSLOduration=1.374285521 podStartE2EDuration="1.374285521s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:16:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:16:57.373023891 +0000 UTC m=+3013.628422987" watchObservedRunningTime="2026-11-02 13:16:57.374285521 +0000 UTC m=+3013.629684616"
Nov 02 13:16:57.684722 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:16:57.684691    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-5f8899/test-model-5f8899-kserve-585d469589-78dz5"
Nov 02 13:17:02.602483 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:17:02.602446    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-361c3a/test-model-361c3a-kserve-5fdb8ffcc-jjfbm"
Nov 02 13:17:02.615580 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:17:02.615553    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-361c3a/test-model-361c3a-kserve-5fdb8ffcc-jjfbm"
Nov 02 13:17:03.422356 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:17:03.422324    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-361c3a/test-model-361c3a-kserve-5fdb8ffcc-jjfbm"
Nov 02 13:17:06.684776 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:17:06.684733    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-5f8899/test-model-5f8899-kserve-585d469589-78dz5"
Nov 02 13:17:06.697807 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:17:06.697783    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-5f8899/test-model-5f8899-kserve-585d469589-78dz5"
Nov 02 13:17:07.457167 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:17:07.457131    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-5f8899/test-model-5f8899-kserve-585d469589-78dz5"
Nov 02 13:17:15.345782 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:17:15.342837    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqcmj" containerName="extractor"
Nov 02 13:17:28.436820 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:17:28.436784    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-740f09/payload-processing-external-model-e2e-extmod-a-740f09-645cjzv9l"]
Nov 02 13:17:30.750085 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:17:30.750054    4559 generic.go:356] "Generic (PLEG): container finished" podID="85ec8b4f-746f-4d0c-9560-7a79f0c64ef6" containerID="07fc6ecc53d115dd90560bf90a6b96e92920642b83d1fb307440c71e204217de" exitCode=0
Nov 02 13:17:30.750613 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:17:30.750127    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-j7rnm" event={"ID":"85ec8b4f-746f-4d0c-9560-7a79f0c64ef6","Type":"ContainerDied","Data":"07fc6ecc53d115dd90560bf90a6b96e92920642b83d1fb307440c71e204217de"}
Nov 02 13:17:30.750613 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:17:30.750565    4559 scope.go:122] "RemoveContainer" containerID="07fc6ecc53d115dd90560bf90a6b96e92920642b83d1fb307440c71e204217de"
Nov 02 13:17:30.752623 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:17:30.752593    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 13:17:30.752717 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:17:30.752683    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 13:17:30.752764 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:17:30.752717    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 13:17:31.759931 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:17:31.759884    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-j7rnm" event={"ID":"85ec8b4f-746f-4d0c-9560-7a79f0c64ef6","Type":"ContainerStarted","Data":"8e8db6f9ffc5e4ded648e00007ee1952e10b6529f23b4af6eac3188ba99d9c03"}
Nov 02 13:17:32.078606 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:17:32.078524    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-j7rnm"
Nov 02 13:17:32.079729 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:17:32.079701    4559 patch_prober.go:29] interesting pod/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-j7rnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused" start-of-body=
Nov 02 13:17:32.079838 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:17:32.079764    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-j7rnm" podUID="85ec8b4f-746f-4d0c-9560-7a79f0c64ef6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused"
Nov 02 13:17:32.474129 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:17:32.474096    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-j7rnm"
Nov 02 13:17:33.079342 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:17:33.079307    4559 patch_prober.go:29] interesting pod/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-j7rnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused" start-of-body=
Nov 02 13:17:33.079857 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:17:33.079363    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-j7rnm" podUID="85ec8b4f-746f-4d0c-9560-7a79f0c64ef6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused"
Nov 02 13:17:33.437091 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:17:33.437019    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-740f09/payload-processing-external-model-e2e-extmod-a-740f09-645cjzv9l" podUID="dc513c22-60e4-4789-90db-4a23974df75c" containerName="payload-processing" containerID="cri-o://3652276d6d67bf6a6efceb4619e8ac12b511ac8ae34e205a8f4b41a22abe235c" gracePeriod=25
Nov 02 13:17:33.765373 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:17:33.765343    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-740f09/payload-processing-external-model-e2e-extmod-a-740f09-645cjzv9l"
Nov 02 13:17:33.776704 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:17:33.776679    4559 generic.go:356] "Generic (PLEG): container finished" podID="dc513c22-60e4-4789-90db-4a23974df75c" containerID="3652276d6d67bf6a6efceb4619e8ac12b511ac8ae34e205a8f4b41a22abe235c" exitCode=0
Nov 02 13:17:33.776828 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:17:33.776731    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-740f09/payload-processing-external-model-e2e-extmod-a-740f09-645cjzv9l"
Nov 02 13:17:33.776828 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:17:33.776761    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-740f09/payload-processing-external-model-e2e-extmod-a-740f09-645cjzv9l" event={"ID":"dc513c22-60e4-4789-90db-4a23974df75c","Type":"ContainerDied","Data":"3652276d6d67bf6a6efceb4619e8ac12b511ac8ae34e205a8f4b41a22abe235c"}
Nov 02 13:17:33.776828 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:17:33.776811    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-740f09/payload-processing-external-model-e2e-extmod-a-740f09-645cjzv9l" event={"ID":"dc513c22-60e4-4789-90db-4a23974df75c","Type":"ContainerDied","Data":"12eb07ab4b877a303e3e9c612f12a2ce90213ace8ada8b6ce1574d7f97b7191e"}
Nov 02 13:17:33.776961 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:17:33.776835    4559 scope.go:122] "RemoveContainer" containerID="3652276d6d67bf6a6efceb4619e8ac12b511ac8ae34e205a8f4b41a22abe235c"
Nov 02 13:17:33.814892 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:17:33.814787    4559 scope.go:122] "RemoveContainer" containerID="3652276d6d67bf6a6efceb4619e8ac12b511ac8ae34e205a8f4b41a22abe235c"
Nov 02 13:17:33.815309 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:17:33.815276    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3652276d6d67bf6a6efceb4619e8ac12b511ac8ae34e205a8f4b41a22abe235c\": container with ID starting with 3652276d6d67bf6a6efceb4619e8ac12b511ac8ae34e205a8f4b41a22abe235c not found: ID does not exist" containerID="3652276d6d67bf6a6efceb4619e8ac12b511ac8ae34e205a8f4b41a22abe235c"
Nov 02 13:17:33.815430 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:17:33.815320    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3652276d6d67bf6a6efceb4619e8ac12b511ac8ae34e205a8f4b41a22abe235c"} err="failed to get container status \"3652276d6d67bf6a6efceb4619e8ac12b511ac8ae34e205a8f4b41a22abe235c\": rpc error: code = NotFound desc = could not find container \"3652276d6d67bf6a6efceb4619e8ac12b511ac8ae34e205a8f4b41a22abe235c\": container with ID starting with 3652276d6d67bf6a6efceb4619e8ac12b511ac8ae34e205a8f4b41a22abe235c not found: ID does not exist"
Nov 02 13:17:33.921492 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:17:33.921465    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc513c22-60e4-4789-90db-4a23974df75c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc513c22-60e4-4789-90db-4a23974df75c-plugins-config-volume\") pod \"dc513c22-60e4-4789-90db-4a23974df75c\" (UID: \"dc513c22-60e4-4789-90db-4a23974df75c\") "
Nov 02 13:17:33.921723 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:17:33.921550    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc513c22-60e4-4789-90db-4a23974df75c-provider-credentials\" (UniqueName: \"kubernetes.io/projected/dc513c22-60e4-4789-90db-4a23974df75c-provider-credentials\") pod \"dc513c22-60e4-4789-90db-4a23974df75c\" (UID: \"dc513c22-60e4-4789-90db-4a23974df75c\") "
Nov 02 13:17:33.921860 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:17:33.921835    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc513c22-60e4-4789-90db-4a23974df75c-plugins-config-volume" pod "dc513c22-60e4-4789-90db-4a23974df75c" (UID: "dc513c22-60e4-4789-90db-4a23974df75c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:17:33.922004 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:17:33.921983    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc513c22-60e4-4789-90db-4a23974df75c-kube-api-access-d2cfp\" (UniqueName: \"kubernetes.io/projected/dc513c22-60e4-4789-90db-4a23974df75c-kube-api-access-d2cfp\") pod \"dc513c22-60e4-4789-90db-4a23974df75c\" (UID: \"dc513c22-60e4-4789-90db-4a23974df75c\") "
Nov 02 13:17:33.922385 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:17:33.922368    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc513c22-60e4-4789-90db-4a23974df75c-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/dc513c22-60e4-4789-90db-4a23974df75c-routing-overlay-volume\") pod \"dc513c22-60e4-4789-90db-4a23974df75c\" (UID: \"dc513c22-60e4-4789-90db-4a23974df75c\") "
Nov 02 13:17:33.923339 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:17:33.923311    4559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc513c22-60e4-4789-90db-4a23974df75c-plugins-config-volume\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:17:33.924009 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:17:33.923987    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc513c22-60e4-4789-90db-4a23974df75c-routing-overlay-volume" pod "dc513c22-60e4-4789-90db-4a23974df75c" (UID: "dc513c22-60e4-4789-90db-4a23974df75c"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:17:33.924097 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:17:33.924011    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc513c22-60e4-4789-90db-4a23974df75c-provider-credentials" pod "dc513c22-60e4-4789-90db-4a23974df75c" (UID: "dc513c22-60e4-4789-90db-4a23974df75c"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:17:33.924164 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:17:33.924147    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc513c22-60e4-4789-90db-4a23974df75c-kube-api-access-d2cfp" pod "dc513c22-60e4-4789-90db-4a23974df75c" (UID: "dc513c22-60e4-4789-90db-4a23974df75c"). InnerVolumeSpecName "kube-api-access-d2cfp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:17:34.024868 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:17:34.024791    4559 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/dc513c22-60e4-4789-90db-4a23974df75c-routing-overlay-volume\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:17:34.024868 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:17:34.024821    4559 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/dc513c22-60e4-4789-90db-4a23974df75c-provider-credentials\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:17:34.024868 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:17:34.024833    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d2cfp\" (UniqueName: \"kubernetes.io/projected/dc513c22-60e4-4789-90db-4a23974df75c-kube-api-access-d2cfp\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:17:34.078926 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:17:34.078881    4559 patch_prober.go:29] interesting pod/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-j7rnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused" start-of-body=
Nov 02 13:17:34.079102 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:17:34.078962    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-j7rnm" podUID="85ec8b4f-746f-4d0c-9560-7a79f0c64ef6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused"
Nov 02 13:17:34.155226 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:17:34.155193    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-740f09/payload-processing-external-model-e2e-extmod-a-740f09-645cjzv9l"]
Nov 02 13:17:34.159102 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:17:34.159080    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-740f09/payload-processing-external-model-e2e-extmod-a-740f09-645cjzv9l"]
Nov 02 13:17:35.078907 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:17:35.078868    4559 patch_prober.go:29] interesting pod/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-j7rnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused" start-of-body=
Nov 02 13:17:35.079091 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:17:35.078923    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-j7rnm" podUID="85ec8b4f-746f-4d0c-9560-7a79f0c64ef6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused"
Nov 02 13:17:35.319741 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:17:35.319626    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-23f720-openshift-default-6c8d4f9ff5-vpdqd"]
Nov 02 13:17:35.320805 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:17:35.320780    4559 cpu_manager.go:414] "containerMap: removing container" podUID="dc513c22-60e4-4789-90db-4a23974df75c" containerName="payload-processing"
Nov 02 13:17:35.320805 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:17:35.320801    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc513c22-60e4-4789-90db-4a23974df75c" containerName="payload-processing"
Nov 02 13:17:35.321081 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:17:35.321064    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc513c22-60e4-4789-90db-4a23974df75c" containerName="payload-processing"
Nov 02 13:17:35.326470 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:17:35.326443    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-23f720-openshift-default-6c8d4f9ff5-vpdqd"
Nov 02 13:17:35.369018 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:17:35.368887    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-23f720-openshift-default-dockercfg-gx99f\""
Nov 02 13:17:35.369176 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:17:35.369150    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d3ed32f-021a-456e-85eb-864596d28c3f-istio-podinfo\") pod \"e2e-inf-a-23f720-openshift-default-6c8d4f9ff5-vpdqd\" (UID: \"2d3ed32f-021a-456e-85eb-864596d28c3f\") " pod="openshift-ingress/e2e-inf-a-23f720-openshift-default-6c8d4f9ff5-vpdqd"
Nov 02 13:17:35.369248 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:17:35.369224    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d3ed32f-021a-456e-85eb-864596d28c3f-workload-certs\") pod \"e2e-inf-a-23f720-openshift-default-6c8d4f9ff5-vpdqd\" (UID: \"2d3ed32f-021a-456e-85eb-864596d28c3f\") " pod="openshift-ingress/e2e-inf-a-23f720-openshift-default-6c8d4f9ff5-vpdqd"
Nov 02 13:17:35.369306 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:17:35.369253    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d3ed32f-021a-456e-85eb-864596d28c3f-credential-socket\") pod \"e2e-inf-a-23f720-openshift-default-6c8d4f9ff5-vpdqd\" (UID: \"2d3ed32f-021a-456e-85eb-864596d28c3f\") " pod="openshift-ingress/e2e-inf-a-23f720-openshift-default-6c8d4f9ff5-vpdqd"
Nov 02 13:17:35.371666 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:17:35.369480    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d3ed32f-021a-456e-85eb-864596d28c3f-istio-envoy\") pod \"e2e-inf-a-23f720-openshift-default-6c8d4f9ff5-vpdqd\" (UID: \"2d3ed32f-021a-456e-85eb-864596d28c3f\") " pod="openshift-ingress/e2e-inf-a-23f720-openshift-default-6c8d4f9ff5-vpdqd"
Nov 02 13:17:35.371666 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:17:35.369556    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d3ed32f-021a-456e-85eb-864596d28c3f-istio-token\") pod \"e2e-inf-a-23f720-openshift-default-6c8d4f9ff5-vpdqd\" (UID: \"2d3ed32f-021a-456e-85eb-864596d28c3f\") " pod="openshift-ingress/e2e-inf-a-23f720-openshift-default-6c8d4f9ff5-vpdqd"
Nov 02 13:17:35.371666 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:17:35.369610    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d3ed32f-021a-456e-85eb-864596d28c3f-istiod-ca-cert\") pod \"e2e-inf-a-23f720-openshift-default-6c8d4f9ff5-vpdqd\" (UID: \"2d3ed32f-021a-456e-85eb-864596d28c3f\") " pod="openshift-ingress/e2e-inf-a-23f720-openshift-default-6c8d4f9ff5-vpdqd"
Nov 02 13:17:35.371666 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:17:35.369669    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d3ed32f-021a-456e-85eb-864596d28c3f-istio-data\") pod \"e2e-inf-a-23f720-openshift-default-6c8d4f9ff5-vpdqd\" (UID: \"2d3ed32f-021a-456e-85eb-864596d28c3f\") " pod="openshift-ingress/e2e-inf-a-23f720-openshift-default-6c8d4f9ff5-vpdqd"
Nov 02 13:17:35.371666 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:17:35.369714    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mc6d9\" (UniqueName: \"kubernetes.io/projected/2d3ed32f-021a-456e-85eb-864596d28c3f-kube-api-access-mc6d9\") pod \"e2e-inf-a-23f720-openshift-default-6c8d4f9ff5-vpdqd\" (UID: \"2d3ed32f-021a-456e-85eb-864596d28c3f\") " pod="openshift-ingress/e2e-inf-a-23f720-openshift-default-6c8d4f9ff5-vpdqd"
Nov 02 13:17:35.371666 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:17:35.369770    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d3ed32f-021a-456e-85eb-864596d28c3f-workload-socket\") pod \"e2e-inf-a-23f720-openshift-default-6c8d4f9ff5-vpdqd\" (UID: \"2d3ed32f-021a-456e-85eb-864596d28c3f\") " pod="openshift-ingress/e2e-inf-a-23f720-openshift-default-6c8d4f9ff5-vpdqd"
Nov 02 13:17:35.380324 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:17:35.380298    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-23f720-openshift-default-6c8d4f9ff5-vpdqd"]
Nov 02 13:17:35.517664 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:17:35.484112    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d3ed32f-021a-456e-85eb-864596d28c3f-istio-token\") pod \"e2e-inf-a-23f720-openshift-default-6c8d4f9ff5-vpdqd\" (UID: \"2d3ed32f-021a-456e-85eb-864596d28c3f\") " pod="openshift-ingress/e2e-inf-a-23f720-openshift-default-6c8d4f9ff5-vpdqd"
Nov 02 13:17:35.517664 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:17:35.484206    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d3ed32f-021a-456e-85eb-864596d28c3f-istiod-ca-cert\") pod \"e2e-inf-a-23f720-openshift-default-6c8d4f9ff5-vpdqd\" (UID: \"2d3ed32f-021a-456e-85eb-864596d28c3f\") " pod="openshift-ingress/e2e-inf-a-23f720-openshift-default-6c8d4f9ff5-vpdqd"
Nov 02 13:17:35.517664 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:17:35.484265    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d3ed32f-021a-456e-85eb-864596d28c3f-istio-data\") pod \"e2e-inf-a-23f720-openshift-default-6c8d4f9ff5-vpdqd\" (UID: \"2d3ed32f-021a-456e-85eb-864596d28c3f\") " pod="openshift-ingress/e2e-inf-a-23f720-openshift-default-6c8d4f9ff5-vpdqd"
Nov 02 13:17:35.517664 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:17:35.484320    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mc6d9\" (UniqueName: \"kubernetes.io/projected/2d3ed32f-021a-456e-85eb-864596d28c3f-kube-api-access-mc6d9\") pod \"e2e-inf-a-23f720-openshift-default-6c8d4f9ff5-vpdqd\" (UID: \"2d3ed32f-021a-456e-85eb-864596d28c3f\") " pod="openshift-ingress/e2e-inf-a-23f720-openshift-default-6c8d4f9ff5-vpdqd"
Nov 02 13:17:35.517664 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:17:35.484393    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d3ed32f-021a-456e-85eb-864596d28c3f-workload-socket\") pod \"e2e-inf-a-23f720-openshift-default-6c8d4f9ff5-vpdqd\" (UID: \"2d3ed32f-021a-456e-85eb-864596d28c3f\") " pod="openshift-ingress/e2e-inf-a-23f720-openshift-default-6c8d4f9ff5-vpdqd"
Nov 02 13:17:35.517664 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:17:35.484539    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d3ed32f-021a-456e-85eb-864596d28c3f-istio-podinfo\") pod \"e2e-inf-a-23f720-openshift-default-6c8d4f9ff5-vpdqd\" (UID: \"2d3ed32f-021a-456e-85eb-864596d28c3f\") " pod="openshift-ingress/e2e-inf-a-23f720-openshift-default-6c8d4f9ff5-vpdqd"
Nov 02 13:17:35.517664 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:17:35.484617    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d3ed32f-021a-456e-85eb-864596d28c3f-workload-certs\") pod \"e2e-inf-a-23f720-openshift-default-6c8d4f9ff5-vpdqd\" (UID: \"2d3ed32f-021a-456e-85eb-864596d28c3f\") " pod="openshift-ingress/e2e-inf-a-23f720-openshift-default-6c8d4f9ff5-vpdqd"
Nov 02 13:17:35.517664 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:17:35.484680    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d3ed32f-021a-456e-85eb-864596d28c3f-credential-socket\") pod \"e2e-inf-a-23f720-openshift-default-6c8d4f9ff5-vpdqd\" (UID: \"2d3ed32f-021a-456e-85eb-864596d28c3f\") " pod="openshift-ingress/e2e-inf-a-23f720-openshift-default-6c8d4f9ff5-vpdqd"
Nov 02 13:17:35.517664 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:17:35.484886    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d3ed32f-021a-456e-85eb-864596d28c3f-istio-envoy\") pod \"e2e-inf-a-23f720-openshift-default-6c8d4f9ff5-vpdqd\" (UID: \"2d3ed32f-021a-456e-85eb-864596d28c3f\") " pod="openshift-ingress/e2e-inf-a-23f720-openshift-default-6c8d4f9ff5-vpdqd"
Nov 02 13:17:35.517664 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:17:35.485582    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d3ed32f-021a-456e-85eb-864596d28c3f-workload-socket\") pod \"e2e-inf-a-23f720-openshift-default-6c8d4f9ff5-vpdqd\" (UID: \"2d3ed32f-021a-456e-85eb-864596d28c3f\") " pod="openshift-ingress/e2e-inf-a-23f720-openshift-default-6c8d4f9ff5-vpdqd"
Nov 02 13:17:35.517664 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:17:35.492651    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d3ed32f-021a-456e-85eb-864596d28c3f-istiod-ca-cert\") pod \"e2e-inf-a-23f720-openshift-default-6c8d4f9ff5-vpdqd\" (UID: \"2d3ed32f-021a-456e-85eb-864596d28c3f\") " pod="openshift-ingress/e2e-inf-a-23f720-openshift-default-6c8d4f9ff5-vpdqd"
Nov 02 13:17:35.517664 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:17:35.492953    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d3ed32f-021a-456e-85eb-864596d28c3f-istio-data\") pod \"e2e-inf-a-23f720-openshift-default-6c8d4f9ff5-vpdqd\" (UID: \"2d3ed32f-021a-456e-85eb-864596d28c3f\") " pod="openshift-ingress/e2e-inf-a-23f720-openshift-default-6c8d4f9ff5-vpdqd"
Nov 02 13:17:35.517664 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:17:35.493196    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d3ed32f-021a-456e-85eb-864596d28c3f-workload-certs\") pod \"e2e-inf-a-23f720-openshift-default-6c8d4f9ff5-vpdqd\" (UID: \"2d3ed32f-021a-456e-85eb-864596d28c3f\") " pod="openshift-ingress/e2e-inf-a-23f720-openshift-default-6c8d4f9ff5-vpdqd"
Nov 02 13:17:35.517664 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:17:35.493440    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d3ed32f-021a-456e-85eb-864596d28c3f-credential-socket\") pod \"e2e-inf-a-23f720-openshift-default-6c8d4f9ff5-vpdqd\" (UID: \"2d3ed32f-021a-456e-85eb-864596d28c3f\") " pod="openshift-ingress/e2e-inf-a-23f720-openshift-default-6c8d4f9ff5-vpdqd"
Nov 02 13:17:35.931138 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:17:35.931107    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d3ed32f-021a-456e-85eb-864596d28c3f-istio-token\") pod \"e2e-inf-a-23f720-openshift-default-6c8d4f9ff5-vpdqd\" (UID: \"2d3ed32f-021a-456e-85eb-864596d28c3f\") " pod="openshift-ingress/e2e-inf-a-23f720-openshift-default-6c8d4f9ff5-vpdqd"
Nov 02 13:17:35.936945 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:17:35.936166    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d3ed32f-021a-456e-85eb-864596d28c3f-istio-podinfo\") pod \"e2e-inf-a-23f720-openshift-default-6c8d4f9ff5-vpdqd\" (UID: \"2d3ed32f-021a-456e-85eb-864596d28c3f\") " pod="openshift-ingress/e2e-inf-a-23f720-openshift-default-6c8d4f9ff5-vpdqd"
Nov 02 13:17:35.937197 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:17:35.937170    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d3ed32f-021a-456e-85eb-864596d28c3f-istio-envoy\") pod \"e2e-inf-a-23f720-openshift-default-6c8d4f9ff5-vpdqd\" (UID: \"2d3ed32f-021a-456e-85eb-864596d28c3f\") " pod="openshift-ingress/e2e-inf-a-23f720-openshift-default-6c8d4f9ff5-vpdqd"
Nov 02 13:17:35.937324 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:17:35.936707    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mc6d9\" (UniqueName: \"kubernetes.io/projected/2d3ed32f-021a-456e-85eb-864596d28c3f-kube-api-access-mc6d9\") pod \"e2e-inf-a-23f720-openshift-default-6c8d4f9ff5-vpdqd\" (UID: \"2d3ed32f-021a-456e-85eb-864596d28c3f\") " pod="openshift-ingress/e2e-inf-a-23f720-openshift-default-6c8d4f9ff5-vpdqd"
Nov 02 13:17:35.948000 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:17:35.947970    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc513c22-60e4-4789-90db-4a23974df75c" path="/var/lib/kubelet/pods/dc513c22-60e4-4789-90db-4a23974df75c/volumes"
Nov 02 13:17:36.042200 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:17:36.042153    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-23f720-openshift-default-6c8d4f9ff5-vpdqd"
Nov 02 13:17:36.079396 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:17:36.079362    4559 patch_prober.go:29] interesting pod/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-j7rnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused" start-of-body=
Nov 02 13:17:36.079553 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:17:36.079428    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-j7rnm" podUID="85ec8b4f-746f-4d0c-9560-7a79f0c64ef6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused"
Nov 02 13:17:36.447808 ip-10-0-128-132 kubenswrapper[4559]: W1102 13:17:36.447774    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d3ed32f_021a_456e_85eb_864596d28c3f.slice/crio-fd74f1a89641aa0f8e697bc6ca22af0d20607295c05b5cda3a4e5672ef93e1af WatchSource:0}: Error finding container fd74f1a89641aa0f8e697bc6ca22af0d20607295c05b5cda3a4e5672ef93e1af: Status 404 returned error can't find the container with id fd74f1a89641aa0f8e697bc6ca22af0d20607295c05b5cda3a4e5672ef93e1af
Nov 02 13:17:36.448965 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:17:36.448943    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-23f720-openshift-default-6c8d4f9ff5-vpdqd"]
Nov 02 13:17:36.452038 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:17:36.451998    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 13:17:36.452221 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:17:36.452201    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 13:17:36.452329 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:17:36.452317    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 13:17:36.955150 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:17:36.955108    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-23f720-openshift-default-6c8d4f9ff5-vpdqd" event={"ID":"2d3ed32f-021a-456e-85eb-864596d28c3f","Type":"ContainerStarted","Data":"20602e8e5be6c5b18679fc49a63b7b5d46ba260dd464be42f7448b7123a1a4cd"}
Nov 02 13:17:36.955328 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:17:36.955165    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-23f720-openshift-default-6c8d4f9ff5-vpdqd" event={"ID":"2d3ed32f-021a-456e-85eb-864596d28c3f","Type":"ContainerStarted","Data":"fd74f1a89641aa0f8e697bc6ca22af0d20607295c05b5cda3a4e5672ef93e1af"}
Nov 02 13:17:37.031710 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:17:37.031651    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-23f720-openshift-default-6c8d4f9ff5-vpdqd" podStartSLOduration=2.031620834 podStartE2EDuration="2.031620834s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:17:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:17:37.025076254 +0000 UTC m=+3053.280475351" watchObservedRunningTime="2026-11-02 13:17:37.031620834 +0000 UTC m=+3053.287019930"
Nov 02 13:17:37.043106 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:17:37.043080    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-23f720-openshift-default-6c8d4f9ff5-vpdqd"
Nov 02 13:17:37.045024 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:17:37.044997    4559 patch_prober.go:29] interesting pod/e2e-inf-a-23f720-openshift-default-6c8d4f9ff5-vpdqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused" start-of-body=
Nov 02 13:17:37.045150 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:17:37.045042    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-23f720-openshift-default-6c8d4f9ff5-vpdqd" podUID="2d3ed32f-021a-456e-85eb-864596d28c3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused"
Nov 02 13:17:37.079238 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:17:37.079208    4559 patch_prober.go:29] interesting pod/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-j7rnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused" start-of-body=
Nov 02 13:17:37.079388 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:17:37.079255    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-j7rnm" podUID="85ec8b4f-746f-4d0c-9560-7a79f0c64ef6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused"
Nov 02 13:17:38.047633 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:17:38.047606    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-23f720-openshift-default-6c8d4f9ff5-vpdqd"
Nov 02 13:17:38.083877 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:17:38.083846    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-j7rnm"
Nov 02 13:17:38.084999 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:17:38.084974    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-j7rnm"
Nov 02 13:17:38.970830 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:17:38.970799    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-23f720-openshift-default-6c8d4f9ff5-vpdqd"
Nov 02 13:17:38.971745 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:17:38.971722    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-23f720-openshift-default-6c8d4f9ff5-vpdqd"
Nov 02 13:17:42.570128 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:17:42.570089    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-361c3a-openshift-default-6f67d958cf-zhhhp"]
Nov 02 13:17:42.576252 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:17:42.576229    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-361c3a-openshift-default-6f67d958cf-zhhhp"
Nov 02 13:17:42.611601 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:17:42.611576    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-361c3a-openshift-default-dockercfg-8nrsz\""
Nov 02 13:17:42.619008 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:17:42.618978    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-361c3a-openshift-default-6f67d958cf-zhhhp"]
Nov 02 13:17:42.661941 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:17:42.661906    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ae3a83d-c3a4-4305-8824-1cde1a6172c0-workload-certs\") pod \"e2e-inf-b-361c3a-openshift-default-6f67d958cf-zhhhp\" (UID: \"5ae3a83d-c3a4-4305-8824-1cde1a6172c0\") " pod="openshift-ingress/e2e-inf-b-361c3a-openshift-default-6f67d958cf-zhhhp"
Nov 02 13:17:42.662085 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:17:42.661996    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ae3a83d-c3a4-4305-8824-1cde1a6172c0-istio-data\") pod \"e2e-inf-b-361c3a-openshift-default-6f67d958cf-zhhhp\" (UID: \"5ae3a83d-c3a4-4305-8824-1cde1a6172c0\") " pod="openshift-ingress/e2e-inf-b-361c3a-openshift-default-6f67d958cf-zhhhp"
Nov 02 13:17:42.662159 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:17:42.662111    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ae3a83d-c3a4-4305-8824-1cde1a6172c0-istio-envoy\") pod \"e2e-inf-b-361c3a-openshift-default-6f67d958cf-zhhhp\" (UID: \"5ae3a83d-c3a4-4305-8824-1cde1a6172c0\") " pod="openshift-ingress/e2e-inf-b-361c3a-openshift-default-6f67d958cf-zhhhp"
Nov 02 13:17:42.662241 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:17:42.662221    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ae3a83d-c3a4-4305-8824-1cde1a6172c0-istiod-ca-cert\") pod \"e2e-inf-b-361c3a-openshift-default-6f67d958cf-zhhhp\" (UID: \"5ae3a83d-c3a4-4305-8824-1cde1a6172c0\") " pod="openshift-ingress/e2e-inf-b-361c3a-openshift-default-6f67d958cf-zhhhp"
Nov 02 13:17:42.662296 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:17:42.662269    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ae3a83d-c3a4-4305-8824-1cde1a6172c0-istio-podinfo\") pod \"e2e-inf-b-361c3a-openshift-default-6f67d958cf-zhhhp\" (UID: \"5ae3a83d-c3a4-4305-8824-1cde1a6172c0\") " pod="openshift-ingress/e2e-inf-b-361c3a-openshift-default-6f67d958cf-zhhhp"
Nov 02 13:17:42.662374 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:17:42.662347    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ae3a83d-c3a4-4305-8824-1cde1a6172c0-credential-socket\") pod \"e2e-inf-b-361c3a-openshift-default-6f67d958cf-zhhhp\" (UID: \"5ae3a83d-c3a4-4305-8824-1cde1a6172c0\") " pod="openshift-ingress/e2e-inf-b-361c3a-openshift-default-6f67d958cf-zhhhp"
Nov 02 13:17:42.662512 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:17:42.662495    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvxzc\" (UniqueName: \"kubernetes.io/projected/5ae3a83d-c3a4-4305-8824-1cde1a6172c0-kube-api-access-bvxzc\") pod \"e2e-inf-b-361c3a-openshift-default-6f67d958cf-zhhhp\" (UID: \"5ae3a83d-c3a4-4305-8824-1cde1a6172c0\") " pod="openshift-ingress/e2e-inf-b-361c3a-openshift-default-6f67d958cf-zhhhp"
Nov 02 13:17:42.662579 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:17:42.662541    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ae3a83d-c3a4-4305-8824-1cde1a6172c0-workload-socket\") pod \"e2e-inf-b-361c3a-openshift-default-6f67d958cf-zhhhp\" (UID: \"5ae3a83d-c3a4-4305-8824-1cde1a6172c0\") " pod="openshift-ingress/e2e-inf-b-361c3a-openshift-default-6f67d958cf-zhhhp"
Nov 02 13:17:42.662689 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:17:42.662662    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ae3a83d-c3a4-4305-8824-1cde1a6172c0-istio-token\") pod \"e2e-inf-b-361c3a-openshift-default-6f67d958cf-zhhhp\" (UID: \"5ae3a83d-c3a4-4305-8824-1cde1a6172c0\") " pod="openshift-ingress/e2e-inf-b-361c3a-openshift-default-6f67d958cf-zhhhp"
Nov 02 13:17:42.764326 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:17:42.764298    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ae3a83d-c3a4-4305-8824-1cde1a6172c0-workload-certs\") pod \"e2e-inf-b-361c3a-openshift-default-6f67d958cf-zhhhp\" (UID: \"5ae3a83d-c3a4-4305-8824-1cde1a6172c0\") " pod="openshift-ingress/e2e-inf-b-361c3a-openshift-default-6f67d958cf-zhhhp"
Nov 02 13:17:42.764471 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:17:42.764352    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ae3a83d-c3a4-4305-8824-1cde1a6172c0-istio-data\") pod \"e2e-inf-b-361c3a-openshift-default-6f67d958cf-zhhhp\" (UID: \"5ae3a83d-c3a4-4305-8824-1cde1a6172c0\") " pod="openshift-ingress/e2e-inf-b-361c3a-openshift-default-6f67d958cf-zhhhp"
Nov 02 13:17:42.764471 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:17:42.764376    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ae3a83d-c3a4-4305-8824-1cde1a6172c0-istio-envoy\") pod \"e2e-inf-b-361c3a-openshift-default-6f67d958cf-zhhhp\" (UID: \"5ae3a83d-c3a4-4305-8824-1cde1a6172c0\") " pod="openshift-ingress/e2e-inf-b-361c3a-openshift-default-6f67d958cf-zhhhp"
Nov 02 13:17:42.764471 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:17:42.764414    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ae3a83d-c3a4-4305-8824-1cde1a6172c0-istiod-ca-cert\") pod \"e2e-inf-b-361c3a-openshift-default-6f67d958cf-zhhhp\" (UID: \"5ae3a83d-c3a4-4305-8824-1cde1a6172c0\") " pod="openshift-ingress/e2e-inf-b-361c3a-openshift-default-6f67d958cf-zhhhp"
Nov 02 13:17:42.764471 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:17:42.764437    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ae3a83d-c3a4-4305-8824-1cde1a6172c0-istio-podinfo\") pod \"e2e-inf-b-361c3a-openshift-default-6f67d958cf-zhhhp\" (UID: \"5ae3a83d-c3a4-4305-8824-1cde1a6172c0\") " pod="openshift-ingress/e2e-inf-b-361c3a-openshift-default-6f67d958cf-zhhhp"
Nov 02 13:17:42.764471 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:17:42.764461    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ae3a83d-c3a4-4305-8824-1cde1a6172c0-credential-socket\") pod \"e2e-inf-b-361c3a-openshift-default-6f67d958cf-zhhhp\" (UID: \"5ae3a83d-c3a4-4305-8824-1cde1a6172c0\") " pod="openshift-ingress/e2e-inf-b-361c3a-openshift-default-6f67d958cf-zhhhp"
Nov 02 13:17:42.764768 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:17:42.764570    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvxzc\" (UniqueName: \"kubernetes.io/projected/5ae3a83d-c3a4-4305-8824-1cde1a6172c0-kube-api-access-bvxzc\") pod \"e2e-inf-b-361c3a-openshift-default-6f67d958cf-zhhhp\" (UID: \"5ae3a83d-c3a4-4305-8824-1cde1a6172c0\") " pod="openshift-ingress/e2e-inf-b-361c3a-openshift-default-6f67d958cf-zhhhp"
Nov 02 13:17:42.764768 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:17:42.764609    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ae3a83d-c3a4-4305-8824-1cde1a6172c0-workload-socket\") pod \"e2e-inf-b-361c3a-openshift-default-6f67d958cf-zhhhp\" (UID: \"5ae3a83d-c3a4-4305-8824-1cde1a6172c0\") " pod="openshift-ingress/e2e-inf-b-361c3a-openshift-default-6f67d958cf-zhhhp"
Nov 02 13:17:42.764768 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:17:42.764681    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ae3a83d-c3a4-4305-8824-1cde1a6172c0-istio-token\") pod \"e2e-inf-b-361c3a-openshift-default-6f67d958cf-zhhhp\" (UID: \"5ae3a83d-c3a4-4305-8824-1cde1a6172c0\") " pod="openshift-ingress/e2e-inf-b-361c3a-openshift-default-6f67d958cf-zhhhp"
Nov 02 13:17:42.764939 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:17:42.764779    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ae3a83d-c3a4-4305-8824-1cde1a6172c0-workload-certs\") pod \"e2e-inf-b-361c3a-openshift-default-6f67d958cf-zhhhp\" (UID: \"5ae3a83d-c3a4-4305-8824-1cde1a6172c0\") " pod="openshift-ingress/e2e-inf-b-361c3a-openshift-default-6f67d958cf-zhhhp"
Nov 02 13:17:42.765102 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:17:42.765072    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ae3a83d-c3a4-4305-8824-1cde1a6172c0-workload-socket\") pod \"e2e-inf-b-361c3a-openshift-default-6f67d958cf-zhhhp\" (UID: \"5ae3a83d-c3a4-4305-8824-1cde1a6172c0\") " pod="openshift-ingress/e2e-inf-b-361c3a-openshift-default-6f67d958cf-zhhhp"
Nov 02 13:17:42.765230 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:17:42.765108    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ae3a83d-c3a4-4305-8824-1cde1a6172c0-istio-data\") pod \"e2e-inf-b-361c3a-openshift-default-6f67d958cf-zhhhp\" (UID: \"5ae3a83d-c3a4-4305-8824-1cde1a6172c0\") " pod="openshift-ingress/e2e-inf-b-361c3a-openshift-default-6f67d958cf-zhhhp"
Nov 02 13:17:42.765230 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:17:42.765155    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ae3a83d-c3a4-4305-8824-1cde1a6172c0-credential-socket\") pod \"e2e-inf-b-361c3a-openshift-default-6f67d958cf-zhhhp\" (UID: \"5ae3a83d-c3a4-4305-8824-1cde1a6172c0\") " pod="openshift-ingress/e2e-inf-b-361c3a-openshift-default-6f67d958cf-zhhhp"
Nov 02 13:17:42.765230 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:17:42.765227    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ae3a83d-c3a4-4305-8824-1cde1a6172c0-istiod-ca-cert\") pod \"e2e-inf-b-361c3a-openshift-default-6f67d958cf-zhhhp\" (UID: \"5ae3a83d-c3a4-4305-8824-1cde1a6172c0\") " pod="openshift-ingress/e2e-inf-b-361c3a-openshift-default-6f67d958cf-zhhhp"
Nov 02 13:17:42.766732 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:17:42.766713    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ae3a83d-c3a4-4305-8824-1cde1a6172c0-istio-envoy\") pod \"e2e-inf-b-361c3a-openshift-default-6f67d958cf-zhhhp\" (UID: \"5ae3a83d-c3a4-4305-8824-1cde1a6172c0\") " pod="openshift-ingress/e2e-inf-b-361c3a-openshift-default-6f67d958cf-zhhhp"
Nov 02 13:17:42.767097 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:17:42.767079    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ae3a83d-c3a4-4305-8824-1cde1a6172c0-istio-podinfo\") pod \"e2e-inf-b-361c3a-openshift-default-6f67d958cf-zhhhp\" (UID: \"5ae3a83d-c3a4-4305-8824-1cde1a6172c0\") " pod="openshift-ingress/e2e-inf-b-361c3a-openshift-default-6f67d958cf-zhhhp"
Nov 02 13:17:42.788209 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:17:42.788186    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ae3a83d-c3a4-4305-8824-1cde1a6172c0-istio-token\") pod \"e2e-inf-b-361c3a-openshift-default-6f67d958cf-zhhhp\" (UID: \"5ae3a83d-c3a4-4305-8824-1cde1a6172c0\") " pod="openshift-ingress/e2e-inf-b-361c3a-openshift-default-6f67d958cf-zhhhp"
Nov 02 13:17:42.788449 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:17:42.788428    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvxzc\" (UniqueName: \"kubernetes.io/projected/5ae3a83d-c3a4-4305-8824-1cde1a6172c0-kube-api-access-bvxzc\") pod \"e2e-inf-b-361c3a-openshift-default-6f67d958cf-zhhhp\" (UID: \"5ae3a83d-c3a4-4305-8824-1cde1a6172c0\") " pod="openshift-ingress/e2e-inf-b-361c3a-openshift-default-6f67d958cf-zhhhp"
Nov 02 13:17:42.892229 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:17:42.892160    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-361c3a-openshift-default-6f67d958cf-zhhhp"
Nov 02 13:17:43.220629 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:17:43.220594    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-361c3a-openshift-default-6f67d958cf-zhhhp"]
Nov 02 13:17:43.222245 ip-10-0-128-132 kubenswrapper[4559]: W1102 13:17:43.222217    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ae3a83d_c3a4_4305_8824_1cde1a6172c0.slice/crio-d228deb82f4000d7f02172ccac56e5992498b0fc56548f1b9ca188ac85fb555a WatchSource:0}: Error finding container d228deb82f4000d7f02172ccac56e5992498b0fc56548f1b9ca188ac85fb555a: Status 404 returned error can't find the container with id d228deb82f4000d7f02172ccac56e5992498b0fc56548f1b9ca188ac85fb555a
Nov 02 13:17:43.225719 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:17:43.225680    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 13:17:43.225823 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:17:43.225753    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 13:17:43.225823 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:17:43.225782    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 13:17:44.023628 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:17:44.023587    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-361c3a-openshift-default-6f67d958cf-zhhhp" event={"ID":"5ae3a83d-c3a4-4305-8824-1cde1a6172c0","Type":"ContainerStarted","Data":"3c67b55c02942fa1365fef1d300ee087b9eaba29eafe6c699b7d1fa9b220e051"}
Nov 02 13:17:44.024149 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:17:44.023790    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-361c3a-openshift-default-6f67d958cf-zhhhp" event={"ID":"5ae3a83d-c3a4-4305-8824-1cde1a6172c0","Type":"ContainerStarted","Data":"d228deb82f4000d7f02172ccac56e5992498b0fc56548f1b9ca188ac85fb555a"}
Nov 02 13:17:44.104663 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:17:44.101680    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-361c3a-openshift-default-6f67d958cf-zhhhp" podStartSLOduration=2.101660424 podStartE2EDuration="2.101660424s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:17:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:17:44.081917589 +0000 UTC m=+3060.337316683" watchObservedRunningTime="2026-11-02 13:17:44.101660424 +0000 UTC m=+3060.357059519"
Nov 02 13:17:44.899664 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:17:44.897895    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-361c3a-openshift-default-6f67d958cf-zhhhp"
Nov 02 13:17:44.899664 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:17:44.898023    4559 patch_prober.go:29] interesting pod/e2e-inf-b-361c3a-openshift-default-6f67d958cf-zhhhp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.246:15021/healthz/ready\": dial tcp 10.133.0.246:15021: connect: connection refused" start-of-body=
Nov 02 13:17:44.899664 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:17:44.898082    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-361c3a-openshift-default-6f67d958cf-zhhhp" podUID="5ae3a83d-c3a4-4305-8824-1cde1a6172c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.246:15021/healthz/ready\": dial tcp 10.133.0.246:15021: connect: connection refused"
Nov 02 13:17:45.114584 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:17:45.114550    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqcmj" containerName="extractor"
Nov 02 13:17:45.899555 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:17:45.899516    4559 patch_prober.go:29] interesting pod/e2e-inf-b-361c3a-openshift-default-6f67d958cf-zhhhp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.246:15021/healthz/ready\": dial tcp 10.133.0.246:15021: connect: connection refused" start-of-body=
Nov 02 13:17:45.901752 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:17:45.901718    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-361c3a-openshift-default-6f67d958cf-zhhhp" podUID="5ae3a83d-c3a4-4305-8824-1cde1a6172c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.246:15021/healthz/ready\": dial tcp 10.133.0.246:15021: connect: connection refused"
Nov 02 13:17:46.901014 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:17:46.900975    4559 patch_prober.go:29] interesting pod/e2e-inf-b-361c3a-openshift-default-6f67d958cf-zhhhp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.246:15021/healthz/ready\": dial tcp 10.133.0.246:15021: connect: connection refused" start-of-body=
Nov 02 13:17:46.901580 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:17:46.901546    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-361c3a-openshift-default-6f67d958cf-zhhhp" podUID="5ae3a83d-c3a4-4305-8824-1cde1a6172c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.246:15021/healthz/ready\": dial tcp 10.133.0.246:15021: connect: connection refused"
Nov 02 13:17:47.906761 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:17:47.906722    4559 patch_prober.go:29] interesting pod/e2e-inf-b-361c3a-openshift-default-6f67d958cf-zhhhp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.246:15021/healthz/ready\": dial tcp 10.133.0.246:15021: connect: connection refused" start-of-body=
Nov 02 13:17:47.907347 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:17:47.907316    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-361c3a-openshift-default-6f67d958cf-zhhhp" podUID="5ae3a83d-c3a4-4305-8824-1cde1a6172c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.246:15021/healthz/ready\": dial tcp 10.133.0.246:15021: connect: connection refused"
Nov 02 13:17:48.907665 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:17:48.907453    4559 patch_prober.go:29] interesting pod/e2e-inf-b-361c3a-openshift-default-6f67d958cf-zhhhp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.246:15021/healthz/ready\": dial tcp 10.133.0.246:15021: connect: connection refused" start-of-body=
Nov 02 13:17:48.907665 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:17:48.907512    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-361c3a-openshift-default-6f67d958cf-zhhhp" podUID="5ae3a83d-c3a4-4305-8824-1cde1a6172c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.246:15021/healthz/ready\": dial tcp 10.133.0.246:15021: connect: connection refused"
Nov 02 13:17:49.967919 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:17:49.967852    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-361c3a-openshift-default-6f67d958cf-zhhhp"
Nov 02 13:17:49.968724 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:17:49.968695    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-361c3a-openshift-default-6f67d958cf-zhhhp"
Nov 02 13:17:49.970397 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:17:49.970371    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-361c3a-openshift-default-6f67d958cf-zhhhp"
Nov 02 13:17:59.199213 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:17:59.199176    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"]
Nov 02 13:18:15.089631 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:18:15.089586    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqcmj" containerName="extractor"
Nov 02 13:18:15.427480 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:15.427449    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-m29cl"]
Nov 02 13:18:15.432484 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:15.432458    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-m29cl"
Nov 02 13:18:15.471743 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:15.471710    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-m29cl"]
Nov 02 13:18:15.477749 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:15.477718    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-2ssw6"]
Nov 02 13:18:15.484382 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:15.484361    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-2ssw6"
Nov 02 13:18:15.502681 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:15.502651    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d60e24f5-ef82-485d-a867-b3e3718b0e65-workload-socket\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-2ssw6\" (UID: \"d60e24f5-ef82-485d-a867-b3e3718b0e65\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-2ssw6"
Nov 02 13:18:15.502922 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:15.502892    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7a083ea-c6dc-4402-aac2-7842d1578ed2-istiod-ca-cert\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-m29cl\" (UID: \"f7a083ea-c6dc-4402-aac2-7842d1578ed2\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-m29cl"
Nov 02 13:18:15.503797 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:15.503023    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d60e24f5-ef82-485d-a867-b3e3718b0e65-istio-envoy\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-2ssw6\" (UID: \"d60e24f5-ef82-485d-a867-b3e3718b0e65\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-2ssw6"
Nov 02 13:18:15.503797 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:15.503079    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7a083ea-c6dc-4402-aac2-7842d1578ed2-credential-socket\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-m29cl\" (UID: \"f7a083ea-c6dc-4402-aac2-7842d1578ed2\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-m29cl"
Nov 02 13:18:15.503797 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:15.503166    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqv7t\" (UniqueName: \"kubernetes.io/projected/f7a083ea-c6dc-4402-aac2-7842d1578ed2-kube-api-access-qqv7t\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-m29cl\" (UID: \"f7a083ea-c6dc-4402-aac2-7842d1578ed2\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-m29cl"
Nov 02 13:18:15.503797 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:15.503195    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d60e24f5-ef82-485d-a867-b3e3718b0e65-istio-token\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-2ssw6\" (UID: \"d60e24f5-ef82-485d-a867-b3e3718b0e65\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-2ssw6"
Nov 02 13:18:15.503797 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:15.503229    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7a083ea-c6dc-4402-aac2-7842d1578ed2-istio-envoy\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-m29cl\" (UID: \"f7a083ea-c6dc-4402-aac2-7842d1578ed2\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-m29cl"
Nov 02 13:18:15.503797 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:15.503244    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7a083ea-c6dc-4402-aac2-7842d1578ed2-workload-certs\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-m29cl\" (UID: \"f7a083ea-c6dc-4402-aac2-7842d1578ed2\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-m29cl"
Nov 02 13:18:15.503797 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:15.503265    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d60e24f5-ef82-485d-a867-b3e3718b0e65-istio-data\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-2ssw6\" (UID: \"d60e24f5-ef82-485d-a867-b3e3718b0e65\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-2ssw6"
Nov 02 13:18:15.503797 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:15.503308    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7a083ea-c6dc-4402-aac2-7842d1578ed2-istio-podinfo\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-m29cl\" (UID: \"f7a083ea-c6dc-4402-aac2-7842d1578ed2\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-m29cl"
Nov 02 13:18:15.503797 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:15.503349    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d60e24f5-ef82-485d-a867-b3e3718b0e65-credential-socket\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-2ssw6\" (UID: \"d60e24f5-ef82-485d-a867-b3e3718b0e65\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-2ssw6"
Nov 02 13:18:15.503797 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:15.503395    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7a083ea-c6dc-4402-aac2-7842d1578ed2-istio-token\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-m29cl\" (UID: \"f7a083ea-c6dc-4402-aac2-7842d1578ed2\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-m29cl"
Nov 02 13:18:15.503797 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:15.503431    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d60e24f5-ef82-485d-a867-b3e3718b0e65-workload-certs\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-2ssw6\" (UID: \"d60e24f5-ef82-485d-a867-b3e3718b0e65\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-2ssw6"
Nov 02 13:18:15.503797 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:15.503480    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7a083ea-c6dc-4402-aac2-7842d1578ed2-istio-data\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-m29cl\" (UID: \"f7a083ea-c6dc-4402-aac2-7842d1578ed2\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-m29cl"
Nov 02 13:18:15.503797 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:15.503507    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jntwf\" (UniqueName: \"kubernetes.io/projected/d60e24f5-ef82-485d-a867-b3e3718b0e65-kube-api-access-jntwf\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-2ssw6\" (UID: \"d60e24f5-ef82-485d-a867-b3e3718b0e65\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-2ssw6"
Nov 02 13:18:15.503797 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:15.503553    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7a083ea-c6dc-4402-aac2-7842d1578ed2-workload-socket\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-m29cl\" (UID: \"f7a083ea-c6dc-4402-aac2-7842d1578ed2\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-m29cl"
Nov 02 13:18:15.503797 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:15.503600    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d60e24f5-ef82-485d-a867-b3e3718b0e65-istiod-ca-cert\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-2ssw6\" (UID: \"d60e24f5-ef82-485d-a867-b3e3718b0e65\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-2ssw6"
Nov 02 13:18:15.503797 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:15.503629    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d60e24f5-ef82-485d-a867-b3e3718b0e65-istio-podinfo\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-2ssw6\" (UID: \"d60e24f5-ef82-485d-a867-b3e3718b0e65\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-2ssw6"
Nov 02 13:18:15.504779 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:15.504567    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-2ssw6"]
Nov 02 13:18:15.605224 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:15.605193    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d60e24f5-ef82-485d-a867-b3e3718b0e65-istio-envoy\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-2ssw6\" (UID: \"d60e24f5-ef82-485d-a867-b3e3718b0e65\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-2ssw6"
Nov 02 13:18:15.605425 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:15.605240    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7a083ea-c6dc-4402-aac2-7842d1578ed2-credential-socket\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-m29cl\" (UID: \"f7a083ea-c6dc-4402-aac2-7842d1578ed2\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-m29cl"
Nov 02 13:18:15.605425 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:15.605301    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqv7t\" (UniqueName: \"kubernetes.io/projected/f7a083ea-c6dc-4402-aac2-7842d1578ed2-kube-api-access-qqv7t\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-m29cl\" (UID: \"f7a083ea-c6dc-4402-aac2-7842d1578ed2\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-m29cl"
Nov 02 13:18:15.605425 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:15.605327    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d60e24f5-ef82-485d-a867-b3e3718b0e65-istio-token\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-2ssw6\" (UID: \"d60e24f5-ef82-485d-a867-b3e3718b0e65\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-2ssw6"
Nov 02 13:18:15.605425 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:15.605362    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7a083ea-c6dc-4402-aac2-7842d1578ed2-istio-envoy\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-m29cl\" (UID: \"f7a083ea-c6dc-4402-aac2-7842d1578ed2\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-m29cl"
Nov 02 13:18:15.605425 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:15.605380    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7a083ea-c6dc-4402-aac2-7842d1578ed2-workload-certs\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-m29cl\" (UID: \"f7a083ea-c6dc-4402-aac2-7842d1578ed2\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-m29cl"
Nov 02 13:18:15.605425 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:15.605407    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d60e24f5-ef82-485d-a867-b3e3718b0e65-istio-data\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-2ssw6\" (UID: \"d60e24f5-ef82-485d-a867-b3e3718b0e65\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-2ssw6"
Nov 02 13:18:15.605833 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:15.605472    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7a083ea-c6dc-4402-aac2-7842d1578ed2-istio-podinfo\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-m29cl\" (UID: \"f7a083ea-c6dc-4402-aac2-7842d1578ed2\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-m29cl"
Nov 02 13:18:15.605833 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:15.605533    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d60e24f5-ef82-485d-a867-b3e3718b0e65-credential-socket\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-2ssw6\" (UID: \"d60e24f5-ef82-485d-a867-b3e3718b0e65\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-2ssw6"
Nov 02 13:18:15.605833 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:15.605614    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7a083ea-c6dc-4402-aac2-7842d1578ed2-istio-token\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-m29cl\" (UID: \"f7a083ea-c6dc-4402-aac2-7842d1578ed2\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-m29cl"
Nov 02 13:18:15.605833 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:15.605679    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d60e24f5-ef82-485d-a867-b3e3718b0e65-workload-certs\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-2ssw6\" (UID: \"d60e24f5-ef82-485d-a867-b3e3718b0e65\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-2ssw6"
Nov 02 13:18:15.605833 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:15.605734    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7a083ea-c6dc-4402-aac2-7842d1578ed2-istio-data\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-m29cl\" (UID: \"f7a083ea-c6dc-4402-aac2-7842d1578ed2\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-m29cl"
Nov 02 13:18:15.605833 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:15.605762    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jntwf\" (UniqueName: \"kubernetes.io/projected/d60e24f5-ef82-485d-a867-b3e3718b0e65-kube-api-access-jntwf\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-2ssw6\" (UID: \"d60e24f5-ef82-485d-a867-b3e3718b0e65\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-2ssw6"
Nov 02 13:18:15.605833 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:15.605669    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7a083ea-c6dc-4402-aac2-7842d1578ed2-credential-socket\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-m29cl\" (UID: \"f7a083ea-c6dc-4402-aac2-7842d1578ed2\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-m29cl"
Nov 02 13:18:15.605833 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:15.605810    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7a083ea-c6dc-4402-aac2-7842d1578ed2-workload-socket\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-m29cl\" (UID: \"f7a083ea-c6dc-4402-aac2-7842d1578ed2\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-m29cl"
Nov 02 13:18:15.606329 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:15.605881    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d60e24f5-ef82-485d-a867-b3e3718b0e65-istiod-ca-cert\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-2ssw6\" (UID: \"d60e24f5-ef82-485d-a867-b3e3718b0e65\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-2ssw6"
Nov 02 13:18:15.606329 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:15.605921    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d60e24f5-ef82-485d-a867-b3e3718b0e65-istio-podinfo\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-2ssw6\" (UID: \"d60e24f5-ef82-485d-a867-b3e3718b0e65\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-2ssw6"
Nov 02 13:18:15.606329 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:15.605935    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7a083ea-c6dc-4402-aac2-7842d1578ed2-workload-certs\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-m29cl\" (UID: \"f7a083ea-c6dc-4402-aac2-7842d1578ed2\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-m29cl"
Nov 02 13:18:15.606329 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:15.605960    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d60e24f5-ef82-485d-a867-b3e3718b0e65-workload-socket\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-2ssw6\" (UID: \"d60e24f5-ef82-485d-a867-b3e3718b0e65\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-2ssw6"
Nov 02 13:18:15.606329 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:15.606016    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7a083ea-c6dc-4402-aac2-7842d1578ed2-istiod-ca-cert\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-m29cl\" (UID: \"f7a083ea-c6dc-4402-aac2-7842d1578ed2\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-m29cl"
Nov 02 13:18:15.606329 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:15.606042    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d60e24f5-ef82-485d-a867-b3e3718b0e65-istio-data\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-2ssw6\" (UID: \"d60e24f5-ef82-485d-a867-b3e3718b0e65\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-2ssw6"
Nov 02 13:18:15.606329 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:15.606178    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7a083ea-c6dc-4402-aac2-7842d1578ed2-workload-socket\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-m29cl\" (UID: \"f7a083ea-c6dc-4402-aac2-7842d1578ed2\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-m29cl"
Nov 02 13:18:15.606329 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:15.606214    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d60e24f5-ef82-485d-a867-b3e3718b0e65-credential-socket\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-2ssw6\" (UID: \"d60e24f5-ef82-485d-a867-b3e3718b0e65\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-2ssw6"
Nov 02 13:18:15.606777 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:15.606359    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7a083ea-c6dc-4402-aac2-7842d1578ed2-istio-data\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-m29cl\" (UID: \"f7a083ea-c6dc-4402-aac2-7842d1578ed2\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-m29cl"
Nov 02 13:18:15.606777 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:15.606476    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d60e24f5-ef82-485d-a867-b3e3718b0e65-workload-socket\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-2ssw6\" (UID: \"d60e24f5-ef82-485d-a867-b3e3718b0e65\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-2ssw6"
Nov 02 13:18:15.606777 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:15.606673    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d60e24f5-ef82-485d-a867-b3e3718b0e65-workload-certs\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-2ssw6\" (UID: \"d60e24f5-ef82-485d-a867-b3e3718b0e65\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-2ssw6"
Nov 02 13:18:15.606922 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:15.606837    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7a083ea-c6dc-4402-aac2-7842d1578ed2-istiod-ca-cert\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-m29cl\" (UID: \"f7a083ea-c6dc-4402-aac2-7842d1578ed2\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-m29cl"
Nov 02 13:18:15.607047 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:15.607022    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d60e24f5-ef82-485d-a867-b3e3718b0e65-istiod-ca-cert\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-2ssw6\" (UID: \"d60e24f5-ef82-485d-a867-b3e3718b0e65\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-2ssw6"
Nov 02 13:18:15.607799 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:15.607773    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d60e24f5-ef82-485d-a867-b3e3718b0e65-istio-envoy\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-2ssw6\" (UID: \"d60e24f5-ef82-485d-a867-b3e3718b0e65\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-2ssw6"
Nov 02 13:18:15.608022 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:15.607997    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7a083ea-c6dc-4402-aac2-7842d1578ed2-istio-envoy\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-m29cl\" (UID: \"f7a083ea-c6dc-4402-aac2-7842d1578ed2\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-m29cl"
Nov 02 13:18:15.608741 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:15.608720    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d60e24f5-ef82-485d-a867-b3e3718b0e65-istio-podinfo\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-2ssw6\" (UID: \"d60e24f5-ef82-485d-a867-b3e3718b0e65\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-2ssw6"
Nov 02 13:18:15.609045 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:15.609020    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7a083ea-c6dc-4402-aac2-7842d1578ed2-istio-podinfo\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-m29cl\" (UID: \"f7a083ea-c6dc-4402-aac2-7842d1578ed2\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-m29cl"
Nov 02 13:18:15.632767 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:15.630350    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d60e24f5-ef82-485d-a867-b3e3718b0e65-istio-token\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-2ssw6\" (UID: \"d60e24f5-ef82-485d-a867-b3e3718b0e65\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-2ssw6"
Nov 02 13:18:15.632767 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:15.630503    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqv7t\" (UniqueName: \"kubernetes.io/projected/f7a083ea-c6dc-4402-aac2-7842d1578ed2-kube-api-access-qqv7t\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-m29cl\" (UID: \"f7a083ea-c6dc-4402-aac2-7842d1578ed2\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-m29cl"
Nov 02 13:18:15.632767 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:15.632384    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7a083ea-c6dc-4402-aac2-7842d1578ed2-istio-token\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-m29cl\" (UID: \"f7a083ea-c6dc-4402-aac2-7842d1578ed2\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-m29cl"
Nov 02 13:18:15.633446 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:15.633404    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jntwf\" (UniqueName: \"kubernetes.io/projected/d60e24f5-ef82-485d-a867-b3e3718b0e65-kube-api-access-jntwf\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-2ssw6\" (UID: \"d60e24f5-ef82-485d-a867-b3e3718b0e65\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-2ssw6"
Nov 02 13:18:15.747883 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:15.747798    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-m29cl"
Nov 02 13:18:15.808622 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:15.808388    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-2ssw6"
Nov 02 13:18:16.147660 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:16.142042    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 13:18:16.147660 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:16.142159    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 13:18:16.147660 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:16.142202    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 13:18:16.152461 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:16.152434    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-m29cl"]
Nov 02 13:18:16.180483 ip-10-0-128-132 kubenswrapper[4559]: W1102 13:18:16.180458    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd60e24f5_ef82_485d_a867_b3e3718b0e65.slice/crio-9410bce39bc8ae8e26b4a926b0e33e9944cfb6caf5ed04ae7eaed264f8442a99 WatchSource:0}: Error finding container 9410bce39bc8ae8e26b4a926b0e33e9944cfb6caf5ed04ae7eaed264f8442a99: Status 404 returned error can't find the container with id 9410bce39bc8ae8e26b4a926b0e33e9944cfb6caf5ed04ae7eaed264f8442a99
Nov 02 13:18:16.181484 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:16.181447    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-2ssw6"]
Nov 02 13:18:16.184331 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:16.184297    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 13:18:16.184426 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:16.184367    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 13:18:16.184426 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:16.184397    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 13:18:16.597145 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:16.597104    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-m29cl" event={"ID":"f7a083ea-c6dc-4402-aac2-7842d1578ed2","Type":"ContainerStarted","Data":"018bc4559c2ded382f5582913c877cf8e38136d64ef2fb188ea95c4ddf51202f"}
Nov 02 13:18:16.597344 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:16.597171    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-m29cl" event={"ID":"f7a083ea-c6dc-4402-aac2-7842d1578ed2","Type":"ContainerStarted","Data":"2ce9546c095d9b10fa79d80a6e0d3b641a3caece4bf3b60261395f43f3c41161"}
Nov 02 13:18:16.599551 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:16.599524    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-2ssw6" event={"ID":"d60e24f5-ef82-485d-a867-b3e3718b0e65","Type":"ContainerStarted","Data":"33623bd9e9155f0968b2aabbe020119796f0dff7fc0d14c533ee50f6599ec6f6"}
Nov 02 13:18:16.599733 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:16.599575    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-2ssw6" event={"ID":"d60e24f5-ef82-485d-a867-b3e3718b0e65","Type":"ContainerStarted","Data":"9410bce39bc8ae8e26b4a926b0e33e9944cfb6caf5ed04ae7eaed264f8442a99"}
Nov 02 13:18:16.650403 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:16.650343    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-m29cl" podStartSLOduration=1.650326288 podStartE2EDuration="1.650326288s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:18:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:18:16.643822656 +0000 UTC m=+3092.899221774" watchObservedRunningTime="2026-11-02 13:18:16.650326288 +0000 UTC m=+3092.905725384"
Nov 02 13:18:16.748909 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:16.748879    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-m29cl"
Nov 02 13:18:16.809283 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:16.809253    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-2ssw6"
Nov 02 13:18:17.750363 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:17.750326    4559 patch_prober.go:29] interesting pod/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-m29cl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused" start-of-body=
Nov 02 13:18:17.750835 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:17.750387    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-m29cl" podUID="f7a083ea-c6dc-4402-aac2-7842d1578ed2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused"
Nov 02 13:18:17.822183 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:17.819787    4559 patch_prober.go:29] interesting pod/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-2ssw6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.248:15021/healthz/ready\": dial tcp 10.133.0.248:15021: connect: connection refused" start-of-body=
Nov 02 13:18:17.822183 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:17.819870    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-2ssw6" podUID="d60e24f5-ef82-485d-a867-b3e3718b0e65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.248:15021/healthz/ready\": dial tcp 10.133.0.248:15021: connect: connection refused"
Nov 02 13:18:18.748683 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:18.748651    4559 patch_prober.go:29] interesting pod/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-m29cl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused" start-of-body=
Nov 02 13:18:18.748867 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:18.748702    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-m29cl" podUID="f7a083ea-c6dc-4402-aac2-7842d1578ed2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused"
Nov 02 13:18:18.815662 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:18.809092    4559 patch_prober.go:29] interesting pod/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-2ssw6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.248:15021/healthz/ready\": dial tcp 10.133.0.248:15021: connect: connection refused" start-of-body=
Nov 02 13:18:18.815662 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:18.809152    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-2ssw6" podUID="d60e24f5-ef82-485d-a867-b3e3718b0e65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.248:15021/healthz/ready\": dial tcp 10.133.0.248:15021: connect: connection refused"
Nov 02 13:18:19.748387 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:19.748349    4559 patch_prober.go:29] interesting pod/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-m29cl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused" start-of-body=
Nov 02 13:18:19.748575 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:19.748403    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-m29cl" podUID="f7a083ea-c6dc-4402-aac2-7842d1578ed2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused"
Nov 02 13:18:19.811662 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:19.809708    4559 patch_prober.go:29] interesting pod/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-2ssw6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.248:15021/healthz/ready\": dial tcp 10.133.0.248:15021: connect: connection refused" start-of-body=
Nov 02 13:18:19.811662 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:19.809767    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-2ssw6" podUID="d60e24f5-ef82-485d-a867-b3e3718b0e65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.248:15021/healthz/ready\": dial tcp 10.133.0.248:15021: connect: connection refused"
Nov 02 13:18:20.748955 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:20.748922    4559 patch_prober.go:29] interesting pod/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-m29cl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused" start-of-body=
Nov 02 13:18:20.749503 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:20.748973    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-m29cl" podUID="f7a083ea-c6dc-4402-aac2-7842d1578ed2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused"
Nov 02 13:18:20.812102 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:20.812060    4559 patch_prober.go:29] interesting pod/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-2ssw6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.248:15021/healthz/ready\": dial tcp 10.133.0.248:15021: connect: connection refused" start-of-body=
Nov 02 13:18:20.812448 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:20.812417    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-2ssw6" podUID="d60e24f5-ef82-485d-a867-b3e3718b0e65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.248:15021/healthz/ready\": dial tcp 10.133.0.248:15021: connect: connection refused"
Nov 02 13:18:21.748515 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:21.748476    4559 patch_prober.go:29] interesting pod/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-m29cl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused" start-of-body=
Nov 02 13:18:21.748725 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:21.748537    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-m29cl" podUID="f7a083ea-c6dc-4402-aac2-7842d1578ed2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused"
Nov 02 13:18:21.812242 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:21.812203    4559 patch_prober.go:29] interesting pod/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-2ssw6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.248:15021/healthz/ready\": dial tcp 10.133.0.248:15021: connect: connection refused" start-of-body=
Nov 02 13:18:21.812860 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:21.812830    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-2ssw6" podUID="d60e24f5-ef82-485d-a867-b3e3718b0e65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.248:15021/healthz/ready\": dial tcp 10.133.0.248:15021: connect: connection refused"
Nov 02 13:18:22.748835 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:22.748800    4559 patch_prober.go:29] interesting pod/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-m29cl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused" start-of-body=
Nov 02 13:18:22.749010 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:22.748855    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-m29cl" podUID="f7a083ea-c6dc-4402-aac2-7842d1578ed2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused"
Nov 02 13:18:22.812018 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:22.811980    4559 patch_prober.go:29] interesting pod/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-2ssw6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.248:15021/healthz/ready\": dial tcp 10.133.0.248:15021: connect: connection refused" start-of-body=
Nov 02 13:18:22.812294 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:22.812265    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-2ssw6" podUID="d60e24f5-ef82-485d-a867-b3e3718b0e65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.248:15021/healthz/ready\": dial tcp 10.133.0.248:15021: connect: connection refused"
Nov 02 13:18:23.768351 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:23.768320    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-m29cl"
Nov 02 13:18:23.769829 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:23.769099    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-m29cl"
Nov 02 13:18:23.770057 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:23.770036    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-m29cl"
Nov 02 13:18:23.813851 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:23.813827    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-2ssw6"
Nov 02 13:18:23.814286 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:23.814197    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-2ssw6"
Nov 02 13:18:23.814952 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:23.814933    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-2ssw6"
Nov 02 13:18:23.848220 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:23.848180    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-2ssw6" podStartSLOduration=8.84816728 podStartE2EDuration="8.84816728s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:18:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:18:16.693920267 +0000 UTC m=+3092.949319362" watchObservedRunningTime="2026-11-02 13:18:23.84816728 +0000 UTC m=+3100.103566375"
Nov 02 13:18:30.916268 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:30.916166    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-23f720-75984ff486-7cn9p"]
Nov 02 13:18:30.922516 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:30.922489    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-23f720-75984ff486-7cn9p"
Nov 02 13:18:30.937894 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:30.937871    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-23f720-serving-cert\""
Nov 02 13:18:30.957807 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:30.957778    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-23f720-75984ff486-7cn9p"]
Nov 02 13:18:31.093696 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:31.093633    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5a9eb160-1c8d-456c-8e37-f2d2e8e234e4-maas-api-tls\") pod \"maas-api-e2e-inf-a-23f720-75984ff486-7cn9p\" (UID: \"5a9eb160-1c8d-456c-8e37-f2d2e8e234e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-23f720-75984ff486-7cn9p"
Nov 02 13:18:31.093899 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:31.093862    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwcvw\" (UniqueName: \"kubernetes.io/projected/5a9eb160-1c8d-456c-8e37-f2d2e8e234e4-kube-api-access-mwcvw\") pod \"maas-api-e2e-inf-a-23f720-75984ff486-7cn9p\" (UID: \"5a9eb160-1c8d-456c-8e37-f2d2e8e234e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-23f720-75984ff486-7cn9p"
Nov 02 13:18:31.093975 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:31.093898    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a9eb160-1c8d-456c-8e37-f2d2e8e234e4-openshift-service-ca\") pod \"maas-api-e2e-inf-a-23f720-75984ff486-7cn9p\" (UID: \"5a9eb160-1c8d-456c-8e37-f2d2e8e234e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-23f720-75984ff486-7cn9p"
Nov 02 13:18:31.094090 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:31.093971    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5a9eb160-1c8d-456c-8e37-f2d2e8e234e4-metrics-certs\") pod \"maas-api-e2e-inf-a-23f720-75984ff486-7cn9p\" (UID: \"5a9eb160-1c8d-456c-8e37-f2d2e8e234e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-23f720-75984ff486-7cn9p"
Nov 02 13:18:31.211064 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:31.211025    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwcvw\" (UniqueName: \"kubernetes.io/projected/5a9eb160-1c8d-456c-8e37-f2d2e8e234e4-kube-api-access-mwcvw\") pod \"maas-api-e2e-inf-a-23f720-75984ff486-7cn9p\" (UID: \"5a9eb160-1c8d-456c-8e37-f2d2e8e234e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-23f720-75984ff486-7cn9p"
Nov 02 13:18:31.211327 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:31.211304    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a9eb160-1c8d-456c-8e37-f2d2e8e234e4-openshift-service-ca\") pod \"maas-api-e2e-inf-a-23f720-75984ff486-7cn9p\" (UID: \"5a9eb160-1c8d-456c-8e37-f2d2e8e234e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-23f720-75984ff486-7cn9p"
Nov 02 13:18:31.211504 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:31.211489    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5a9eb160-1c8d-456c-8e37-f2d2e8e234e4-metrics-certs\") pod \"maas-api-e2e-inf-a-23f720-75984ff486-7cn9p\" (UID: \"5a9eb160-1c8d-456c-8e37-f2d2e8e234e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-23f720-75984ff486-7cn9p"
Nov 02 13:18:31.211830 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:31.211814    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5a9eb160-1c8d-456c-8e37-f2d2e8e234e4-maas-api-tls\") pod \"maas-api-e2e-inf-a-23f720-75984ff486-7cn9p\" (UID: \"5a9eb160-1c8d-456c-8e37-f2d2e8e234e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-23f720-75984ff486-7cn9p"
Nov 02 13:18:31.221622 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:31.221581    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a9eb160-1c8d-456c-8e37-f2d2e8e234e4-openshift-service-ca\") pod \"maas-api-e2e-inf-a-23f720-75984ff486-7cn9p\" (UID: \"5a9eb160-1c8d-456c-8e37-f2d2e8e234e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-23f720-75984ff486-7cn9p"
Nov 02 13:18:31.222217 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:18:31.222188    4559 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-23f720-serving-cert: secret "maas-api-e2e-inf-a-23f720-serving-cert" not found
Nov 02 13:18:31.236981 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:18:31.236957    4559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a9eb160-1c8d-456c-8e37-f2d2e8e234e4-maas-api-tls podName:5a9eb160-1c8d-456c-8e37-f2d2e8e234e4 nodeName:}" failed. No retries permitted until 2026-11-02 13:18:31.736931 +0000 UTC m=+3107.992330086 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5a9eb160-1c8d-456c-8e37-f2d2e8e234e4-maas-api-tls") pod "maas-api-e2e-inf-a-23f720-75984ff486-7cn9p" (UID: "5a9eb160-1c8d-456c-8e37-f2d2e8e234e4") : secret "maas-api-e2e-inf-a-23f720-serving-cert" not found
Nov 02 13:18:31.459624 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:31.459523    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5a9eb160-1c8d-456c-8e37-f2d2e8e234e4-metrics-certs\") pod \"maas-api-e2e-inf-a-23f720-75984ff486-7cn9p\" (UID: \"5a9eb160-1c8d-456c-8e37-f2d2e8e234e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-23f720-75984ff486-7cn9p"
Nov 02 13:18:31.470712 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:31.467809    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwcvw\" (UniqueName: \"kubernetes.io/projected/5a9eb160-1c8d-456c-8e37-f2d2e8e234e4-kube-api-access-mwcvw\") pod \"maas-api-e2e-inf-a-23f720-75984ff486-7cn9p\" (UID: \"5a9eb160-1c8d-456c-8e37-f2d2e8e234e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-23f720-75984ff486-7cn9p"
Nov 02 13:18:31.787716 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:31.777983    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5a9eb160-1c8d-456c-8e37-f2d2e8e234e4-maas-api-tls\") pod \"maas-api-e2e-inf-a-23f720-75984ff486-7cn9p\" (UID: \"5a9eb160-1c8d-456c-8e37-f2d2e8e234e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-23f720-75984ff486-7cn9p"
Nov 02 13:18:32.036599 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:32.036563    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5a9eb160-1c8d-456c-8e37-f2d2e8e234e4-maas-api-tls\") pod \"maas-api-e2e-inf-a-23f720-75984ff486-7cn9p\" (UID: \"5a9eb160-1c8d-456c-8e37-f2d2e8e234e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-23f720-75984ff486-7cn9p"
Nov 02 13:18:32.170712 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:32.169068    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-23f720-75984ff486-7cn9p"
Nov 02 13:18:32.582847 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:32.582795    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-61e341/distinct-2-w1-61e341-kserve-954cf956f-n7jmg" podUID="c5e1a5f8-97b4-469d-be9a-eae7ca995283" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded"
Nov 02 13:18:33.381388 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:33.381334    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-23f720-75984ff486-7cn9p"]
Nov 02 13:18:33.399854 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:33.399827    4559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 13:18:33.873208 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:33.873175    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-23f720-75984ff486-7cn9p" event={"ID":"5a9eb160-1c8d-456c-8e37-f2d2e8e234e4","Type":"ContainerStarted","Data":"4ddb4722c84cf14a6fce18487755734ed8820a0daf412c73d18508f2759dafcb"}
Nov 02 13:18:33.873389 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:33.873235    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-23f720-75984ff486-7cn9p" event={"ID":"5a9eb160-1c8d-456c-8e37-f2d2e8e234e4","Type":"ContainerStarted","Data":"87f823483cdda29e232cf1c7a57b10bfd1048a13bc97b92a6dfeea694c3b17d1"}
Nov 02 13:18:33.873389 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:33.873270    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-23f720-75984ff486-7cn9p"
Nov 02 13:18:33.921880 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:33.921826    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-23f720-75984ff486-7cn9p" podStartSLOduration=3.591506774 podStartE2EDuration="3.92180641s" totalImagesPullingTime="330.299636ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:18:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:18:33.918141536 +0000 UTC m=+3110.173540635" watchObservedRunningTime="2026-11-02 13:18:33.92180641 +0000 UTC m=+3110.177205506"
Nov 02 13:18:34.010826 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:34.010762    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-1e14d4-openshift-default-6dfdf7978c-jdj7d"]
Nov 02 13:18:34.010990 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:34.010953    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-1e14d4-openshift-default-6dfdf7978c-jdj7d" podUID="4e103054-2a54-43b1-b73d-a50a44c6492a" containerName="istio-proxy" containerID="cri-o://31e13065cd3b65f6ca84a37466fe3229b7b91b31ce7ee35d6c1ef1b655cabfe6" gracePeriod=30
Nov 02 13:18:39.168401 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:39.168361    4559 patch_prober.go:29] interesting pod/e2e-shared-b-w2-1e14d4-openshift-default-6dfdf7978c-jdj7d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body=
Nov 02 13:18:39.168950 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:39.168428    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-1e14d4-openshift-default-6dfdf7978c-jdj7d" podUID="4e103054-2a54-43b1-b73d-a50a44c6492a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused"
Nov 02 13:18:39.319907 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:39.319873    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-1e14d4-openshift-default-6dfdf7978c-jdj7d"
Nov 02 13:18:39.491963 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:39.491934    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e103054-2a54-43b1-b73d-a50a44c6492a-kube-api-access-nkj2c\" (UniqueName: \"kubernetes.io/projected/4e103054-2a54-43b1-b73d-a50a44c6492a-kube-api-access-nkj2c\") pod \"4e103054-2a54-43b1-b73d-a50a44c6492a\" (UID: \"4e103054-2a54-43b1-b73d-a50a44c6492a\") "
Nov 02 13:18:39.492144 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:39.491994    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4e103054-2a54-43b1-b73d-a50a44c6492a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e103054-2a54-43b1-b73d-a50a44c6492a-istio-podinfo\") pod \"4e103054-2a54-43b1-b73d-a50a44c6492a\" (UID: \"4e103054-2a54-43b1-b73d-a50a44c6492a\") "
Nov 02 13:18:39.492144 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:39.492072    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e103054-2a54-43b1-b73d-a50a44c6492a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e103054-2a54-43b1-b73d-a50a44c6492a-istio-envoy\") pod \"4e103054-2a54-43b1-b73d-a50a44c6492a\" (UID: \"4e103054-2a54-43b1-b73d-a50a44c6492a\") "
Nov 02 13:18:39.492144 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:39.492124    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e103054-2a54-43b1-b73d-a50a44c6492a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e103054-2a54-43b1-b73d-a50a44c6492a-istiod-ca-cert\") pod \"4e103054-2a54-43b1-b73d-a50a44c6492a\" (UID: \"4e103054-2a54-43b1-b73d-a50a44c6492a\") "
Nov 02 13:18:39.492355 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:39.492155    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e103054-2a54-43b1-b73d-a50a44c6492a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e103054-2a54-43b1-b73d-a50a44c6492a-credential-socket\") pod \"4e103054-2a54-43b1-b73d-a50a44c6492a\" (UID: \"4e103054-2a54-43b1-b73d-a50a44c6492a\") "
Nov 02 13:18:39.492355 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:39.492270    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e103054-2a54-43b1-b73d-a50a44c6492a-istio-token\" (UniqueName: \"kubernetes.io/projected/4e103054-2a54-43b1-b73d-a50a44c6492a-istio-token\") pod \"4e103054-2a54-43b1-b73d-a50a44c6492a\" (UID: \"4e103054-2a54-43b1-b73d-a50a44c6492a\") "
Nov 02 13:18:39.492468 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:39.492377    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e103054-2a54-43b1-b73d-a50a44c6492a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e103054-2a54-43b1-b73d-a50a44c6492a-workload-certs\") pod \"4e103054-2a54-43b1-b73d-a50a44c6492a\" (UID: \"4e103054-2a54-43b1-b73d-a50a44c6492a\") "
Nov 02 13:18:39.492468 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:39.492441    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e103054-2a54-43b1-b73d-a50a44c6492a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e103054-2a54-43b1-b73d-a50a44c6492a-istio-data\") pod \"4e103054-2a54-43b1-b73d-a50a44c6492a\" (UID: \"4e103054-2a54-43b1-b73d-a50a44c6492a\") "
Nov 02 13:18:39.492468 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:39.492446    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e103054-2a54-43b1-b73d-a50a44c6492a-credential-socket" pod "4e103054-2a54-43b1-b73d-a50a44c6492a" (UID: "4e103054-2a54-43b1-b73d-a50a44c6492a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:18:39.496187 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:39.492511    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e103054-2a54-43b1-b73d-a50a44c6492a-istiod-ca-cert" pod "4e103054-2a54-43b1-b73d-a50a44c6492a" (UID: "4e103054-2a54-43b1-b73d-a50a44c6492a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:18:39.496187 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:39.492523    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e103054-2a54-43b1-b73d-a50a44c6492a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e103054-2a54-43b1-b73d-a50a44c6492a-workload-socket\") pod \"4e103054-2a54-43b1-b73d-a50a44c6492a\" (UID: \"4e103054-2a54-43b1-b73d-a50a44c6492a\") "
Nov 02 13:18:39.496187 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:39.493460    4559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e103054-2a54-43b1-b73d-a50a44c6492a-istiod-ca-cert\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:18:39.496187 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:39.493484    4559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e103054-2a54-43b1-b73d-a50a44c6492a-credential-socket\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:18:39.497209 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:39.497179    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e103054-2a54-43b1-b73d-a50a44c6492a-istio-data" pod "4e103054-2a54-43b1-b73d-a50a44c6492a" (UID: "4e103054-2a54-43b1-b73d-a50a44c6492a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:18:39.509806 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:39.497120    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e103054-2a54-43b1-b73d-a50a44c6492a-workload-socket" pod "4e103054-2a54-43b1-b73d-a50a44c6492a" (UID: "4e103054-2a54-43b1-b73d-a50a44c6492a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:18:39.509806 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:39.497285    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e103054-2a54-43b1-b73d-a50a44c6492a-workload-certs" pod "4e103054-2a54-43b1-b73d-a50a44c6492a" (UID: "4e103054-2a54-43b1-b73d-a50a44c6492a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:18:39.509806 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:39.502788    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e103054-2a54-43b1-b73d-a50a44c6492a-istio-envoy" pod "4e103054-2a54-43b1-b73d-a50a44c6492a" (UID: "4e103054-2a54-43b1-b73d-a50a44c6492a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:18:39.509806 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:39.502844    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4e103054-2a54-43b1-b73d-a50a44c6492a-istio-podinfo" pod "4e103054-2a54-43b1-b73d-a50a44c6492a" (UID: "4e103054-2a54-43b1-b73d-a50a44c6492a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 13:18:39.509806 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:39.503960    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e103054-2a54-43b1-b73d-a50a44c6492a-istio-token" pod "4e103054-2a54-43b1-b73d-a50a44c6492a" (UID: "4e103054-2a54-43b1-b73d-a50a44c6492a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:18:39.511251 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:39.511226    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e103054-2a54-43b1-b73d-a50a44c6492a-kube-api-access-nkj2c" pod "4e103054-2a54-43b1-b73d-a50a44c6492a" (UID: "4e103054-2a54-43b1-b73d-a50a44c6492a"). InnerVolumeSpecName "kube-api-access-nkj2c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:18:39.595686 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:39.595601    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nkj2c\" (UniqueName: \"kubernetes.io/projected/4e103054-2a54-43b1-b73d-a50a44c6492a-kube-api-access-nkj2c\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:18:39.595686 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:39.595633    4559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e103054-2a54-43b1-b73d-a50a44c6492a-istio-podinfo\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:18:39.595686 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:39.595662    4559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e103054-2a54-43b1-b73d-a50a44c6492a-istio-envoy\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:18:39.595686 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:39.595674    4559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e103054-2a54-43b1-b73d-a50a44c6492a-istio-token\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:18:39.595686 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:39.595690    4559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e103054-2a54-43b1-b73d-a50a44c6492a-workload-certs\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:18:39.595939 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:39.595701    4559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e103054-2a54-43b1-b73d-a50a44c6492a-istio-data\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:18:39.595939 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:39.595713    4559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e103054-2a54-43b1-b73d-a50a44c6492a-workload-socket\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:18:39.895775 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:39.895702    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-23f720-75984ff486-7cn9p"
Nov 02 13:18:39.923396 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:39.923365    4559 generic.go:356] "Generic (PLEG): container finished" podID="4e103054-2a54-43b1-b73d-a50a44c6492a" containerID="31e13065cd3b65f6ca84a37466fe3229b7b91b31ce7ee35d6c1ef1b655cabfe6" exitCode=0
Nov 02 13:18:39.923537 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:39.923414    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-1e14d4-openshift-default-6dfdf7978c-jdj7d"
Nov 02 13:18:39.923537 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:39.923447    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-1e14d4-openshift-default-6dfdf7978c-jdj7d" event={"ID":"4e103054-2a54-43b1-b73d-a50a44c6492a","Type":"ContainerDied","Data":"31e13065cd3b65f6ca84a37466fe3229b7b91b31ce7ee35d6c1ef1b655cabfe6"}
Nov 02 13:18:39.923537 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:39.923501    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-1e14d4-openshift-default-6dfdf7978c-jdj7d" event={"ID":"4e103054-2a54-43b1-b73d-a50a44c6492a","Type":"ContainerDied","Data":"295b1c068adc8c3eac7ec798eed201ae82f7c62bc31eb86c14cb0e9518b139df"}
Nov 02 13:18:39.923537 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:39.923533    4559 scope.go:122] "RemoveContainer" containerID="31e13065cd3b65f6ca84a37466fe3229b7b91b31ce7ee35d6c1ef1b655cabfe6"
Nov 02 13:18:39.934736 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:39.934719    4559 scope.go:122] "RemoveContainer" containerID="31e13065cd3b65f6ca84a37466fe3229b7b91b31ce7ee35d6c1ef1b655cabfe6"
Nov 02 13:18:39.935027 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:18:39.935000    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31e13065cd3b65f6ca84a37466fe3229b7b91b31ce7ee35d6c1ef1b655cabfe6\": container with ID starting with 31e13065cd3b65f6ca84a37466fe3229b7b91b31ce7ee35d6c1ef1b655cabfe6 not found: ID does not exist" containerID="31e13065cd3b65f6ca84a37466fe3229b7b91b31ce7ee35d6c1ef1b655cabfe6"
Nov 02 13:18:39.935079 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:39.935034    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31e13065cd3b65f6ca84a37466fe3229b7b91b31ce7ee35d6c1ef1b655cabfe6"} err="failed to get container status \"31e13065cd3b65f6ca84a37466fe3229b7b91b31ce7ee35d6c1ef1b655cabfe6\": rpc error: code = NotFound desc = could not find container \"31e13065cd3b65f6ca84a37466fe3229b7b91b31ce7ee35d6c1ef1b655cabfe6\": container with ID starting with 31e13065cd3b65f6ca84a37466fe3229b7b91b31ce7ee35d6c1ef1b655cabfe6 not found: ID does not exist"
Nov 02 13:18:39.988599 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:39.988562    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-1e14d4-openshift-default-6dfdf7978c-jdj7d"]
Nov 02 13:18:39.993332 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:39.993303    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-1e14d4-openshift-default-6dfdf7978c-jdj7d"]
Nov 02 13:18:41.896038 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:41.896007    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e103054-2a54-43b1-b73d-a50a44c6492a" path="/var/lib/kubelet/pods/4e103054-2a54-43b1-b73d-a50a44c6492a/volumes"
Nov 02 13:18:42.561180 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:42.561142    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-a6f2c0-openshift-default-7f47858fbf-x45sj"]
Nov 02 13:18:42.561429 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:42.561407    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-a6f2c0-openshift-default-7f47858fbf-x45sj" podUID="27dd21d8-102d-45c3-8e54-3c1e846829de" containerName="istio-proxy" containerID="cri-o://d8d8080416b5f46c65cb189efafd288d2033b0c353dbc7c7e54ba0319aaba737" gracePeriod=30
Nov 02 13:18:45.111847 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:18:45.111815    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqcmj" containerName="extractor"
Nov 02 13:18:47.854118 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:47.854092    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-a6f2c0-openshift-default-7f47858fbf-x45sj"
Nov 02 13:18:47.959518 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:47.959480    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27dd21d8-102d-45c3-8e54-3c1e846829de-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27dd21d8-102d-45c3-8e54-3c1e846829de-workload-socket\") pod \"27dd21d8-102d-45c3-8e54-3c1e846829de\" (UID: \"27dd21d8-102d-45c3-8e54-3c1e846829de\") "
Nov 02 13:18:47.959518 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:47.959518    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27dd21d8-102d-45c3-8e54-3c1e846829de-kube-api-access-ffhrw\" (UniqueName: \"kubernetes.io/projected/27dd21d8-102d-45c3-8e54-3c1e846829de-kube-api-access-ffhrw\") pod \"27dd21d8-102d-45c3-8e54-3c1e846829de\" (UID: \"27dd21d8-102d-45c3-8e54-3c1e846829de\") "
Nov 02 13:18:47.959841 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:47.959602    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27dd21d8-102d-45c3-8e54-3c1e846829de-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27dd21d8-102d-45c3-8e54-3c1e846829de-workload-certs\") pod \"27dd21d8-102d-45c3-8e54-3c1e846829de\" (UID: \"27dd21d8-102d-45c3-8e54-3c1e846829de\") "
Nov 02 13:18:47.959841 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:47.959717    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/27dd21d8-102d-45c3-8e54-3c1e846829de-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27dd21d8-102d-45c3-8e54-3c1e846829de-istio-podinfo\") pod \"27dd21d8-102d-45c3-8e54-3c1e846829de\" (UID: \"27dd21d8-102d-45c3-8e54-3c1e846829de\") "
Nov 02 13:18:47.959841 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:47.959801    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27dd21d8-102d-45c3-8e54-3c1e846829de-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27dd21d8-102d-45c3-8e54-3c1e846829de-istiod-ca-cert\") pod \"27dd21d8-102d-45c3-8e54-3c1e846829de\" (UID: \"27dd21d8-102d-45c3-8e54-3c1e846829de\") "
Nov 02 13:18:47.959841 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:47.959807    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27dd21d8-102d-45c3-8e54-3c1e846829de-workload-socket" pod "27dd21d8-102d-45c3-8e54-3c1e846829de" (UID: "27dd21d8-102d-45c3-8e54-3c1e846829de"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:18:47.959841 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:47.959830    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27dd21d8-102d-45c3-8e54-3c1e846829de-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27dd21d8-102d-45c3-8e54-3c1e846829de-credential-socket\") pod \"27dd21d8-102d-45c3-8e54-3c1e846829de\" (UID: \"27dd21d8-102d-45c3-8e54-3c1e846829de\") "
Nov 02 13:18:47.960192 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:47.959904    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27dd21d8-102d-45c3-8e54-3c1e846829de-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27dd21d8-102d-45c3-8e54-3c1e846829de-istio-data\") pod \"27dd21d8-102d-45c3-8e54-3c1e846829de\" (UID: \"27dd21d8-102d-45c3-8e54-3c1e846829de\") "
Nov 02 13:18:47.960192 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:47.959995    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27dd21d8-102d-45c3-8e54-3c1e846829de-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27dd21d8-102d-45c3-8e54-3c1e846829de-istio-envoy\") pod \"27dd21d8-102d-45c3-8e54-3c1e846829de\" (UID: \"27dd21d8-102d-45c3-8e54-3c1e846829de\") "
Nov 02 13:18:47.960192 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:47.960090    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27dd21d8-102d-45c3-8e54-3c1e846829de-istio-token\" (UniqueName: \"kubernetes.io/projected/27dd21d8-102d-45c3-8e54-3c1e846829de-istio-token\") pod \"27dd21d8-102d-45c3-8e54-3c1e846829de\" (UID: \"27dd21d8-102d-45c3-8e54-3c1e846829de\") "
Nov 02 13:18:47.960365 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:47.960192    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27dd21d8-102d-45c3-8e54-3c1e846829de-istiod-ca-cert" pod "27dd21d8-102d-45c3-8e54-3c1e846829de" (UID: "27dd21d8-102d-45c3-8e54-3c1e846829de"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:18:47.960365 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:47.960188    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27dd21d8-102d-45c3-8e54-3c1e846829de-credential-socket" pod "27dd21d8-102d-45c3-8e54-3c1e846829de" (UID: "27dd21d8-102d-45c3-8e54-3c1e846829de"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:18:47.960365 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:47.960300    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27dd21d8-102d-45c3-8e54-3c1e846829de-istio-data" pod "27dd21d8-102d-45c3-8e54-3c1e846829de" (UID: "27dd21d8-102d-45c3-8e54-3c1e846829de"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:18:47.960833 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:47.960800    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27dd21d8-102d-45c3-8e54-3c1e846829de-workload-certs" pod "27dd21d8-102d-45c3-8e54-3c1e846829de" (UID: "27dd21d8-102d-45c3-8e54-3c1e846829de"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:18:47.961049 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:47.960958    4559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27dd21d8-102d-45c3-8e54-3c1e846829de-istiod-ca-cert\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:18:47.961049 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:47.960987    4559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27dd21d8-102d-45c3-8e54-3c1e846829de-credential-socket\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:18:47.961049 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:47.961002    4559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27dd21d8-102d-45c3-8e54-3c1e846829de-istio-data\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:18:47.961049 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:47.961016    4559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27dd21d8-102d-45c3-8e54-3c1e846829de-workload-socket\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:18:47.961049 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:47.961028    4559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27dd21d8-102d-45c3-8e54-3c1e846829de-workload-certs\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:18:47.962506 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:47.962478    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27dd21d8-102d-45c3-8e54-3c1e846829de-kube-api-access-ffhrw" pod "27dd21d8-102d-45c3-8e54-3c1e846829de" (UID: "27dd21d8-102d-45c3-8e54-3c1e846829de"). InnerVolumeSpecName "kube-api-access-ffhrw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:18:47.962738 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:47.962713    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27dd21d8-102d-45c3-8e54-3c1e846829de-istio-envoy" pod "27dd21d8-102d-45c3-8e54-3c1e846829de" (UID: "27dd21d8-102d-45c3-8e54-3c1e846829de"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:18:47.963121 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:47.963095    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/27dd21d8-102d-45c3-8e54-3c1e846829de-istio-podinfo" pod "27dd21d8-102d-45c3-8e54-3c1e846829de" (UID: "27dd21d8-102d-45c3-8e54-3c1e846829de"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 13:18:47.963217 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:47.963197    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27dd21d8-102d-45c3-8e54-3c1e846829de-istio-token" pod "27dd21d8-102d-45c3-8e54-3c1e846829de" (UID: "27dd21d8-102d-45c3-8e54-3c1e846829de"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:18:47.991402 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:47.991377    4559 generic.go:356] "Generic (PLEG): container finished" podID="27dd21d8-102d-45c3-8e54-3c1e846829de" containerID="d8d8080416b5f46c65cb189efafd288d2033b0c353dbc7c7e54ba0319aaba737" exitCode=0
Nov 02 13:18:47.991496 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:47.991450    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-a6f2c0-openshift-default-7f47858fbf-x45sj"
Nov 02 13:18:47.991563 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:47.991449    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-a6f2c0-openshift-default-7f47858fbf-x45sj" event={"ID":"27dd21d8-102d-45c3-8e54-3c1e846829de","Type":"ContainerDied","Data":"d8d8080416b5f46c65cb189efafd288d2033b0c353dbc7c7e54ba0319aaba737"}
Nov 02 13:18:47.991614 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:47.991567    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-a6f2c0-openshift-default-7f47858fbf-x45sj" event={"ID":"27dd21d8-102d-45c3-8e54-3c1e846829de","Type":"ContainerDied","Data":"c1b558c51d1777b1ca557894c40688c6d9ec421be9f21dfb05111c5794390baf"}
Nov 02 13:18:47.991614 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:47.991595    4559 scope.go:122] "RemoveContainer" containerID="d8d8080416b5f46c65cb189efafd288d2033b0c353dbc7c7e54ba0319aaba737"
Nov 02 13:18:48.016222 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:48.016199    4559 scope.go:122] "RemoveContainer" containerID="214bc758f94617b4bb60976c80c20ad95dc5e3249685f127b36f6acc044ddbbf"
Nov 02 13:18:48.028246 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:48.028198    4559 scope.go:122] "RemoveContainer" containerID="d8d8080416b5f46c65cb189efafd288d2033b0c353dbc7c7e54ba0319aaba737"
Nov 02 13:18:48.028538 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:18:48.028499    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8d8080416b5f46c65cb189efafd288d2033b0c353dbc7c7e54ba0319aaba737\": container with ID starting with d8d8080416b5f46c65cb189efafd288d2033b0c353dbc7c7e54ba0319aaba737 not found: ID does not exist" containerID="d8d8080416b5f46c65cb189efafd288d2033b0c353dbc7c7e54ba0319aaba737"
Nov 02 13:18:48.028630 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:48.028548    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8d8080416b5f46c65cb189efafd288d2033b0c353dbc7c7e54ba0319aaba737"} err="failed to get container status \"d8d8080416b5f46c65cb189efafd288d2033b0c353dbc7c7e54ba0319aaba737\": rpc error: code = NotFound desc = could not find container \"d8d8080416b5f46c65cb189efafd288d2033b0c353dbc7c7e54ba0319aaba737\": container with ID starting with d8d8080416b5f46c65cb189efafd288d2033b0c353dbc7c7e54ba0319aaba737 not found: ID does not exist"
Nov 02 13:18:48.028630 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:48.028569    4559 scope.go:122] "RemoveContainer" containerID="214bc758f94617b4bb60976c80c20ad95dc5e3249685f127b36f6acc044ddbbf"
Nov 02 13:18:48.028910 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:18:48.028882    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"214bc758f94617b4bb60976c80c20ad95dc5e3249685f127b36f6acc044ddbbf\": container with ID starting with 214bc758f94617b4bb60976c80c20ad95dc5e3249685f127b36f6acc044ddbbf not found: ID does not exist" containerID="214bc758f94617b4bb60976c80c20ad95dc5e3249685f127b36f6acc044ddbbf"
Nov 02 13:18:48.028984 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:48.028919    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"214bc758f94617b4bb60976c80c20ad95dc5e3249685f127b36f6acc044ddbbf"} err="failed to get container status \"214bc758f94617b4bb60976c80c20ad95dc5e3249685f127b36f6acc044ddbbf\": rpc error: code = NotFound desc = could not find container \"214bc758f94617b4bb60976c80c20ad95dc5e3249685f127b36f6acc044ddbbf\": container with ID starting with 214bc758f94617b4bb60976c80c20ad95dc5e3249685f127b36f6acc044ddbbf not found: ID does not exist"
Nov 02 13:18:48.057768 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:48.057746    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-a6f2c0-openshift-default-7f47858fbf-x45sj"]
Nov 02 13:18:48.062343 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:48.062318    4559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27dd21d8-102d-45c3-8e54-3c1e846829de-istio-envoy\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:18:48.062343 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:48.062342    4559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27dd21d8-102d-45c3-8e54-3c1e846829de-istio-token\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:18:48.062529 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:48.062351    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ffhrw\" (UniqueName: \"kubernetes.io/projected/27dd21d8-102d-45c3-8e54-3c1e846829de-kube-api-access-ffhrw\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:18:48.062529 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:48.062360    4559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27dd21d8-102d-45c3-8e54-3c1e846829de-istio-podinfo\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:18:48.066595 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:48.066573    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-a6f2c0-openshift-default-7f47858fbf-x45sj"]
Nov 02 13:18:49.895701 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:18:49.895667    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27dd21d8-102d-45c3-8e54-3c1e846829de" path="/var/lib/kubelet/pods/27dd21d8-102d-45c3-8e54-3c1e846829de/volumes"
Nov 02 13:19:15.086735 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:19:15.086698    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqcmj" containerName="extractor"
Nov 02 13:19:16.194296 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:19:16.194260    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"]
Nov 02 13:19:25.011836 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:19:25.011794    4559 patch_prober.go:29] interesting pod/e2e-inf-a-23f720-openshift-default-6c8d4f9ff5-vpdqd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.245:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 13:19:25.012385 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:19:25.011904    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-23f720-openshift-default-6c8d4f9ff5-vpdqd" podUID="2d3ed32f-021a-456e-85eb-864596d28c3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.245:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:19:30.867739 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:19:30.867616    4559 generic.go:356] "Generic (PLEG): container finished" podID="2d3ed32f-021a-456e-85eb-864596d28c3f" containerID="20602e8e5be6c5b18679fc49a63b7b5d46ba260dd464be42f7448b7123a1a4cd" exitCode=0
Nov 02 13:19:30.867739 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:19:30.867698    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-23f720-openshift-default-6c8d4f9ff5-vpdqd" event={"ID":"2d3ed32f-021a-456e-85eb-864596d28c3f","Type":"ContainerDied","Data":"20602e8e5be6c5b18679fc49a63b7b5d46ba260dd464be42f7448b7123a1a4cd"}
Nov 02 13:19:30.868192 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:19:30.868141    4559 scope.go:122] "RemoveContainer" containerID="20602e8e5be6c5b18679fc49a63b7b5d46ba260dd464be42f7448b7123a1a4cd"
Nov 02 13:19:30.870011 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:19:30.869982    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 13:19:30.870107 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:19:30.870053    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 13:19:30.870107 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:19:30.870086    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 13:19:31.877280 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:19:31.877249    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-23f720-openshift-default-6c8d4f9ff5-vpdqd" event={"ID":"2d3ed32f-021a-456e-85eb-864596d28c3f","Type":"ContainerStarted","Data":"fa17bc1362dc89348a5ceaadb90ebc414eafa81682f2b06cf0eb0f16f3854aa4"}
Nov 02 13:19:32.042704 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:19:32.042673    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-23f720-openshift-default-6c8d4f9ff5-vpdqd"
Nov 02 13:19:32.042901 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:19:32.042836    4559 patch_prober.go:29] interesting pod/e2e-inf-a-23f720-openshift-default-6c8d4f9ff5-vpdqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused" start-of-body=
Nov 02 13:19:32.042901 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:19:32.042888    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-23f720-openshift-default-6c8d4f9ff5-vpdqd" podUID="2d3ed32f-021a-456e-85eb-864596d28c3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused"
Nov 02 13:19:33.043097 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:19:33.043066    4559 patch_prober.go:29] interesting pod/e2e-inf-a-23f720-openshift-default-6c8d4f9ff5-vpdqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused" start-of-body=
Nov 02 13:19:33.043481 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:19:33.043115    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-23f720-openshift-default-6c8d4f9ff5-vpdqd" podUID="2d3ed32f-021a-456e-85eb-864596d28c3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused"
Nov 02 13:19:34.043663 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:19:34.043203    4559 patch_prober.go:29] interesting pod/e2e-inf-a-23f720-openshift-default-6c8d4f9ff5-vpdqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused" start-of-body=
Nov 02 13:19:34.043663 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:19:34.043267    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-23f720-openshift-default-6c8d4f9ff5-vpdqd" podUID="2d3ed32f-021a-456e-85eb-864596d28c3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused"
Nov 02 13:19:35.043264 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:19:35.043229    4559 patch_prober.go:29] interesting pod/e2e-inf-a-23f720-openshift-default-6c8d4f9ff5-vpdqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused" start-of-body=
Nov 02 13:19:35.043434 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:19:35.043291    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-23f720-openshift-default-6c8d4f9ff5-vpdqd" podUID="2d3ed32f-021a-456e-85eb-864596d28c3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused"
Nov 02 13:19:36.042972 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:19:36.042935    4559 patch_prober.go:29] interesting pod/e2e-inf-a-23f720-openshift-default-6c8d4f9ff5-vpdqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused" start-of-body=
Nov 02 13:19:36.043550 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:19:36.043014    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-23f720-openshift-default-6c8d4f9ff5-vpdqd" podUID="2d3ed32f-021a-456e-85eb-864596d28c3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused"
Nov 02 13:19:37.043456 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:19:37.043423    4559 patch_prober.go:29] interesting pod/e2e-inf-a-23f720-openshift-default-6c8d4f9ff5-vpdqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused" start-of-body=
Nov 02 13:19:37.043891 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:19:37.043481    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-23f720-openshift-default-6c8d4f9ff5-vpdqd" podUID="2d3ed32f-021a-456e-85eb-864596d28c3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused"
Nov 02 13:19:38.080214 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:19:38.080180    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-23f720-openshift-default-6c8d4f9ff5-vpdqd"
Nov 02 13:19:38.080793 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:19:38.080749    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-23f720-openshift-default-6c8d4f9ff5-vpdqd"
Nov 02 13:19:38.081748 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:19:38.081725    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-23f720-openshift-default-6c8d4f9ff5-vpdqd"
Nov 02 13:19:43.345229 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:19:43.345195    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-bbb7d0-6bcb48d58c-h8gsp"]
Nov 02 13:19:43.380017 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:19:43.379979    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-bbb7d0-7cddbf6b47-z2q7w"]
Nov 02 13:19:45.123604 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:19:45.120743    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqcmj" containerName="extractor"
Nov 02 13:19:48.346483 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:19:48.346433    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-bbb7d0-6bcb48d58c-h8gsp" podUID="1b3239ce-770b-4f22-9461-89f8c62f850a" containerName="payload-processing" containerID="cri-o://b6dceb868d01a25c8ae20abe82d66e591fbb1e19f352dbdcab4bc1ad6ca97c69" gracePeriod=25
Nov 02 13:19:48.380321 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:19:48.380285    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bbb7d0-7cddbf6b47-z2q7w" podUID="affbe914-1335-4829-b1d5-c31bf0bd44ea" containerName="payload-pre-processing" containerID="cri-o://051e6af572a8d3d0d97733c531e6f5ca206ee4c0db728d65036300d25b90f83a" gracePeriod=25
Nov 02 13:19:54.246660 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:19:54.246595    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bbb7d0-7cddbf6b47-z2q7w" podUID="affbe914-1335-4829-b1d5-c31bf0bd44ea" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 13:19:54.247092 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:19:54.246614    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-bbb7d0-6bcb48d58c-h8gsp" podUID="1b3239ce-770b-4f22-9461-89f8c62f850a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 13:20:04.245447 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:20:04.245355    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bbb7d0-7cddbf6b47-z2q7w" podUID="affbe914-1335-4829-b1d5-c31bf0bd44ea" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 13:20:04.245957 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:20:04.245457    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-bbb7d0-6bcb48d58c-h8gsp" podUID="1b3239ce-770b-4f22-9461-89f8c62f850a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 13:20:08.723762 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:20:08.723734    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bbb7d0-7cddbf6b47-z2q7w"
Nov 02 13:20:08.727527 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:20:08.727507    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-bbb7d0-6bcb48d58c-h8gsp"
Nov 02 13:20:08.771936 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:20:08.771908    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b3239ce-770b-4f22-9461-89f8c62f850a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b3239ce-770b-4f22-9461-89f8c62f850a-plugins-config-volume\") pod \"1b3239ce-770b-4f22-9461-89f8c62f850a\" (UID: \"1b3239ce-770b-4f22-9461-89f8c62f850a\") "
Nov 02 13:20:08.772237 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:20:08.772206    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/affbe914-1335-4829-b1d5-c31bf0bd44ea-kube-api-access-rvscc\" (UniqueName: \"kubernetes.io/projected/affbe914-1335-4829-b1d5-c31bf0bd44ea-kube-api-access-rvscc\") pod \"affbe914-1335-4829-b1d5-c31bf0bd44ea\" (UID: \"affbe914-1335-4829-b1d5-c31bf0bd44ea\") "
Nov 02 13:20:08.772322 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:20:08.772260    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b3239ce-770b-4f22-9461-89f8c62f850a-kube-api-access-rmdr5\" (UniqueName: \"kubernetes.io/projected/1b3239ce-770b-4f22-9461-89f8c62f850a-kube-api-access-rmdr5\") pod \"1b3239ce-770b-4f22-9461-89f8c62f850a\" (UID: \"1b3239ce-770b-4f22-9461-89f8c62f850a\") "
Nov 02 13:20:08.772322 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:20:08.772265    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b3239ce-770b-4f22-9461-89f8c62f850a-plugins-config-volume" pod "1b3239ce-770b-4f22-9461-89f8c62f850a" (UID: "1b3239ce-770b-4f22-9461-89f8c62f850a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:20:08.772412 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:20:08.772334    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/affbe914-1335-4829-b1d5-c31bf0bd44ea-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/affbe914-1335-4829-b1d5-c31bf0bd44ea-plugins-config-volume\") pod \"affbe914-1335-4829-b1d5-c31bf0bd44ea\" (UID: \"affbe914-1335-4829-b1d5-c31bf0bd44ea\") "
Nov 02 13:20:08.772772 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:20:08.772748    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/affbe914-1335-4829-b1d5-c31bf0bd44ea-plugins-config-volume" pod "affbe914-1335-4829-b1d5-c31bf0bd44ea" (UID: "affbe914-1335-4829-b1d5-c31bf0bd44ea"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:20:08.773165 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:20:08.773142    4559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/affbe914-1335-4829-b1d5-c31bf0bd44ea-plugins-config-volume\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:20:08.773240 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:20:08.773172    4559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b3239ce-770b-4f22-9461-89f8c62f850a-plugins-config-volume\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:20:08.774872 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:20:08.774851    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b3239ce-770b-4f22-9461-89f8c62f850a-kube-api-access-rmdr5" pod "1b3239ce-770b-4f22-9461-89f8c62f850a" (UID: "1b3239ce-770b-4f22-9461-89f8c62f850a"). InnerVolumeSpecName "kube-api-access-rmdr5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:20:08.774972 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:20:08.774881    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/affbe914-1335-4829-b1d5-c31bf0bd44ea-kube-api-access-rvscc" pod "affbe914-1335-4829-b1d5-c31bf0bd44ea" (UID: "affbe914-1335-4829-b1d5-c31bf0bd44ea"). InnerVolumeSpecName "kube-api-access-rvscc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:20:08.875005 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:20:08.874976    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvscc\" (UniqueName: \"kubernetes.io/projected/affbe914-1335-4829-b1d5-c31bf0bd44ea-kube-api-access-rvscc\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:20:08.875005 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:20:08.875002    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rmdr5\" (UniqueName: \"kubernetes.io/projected/1b3239ce-770b-4f22-9461-89f8c62f850a-kube-api-access-rmdr5\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:20:09.397158 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:20:09.397126    4559 generic.go:356] "Generic (PLEG): container finished" podID="affbe914-1335-4829-b1d5-c31bf0bd44ea" containerID="051e6af572a8d3d0d97733c531e6f5ca206ee4c0db728d65036300d25b90f83a" exitCode=0
Nov 02 13:20:09.397330 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:20:09.397171    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bbb7d0-7cddbf6b47-z2q7w"
Nov 02 13:20:09.397330 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:20:09.397210    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bbb7d0-7cddbf6b47-z2q7w" event={"ID":"affbe914-1335-4829-b1d5-c31bf0bd44ea","Type":"ContainerDied","Data":"051e6af572a8d3d0d97733c531e6f5ca206ee4c0db728d65036300d25b90f83a"}
Nov 02 13:20:09.397330 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:20:09.397267    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bbb7d0-7cddbf6b47-z2q7w" event={"ID":"affbe914-1335-4829-b1d5-c31bf0bd44ea","Type":"ContainerDied","Data":"7bb85d058ffcd7f77ac7c6735699b541ed47551fc4807d6ebd0ada9c6bf34315"}
Nov 02 13:20:09.397330 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:20:09.397298    4559 scope.go:122] "RemoveContainer" containerID="051e6af572a8d3d0d97733c531e6f5ca206ee4c0db728d65036300d25b90f83a"
Nov 02 13:20:09.398827 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:20:09.398795    4559 generic.go:356] "Generic (PLEG): container finished" podID="1b3239ce-770b-4f22-9461-89f8c62f850a" containerID="b6dceb868d01a25c8ae20abe82d66e591fbb1e19f352dbdcab4bc1ad6ca97c69" exitCode=0
Nov 02 13:20:09.399066 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:20:09.398833    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-bbb7d0-6bcb48d58c-h8gsp"
Nov 02 13:20:09.399066 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:20:09.398845    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-bbb7d0-6bcb48d58c-h8gsp" event={"ID":"1b3239ce-770b-4f22-9461-89f8c62f850a","Type":"ContainerDied","Data":"b6dceb868d01a25c8ae20abe82d66e591fbb1e19f352dbdcab4bc1ad6ca97c69"}
Nov 02 13:20:09.399066 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:20:09.398883    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-bbb7d0-6bcb48d58c-h8gsp" event={"ID":"1b3239ce-770b-4f22-9461-89f8c62f850a","Type":"ContainerDied","Data":"84e01fcc7e6fad6a13d28aa29e46ef9d2609500039e021277e14d711395675f9"}
Nov 02 13:20:09.412005 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:20:09.411968    4559 scope.go:122] "RemoveContainer" containerID="051e6af572a8d3d0d97733c531e6f5ca206ee4c0db728d65036300d25b90f83a"
Nov 02 13:20:09.412478 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:20:09.412445    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"051e6af572a8d3d0d97733c531e6f5ca206ee4c0db728d65036300d25b90f83a\": container with ID starting with 051e6af572a8d3d0d97733c531e6f5ca206ee4c0db728d65036300d25b90f83a not found: ID does not exist" containerID="051e6af572a8d3d0d97733c531e6f5ca206ee4c0db728d65036300d25b90f83a"
Nov 02 13:20:09.412569 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:20:09.412490    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"051e6af572a8d3d0d97733c531e6f5ca206ee4c0db728d65036300d25b90f83a"} err="failed to get container status \"051e6af572a8d3d0d97733c531e6f5ca206ee4c0db728d65036300d25b90f83a\": rpc error: code = NotFound desc = could not find container \"051e6af572a8d3d0d97733c531e6f5ca206ee4c0db728d65036300d25b90f83a\": container with ID starting with 051e6af572a8d3d0d97733c531e6f5ca206ee4c0db728d65036300d25b90f83a not found: ID does not exist"
Nov 02 13:20:09.412569 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:20:09.412516    4559 scope.go:122] "RemoveContainer" containerID="b6dceb868d01a25c8ae20abe82d66e591fbb1e19f352dbdcab4bc1ad6ca97c69"
Nov 02 13:20:09.428025 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:20:09.428005    4559 scope.go:122] "RemoveContainer" containerID="b6dceb868d01a25c8ae20abe82d66e591fbb1e19f352dbdcab4bc1ad6ca97c69"
Nov 02 13:20:09.428321 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:20:09.428296    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6dceb868d01a25c8ae20abe82d66e591fbb1e19f352dbdcab4bc1ad6ca97c69\": container with ID starting with b6dceb868d01a25c8ae20abe82d66e591fbb1e19f352dbdcab4bc1ad6ca97c69 not found: ID does not exist" containerID="b6dceb868d01a25c8ae20abe82d66e591fbb1e19f352dbdcab4bc1ad6ca97c69"
Nov 02 13:20:09.428428 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:20:09.428324    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6dceb868d01a25c8ae20abe82d66e591fbb1e19f352dbdcab4bc1ad6ca97c69"} err="failed to get container status \"b6dceb868d01a25c8ae20abe82d66e591fbb1e19f352dbdcab4bc1ad6ca97c69\": rpc error: code = NotFound desc = could not find container \"b6dceb868d01a25c8ae20abe82d66e591fbb1e19f352dbdcab4bc1ad6ca97c69\": container with ID starting with b6dceb868d01a25c8ae20abe82d66e591fbb1e19f352dbdcab4bc1ad6ca97c69 not found: ID does not exist"
Nov 02 13:20:09.454725 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:20:09.454692    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-bbb7d0-7cddbf6b47-z2q7w"]
Nov 02 13:20:09.466434 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:20:09.466410    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-bbb7d0-7cddbf6b47-z2q7w"]
Nov 02 13:20:09.517906 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:20:09.517879    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-bbb7d0-6bcb48d58c-h8gsp"]
Nov 02 13:20:09.521665 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:20:09.521630    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-bbb7d0-6bcb48d58c-h8gsp"]
Nov 02 13:20:09.894656 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:20:09.894606    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b3239ce-770b-4f22-9461-89f8c62f850a" path="/var/lib/kubelet/pods/1b3239ce-770b-4f22-9461-89f8c62f850a/volumes"
Nov 02 13:20:09.895268 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:20:09.895245    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="affbe914-1335-4829-b1d5-c31bf0bd44ea" path="/var/lib/kubelet/pods/affbe914-1335-4829-b1d5-c31bf0bd44ea/volumes"
Nov 02 13:20:15.088125 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:20:15.088079    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqcmj" containerName="extractor"
Nov 02 13:20:23.721029 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:20:23.720998    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-bbb7d0-84c974d75-jlqqq"]
Nov 02 13:20:23.721500 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:20:23.721203    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bbb7d0-84c974d75-jlqqq" podUID="4436e381-1572-416e-ad03-2805362dde64" containerName="maas-api" containerID="cri-o://6ebaad0b6c4564b1bb0480b10003ff6f0898fbd29c966e61317a5ec30282d8bb" gracePeriod=30
Nov 02 13:20:24.313708 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:20:24.308996    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bbb7d0-84c974d75-jlqqq"
Nov 02 13:20:24.420661 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:20:24.420387    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4436e381-1572-416e-ad03-2805362dde64-kube-api-access-zj8fz\" (UniqueName: \"kubernetes.io/projected/4436e381-1572-416e-ad03-2805362dde64-kube-api-access-zj8fz\") pod \"4436e381-1572-416e-ad03-2805362dde64\" (UID: \"4436e381-1572-416e-ad03-2805362dde64\") "
Nov 02 13:20:24.420661 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:20:24.420483    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4436e381-1572-416e-ad03-2805362dde64-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4436e381-1572-416e-ad03-2805362dde64-openshift-service-ca\") pod \"4436e381-1572-416e-ad03-2805362dde64\" (UID: \"4436e381-1572-416e-ad03-2805362dde64\") "
Nov 02 13:20:24.420661 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:20:24.420556    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4436e381-1572-416e-ad03-2805362dde64-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4436e381-1572-416e-ad03-2805362dde64-maas-api-tls\") pod \"4436e381-1572-416e-ad03-2805362dde64\" (UID: \"4436e381-1572-416e-ad03-2805362dde64\") "
Nov 02 13:20:24.420661 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:20:24.420625    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4436e381-1572-416e-ad03-2805362dde64-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4436e381-1572-416e-ad03-2805362dde64-metrics-certs\") pod \"4436e381-1572-416e-ad03-2805362dde64\" (UID: \"4436e381-1572-416e-ad03-2805362dde64\") "
Nov 02 13:20:24.433524 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:20:24.431090    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4436e381-1572-416e-ad03-2805362dde64-metrics-certs" pod "4436e381-1572-416e-ad03-2805362dde64" (UID: "4436e381-1572-416e-ad03-2805362dde64"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:20:24.446663 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:20:24.433881    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4436e381-1572-416e-ad03-2805362dde64-openshift-service-ca" pod "4436e381-1572-416e-ad03-2805362dde64" (UID: "4436e381-1572-416e-ad03-2805362dde64"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:20:24.446663 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:20:24.446469    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4436e381-1572-416e-ad03-2805362dde64-maas-api-tls" pod "4436e381-1572-416e-ad03-2805362dde64" (UID: "4436e381-1572-416e-ad03-2805362dde64"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:20:24.463664 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:20:24.450141    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4436e381-1572-416e-ad03-2805362dde64-kube-api-access-zj8fz" pod "4436e381-1572-416e-ad03-2805362dde64" (UID: "4436e381-1572-416e-ad03-2805362dde64"). InnerVolumeSpecName "kube-api-access-zj8fz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:20:24.523568 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:20:24.523519    4559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4436e381-1572-416e-ad03-2805362dde64-openshift-service-ca\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:20:24.523568 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:20:24.523559    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4436e381-1572-416e-ad03-2805362dde64-maas-api-tls\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:20:24.523863 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:20:24.523576    4559 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4436e381-1572-416e-ad03-2805362dde64-metrics-certs\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:20:24.523863 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:20:24.523591    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zj8fz\" (UniqueName: \"kubernetes.io/projected/4436e381-1572-416e-ad03-2805362dde64-kube-api-access-zj8fz\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:20:24.608313 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:20:24.608213    4559 generic.go:356] "Generic (PLEG): container finished" podID="4436e381-1572-416e-ad03-2805362dde64" containerID="6ebaad0b6c4564b1bb0480b10003ff6f0898fbd29c966e61317a5ec30282d8bb" exitCode=0
Nov 02 13:20:24.608508 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:20:24.608354    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bbb7d0-84c974d75-jlqqq" event={"ID":"4436e381-1572-416e-ad03-2805362dde64","Type":"ContainerDied","Data":"6ebaad0b6c4564b1bb0480b10003ff6f0898fbd29c966e61317a5ec30282d8bb"}
Nov 02 13:20:24.608508 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:20:24.608405    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bbb7d0-84c974d75-jlqqq" event={"ID":"4436e381-1572-416e-ad03-2805362dde64","Type":"ContainerDied","Data":"256ffb7640772b7d6a503acf12150fd1fbb24bd77f963bd6787538916e18d9c9"}
Nov 02 13:20:24.608508 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:20:24.608441    4559 scope.go:122] "RemoveContainer" containerID="6ebaad0b6c4564b1bb0480b10003ff6f0898fbd29c966e61317a5ec30282d8bb"
Nov 02 13:20:24.608695 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:20:24.608608    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bbb7d0-84c974d75-jlqqq"
Nov 02 13:20:24.634381 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:20:24.634354    4559 scope.go:122] "RemoveContainer" containerID="6ebaad0b6c4564b1bb0480b10003ff6f0898fbd29c966e61317a5ec30282d8bb"
Nov 02 13:20:24.634834 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:20:24.634807    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ebaad0b6c4564b1bb0480b10003ff6f0898fbd29c966e61317a5ec30282d8bb\": container with ID starting with 6ebaad0b6c4564b1bb0480b10003ff6f0898fbd29c966e61317a5ec30282d8bb not found: ID does not exist" containerID="6ebaad0b6c4564b1bb0480b10003ff6f0898fbd29c966e61317a5ec30282d8bb"
Nov 02 13:20:24.634935 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:20:24.634843    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ebaad0b6c4564b1bb0480b10003ff6f0898fbd29c966e61317a5ec30282d8bb"} err="failed to get container status \"6ebaad0b6c4564b1bb0480b10003ff6f0898fbd29c966e61317a5ec30282d8bb\": rpc error: code = NotFound desc = could not find container \"6ebaad0b6c4564b1bb0480b10003ff6f0898fbd29c966e61317a5ec30282d8bb\": container with ID starting with 6ebaad0b6c4564b1bb0480b10003ff6f0898fbd29c966e61317a5ec30282d8bb not found: ID does not exist"
Nov 02 13:20:24.675707 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:20:24.675675    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-bbb7d0-84c974d75-jlqqq"]
Nov 02 13:20:24.683136 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:20:24.683107    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-bbb7d0-84c974d75-jlqqq"]
Nov 02 13:20:24.708172 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:20:24.708143    4559 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4436e381_1572_416e_ad03_2805362dde64.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4436e381_1572_416e_ad03_2805362dde64.slice/crio-256ffb7640772b7d6a503acf12150fd1fbb24bd77f963bd6787538916e18d9c9\": RecentStats: unable to find data in memory cache]"
Nov 02 13:20:25.134674 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:20:25.132782    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"]
Nov 02 13:20:26.002665 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:20:25.999563    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4436e381-1572-416e-ad03-2805362dde64" path="/var/lib/kubelet/pods/4436e381-1572-416e-ad03-2805362dde64/volumes"
Nov 02 13:20:45.088541 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:20:45.088509    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqcmj" containerName="extractor"
Nov 02 13:21:15.088606 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:21:15.088570    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqcmj" containerName="extractor"
Nov 02 13:21:31.210073 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:21:31.209994    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-9139ec-74b46d9789-dtps2"]
Nov 02 13:21:31.210599 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:21:31.210242    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9139ec-74b46d9789-dtps2" podUID="004721a0-56e6-42d6-bca7-c9cb11cf0e5f" containerName="maas-api" containerID="cri-o://3097121f84d7c13654d14342b17b4b3beca591f7d7dad6e84bcf0045fe2c033e" gracePeriod=30
Nov 02 13:21:31.385778 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:21:31.385746    4559 generic.go:356] "Generic (PLEG): container finished" podID="004721a0-56e6-42d6-bca7-c9cb11cf0e5f" containerID="3097121f84d7c13654d14342b17b4b3beca591f7d7dad6e84bcf0045fe2c033e" exitCode=0
Nov 02 13:21:31.385937 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:21:31.385818    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9139ec-74b46d9789-dtps2" event={"ID":"004721a0-56e6-42d6-bca7-c9cb11cf0e5f","Type":"ContainerDied","Data":"3097121f84d7c13654d14342b17b4b3beca591f7d7dad6e84bcf0045fe2c033e"}
Nov 02 13:21:31.867662 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:21:31.867173    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9139ec-74b46d9789-dtps2"
Nov 02 13:21:32.008742 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:21:32.008708    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/004721a0-56e6-42d6-bca7-c9cb11cf0e5f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/004721a0-56e6-42d6-bca7-c9cb11cf0e5f-maas-api-tls\") pod \"004721a0-56e6-42d6-bca7-c9cb11cf0e5f\" (UID: \"004721a0-56e6-42d6-bca7-c9cb11cf0e5f\") "
Nov 02 13:21:32.008960 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:21:32.008903    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/004721a0-56e6-42d6-bca7-c9cb11cf0e5f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/004721a0-56e6-42d6-bca7-c9cb11cf0e5f-openshift-service-ca\") pod \"004721a0-56e6-42d6-bca7-c9cb11cf0e5f\" (UID: \"004721a0-56e6-42d6-bca7-c9cb11cf0e5f\") "
Nov 02 13:21:32.009157 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:21:32.009133    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/004721a0-56e6-42d6-bca7-c9cb11cf0e5f-kube-api-access-cptfx\" (UniqueName: \"kubernetes.io/projected/004721a0-56e6-42d6-bca7-c9cb11cf0e5f-kube-api-access-cptfx\") pod \"004721a0-56e6-42d6-bca7-c9cb11cf0e5f\" (UID: \"004721a0-56e6-42d6-bca7-c9cb11cf0e5f\") "
Nov 02 13:21:32.009316 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:21:32.009182    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/004721a0-56e6-42d6-bca7-c9cb11cf0e5f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/004721a0-56e6-42d6-bca7-c9cb11cf0e5f-metrics-certs\") pod \"004721a0-56e6-42d6-bca7-c9cb11cf0e5f\" (UID: \"004721a0-56e6-42d6-bca7-c9cb11cf0e5f\") "
Nov 02 13:21:32.009391 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:21:32.009312    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/004721a0-56e6-42d6-bca7-c9cb11cf0e5f-openshift-service-ca" pod "004721a0-56e6-42d6-bca7-c9cb11cf0e5f" (UID: "004721a0-56e6-42d6-bca7-c9cb11cf0e5f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:21:32.010099 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:21:32.010070    4559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/004721a0-56e6-42d6-bca7-c9cb11cf0e5f-openshift-service-ca\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:21:32.011072 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:21:32.011049    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/004721a0-56e6-42d6-bca7-c9cb11cf0e5f-maas-api-tls" pod "004721a0-56e6-42d6-bca7-c9cb11cf0e5f" (UID: "004721a0-56e6-42d6-bca7-c9cb11cf0e5f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:21:32.011454 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:21:32.011432    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/004721a0-56e6-42d6-bca7-c9cb11cf0e5f-kube-api-access-cptfx" pod "004721a0-56e6-42d6-bca7-c9cb11cf0e5f" (UID: "004721a0-56e6-42d6-bca7-c9cb11cf0e5f"). InnerVolumeSpecName "kube-api-access-cptfx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:21:32.011847 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:21:32.011829    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/004721a0-56e6-42d6-bca7-c9cb11cf0e5f-metrics-certs" pod "004721a0-56e6-42d6-bca7-c9cb11cf0e5f" (UID: "004721a0-56e6-42d6-bca7-c9cb11cf0e5f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:21:32.117708 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:21:32.113194    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cptfx\" (UniqueName: \"kubernetes.io/projected/004721a0-56e6-42d6-bca7-c9cb11cf0e5f-kube-api-access-cptfx\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:21:32.117708 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:21:32.113244    4559 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/004721a0-56e6-42d6-bca7-c9cb11cf0e5f-metrics-certs\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:21:32.117708 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:21:32.113260    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/004721a0-56e6-42d6-bca7-c9cb11cf0e5f-maas-api-tls\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:21:32.396146 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:21:32.396115    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9139ec-74b46d9789-dtps2" event={"ID":"004721a0-56e6-42d6-bca7-c9cb11cf0e5f","Type":"ContainerDied","Data":"fd3f1a557070281ad707d6253fd4a9dcf4d418515688e63c228234bb98e9776d"}
Nov 02 13:21:32.396560 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:21:32.396155    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9139ec-74b46d9789-dtps2"
Nov 02 13:21:32.396560 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:21:32.396173    4559 scope.go:122] "RemoveContainer" containerID="3097121f84d7c13654d14342b17b4b3beca591f7d7dad6e84bcf0045fe2c033e"
Nov 02 13:21:32.453769 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:21:32.453743    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-9139ec-74b46d9789-dtps2"]
Nov 02 13:21:32.465803 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:21:32.465776    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-9139ec-74b46d9789-dtps2"]
Nov 02 13:21:33.904880 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:21:33.904846    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="004721a0-56e6-42d6-bca7-c9cb11cf0e5f" path="/var/lib/kubelet/pods/004721a0-56e6-42d6-bca7-c9cb11cf0e5f/volumes"
Nov 02 13:21:40.837475 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:21:40.837426    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-5f8899/test-model-5f8899-kserve-585d469589-78dz5"]
Nov 02 13:21:41.811825 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:21:41.811789    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-361c3a/test-model-361c3a-kserve-5fdb8ffcc-jjfbm"]
Nov 02 13:21:43.495420 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:21:43.495387    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"]
Nov 02 13:21:45.255600 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:21:45.255528    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqcmj" containerName="extractor"
Nov 02 13:21:55.883435 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:21:55.883388    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-5f8899/test-model-5f8899-kserve-585d469589-78dz5" podUID="3799b3ab-6927-4a3c-a610-2d7eb7fe507c" containerName="main" containerID="cri-o://9eb7ef16d099bb488fd4418cf51e91a4be81a35ca555dde1dbcfd0cfd7f370ee" gracePeriod=45
Nov 02 13:21:56.249332 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:21:56.249304    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-5f8899/test-model-5f8899-kserve-585d469589-78dz5"
Nov 02 13:21:56.369960 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:21:56.369927    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3799b3ab-6927-4a3c-a610-2d7eb7fe507c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3799b3ab-6927-4a3c-a610-2d7eb7fe507c-model-cache\") pod \"3799b3ab-6927-4a3c-a610-2d7eb7fe507c\" (UID: \"3799b3ab-6927-4a3c-a610-2d7eb7fe507c\") "
Nov 02 13:21:56.369960 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:21:56.369963    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3799b3ab-6927-4a3c-a610-2d7eb7fe507c-home\" (UniqueName: \"kubernetes.io/empty-dir/3799b3ab-6927-4a3c-a610-2d7eb7fe507c-home\") pod \"3799b3ab-6927-4a3c-a610-2d7eb7fe507c\" (UID: \"3799b3ab-6927-4a3c-a610-2d7eb7fe507c\") "
Nov 02 13:21:56.370171 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:21:56.370060    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3799b3ab-6927-4a3c-a610-2d7eb7fe507c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3799b3ab-6927-4a3c-a610-2d7eb7fe507c-tmp-dir\") pod \"3799b3ab-6927-4a3c-a610-2d7eb7fe507c\" (UID: \"3799b3ab-6927-4a3c-a610-2d7eb7fe507c\") "
Nov 02 13:21:56.370171 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:21:56.370139    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3799b3ab-6927-4a3c-a610-2d7eb7fe507c-kube-api-access-5w9ng\" (UniqueName: \"kubernetes.io/projected/3799b3ab-6927-4a3c-a610-2d7eb7fe507c-kube-api-access-5w9ng\") pod \"3799b3ab-6927-4a3c-a610-2d7eb7fe507c\" (UID: \"3799b3ab-6927-4a3c-a610-2d7eb7fe507c\") "
Nov 02 13:21:56.370288 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:21:56.370240    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3799b3ab-6927-4a3c-a610-2d7eb7fe507c-model-cache" pod "3799b3ab-6927-4a3c-a610-2d7eb7fe507c" (UID: "3799b3ab-6927-4a3c-a610-2d7eb7fe507c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:21:56.370288 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:21:56.370257    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3799b3ab-6927-4a3c-a610-2d7eb7fe507c-home" pod "3799b3ab-6927-4a3c-a610-2d7eb7fe507c" (UID: "3799b3ab-6927-4a3c-a610-2d7eb7fe507c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:21:56.370487 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:21:56.370356    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3799b3ab-6927-4a3c-a610-2d7eb7fe507c-tmp-dir" pod "3799b3ab-6927-4a3c-a610-2d7eb7fe507c" (UID: "3799b3ab-6927-4a3c-a610-2d7eb7fe507c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:21:56.370487 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:21:56.370364    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3799b3ab-6927-4a3c-a610-2d7eb7fe507c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3799b3ab-6927-4a3c-a610-2d7eb7fe507c-dshm\") pod \"3799b3ab-6927-4a3c-a610-2d7eb7fe507c\" (UID: \"3799b3ab-6927-4a3c-a610-2d7eb7fe507c\") "
Nov 02 13:21:56.370487 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:21:56.370423    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3799b3ab-6927-4a3c-a610-2d7eb7fe507c-tls-certs\" (UniqueName: \"kubernetes.io/secret/3799b3ab-6927-4a3c-a610-2d7eb7fe507c-tls-certs\") pod \"3799b3ab-6927-4a3c-a610-2d7eb7fe507c\" (UID: \"3799b3ab-6927-4a3c-a610-2d7eb7fe507c\") "
Nov 02 13:21:56.371288 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:21:56.371263    4559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3799b3ab-6927-4a3c-a610-2d7eb7fe507c-model-cache\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:21:56.371393 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:21:56.371294    4559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3799b3ab-6927-4a3c-a610-2d7eb7fe507c-home\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:21:56.371393 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:21:56.371313    4559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3799b3ab-6927-4a3c-a610-2d7eb7fe507c-tmp-dir\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:21:56.372561 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:21:56.372541    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3799b3ab-6927-4a3c-a610-2d7eb7fe507c-dshm" pod "3799b3ab-6927-4a3c-a610-2d7eb7fe507c" (UID: "3799b3ab-6927-4a3c-a610-2d7eb7fe507c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:21:56.372738 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:21:56.372715    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3799b3ab-6927-4a3c-a610-2d7eb7fe507c-tls-certs" pod "3799b3ab-6927-4a3c-a610-2d7eb7fe507c" (UID: "3799b3ab-6927-4a3c-a610-2d7eb7fe507c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:21:56.372855 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:21:56.372826    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3799b3ab-6927-4a3c-a610-2d7eb7fe507c-kube-api-access-5w9ng" pod "3799b3ab-6927-4a3c-a610-2d7eb7fe507c" (UID: "3799b3ab-6927-4a3c-a610-2d7eb7fe507c"). InnerVolumeSpecName "kube-api-access-5w9ng". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:21:56.472949 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:21:56.472886    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5w9ng\" (UniqueName: \"kubernetes.io/projected/3799b3ab-6927-4a3c-a610-2d7eb7fe507c-kube-api-access-5w9ng\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:21:56.472949 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:21:56.472910    4559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3799b3ab-6927-4a3c-a610-2d7eb7fe507c-dshm\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:21:56.472949 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:21:56.472919    4559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3799b3ab-6927-4a3c-a610-2d7eb7fe507c-tls-certs\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:21:56.686503 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:21:56.686477    4559 generic.go:356] "Generic (PLEG): container finished" podID="3799b3ab-6927-4a3c-a610-2d7eb7fe507c" containerID="9eb7ef16d099bb488fd4418cf51e91a4be81a35ca555dde1dbcfd0cfd7f370ee" exitCode=0
Nov 02 13:21:56.686705 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:21:56.686521    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-5f8899/test-model-5f8899-kserve-585d469589-78dz5"
Nov 02 13:21:56.686705 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:21:56.686554    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-5f8899/test-model-5f8899-kserve-585d469589-78dz5" event={"ID":"3799b3ab-6927-4a3c-a610-2d7eb7fe507c","Type":"ContainerDied","Data":"9eb7ef16d099bb488fd4418cf51e91a4be81a35ca555dde1dbcfd0cfd7f370ee"}
Nov 02 13:21:56.686705 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:21:56.686602    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-5f8899/test-model-5f8899-kserve-585d469589-78dz5" event={"ID":"3799b3ab-6927-4a3c-a610-2d7eb7fe507c","Type":"ContainerDied","Data":"b902b6e1348a2b4181060a980e4c7f9cb2803d38d5f58555ca2c70be52cb74af"}
Nov 02 13:21:56.686705 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:21:56.686624    4559 scope.go:122] "RemoveContainer" containerID="9eb7ef16d099bb488fd4418cf51e91a4be81a35ca555dde1dbcfd0cfd7f370ee"
Nov 02 13:21:56.698248 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:21:56.698231    4559 scope.go:122] "RemoveContainer" containerID="9eb7ef16d099bb488fd4418cf51e91a4be81a35ca555dde1dbcfd0cfd7f370ee"
Nov 02 13:21:56.698532 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:21:56.698505    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9eb7ef16d099bb488fd4418cf51e91a4be81a35ca555dde1dbcfd0cfd7f370ee\": container with ID starting with 9eb7ef16d099bb488fd4418cf51e91a4be81a35ca555dde1dbcfd0cfd7f370ee not found: ID does not exist" containerID="9eb7ef16d099bb488fd4418cf51e91a4be81a35ca555dde1dbcfd0cfd7f370ee"
Nov 02 13:21:56.698602 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:21:56.698531    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9eb7ef16d099bb488fd4418cf51e91a4be81a35ca555dde1dbcfd0cfd7f370ee"} err="failed to get container status \"9eb7ef16d099bb488fd4418cf51e91a4be81a35ca555dde1dbcfd0cfd7f370ee\": rpc error: code = NotFound desc = could not find container \"9eb7ef16d099bb488fd4418cf51e91a4be81a35ca555dde1dbcfd0cfd7f370ee\": container with ID starting with 9eb7ef16d099bb488fd4418cf51e91a4be81a35ca555dde1dbcfd0cfd7f370ee not found: ID does not exist"
Nov 02 13:21:56.743716 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:21:56.743657    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-5f8899/test-model-5f8899-kserve-585d469589-78dz5"]
Nov 02 13:21:56.755192 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:21:56.755172    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-5f8899/test-model-5f8899-kserve-585d469589-78dz5"]
Nov 02 13:21:56.835740 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:21:56.835680    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-361c3a/test-model-361c3a-kserve-5fdb8ffcc-jjfbm" podUID="2f43f207-7f92-4aab-a554-903c5d18c8f3" containerName="main" containerID="cri-o://b32179fe53f896ba73cfa06d9a9d8814520e60af1f1c6c944b3b8a2b3b01ec7d" gracePeriod=45
Nov 02 13:21:57.117942 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:21:57.117917    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-361c3a/test-model-361c3a-kserve-5fdb8ffcc-jjfbm"
Nov 02 13:21:57.285129 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:21:57.285054    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f43f207-7f92-4aab-a554-903c5d18c8f3-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f43f207-7f92-4aab-a554-903c5d18c8f3-tmp-dir\") pod \"2f43f207-7f92-4aab-a554-903c5d18c8f3\" (UID: \"2f43f207-7f92-4aab-a554-903c5d18c8f3\") "
Nov 02 13:21:57.285274 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:21:57.285169    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f43f207-7f92-4aab-a554-903c5d18c8f3-kube-api-access-bjc2x\" (UniqueName: \"kubernetes.io/projected/2f43f207-7f92-4aab-a554-903c5d18c8f3-kube-api-access-bjc2x\") pod \"2f43f207-7f92-4aab-a554-903c5d18c8f3\" (UID: \"2f43f207-7f92-4aab-a554-903c5d18c8f3\") "
Nov 02 13:21:57.285274 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:21:57.285234    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f43f207-7f92-4aab-a554-903c5d18c8f3-home\" (UniqueName: \"kubernetes.io/empty-dir/2f43f207-7f92-4aab-a554-903c5d18c8f3-home\") pod \"2f43f207-7f92-4aab-a554-903c5d18c8f3\" (UID: \"2f43f207-7f92-4aab-a554-903c5d18c8f3\") "
Nov 02 13:21:57.285353 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:21:57.285336    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f43f207-7f92-4aab-a554-903c5d18c8f3-tmp-dir" pod "2f43f207-7f92-4aab-a554-903c5d18c8f3" (UID: "2f43f207-7f92-4aab-a554-903c5d18c8f3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:21:57.285407 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:21:57.285392    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2f43f207-7f92-4aab-a554-903c5d18c8f3-tls-certs\" (UniqueName: \"kubernetes.io/secret/2f43f207-7f92-4aab-a554-903c5d18c8f3-tls-certs\") pod \"2f43f207-7f92-4aab-a554-903c5d18c8f3\" (UID: \"2f43f207-7f92-4aab-a554-903c5d18c8f3\") "
Nov 02 13:21:57.285466 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:21:57.285447    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f43f207-7f92-4aab-a554-903c5d18c8f3-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f43f207-7f92-4aab-a554-903c5d18c8f3-model-cache\") pod \"2f43f207-7f92-4aab-a554-903c5d18c8f3\" (UID: \"2f43f207-7f92-4aab-a554-903c5d18c8f3\") "
Nov 02 13:21:57.285531 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:21:57.285505    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f43f207-7f92-4aab-a554-903c5d18c8f3-home" pod "2f43f207-7f92-4aab-a554-903c5d18c8f3" (UID: "2f43f207-7f92-4aab-a554-903c5d18c8f3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:21:57.285565 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:21:57.285514    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f43f207-7f92-4aab-a554-903c5d18c8f3-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f43f207-7f92-4aab-a554-903c5d18c8f3-dshm\") pod \"2f43f207-7f92-4aab-a554-903c5d18c8f3\" (UID: \"2f43f207-7f92-4aab-a554-903c5d18c8f3\") "
Nov 02 13:21:57.285784 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:21:57.285760    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f43f207-7f92-4aab-a554-903c5d18c8f3-model-cache" pod "2f43f207-7f92-4aab-a554-903c5d18c8f3" (UID: "2f43f207-7f92-4aab-a554-903c5d18c8f3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:21:57.286170 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:21:57.286152    4559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f43f207-7f92-4aab-a554-903c5d18c8f3-model-cache\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:21:57.286255 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:21:57.286173    4559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f43f207-7f92-4aab-a554-903c5d18c8f3-tmp-dir\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:21:57.286255 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:21:57.286181    4559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f43f207-7f92-4aab-a554-903c5d18c8f3-home\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:21:57.287690 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:21:57.287663    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f43f207-7f92-4aab-a554-903c5d18c8f3-dshm" pod "2f43f207-7f92-4aab-a554-903c5d18c8f3" (UID: "2f43f207-7f92-4aab-a554-903c5d18c8f3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:21:57.287813 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:21:57.287701    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f43f207-7f92-4aab-a554-903c5d18c8f3-tls-certs" pod "2f43f207-7f92-4aab-a554-903c5d18c8f3" (UID: "2f43f207-7f92-4aab-a554-903c5d18c8f3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:21:57.287882 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:21:57.287845    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f43f207-7f92-4aab-a554-903c5d18c8f3-kube-api-access-bjc2x" pod "2f43f207-7f92-4aab-a554-903c5d18c8f3" (UID: "2f43f207-7f92-4aab-a554-903c5d18c8f3"). InnerVolumeSpecName "kube-api-access-bjc2x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:21:57.387903 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:21:57.387864    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bjc2x\" (UniqueName: \"kubernetes.io/projected/2f43f207-7f92-4aab-a554-903c5d18c8f3-kube-api-access-bjc2x\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:21:57.387903 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:21:57.387893    4559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f43f207-7f92-4aab-a554-903c5d18c8f3-tls-certs\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:21:57.387903 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:21:57.387906    4559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f43f207-7f92-4aab-a554-903c5d18c8f3-dshm\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:21:57.696182 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:21:57.696146    4559 generic.go:356] "Generic (PLEG): container finished" podID="2f43f207-7f92-4aab-a554-903c5d18c8f3" containerID="b32179fe53f896ba73cfa06d9a9d8814520e60af1f1c6c944b3b8a2b3b01ec7d" exitCode=0
Nov 02 13:21:57.696388 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:21:57.696215    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-361c3a/test-model-361c3a-kserve-5fdb8ffcc-jjfbm"
Nov 02 13:21:57.696388 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:21:57.696244    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-361c3a/test-model-361c3a-kserve-5fdb8ffcc-jjfbm" event={"ID":"2f43f207-7f92-4aab-a554-903c5d18c8f3","Type":"ContainerDied","Data":"b32179fe53f896ba73cfa06d9a9d8814520e60af1f1c6c944b3b8a2b3b01ec7d"}
Nov 02 13:21:57.696388 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:21:57.696303    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-361c3a/test-model-361c3a-kserve-5fdb8ffcc-jjfbm" event={"ID":"2f43f207-7f92-4aab-a554-903c5d18c8f3","Type":"ContainerDied","Data":"01950ec6304c475264cf9172ca4b9a14fd1da0088b2b75f7de6ec9efa860202d"}
Nov 02 13:21:57.696388 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:21:57.696321    4559 scope.go:122] "RemoveContainer" containerID="b32179fe53f896ba73cfa06d9a9d8814520e60af1f1c6c944b3b8a2b3b01ec7d"
Nov 02 13:21:57.709279 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:21:57.709262    4559 scope.go:122] "RemoveContainer" containerID="b32179fe53f896ba73cfa06d9a9d8814520e60af1f1c6c944b3b8a2b3b01ec7d"
Nov 02 13:21:57.709560 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:21:57.709531    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b32179fe53f896ba73cfa06d9a9d8814520e60af1f1c6c944b3b8a2b3b01ec7d\": container with ID starting with b32179fe53f896ba73cfa06d9a9d8814520e60af1f1c6c944b3b8a2b3b01ec7d not found: ID does not exist" containerID="b32179fe53f896ba73cfa06d9a9d8814520e60af1f1c6c944b3b8a2b3b01ec7d"
Nov 02 13:21:57.709733 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:21:57.709570    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b32179fe53f896ba73cfa06d9a9d8814520e60af1f1c6c944b3b8a2b3b01ec7d"} err="failed to get container status \"b32179fe53f896ba73cfa06d9a9d8814520e60af1f1c6c944b3b8a2b3b01ec7d\": rpc error: code = NotFound desc = could not find container \"b32179fe53f896ba73cfa06d9a9d8814520e60af1f1c6c944b3b8a2b3b01ec7d\": container with ID starting with b32179fe53f896ba73cfa06d9a9d8814520e60af1f1c6c944b3b8a2b3b01ec7d not found: ID does not exist"
Nov 02 13:21:57.755914 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:21:57.755887    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-361c3a/test-model-361c3a-kserve-5fdb8ffcc-jjfbm"]
Nov 02 13:21:57.767479 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:21:57.767456    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-361c3a/test-model-361c3a-kserve-5fdb8ffcc-jjfbm"]
Nov 02 13:21:57.895345 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:21:57.895315    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f43f207-7f92-4aab-a554-903c5d18c8f3" path="/var/lib/kubelet/pods/2f43f207-7f92-4aab-a554-903c5d18c8f3/volumes"
Nov 02 13:21:57.895841 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:21:57.895826    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3799b3ab-6927-4a3c-a610-2d7eb7fe507c" path="/var/lib/kubelet/pods/3799b3ab-6927-4a3c-a610-2d7eb7fe507c/volumes"
Nov 02 13:22:00.148770 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:00.148739    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-5f8899-59548dd96c-5p585"]
Nov 02 13:22:02.861517 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:02.860652    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-5f8899-69b5fb7989-zz4xv"]
Nov 02 13:22:02.861517 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:02.860915    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5f8899-69b5fb7989-zz4xv" podUID="ada43a30-3953-47ab-9bb7-d623b8025e13" containerName="maas-api" containerID="cri-o://7e29594fb9ab173d1c431e732f04f7efa4ae94ebdb8ee39d4d7576be8b982318" gracePeriod=30
Nov 02 13:22:03.788660 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:03.786125    4559 generic.go:356] "Generic (PLEG): container finished" podID="ada43a30-3953-47ab-9bb7-d623b8025e13" containerID="7e29594fb9ab173d1c431e732f04f7efa4ae94ebdb8ee39d4d7576be8b982318" exitCode=0
Nov 02 13:22:03.788660 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:03.786434    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5f8899-69b5fb7989-zz4xv" event={"ID":"ada43a30-3953-47ab-9bb7-d623b8025e13","Type":"ContainerDied","Data":"7e29594fb9ab173d1c431e732f04f7efa4ae94ebdb8ee39d4d7576be8b982318"}
Nov 02 13:22:03.931553 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:03.931529    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5f8899-69b5fb7989-zz4xv"
Nov 02 13:22:04.070926 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:04.070848    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ada43a30-3953-47ab-9bb7-d623b8025e13-kube-api-access-kqvcm\" (UniqueName: \"kubernetes.io/projected/ada43a30-3953-47ab-9bb7-d623b8025e13-kube-api-access-kqvcm\") pod \"ada43a30-3953-47ab-9bb7-d623b8025e13\" (UID: \"ada43a30-3953-47ab-9bb7-d623b8025e13\") "
Nov 02 13:22:04.071068 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:04.070981    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ada43a30-3953-47ab-9bb7-d623b8025e13-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ada43a30-3953-47ab-9bb7-d623b8025e13-metrics-certs\") pod \"ada43a30-3953-47ab-9bb7-d623b8025e13\" (UID: \"ada43a30-3953-47ab-9bb7-d623b8025e13\") "
Nov 02 13:22:04.071068 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:04.071009    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ada43a30-3953-47ab-9bb7-d623b8025e13-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ada43a30-3953-47ab-9bb7-d623b8025e13-maas-api-tls\") pod \"ada43a30-3953-47ab-9bb7-d623b8025e13\" (UID: \"ada43a30-3953-47ab-9bb7-d623b8025e13\") "
Nov 02 13:22:04.071161 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:04.071075    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ada43a30-3953-47ab-9bb7-d623b8025e13-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ada43a30-3953-47ab-9bb7-d623b8025e13-openshift-service-ca\") pod \"ada43a30-3953-47ab-9bb7-d623b8025e13\" (UID: \"ada43a30-3953-47ab-9bb7-d623b8025e13\") "
Nov 02 13:22:04.071512 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:04.071485    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ada43a30-3953-47ab-9bb7-d623b8025e13-openshift-service-ca" pod "ada43a30-3953-47ab-9bb7-d623b8025e13" (UID: "ada43a30-3953-47ab-9bb7-d623b8025e13"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:22:04.071941 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:04.071916    4559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ada43a30-3953-47ab-9bb7-d623b8025e13-openshift-service-ca\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:22:04.073017 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:04.072988    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ada43a30-3953-47ab-9bb7-d623b8025e13-kube-api-access-kqvcm" pod "ada43a30-3953-47ab-9bb7-d623b8025e13" (UID: "ada43a30-3953-47ab-9bb7-d623b8025e13"). InnerVolumeSpecName "kube-api-access-kqvcm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:22:04.073148 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:04.073127    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ada43a30-3953-47ab-9bb7-d623b8025e13-metrics-certs" pod "ada43a30-3953-47ab-9bb7-d623b8025e13" (UID: "ada43a30-3953-47ab-9bb7-d623b8025e13"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:22:04.073199 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:04.073184    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ada43a30-3953-47ab-9bb7-d623b8025e13-maas-api-tls" pod "ada43a30-3953-47ab-9bb7-d623b8025e13" (UID: "ada43a30-3953-47ab-9bb7-d623b8025e13"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:22:04.173721 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:04.173681    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kqvcm\" (UniqueName: \"kubernetes.io/projected/ada43a30-3953-47ab-9bb7-d623b8025e13-kube-api-access-kqvcm\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:22:04.173721 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:04.173712    4559 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ada43a30-3953-47ab-9bb7-d623b8025e13-metrics-certs\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:22:04.173721 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:04.173725    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ada43a30-3953-47ab-9bb7-d623b8025e13-maas-api-tls\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:22:04.795156 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:04.795118    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5f8899-69b5fb7989-zz4xv" event={"ID":"ada43a30-3953-47ab-9bb7-d623b8025e13","Type":"ContainerDied","Data":"1c30bdc437bb1b01e6a2059cb710f9ae781e3bba96bfaf6698fe06bd8e24df69"}
Nov 02 13:22:04.795352 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:04.795185    4559 scope.go:122] "RemoveContainer" containerID="7e29594fb9ab173d1c431e732f04f7efa4ae94ebdb8ee39d4d7576be8b982318"
Nov 02 13:22:04.795352 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:04.795139    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5f8899-69b5fb7989-zz4xv"
Nov 02 13:22:04.851811 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:04.851787    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-5f8899-69b5fb7989-zz4xv"]
Nov 02 13:22:04.864629 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:04.864605    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-5f8899-69b5fb7989-zz4xv"]
Nov 02 13:22:05.149570 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:05.149488    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5f8899-59548dd96c-5p585" podUID="da6dd2b6-1e1c-4128-a988-cf86451095e1" containerName="payload-pre-processing" containerID="cri-o://31e8306d86f94bf7ad0f16e6dca4a5bc07d2df8c306d0ec8455f4152832b2d61" gracePeriod=25
Nov 02 13:22:05.894948 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:05.894908    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ada43a30-3953-47ab-9bb7-d623b8025e13" path="/var/lib/kubelet/pods/ada43a30-3953-47ab-9bb7-d623b8025e13/volumes"
Nov 02 13:22:08.030580 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:08.030538    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5f8899-59548dd96c-5p585" podUID="da6dd2b6-1e1c-4128-a988-cf86451095e1" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 13:22:08.954839 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:08.954804    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-bmcxg"]
Nov 02 13:22:08.955092 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:08.955061    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-bmcxg" podUID="185fcba5-401f-4a43-bad1-f82f845221b8" containerName="istio-proxy" containerID="cri-o://190aeadec88fade680508bb06145e8268d599a5ce8bcfb0c305de74114cb52ff" gracePeriod=30
Nov 02 13:22:08.965873 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:08.965847    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-q77g2"]
Nov 02 13:22:08.966091 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:08.966067    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-q77g2" podUID="279e09ef-b780-457c-a0b0-66948ee0f699" containerName="istio-proxy" containerID="cri-o://6611f45b056f82ddcaaac585cbe7560a0cd621b022bbf602a9c07b50152dcad5" gracePeriod=30
Nov 02 13:22:10.851346 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:10.851312    4559 patch_prober.go:29] interesting pod/e2e-inf-a-5f8899-openshift-default-67cc5d8789-bmcxg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 13:22:10.851864 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:10.851365    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-bmcxg" podUID="185fcba5-401f-4a43-bad1-f82f845221b8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 13:22:11.692848 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:11.692809    4559 patch_prober.go:29] interesting pod/e2e-inf-a-5f8899-openshift-default-67cc5d8789-q77g2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 13:22:11.693018 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:11.692876    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-q77g2" podUID="279e09ef-b780-457c-a0b0-66948ee0f699" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 13:22:14.272957 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:14.272928    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-bmcxg"
Nov 02 13:22:14.306476 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:14.306450    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-q77g2"
Nov 02 13:22:14.336593 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:14.336494    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/185fcba5-401f-4a43-bad1-f82f845221b8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/185fcba5-401f-4a43-bad1-f82f845221b8-credential-socket\") pod \"185fcba5-401f-4a43-bad1-f82f845221b8\" (UID: \"185fcba5-401f-4a43-bad1-f82f845221b8\") "
Nov 02 13:22:14.336804 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:14.336660    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/185fcba5-401f-4a43-bad1-f82f845221b8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/185fcba5-401f-4a43-bad1-f82f845221b8-istiod-ca-cert\") pod \"185fcba5-401f-4a43-bad1-f82f845221b8\" (UID: \"185fcba5-401f-4a43-bad1-f82f845221b8\") "
Nov 02 13:22:14.336804 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:14.336724    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/185fcba5-401f-4a43-bad1-f82f845221b8-istio-token\" (UniqueName: \"kubernetes.io/projected/185fcba5-401f-4a43-bad1-f82f845221b8-istio-token\") pod \"185fcba5-401f-4a43-bad1-f82f845221b8\" (UID: \"185fcba5-401f-4a43-bad1-f82f845221b8\") "
Nov 02 13:22:14.336804 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:14.336756    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/185fcba5-401f-4a43-bad1-f82f845221b8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/185fcba5-401f-4a43-bad1-f82f845221b8-istio-data\") pod \"185fcba5-401f-4a43-bad1-f82f845221b8\" (UID: \"185fcba5-401f-4a43-bad1-f82f845221b8\") "
Nov 02 13:22:14.337054 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:14.336833    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/185fcba5-401f-4a43-bad1-f82f845221b8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/185fcba5-401f-4a43-bad1-f82f845221b8-workload-certs\") pod \"185fcba5-401f-4a43-bad1-f82f845221b8\" (UID: \"185fcba5-401f-4a43-bad1-f82f845221b8\") "
Nov 02 13:22:14.337054 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:14.336893    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/185fcba5-401f-4a43-bad1-f82f845221b8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/185fcba5-401f-4a43-bad1-f82f845221b8-workload-socket\") pod \"185fcba5-401f-4a43-bad1-f82f845221b8\" (UID: \"185fcba5-401f-4a43-bad1-f82f845221b8\") "
Nov 02 13:22:14.337054 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:14.336969    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/185fcba5-401f-4a43-bad1-f82f845221b8-kube-api-access-lbzgw\" (UniqueName: \"kubernetes.io/projected/185fcba5-401f-4a43-bad1-f82f845221b8-kube-api-access-lbzgw\") pod \"185fcba5-401f-4a43-bad1-f82f845221b8\" (UID: \"185fcba5-401f-4a43-bad1-f82f845221b8\") "
Nov 02 13:22:14.337054 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:14.337002    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/185fcba5-401f-4a43-bad1-f82f845221b8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/185fcba5-401f-4a43-bad1-f82f845221b8-istio-envoy\") pod \"185fcba5-401f-4a43-bad1-f82f845221b8\" (UID: \"185fcba5-401f-4a43-bad1-f82f845221b8\") "
Nov 02 13:22:14.337283 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:14.337081    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/185fcba5-401f-4a43-bad1-f82f845221b8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/185fcba5-401f-4a43-bad1-f82f845221b8-istio-podinfo\") pod \"185fcba5-401f-4a43-bad1-f82f845221b8\" (UID: \"185fcba5-401f-4a43-bad1-f82f845221b8\") "
Nov 02 13:22:14.337877 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:14.337608    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/185fcba5-401f-4a43-bad1-f82f845221b8-workload-certs" pod "185fcba5-401f-4a43-bad1-f82f845221b8" (UID: "185fcba5-401f-4a43-bad1-f82f845221b8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:22:14.338314 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:14.338017    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/185fcba5-401f-4a43-bad1-f82f845221b8-istiod-ca-cert" pod "185fcba5-401f-4a43-bad1-f82f845221b8" (UID: "185fcba5-401f-4a43-bad1-f82f845221b8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:22:14.338314 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:14.338227    4559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/185fcba5-401f-4a43-bad1-f82f845221b8-istiod-ca-cert\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:22:14.338314 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:14.338253    4559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/185fcba5-401f-4a43-bad1-f82f845221b8-workload-certs\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:22:14.338314 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:14.338274    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/185fcba5-401f-4a43-bad1-f82f845221b8-workload-socket" pod "185fcba5-401f-4a43-bad1-f82f845221b8" (UID: "185fcba5-401f-4a43-bad1-f82f845221b8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:22:14.338314 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:14.338286    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/185fcba5-401f-4a43-bad1-f82f845221b8-istio-data" pod "185fcba5-401f-4a43-bad1-f82f845221b8" (UID: "185fcba5-401f-4a43-bad1-f82f845221b8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:22:14.338734 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:14.338430    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/185fcba5-401f-4a43-bad1-f82f845221b8-credential-socket" pod "185fcba5-401f-4a43-bad1-f82f845221b8" (UID: "185fcba5-401f-4a43-bad1-f82f845221b8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:22:14.340420 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:14.340392    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/185fcba5-401f-4a43-bad1-f82f845221b8-istio-token" pod "185fcba5-401f-4a43-bad1-f82f845221b8" (UID: "185fcba5-401f-4a43-bad1-f82f845221b8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:22:14.340420 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:14.340402    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/185fcba5-401f-4a43-bad1-f82f845221b8-istio-podinfo" pod "185fcba5-401f-4a43-bad1-f82f845221b8" (UID: "185fcba5-401f-4a43-bad1-f82f845221b8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 13:22:14.342010 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:14.341974    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/185fcba5-401f-4a43-bad1-f82f845221b8-istio-envoy" pod "185fcba5-401f-4a43-bad1-f82f845221b8" (UID: "185fcba5-401f-4a43-bad1-f82f845221b8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:22:14.349044 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:14.349020    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/185fcba5-401f-4a43-bad1-f82f845221b8-kube-api-access-lbzgw" pod "185fcba5-401f-4a43-bad1-f82f845221b8" (UID: "185fcba5-401f-4a43-bad1-f82f845221b8"). InnerVolumeSpecName "kube-api-access-lbzgw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:22:14.438997 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:14.438965    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/279e09ef-b780-457c-a0b0-66948ee0f699-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/279e09ef-b780-457c-a0b0-66948ee0f699-istiod-ca-cert\") pod \"279e09ef-b780-457c-a0b0-66948ee0f699\" (UID: \"279e09ef-b780-457c-a0b0-66948ee0f699\") "
Nov 02 13:22:14.439239 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:14.439015    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/279e09ef-b780-457c-a0b0-66948ee0f699-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/279e09ef-b780-457c-a0b0-66948ee0f699-istio-envoy\") pod \"279e09ef-b780-457c-a0b0-66948ee0f699\" (UID: \"279e09ef-b780-457c-a0b0-66948ee0f699\") "
Nov 02 13:22:14.439239 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:14.439113    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/279e09ef-b780-457c-a0b0-66948ee0f699-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/279e09ef-b780-457c-a0b0-66948ee0f699-credential-socket\") pod \"279e09ef-b780-457c-a0b0-66948ee0f699\" (UID: \"279e09ef-b780-457c-a0b0-66948ee0f699\") "
Nov 02 13:22:14.439239 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:14.439142    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/279e09ef-b780-457c-a0b0-66948ee0f699-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/279e09ef-b780-457c-a0b0-66948ee0f699-workload-certs\") pod \"279e09ef-b780-457c-a0b0-66948ee0f699\" (UID: \"279e09ef-b780-457c-a0b0-66948ee0f699\") "
Nov 02 13:22:14.439239 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:14.439210    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/279e09ef-b780-457c-a0b0-66948ee0f699-istio-token\" (UniqueName: \"kubernetes.io/projected/279e09ef-b780-457c-a0b0-66948ee0f699-istio-token\") pod \"279e09ef-b780-457c-a0b0-66948ee0f699\" (UID: \"279e09ef-b780-457c-a0b0-66948ee0f699\") "
Nov 02 13:22:14.439512 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:14.439258    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/279e09ef-b780-457c-a0b0-66948ee0f699-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/279e09ef-b780-457c-a0b0-66948ee0f699-workload-socket\") pod \"279e09ef-b780-457c-a0b0-66948ee0f699\" (UID: \"279e09ef-b780-457c-a0b0-66948ee0f699\") "
Nov 02 13:22:14.439512 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:14.439366    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/279e09ef-b780-457c-a0b0-66948ee0f699-kube-api-access-x7tqn\" (UniqueName: \"kubernetes.io/projected/279e09ef-b780-457c-a0b0-66948ee0f699-kube-api-access-x7tqn\") pod \"279e09ef-b780-457c-a0b0-66948ee0f699\" (UID: \"279e09ef-b780-457c-a0b0-66948ee0f699\") "
Nov 02 13:22:14.439512 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:14.439436    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/279e09ef-b780-457c-a0b0-66948ee0f699-credential-socket" pod "279e09ef-b780-457c-a0b0-66948ee0f699" (UID: "279e09ef-b780-457c-a0b0-66948ee0f699"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:22:14.439512 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:14.439346    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/279e09ef-b780-457c-a0b0-66948ee0f699-istiod-ca-cert" pod "279e09ef-b780-457c-a0b0-66948ee0f699" (UID: "279e09ef-b780-457c-a0b0-66948ee0f699"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:22:14.439512 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:14.439460    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/279e09ef-b780-457c-a0b0-66948ee0f699-workload-certs" pod "279e09ef-b780-457c-a0b0-66948ee0f699" (UID: "279e09ef-b780-457c-a0b0-66948ee0f699"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:22:14.439512 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:14.439496    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/279e09ef-b780-457c-a0b0-66948ee0f699-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/279e09ef-b780-457c-a0b0-66948ee0f699-istio-data\") pod \"279e09ef-b780-457c-a0b0-66948ee0f699\" (UID: \"279e09ef-b780-457c-a0b0-66948ee0f699\") "
Nov 02 13:22:14.439879 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:14.439557    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/279e09ef-b780-457c-a0b0-66948ee0f699-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/279e09ef-b780-457c-a0b0-66948ee0f699-istio-podinfo\") pod \"279e09ef-b780-457c-a0b0-66948ee0f699\" (UID: \"279e09ef-b780-457c-a0b0-66948ee0f699\") "
Nov 02 13:22:14.439879 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:14.439633    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/279e09ef-b780-457c-a0b0-66948ee0f699-workload-socket" pod "279e09ef-b780-457c-a0b0-66948ee0f699" (UID: "279e09ef-b780-457c-a0b0-66948ee0f699"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:22:14.439879 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:14.439769    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/279e09ef-b780-457c-a0b0-66948ee0f699-istio-data" pod "279e09ef-b780-457c-a0b0-66948ee0f699" (UID: "279e09ef-b780-457c-a0b0-66948ee0f699"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:22:14.440332 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:14.440307    4559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/279e09ef-b780-457c-a0b0-66948ee0f699-credential-socket\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:22:14.440481 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:14.440337    4559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/279e09ef-b780-457c-a0b0-66948ee0f699-workload-certs\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:22:14.440481 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:14.440355    4559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/185fcba5-401f-4a43-bad1-f82f845221b8-istio-token\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:22:14.440481 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:14.440368    4559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/185fcba5-401f-4a43-bad1-f82f845221b8-istio-data\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:22:14.440481 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:14.440380    4559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/279e09ef-b780-457c-a0b0-66948ee0f699-workload-socket\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:22:14.440481 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:14.440396    4559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/185fcba5-401f-4a43-bad1-f82f845221b8-workload-socket\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:22:14.440481 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:14.440414    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lbzgw\" (UniqueName: \"kubernetes.io/projected/185fcba5-401f-4a43-bad1-f82f845221b8-kube-api-access-lbzgw\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:22:14.440481 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:14.440429    4559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/185fcba5-401f-4a43-bad1-f82f845221b8-istio-envoy\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:22:14.440481 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:14.440443    4559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/185fcba5-401f-4a43-bad1-f82f845221b8-istio-podinfo\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:22:14.440481 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:14.440456    4559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/279e09ef-b780-457c-a0b0-66948ee0f699-istio-data\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:22:14.440481 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:14.440469    4559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/279e09ef-b780-457c-a0b0-66948ee0f699-istiod-ca-cert\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:22:14.440481 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:14.440482    4559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/185fcba5-401f-4a43-bad1-f82f845221b8-credential-socket\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:22:14.441556 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:14.441528    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/279e09ef-b780-457c-a0b0-66948ee0f699-istio-envoy" pod "279e09ef-b780-457c-a0b0-66948ee0f699" (UID: "279e09ef-b780-457c-a0b0-66948ee0f699"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:22:14.441736 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:14.441712    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/279e09ef-b780-457c-a0b0-66948ee0f699-kube-api-access-x7tqn" pod "279e09ef-b780-457c-a0b0-66948ee0f699" (UID: "279e09ef-b780-457c-a0b0-66948ee0f699"). InnerVolumeSpecName "kube-api-access-x7tqn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:22:14.442042 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:14.442021    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/279e09ef-b780-457c-a0b0-66948ee0f699-istio-token" pod "279e09ef-b780-457c-a0b0-66948ee0f699" (UID: "279e09ef-b780-457c-a0b0-66948ee0f699"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:22:14.442102 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:14.442055    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/279e09ef-b780-457c-a0b0-66948ee0f699-istio-podinfo" pod "279e09ef-b780-457c-a0b0-66948ee0f699" (UID: "279e09ef-b780-457c-a0b0-66948ee0f699"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 13:22:14.542196 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:14.542133    4559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/279e09ef-b780-457c-a0b0-66948ee0f699-istio-token\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:22:14.542196 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:14.542158    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x7tqn\" (UniqueName: \"kubernetes.io/projected/279e09ef-b780-457c-a0b0-66948ee0f699-kube-api-access-x7tqn\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:22:14.542196 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:14.542167    4559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/279e09ef-b780-457c-a0b0-66948ee0f699-istio-podinfo\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:22:14.542196 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:14.542177    4559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/279e09ef-b780-457c-a0b0-66948ee0f699-istio-envoy\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:22:14.893549 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:14.893448    4559 generic.go:356] "Generic (PLEG): container finished" podID="185fcba5-401f-4a43-bad1-f82f845221b8" containerID="190aeadec88fade680508bb06145e8268d599a5ce8bcfb0c305de74114cb52ff" exitCode=0
Nov 02 13:22:14.893811 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:14.893553    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-bmcxg"
Nov 02 13:22:14.893811 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:14.893705    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-bmcxg" event={"ID":"185fcba5-401f-4a43-bad1-f82f845221b8","Type":"ContainerDied","Data":"190aeadec88fade680508bb06145e8268d599a5ce8bcfb0c305de74114cb52ff"}
Nov 02 13:22:14.893811 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:14.893754    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-bmcxg" event={"ID":"185fcba5-401f-4a43-bad1-f82f845221b8","Type":"ContainerDied","Data":"f00875f9f6cdf35bf678f81f0754b90a53b9caf8c5c95855266cd52e64f71f45"}
Nov 02 13:22:14.893811 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:14.893786    4559 scope.go:122] "RemoveContainer" containerID="190aeadec88fade680508bb06145e8268d599a5ce8bcfb0c305de74114cb52ff"
Nov 02 13:22:14.895767 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:14.895740    4559 generic.go:356] "Generic (PLEG): container finished" podID="279e09ef-b780-457c-a0b0-66948ee0f699" containerID="6611f45b056f82ddcaaac585cbe7560a0cd621b022bbf602a9c07b50152dcad5" exitCode=0
Nov 02 13:22:14.895923 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:14.895902    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-q77g2" event={"ID":"279e09ef-b780-457c-a0b0-66948ee0f699","Type":"ContainerDied","Data":"6611f45b056f82ddcaaac585cbe7560a0cd621b022bbf602a9c07b50152dcad5"}
Nov 02 13:22:14.896036 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:14.895956    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-q77g2" event={"ID":"279e09ef-b780-457c-a0b0-66948ee0f699","Type":"ContainerDied","Data":"33bf23a44a39e9a239aea3f710edbc6478151d4d5d04aa88cb03a231d3651467"}
Nov 02 13:22:14.896036 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:14.896029    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-q77g2"
Nov 02 13:22:14.916524 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:14.916502    4559 scope.go:122] "RemoveContainer" containerID="190aeadec88fade680508bb06145e8268d599a5ce8bcfb0c305de74114cb52ff"
Nov 02 13:22:14.917588 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:22:14.917146    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"190aeadec88fade680508bb06145e8268d599a5ce8bcfb0c305de74114cb52ff\": container with ID starting with 190aeadec88fade680508bb06145e8268d599a5ce8bcfb0c305de74114cb52ff not found: ID does not exist" containerID="190aeadec88fade680508bb06145e8268d599a5ce8bcfb0c305de74114cb52ff"
Nov 02 13:22:14.917588 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:14.917174    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"190aeadec88fade680508bb06145e8268d599a5ce8bcfb0c305de74114cb52ff"} err="failed to get container status \"190aeadec88fade680508bb06145e8268d599a5ce8bcfb0c305de74114cb52ff\": rpc error: code = NotFound desc = could not find container \"190aeadec88fade680508bb06145e8268d599a5ce8bcfb0c305de74114cb52ff\": container with ID starting with 190aeadec88fade680508bb06145e8268d599a5ce8bcfb0c305de74114cb52ff not found: ID does not exist"
Nov 02 13:22:14.917588 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:14.917195    4559 scope.go:122] "RemoveContainer" containerID="6611f45b056f82ddcaaac585cbe7560a0cd621b022bbf602a9c07b50152dcad5"
Nov 02 13:22:14.927395 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:14.927370    4559 scope.go:122] "RemoveContainer" containerID="6611f45b056f82ddcaaac585cbe7560a0cd621b022bbf602a9c07b50152dcad5"
Nov 02 13:22:14.927670 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:22:14.927618    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6611f45b056f82ddcaaac585cbe7560a0cd621b022bbf602a9c07b50152dcad5\": container with ID starting with 6611f45b056f82ddcaaac585cbe7560a0cd621b022bbf602a9c07b50152dcad5 not found: ID does not exist" containerID="6611f45b056f82ddcaaac585cbe7560a0cd621b022bbf602a9c07b50152dcad5"
Nov 02 13:22:14.927739 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:14.927684    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6611f45b056f82ddcaaac585cbe7560a0cd621b022bbf602a9c07b50152dcad5"} err="failed to get container status \"6611f45b056f82ddcaaac585cbe7560a0cd621b022bbf602a9c07b50152dcad5\": rpc error: code = NotFound desc = could not find container \"6611f45b056f82ddcaaac585cbe7560a0cd621b022bbf602a9c07b50152dcad5\": container with ID starting with 6611f45b056f82ddcaaac585cbe7560a0cd621b022bbf602a9c07b50152dcad5 not found: ID does not exist"
Nov 02 13:22:14.966605 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:14.966578    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-bmcxg"]
Nov 02 13:22:14.980917 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:14.980894    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-bmcxg"]
Nov 02 13:22:15.016025 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:15.015981    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-q77g2"]
Nov 02 13:22:15.021888 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:15.021867    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-q77g2"]
Nov 02 13:22:15.087472 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:22:15.087438    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqcmj" containerName="extractor"
Nov 02 13:22:15.895252 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:15.895219    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="185fcba5-401f-4a43-bad1-f82f845221b8" path="/var/lib/kubelet/pods/185fcba5-401f-4a43-bad1-f82f845221b8/volumes"
Nov 02 13:22:15.895974 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:15.895955    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="279e09ef-b780-457c-a0b0-66948ee0f699" path="/var/lib/kubelet/pods/279e09ef-b780-457c-a0b0-66948ee0f699/volumes"
Nov 02 13:22:16.056855 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:16.056821    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-48d74e-57f8487987-np4k8"]
Nov 02 13:22:16.091814 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:16.091781    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-48d74e-77f759984c-j6jsk"]
Nov 02 13:22:18.029795 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:18.029758    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5f8899-59548dd96c-5p585" podUID="da6dd2b6-1e1c-4128-a988-cf86451095e1" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 13:22:21.058124 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:21.058083    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-48d74e-57f8487987-np4k8" podUID="4f71b7db-e5b6-4fb6-b489-ea87e96450a7" containerName="payload-processing" containerID="cri-o://59932a3d203e08813bfa9712eb627a678d24f71a13a0a4db874e73375caf8b24" gracePeriod=25
Nov 02 13:22:21.092525 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:21.092484    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-48d74e-77f759984c-j6jsk" podUID="e74c7087-2b2c-44c5-b480-8d692b91d134" containerName="payload-pre-processing" containerID="cri-o://2303c6e317d118f062da5202a64703c41c14837340f2f762686ee91c2bda8580" gracePeriod=25
Nov 02 13:22:25.516919 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:25.516886    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5f8899-59548dd96c-5p585"
Nov 02 13:22:25.608226 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:25.608193    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da6dd2b6-1e1c-4128-a988-cf86451095e1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da6dd2b6-1e1c-4128-a988-cf86451095e1-plugins-config-volume\") pod \"da6dd2b6-1e1c-4128-a988-cf86451095e1\" (UID: \"da6dd2b6-1e1c-4128-a988-cf86451095e1\") "
Nov 02 13:22:25.608410 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:25.608395    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da6dd2b6-1e1c-4128-a988-cf86451095e1-kube-api-access-s8qkp\" (UniqueName: \"kubernetes.io/projected/da6dd2b6-1e1c-4128-a988-cf86451095e1-kube-api-access-s8qkp\") pod \"da6dd2b6-1e1c-4128-a988-cf86451095e1\" (UID: \"da6dd2b6-1e1c-4128-a988-cf86451095e1\") "
Nov 02 13:22:25.608688 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:25.608656    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da6dd2b6-1e1c-4128-a988-cf86451095e1-plugins-config-volume" pod "da6dd2b6-1e1c-4128-a988-cf86451095e1" (UID: "da6dd2b6-1e1c-4128-a988-cf86451095e1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:22:25.609136 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:25.609113    4559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da6dd2b6-1e1c-4128-a988-cf86451095e1-plugins-config-volume\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:22:25.610343 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:25.610318    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da6dd2b6-1e1c-4128-a988-cf86451095e1-kube-api-access-s8qkp" pod "da6dd2b6-1e1c-4128-a988-cf86451095e1" (UID: "da6dd2b6-1e1c-4128-a988-cf86451095e1"). InnerVolumeSpecName "kube-api-access-s8qkp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:22:25.710683 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:25.710653    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s8qkp\" (UniqueName: \"kubernetes.io/projected/da6dd2b6-1e1c-4128-a988-cf86451095e1-kube-api-access-s8qkp\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:22:26.029187 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:26.029109    4559 generic.go:356] "Generic (PLEG): container finished" podID="da6dd2b6-1e1c-4128-a988-cf86451095e1" containerID="31e8306d86f94bf7ad0f16e6dca4a5bc07d2df8c306d0ec8455f4152832b2d61" exitCode=0
Nov 02 13:22:26.029187 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:26.029167    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5f8899-59548dd96c-5p585"
Nov 02 13:22:26.029409 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:26.029185    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5f8899-59548dd96c-5p585" event={"ID":"da6dd2b6-1e1c-4128-a988-cf86451095e1","Type":"ContainerDied","Data":"31e8306d86f94bf7ad0f16e6dca4a5bc07d2df8c306d0ec8455f4152832b2d61"}
Nov 02 13:22:26.029409 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:26.029233    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5f8899-59548dd96c-5p585" event={"ID":"da6dd2b6-1e1c-4128-a988-cf86451095e1","Type":"ContainerDied","Data":"085fffe8b6ded69c4b42668a1f0862be37cb6c2efbc60b24eacdec6b261b5864"}
Nov 02 13:22:26.029409 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:26.029255    4559 scope.go:122] "RemoveContainer" containerID="31e8306d86f94bf7ad0f16e6dca4a5bc07d2df8c306d0ec8455f4152832b2d61"
Nov 02 13:22:26.040892 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:26.040873    4559 scope.go:122] "RemoveContainer" containerID="31e8306d86f94bf7ad0f16e6dca4a5bc07d2df8c306d0ec8455f4152832b2d61"
Nov 02 13:22:26.041148 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:22:26.041122    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31e8306d86f94bf7ad0f16e6dca4a5bc07d2df8c306d0ec8455f4152832b2d61\": container with ID starting with 31e8306d86f94bf7ad0f16e6dca4a5bc07d2df8c306d0ec8455f4152832b2d61 not found: ID does not exist" containerID="31e8306d86f94bf7ad0f16e6dca4a5bc07d2df8c306d0ec8455f4152832b2d61"
Nov 02 13:22:26.041209 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:26.041151    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31e8306d86f94bf7ad0f16e6dca4a5bc07d2df8c306d0ec8455f4152832b2d61"} err="failed to get container status \"31e8306d86f94bf7ad0f16e6dca4a5bc07d2df8c306d0ec8455f4152832b2d61\": rpc error: code = NotFound desc = could not find container \"31e8306d86f94bf7ad0f16e6dca4a5bc07d2df8c306d0ec8455f4152832b2d61\": container with ID starting with 31e8306d86f94bf7ad0f16e6dca4a5bc07d2df8c306d0ec8455f4152832b2d61 not found: ID does not exist"
Nov 02 13:22:26.085037 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:26.085009    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-5f8899-59548dd96c-5p585"]
Nov 02 13:22:26.096749 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:26.096726    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-5f8899-59548dd96c-5p585"]
Nov 02 13:22:27.895907 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:27.895876    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da6dd2b6-1e1c-4128-a988-cf86451095e1" path="/var/lib/kubelet/pods/da6dd2b6-1e1c-4128-a988-cf86451095e1/volumes"
Nov 02 13:22:30.136870 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:30.136829    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-48d74e-77f759984c-j6jsk" podUID="e74c7087-2b2c-44c5-b480-8d692b91d134" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 13:22:30.140444 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:30.140412    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-48d74e-57f8487987-np4k8" podUID="4f71b7db-e5b6-4fb6-b489-ea87e96450a7" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 13:22:32.266411 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:32.266368    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-j7rnm"]
Nov 02 13:22:32.266969 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:32.266673    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-j7rnm" podUID="85ec8b4f-746f-4d0c-9560-7a79f0c64ef6" containerName="istio-proxy" containerID="cri-o://8e8db6f9ffc5e4ded648e00007ee1952e10b6529f23b4af6eac3188ba99d9c03" gracePeriod=30
Nov 02 13:22:32.281319 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:32.281288    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-m29cl"]
Nov 02 13:22:32.281550 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:32.281511    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-m29cl" podUID="f7a083ea-c6dc-4402-aac2-7842d1578ed2" containerName="istio-proxy" containerID="cri-o://018bc4559c2ded382f5582913c877cf8e38136d64ef2fb188ea95c4ddf51202f" gracePeriod=30
Nov 02 13:22:32.340663 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:32.333218    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-2ssw6"]
Nov 02 13:22:32.340663 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:32.333461    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-2ssw6" podUID="d60e24f5-ef82-485d-a867-b3e3718b0e65" containerName="istio-proxy" containerID="cri-o://33623bd9e9155f0968b2aabbe020119796f0dff7fc0d14c533ee50f6599ec6f6" gracePeriod=30
Nov 02 13:22:37.495193 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:37.495163    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-m29cl"
Nov 02 13:22:37.582277 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:37.582197    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f7a083ea-c6dc-4402-aac2-7842d1578ed2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7a083ea-c6dc-4402-aac2-7842d1578ed2-istio-podinfo\") pod \"f7a083ea-c6dc-4402-aac2-7842d1578ed2\" (UID: \"f7a083ea-c6dc-4402-aac2-7842d1578ed2\") "
Nov 02 13:22:37.582277 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:37.582239    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7a083ea-c6dc-4402-aac2-7842d1578ed2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7a083ea-c6dc-4402-aac2-7842d1578ed2-istio-data\") pod \"f7a083ea-c6dc-4402-aac2-7842d1578ed2\" (UID: \"f7a083ea-c6dc-4402-aac2-7842d1578ed2\") "
Nov 02 13:22:37.582492 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:37.582286    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7a083ea-c6dc-4402-aac2-7842d1578ed2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7a083ea-c6dc-4402-aac2-7842d1578ed2-credential-socket\") pod \"f7a083ea-c6dc-4402-aac2-7842d1578ed2\" (UID: \"f7a083ea-c6dc-4402-aac2-7842d1578ed2\") "
Nov 02 13:22:37.582492 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:37.582348    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7a083ea-c6dc-4402-aac2-7842d1578ed2-kube-api-access-qqv7t\" (UniqueName: \"kubernetes.io/projected/f7a083ea-c6dc-4402-aac2-7842d1578ed2-kube-api-access-qqv7t\") pod \"f7a083ea-c6dc-4402-aac2-7842d1578ed2\" (UID: \"f7a083ea-c6dc-4402-aac2-7842d1578ed2\") "
Nov 02 13:22:37.583142 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:37.582628    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7a083ea-c6dc-4402-aac2-7842d1578ed2-istio-data" pod "f7a083ea-c6dc-4402-aac2-7842d1578ed2" (UID: "f7a083ea-c6dc-4402-aac2-7842d1578ed2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:22:37.583142 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:37.582729    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7a083ea-c6dc-4402-aac2-7842d1578ed2-credential-socket" pod "f7a083ea-c6dc-4402-aac2-7842d1578ed2" (UID: "f7a083ea-c6dc-4402-aac2-7842d1578ed2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:22:37.583142 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:37.582830    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7a083ea-c6dc-4402-aac2-7842d1578ed2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7a083ea-c6dc-4402-aac2-7842d1578ed2-workload-certs\") pod \"f7a083ea-c6dc-4402-aac2-7842d1578ed2\" (UID: \"f7a083ea-c6dc-4402-aac2-7842d1578ed2\") "
Nov 02 13:22:37.583142 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:37.583006    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7a083ea-c6dc-4402-aac2-7842d1578ed2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7a083ea-c6dc-4402-aac2-7842d1578ed2-istiod-ca-cert\") pod \"f7a083ea-c6dc-4402-aac2-7842d1578ed2\" (UID: \"f7a083ea-c6dc-4402-aac2-7842d1578ed2\") "
Nov 02 13:22:37.583142 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:37.583030    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7a083ea-c6dc-4402-aac2-7842d1578ed2-workload-certs" pod "f7a083ea-c6dc-4402-aac2-7842d1578ed2" (UID: "f7a083ea-c6dc-4402-aac2-7842d1578ed2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:22:37.583142 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:37.583037    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7a083ea-c6dc-4402-aac2-7842d1578ed2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7a083ea-c6dc-4402-aac2-7842d1578ed2-workload-socket\") pod \"f7a083ea-c6dc-4402-aac2-7842d1578ed2\" (UID: \"f7a083ea-c6dc-4402-aac2-7842d1578ed2\") "
Nov 02 13:22:37.583142 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:37.583091    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7a083ea-c6dc-4402-aac2-7842d1578ed2-istio-token\" (UniqueName: \"kubernetes.io/projected/f7a083ea-c6dc-4402-aac2-7842d1578ed2-istio-token\") pod \"f7a083ea-c6dc-4402-aac2-7842d1578ed2\" (UID: \"f7a083ea-c6dc-4402-aac2-7842d1578ed2\") "
Nov 02 13:22:37.583661 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:37.583149    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7a083ea-c6dc-4402-aac2-7842d1578ed2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7a083ea-c6dc-4402-aac2-7842d1578ed2-istio-envoy\") pod \"f7a083ea-c6dc-4402-aac2-7842d1578ed2\" (UID: \"f7a083ea-c6dc-4402-aac2-7842d1578ed2\") "
Nov 02 13:22:37.583661 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:37.583303    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7a083ea-c6dc-4402-aac2-7842d1578ed2-istiod-ca-cert" pod "f7a083ea-c6dc-4402-aac2-7842d1578ed2" (UID: "f7a083ea-c6dc-4402-aac2-7842d1578ed2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:22:37.583661 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:37.583519    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7a083ea-c6dc-4402-aac2-7842d1578ed2-workload-socket" pod "f7a083ea-c6dc-4402-aac2-7842d1578ed2" (UID: "f7a083ea-c6dc-4402-aac2-7842d1578ed2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:22:37.584020 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:37.583991    4559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7a083ea-c6dc-4402-aac2-7842d1578ed2-istiod-ca-cert\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:22:37.584125 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:37.584028    4559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7a083ea-c6dc-4402-aac2-7842d1578ed2-workload-socket\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:22:37.584125 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:37.584042    4559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7a083ea-c6dc-4402-aac2-7842d1578ed2-istio-data\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:22:37.584125 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:37.584055    4559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7a083ea-c6dc-4402-aac2-7842d1578ed2-credential-socket\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:22:37.584125 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:37.584068    4559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7a083ea-c6dc-4402-aac2-7842d1578ed2-workload-certs\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:22:37.584719 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:37.584695    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f7a083ea-c6dc-4402-aac2-7842d1578ed2-istio-podinfo" pod "f7a083ea-c6dc-4402-aac2-7842d1578ed2" (UID: "f7a083ea-c6dc-4402-aac2-7842d1578ed2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 13:22:37.585571 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:37.585549    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7a083ea-c6dc-4402-aac2-7842d1578ed2-istio-token" pod "f7a083ea-c6dc-4402-aac2-7842d1578ed2" (UID: "f7a083ea-c6dc-4402-aac2-7842d1578ed2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:22:37.585800 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:37.585779    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7a083ea-c6dc-4402-aac2-7842d1578ed2-istio-envoy" pod "f7a083ea-c6dc-4402-aac2-7842d1578ed2" (UID: "f7a083ea-c6dc-4402-aac2-7842d1578ed2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:22:37.586988 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:37.586966    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7a083ea-c6dc-4402-aac2-7842d1578ed2-kube-api-access-qqv7t" pod "f7a083ea-c6dc-4402-aac2-7842d1578ed2" (UID: "f7a083ea-c6dc-4402-aac2-7842d1578ed2"). InnerVolumeSpecName "kube-api-access-qqv7t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:22:37.685569 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:37.685534    4559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7a083ea-c6dc-4402-aac2-7842d1578ed2-istio-token\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:22:37.685569 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:37.685565    4559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7a083ea-c6dc-4402-aac2-7842d1578ed2-istio-envoy\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:22:37.685569 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:37.685574    4559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7a083ea-c6dc-4402-aac2-7842d1578ed2-istio-podinfo\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:22:37.685828 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:37.685584    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqv7t\" (UniqueName: \"kubernetes.io/projected/f7a083ea-c6dc-4402-aac2-7842d1578ed2-kube-api-access-qqv7t\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:22:37.976789 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:37.976758    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-j7rnm"
Nov 02 13:22:38.046629 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:38.046597    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-2ssw6"
Nov 02 13:22:38.091577 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:38.091549    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d60e24f5-ef82-485d-a867-b3e3718b0e65-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d60e24f5-ef82-485d-a867-b3e3718b0e65-credential-socket\") pod \"d60e24f5-ef82-485d-a867-b3e3718b0e65\" (UID: \"d60e24f5-ef82-485d-a867-b3e3718b0e65\") "
Nov 02 13:22:38.091792 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:38.091615    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d60e24f5-ef82-485d-a867-b3e3718b0e65-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d60e24f5-ef82-485d-a867-b3e3718b0e65-istio-data\") pod \"d60e24f5-ef82-485d-a867-b3e3718b0e65\" (UID: \"d60e24f5-ef82-485d-a867-b3e3718b0e65\") "
Nov 02 13:22:38.091792 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:38.091678    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85ec8b4f-746f-4d0c-9560-7a79f0c64ef6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85ec8b4f-746f-4d0c-9560-7a79f0c64ef6-istio-data\") pod \"85ec8b4f-746f-4d0c-9560-7a79f0c64ef6\" (UID: \"85ec8b4f-746f-4d0c-9560-7a79f0c64ef6\") "
Nov 02 13:22:38.091792 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:38.091708    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85ec8b4f-746f-4d0c-9560-7a79f0c64ef6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85ec8b4f-746f-4d0c-9560-7a79f0c64ef6-credential-socket\") pod \"85ec8b4f-746f-4d0c-9560-7a79f0c64ef6\" (UID: \"85ec8b4f-746f-4d0c-9560-7a79f0c64ef6\") "
Nov 02 13:22:38.091792 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:38.091791    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85ec8b4f-746f-4d0c-9560-7a79f0c64ef6-istio-token\" (UniqueName: \"kubernetes.io/projected/85ec8b4f-746f-4d0c-9560-7a79f0c64ef6-istio-token\") pod \"85ec8b4f-746f-4d0c-9560-7a79f0c64ef6\" (UID: \"85ec8b4f-746f-4d0c-9560-7a79f0c64ef6\") "
Nov 02 13:22:38.092039 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:38.091844    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d60e24f5-ef82-485d-a867-b3e3718b0e65-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d60e24f5-ef82-485d-a867-b3e3718b0e65-istio-podinfo\") pod \"d60e24f5-ef82-485d-a867-b3e3718b0e65\" (UID: \"d60e24f5-ef82-485d-a867-b3e3718b0e65\") "
Nov 02 13:22:38.092039 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:38.091886    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d60e24f5-ef82-485d-a867-b3e3718b0e65-credential-socket" pod "d60e24f5-ef82-485d-a867-b3e3718b0e65" (UID: "d60e24f5-ef82-485d-a867-b3e3718b0e65"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:22:38.092039 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:38.091910    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d60e24f5-ef82-485d-a867-b3e3718b0e65-istio-data" pod "d60e24f5-ef82-485d-a867-b3e3718b0e65" (UID: "d60e24f5-ef82-485d-a867-b3e3718b0e65"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:22:38.092039 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:38.091952    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/85ec8b4f-746f-4d0c-9560-7a79f0c64ef6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85ec8b4f-746f-4d0c-9560-7a79f0c64ef6-istio-podinfo\") pod \"85ec8b4f-746f-4d0c-9560-7a79f0c64ef6\" (UID: \"85ec8b4f-746f-4d0c-9560-7a79f0c64ef6\") "
Nov 02 13:22:38.092039 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:38.091989    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85ec8b4f-746f-4d0c-9560-7a79f0c64ef6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85ec8b4f-746f-4d0c-9560-7a79f0c64ef6-istiod-ca-cert\") pod \"85ec8b4f-746f-4d0c-9560-7a79f0c64ef6\" (UID: \"85ec8b4f-746f-4d0c-9560-7a79f0c64ef6\") "
Nov 02 13:22:38.092039 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:38.092003    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85ec8b4f-746f-4d0c-9560-7a79f0c64ef6-istio-data" pod "85ec8b4f-746f-4d0c-9560-7a79f0c64ef6" (UID: "85ec8b4f-746f-4d0c-9560-7a79f0c64ef6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:22:38.092039 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:38.092010    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d60e24f5-ef82-485d-a867-b3e3718b0e65-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d60e24f5-ef82-485d-a867-b3e3718b0e65-istiod-ca-cert\") pod \"d60e24f5-ef82-485d-a867-b3e3718b0e65\" (UID: \"d60e24f5-ef82-485d-a867-b3e3718b0e65\") "
Nov 02 13:22:38.092388 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:38.092078    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d60e24f5-ef82-485d-a867-b3e3718b0e65-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d60e24f5-ef82-485d-a867-b3e3718b0e65-istio-envoy\") pod \"d60e24f5-ef82-485d-a867-b3e3718b0e65\" (UID: \"d60e24f5-ef82-485d-a867-b3e3718b0e65\") "
Nov 02 13:22:38.092388 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:38.092113    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85ec8b4f-746f-4d0c-9560-7a79f0c64ef6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85ec8b4f-746f-4d0c-9560-7a79f0c64ef6-workload-socket\") pod \"85ec8b4f-746f-4d0c-9560-7a79f0c64ef6\" (UID: \"85ec8b4f-746f-4d0c-9560-7a79f0c64ef6\") "
Nov 02 13:22:38.092388 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:38.092196    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d60e24f5-ef82-485d-a867-b3e3718b0e65-kube-api-access-jntwf\" (UniqueName: \"kubernetes.io/projected/d60e24f5-ef82-485d-a867-b3e3718b0e65-kube-api-access-jntwf\") pod \"d60e24f5-ef82-485d-a867-b3e3718b0e65\" (UID: \"d60e24f5-ef82-485d-a867-b3e3718b0e65\") "
Nov 02 13:22:38.092388 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:38.092233    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d60e24f5-ef82-485d-a867-b3e3718b0e65-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d60e24f5-ef82-485d-a867-b3e3718b0e65-workload-certs\") pod \"d60e24f5-ef82-485d-a867-b3e3718b0e65\" (UID: \"d60e24f5-ef82-485d-a867-b3e3718b0e65\") "
Nov 02 13:22:38.092388 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:38.092267    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d60e24f5-ef82-485d-a867-b3e3718b0e65-istio-token\" (UniqueName: \"kubernetes.io/projected/d60e24f5-ef82-485d-a867-b3e3718b0e65-istio-token\") pod \"d60e24f5-ef82-485d-a867-b3e3718b0e65\" (UID: \"d60e24f5-ef82-485d-a867-b3e3718b0e65\") "
Nov 02 13:22:38.092388 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:38.092310    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85ec8b4f-746f-4d0c-9560-7a79f0c64ef6-kube-api-access-fpwfm\" (UniqueName: \"kubernetes.io/projected/85ec8b4f-746f-4d0c-9560-7a79f0c64ef6-kube-api-access-fpwfm\") pod \"85ec8b4f-746f-4d0c-9560-7a79f0c64ef6\" (UID: \"85ec8b4f-746f-4d0c-9560-7a79f0c64ef6\") "
Nov 02 13:22:38.092388 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:38.092359    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d60e24f5-ef82-485d-a867-b3e3718b0e65-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d60e24f5-ef82-485d-a867-b3e3718b0e65-workload-socket\") pod \"d60e24f5-ef82-485d-a867-b3e3718b0e65\" (UID: \"d60e24f5-ef82-485d-a867-b3e3718b0e65\") "
Nov 02 13:22:38.093102 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:38.092403    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85ec8b4f-746f-4d0c-9560-7a79f0c64ef6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85ec8b4f-746f-4d0c-9560-7a79f0c64ef6-workload-certs\") pod \"85ec8b4f-746f-4d0c-9560-7a79f0c64ef6\" (UID: \"85ec8b4f-746f-4d0c-9560-7a79f0c64ef6\") "
Nov 02 13:22:38.093102 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:38.092440    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85ec8b4f-746f-4d0c-9560-7a79f0c64ef6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85ec8b4f-746f-4d0c-9560-7a79f0c64ef6-istio-envoy\") pod \"85ec8b4f-746f-4d0c-9560-7a79f0c64ef6\" (UID: \"85ec8b4f-746f-4d0c-9560-7a79f0c64ef6\") "
Nov 02 13:22:38.093102 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:38.092435    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d60e24f5-ef82-485d-a867-b3e3718b0e65-istiod-ca-cert" pod "d60e24f5-ef82-485d-a867-b3e3718b0e65" (UID: "d60e24f5-ef82-485d-a867-b3e3718b0e65"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:22:38.093102 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:38.092483    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85ec8b4f-746f-4d0c-9560-7a79f0c64ef6-credential-socket" pod "85ec8b4f-746f-4d0c-9560-7a79f0c64ef6" (UID: "85ec8b4f-746f-4d0c-9560-7a79f0c64ef6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:22:38.093518 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:38.093325    4559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d60e24f5-ef82-485d-a867-b3e3718b0e65-credential-socket\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:22:38.093518 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:38.093362    4559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d60e24f5-ef82-485d-a867-b3e3718b0e65-istio-data\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:22:38.093518 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:38.093376    4559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85ec8b4f-746f-4d0c-9560-7a79f0c64ef6-istio-data\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:22:38.093518 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:38.093390    4559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85ec8b4f-746f-4d0c-9560-7a79f0c64ef6-credential-socket\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:22:38.093518 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:38.093403    4559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d60e24f5-ef82-485d-a867-b3e3718b0e65-istiod-ca-cert\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:22:38.094018 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:38.093738    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85ec8b4f-746f-4d0c-9560-7a79f0c64ef6-istiod-ca-cert" pod "85ec8b4f-746f-4d0c-9560-7a79f0c64ef6" (UID: "85ec8b4f-746f-4d0c-9560-7a79f0c64ef6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:22:38.094018 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:38.094006    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d60e24f5-ef82-485d-a867-b3e3718b0e65-workload-socket" pod "d60e24f5-ef82-485d-a867-b3e3718b0e65" (UID: "d60e24f5-ef82-485d-a867-b3e3718b0e65"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:22:38.094191 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:38.094124    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85ec8b4f-746f-4d0c-9560-7a79f0c64ef6-workload-certs" pod "85ec8b4f-746f-4d0c-9560-7a79f0c64ef6" (UID: "85ec8b4f-746f-4d0c-9560-7a79f0c64ef6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:22:38.094276 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:38.094224    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d60e24f5-ef82-485d-a867-b3e3718b0e65-workload-certs" pod "d60e24f5-ef82-485d-a867-b3e3718b0e65" (UID: "d60e24f5-ef82-485d-a867-b3e3718b0e65"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:22:38.094365 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:38.094337    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85ec8b4f-746f-4d0c-9560-7a79f0c64ef6-workload-socket" pod "85ec8b4f-746f-4d0c-9560-7a79f0c64ef6" (UID: "85ec8b4f-746f-4d0c-9560-7a79f0c64ef6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:22:38.096561 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:38.096527    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d60e24f5-ef82-485d-a867-b3e3718b0e65-istio-token" pod "d60e24f5-ef82-485d-a867-b3e3718b0e65" (UID: "d60e24f5-ef82-485d-a867-b3e3718b0e65"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:22:38.096702 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:38.096687    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85ec8b4f-746f-4d0c-9560-7a79f0c64ef6-istio-envoy" pod "85ec8b4f-746f-4d0c-9560-7a79f0c64ef6" (UID: "85ec8b4f-746f-4d0c-9560-7a79f0c64ef6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:22:38.096921 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:38.096793    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d60e24f5-ef82-485d-a867-b3e3718b0e65-istio-podinfo" pod "d60e24f5-ef82-485d-a867-b3e3718b0e65" (UID: "d60e24f5-ef82-485d-a867-b3e3718b0e65"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 13:22:38.097185 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:38.097124    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d60e24f5-ef82-485d-a867-b3e3718b0e65-kube-api-access-jntwf" pod "d60e24f5-ef82-485d-a867-b3e3718b0e65" (UID: "d60e24f5-ef82-485d-a867-b3e3718b0e65"). InnerVolumeSpecName "kube-api-access-jntwf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:22:38.097543 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:38.097514    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d60e24f5-ef82-485d-a867-b3e3718b0e65-istio-envoy" pod "d60e24f5-ef82-485d-a867-b3e3718b0e65" (UID: "d60e24f5-ef82-485d-a867-b3e3718b0e65"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:22:38.098240 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:38.098205    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85ec8b4f-746f-4d0c-9560-7a79f0c64ef6-istio-token" pod "85ec8b4f-746f-4d0c-9560-7a79f0c64ef6" (UID: "85ec8b4f-746f-4d0c-9560-7a79f0c64ef6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:22:38.098566 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:38.098542    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85ec8b4f-746f-4d0c-9560-7a79f0c64ef6-kube-api-access-fpwfm" pod "85ec8b4f-746f-4d0c-9560-7a79f0c64ef6" (UID: "85ec8b4f-746f-4d0c-9560-7a79f0c64ef6"). InnerVolumeSpecName "kube-api-access-fpwfm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:22:38.099034 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:38.099010    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/85ec8b4f-746f-4d0c-9560-7a79f0c64ef6-istio-podinfo" pod "85ec8b4f-746f-4d0c-9560-7a79f0c64ef6" (UID: "85ec8b4f-746f-4d0c-9560-7a79f0c64ef6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 13:22:38.218313 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:38.206304    4559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85ec8b4f-746f-4d0c-9560-7a79f0c64ef6-istio-podinfo\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:22:38.218313 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:38.206342    4559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85ec8b4f-746f-4d0c-9560-7a79f0c64ef6-istiod-ca-cert\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:22:38.218313 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:38.206361    4559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d60e24f5-ef82-485d-a867-b3e3718b0e65-istio-envoy\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:22:38.218313 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:38.206378    4559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85ec8b4f-746f-4d0c-9560-7a79f0c64ef6-workload-socket\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:22:38.218313 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:38.206394    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jntwf\" (UniqueName: \"kubernetes.io/projected/d60e24f5-ef82-485d-a867-b3e3718b0e65-kube-api-access-jntwf\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:22:38.218313 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:38.206411    4559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d60e24f5-ef82-485d-a867-b3e3718b0e65-workload-certs\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:22:38.218313 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:38.206426    4559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d60e24f5-ef82-485d-a867-b3e3718b0e65-istio-token\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:22:38.218313 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:38.206441    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fpwfm\" (UniqueName: \"kubernetes.io/projected/85ec8b4f-746f-4d0c-9560-7a79f0c64ef6-kube-api-access-fpwfm\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:22:38.218313 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:38.206457    4559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d60e24f5-ef82-485d-a867-b3e3718b0e65-workload-socket\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:22:38.218313 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:38.206473    4559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85ec8b4f-746f-4d0c-9560-7a79f0c64ef6-workload-certs\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:22:38.218313 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:38.206487    4559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85ec8b4f-746f-4d0c-9560-7a79f0c64ef6-istio-envoy\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:22:38.218313 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:38.206501    4559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85ec8b4f-746f-4d0c-9560-7a79f0c64ef6-istio-token\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:22:38.218313 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:38.206514    4559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d60e24f5-ef82-485d-a867-b3e3718b0e65-istio-podinfo\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:22:38.264493 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:38.264465    4559 generic.go:356] "Generic (PLEG): container finished" podID="85ec8b4f-746f-4d0c-9560-7a79f0c64ef6" containerID="8e8db6f9ffc5e4ded648e00007ee1952e10b6529f23b4af6eac3188ba99d9c03" exitCode=0
Nov 02 13:22:38.264598 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:38.264493    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-j7rnm" event={"ID":"85ec8b4f-746f-4d0c-9560-7a79f0c64ef6","Type":"ContainerDied","Data":"8e8db6f9ffc5e4ded648e00007ee1952e10b6529f23b4af6eac3188ba99d9c03"}
Nov 02 13:22:38.264598 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:38.264519    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-j7rnm"
Nov 02 13:22:38.264598 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:38.264546    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-j7rnm" event={"ID":"85ec8b4f-746f-4d0c-9560-7a79f0c64ef6","Type":"ContainerDied","Data":"161f787b1c7a84e4c93d97b63575af60d73efbf09faf5293b33884c10646b849"}
Nov 02 13:22:38.264598 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:38.264575    4559 scope.go:122] "RemoveContainer" containerID="8e8db6f9ffc5e4ded648e00007ee1952e10b6529f23b4af6eac3188ba99d9c03"
Nov 02 13:22:38.266323 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:38.266297    4559 generic.go:356] "Generic (PLEG): container finished" podID="f7a083ea-c6dc-4402-aac2-7842d1578ed2" containerID="018bc4559c2ded382f5582913c877cf8e38136d64ef2fb188ea95c4ddf51202f" exitCode=0
Nov 02 13:22:38.266415 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:38.266348    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-m29cl"
Nov 02 13:22:38.266415 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:38.266370    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-m29cl" event={"ID":"f7a083ea-c6dc-4402-aac2-7842d1578ed2","Type":"ContainerDied","Data":"018bc4559c2ded382f5582913c877cf8e38136d64ef2fb188ea95c4ddf51202f"}
Nov 02 13:22:38.266516 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:38.266426    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-m29cl" event={"ID":"f7a083ea-c6dc-4402-aac2-7842d1578ed2","Type":"ContainerDied","Data":"2ce9546c095d9b10fa79d80a6e0d3b641a3caece4bf3b60261395f43f3c41161"}
Nov 02 13:22:38.268029 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:38.267997    4559 generic.go:356] "Generic (PLEG): container finished" podID="d60e24f5-ef82-485d-a867-b3e3718b0e65" containerID="33623bd9e9155f0968b2aabbe020119796f0dff7fc0d14c533ee50f6599ec6f6" exitCode=0
Nov 02 13:22:38.268147 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:38.268041    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-2ssw6"
Nov 02 13:22:38.268147 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:38.268062    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-2ssw6" event={"ID":"d60e24f5-ef82-485d-a867-b3e3718b0e65","Type":"ContainerDied","Data":"33623bd9e9155f0968b2aabbe020119796f0dff7fc0d14c533ee50f6599ec6f6"}
Nov 02 13:22:38.268147 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:38.268094    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-2ssw6" event={"ID":"d60e24f5-ef82-485d-a867-b3e3718b0e65","Type":"ContainerDied","Data":"9410bce39bc8ae8e26b4a926b0e33e9944cfb6caf5ed04ae7eaed264f8442a99"}
Nov 02 13:22:38.276530 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:38.276513    4559 scope.go:122] "RemoveContainer" containerID="07fc6ecc53d115dd90560bf90a6b96e92920642b83d1fb307440c71e204217de"
Nov 02 13:22:38.310978 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:38.310959    4559 scope.go:122] "RemoveContainer" containerID="8e8db6f9ffc5e4ded648e00007ee1952e10b6529f23b4af6eac3188ba99d9c03"
Nov 02 13:22:38.311297 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:22:38.311270    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e8db6f9ffc5e4ded648e00007ee1952e10b6529f23b4af6eac3188ba99d9c03\": container with ID starting with 8e8db6f9ffc5e4ded648e00007ee1952e10b6529f23b4af6eac3188ba99d9c03 not found: ID does not exist" containerID="8e8db6f9ffc5e4ded648e00007ee1952e10b6529f23b4af6eac3188ba99d9c03"
Nov 02 13:22:38.311407 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:38.311302    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e8db6f9ffc5e4ded648e00007ee1952e10b6529f23b4af6eac3188ba99d9c03"} err="failed to get container status \"8e8db6f9ffc5e4ded648e00007ee1952e10b6529f23b4af6eac3188ba99d9c03\": rpc error: code = NotFound desc = could not find container \"8e8db6f9ffc5e4ded648e00007ee1952e10b6529f23b4af6eac3188ba99d9c03\": container with ID starting with 8e8db6f9ffc5e4ded648e00007ee1952e10b6529f23b4af6eac3188ba99d9c03 not found: ID does not exist"
Nov 02 13:22:38.311407 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:38.311319    4559 scope.go:122] "RemoveContainer" containerID="07fc6ecc53d115dd90560bf90a6b96e92920642b83d1fb307440c71e204217de"
Nov 02 13:22:38.311561 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:22:38.311536    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07fc6ecc53d115dd90560bf90a6b96e92920642b83d1fb307440c71e204217de\": container with ID starting with 07fc6ecc53d115dd90560bf90a6b96e92920642b83d1fb307440c71e204217de not found: ID does not exist" containerID="07fc6ecc53d115dd90560bf90a6b96e92920642b83d1fb307440c71e204217de"
Nov 02 13:22:38.311650 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:38.311573    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07fc6ecc53d115dd90560bf90a6b96e92920642b83d1fb307440c71e204217de"} err="failed to get container status \"07fc6ecc53d115dd90560bf90a6b96e92920642b83d1fb307440c71e204217de\": rpc error: code = NotFound desc = could not find container \"07fc6ecc53d115dd90560bf90a6b96e92920642b83d1fb307440c71e204217de\": container with ID starting with 07fc6ecc53d115dd90560bf90a6b96e92920642b83d1fb307440c71e204217de not found: ID does not exist"
Nov 02 13:22:38.311650 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:38.311596    4559 scope.go:122] "RemoveContainer" containerID="018bc4559c2ded382f5582913c877cf8e38136d64ef2fb188ea95c4ddf51202f"
Nov 02 13:22:38.321491 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:38.321469    4559 scope.go:122] "RemoveContainer" containerID="018bc4559c2ded382f5582913c877cf8e38136d64ef2fb188ea95c4ddf51202f"
Nov 02 13:22:38.321759 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:22:38.321739    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"018bc4559c2ded382f5582913c877cf8e38136d64ef2fb188ea95c4ddf51202f\": container with ID starting with 018bc4559c2ded382f5582913c877cf8e38136d64ef2fb188ea95c4ddf51202f not found: ID does not exist" containerID="018bc4559c2ded382f5582913c877cf8e38136d64ef2fb188ea95c4ddf51202f"
Nov 02 13:22:38.321814 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:38.321765    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"018bc4559c2ded382f5582913c877cf8e38136d64ef2fb188ea95c4ddf51202f"} err="failed to get container status \"018bc4559c2ded382f5582913c877cf8e38136d64ef2fb188ea95c4ddf51202f\": rpc error: code = NotFound desc = could not find container \"018bc4559c2ded382f5582913c877cf8e38136d64ef2fb188ea95c4ddf51202f\": container with ID starting with 018bc4559c2ded382f5582913c877cf8e38136d64ef2fb188ea95c4ddf51202f not found: ID does not exist"
Nov 02 13:22:38.321814 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:38.321781    4559 scope.go:122] "RemoveContainer" containerID="33623bd9e9155f0968b2aabbe020119796f0dff7fc0d14c533ee50f6599ec6f6"
Nov 02 13:22:38.331006 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:38.330987    4559 scope.go:122] "RemoveContainer" containerID="33623bd9e9155f0968b2aabbe020119796f0dff7fc0d14c533ee50f6599ec6f6"
Nov 02 13:22:38.331266 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:22:38.331249    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33623bd9e9155f0968b2aabbe020119796f0dff7fc0d14c533ee50f6599ec6f6\": container with ID starting with 33623bd9e9155f0968b2aabbe020119796f0dff7fc0d14c533ee50f6599ec6f6 not found: ID does not exist" containerID="33623bd9e9155f0968b2aabbe020119796f0dff7fc0d14c533ee50f6599ec6f6"
Nov 02 13:22:38.331331 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:38.331271    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33623bd9e9155f0968b2aabbe020119796f0dff7fc0d14c533ee50f6599ec6f6"} err="failed to get container status \"33623bd9e9155f0968b2aabbe020119796f0dff7fc0d14c533ee50f6599ec6f6\": rpc error: code = NotFound desc = could not find container \"33623bd9e9155f0968b2aabbe020119796f0dff7fc0d14c533ee50f6599ec6f6\": container with ID starting with 33623bd9e9155f0968b2aabbe020119796f0dff7fc0d14c533ee50f6599ec6f6 not found: ID does not exist"
Nov 02 13:22:38.336129 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:38.336106    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-j7rnm"]
Nov 02 13:22:38.340411 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:38.340390    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-j7rnm"]
Nov 02 13:22:38.376406 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:38.376328    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-m29cl"]
Nov 02 13:22:38.381994 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:38.381971    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-m29cl"]
Nov 02 13:22:38.415903 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:38.415880    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-2ssw6"]
Nov 02 13:22:38.419975 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:38.419951    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-2ssw6"]
Nov 02 13:22:39.895525 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:39.895491    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85ec8b4f-746f-4d0c-9560-7a79f0c64ef6" path="/var/lib/kubelet/pods/85ec8b4f-746f-4d0c-9560-7a79f0c64ef6/volumes"
Nov 02 13:22:39.896079 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:39.896063    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d60e24f5-ef82-485d-a867-b3e3718b0e65" path="/var/lib/kubelet/pods/d60e24f5-ef82-485d-a867-b3e3718b0e65/volumes"
Nov 02 13:22:39.896523 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:39.896510    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7a083ea-c6dc-4402-aac2-7842d1578ed2" path="/var/lib/kubelet/pods/f7a083ea-c6dc-4402-aac2-7842d1578ed2/volumes"
Nov 02 13:22:40.111203 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:40.111166    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-48d74e-77f759984c-j6jsk" podUID="e74c7087-2b2c-44c5-b480-8d692b91d134" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 13:22:40.111373 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:40.111174    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-48d74e-57f8487987-np4k8" podUID="4f71b7db-e5b6-4fb6-b489-ea87e96450a7" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 13:22:41.316325 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:41.316232    4559 generic.go:356] "Generic (PLEG): container finished" podID="4f71b7db-e5b6-4fb6-b489-ea87e96450a7" containerID="59932a3d203e08813bfa9712eb627a678d24f71a13a0a4db874e73375caf8b24" exitCode=0
Nov 02 13:22:41.316799 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:41.316343    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-48d74e-57f8487987-np4k8" event={"ID":"4f71b7db-e5b6-4fb6-b489-ea87e96450a7","Type":"ContainerDied","Data":"59932a3d203e08813bfa9712eb627a678d24f71a13a0a4db874e73375caf8b24"}
Nov 02 13:22:41.318409 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:41.318376    4559 generic.go:356] "Generic (PLEG): container finished" podID="e74c7087-2b2c-44c5-b480-8d692b91d134" containerID="2303c6e317d118f062da5202a64703c41c14837340f2f762686ee91c2bda8580" exitCode=0
Nov 02 13:22:41.318522 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:41.318470    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-48d74e-77f759984c-j6jsk" event={"ID":"e74c7087-2b2c-44c5-b480-8d692b91d134","Type":"ContainerDied","Data":"2303c6e317d118f062da5202a64703c41c14837340f2f762686ee91c2bda8580"}
Nov 02 13:22:41.374549 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:41.374500    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-48d74e-57f8487987-np4k8"
Nov 02 13:22:41.430633 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:41.430606    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-48d74e-77f759984c-j6jsk"
Nov 02 13:22:41.466622 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:41.466574    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f71b7db-e5b6-4fb6-b489-ea87e96450a7-kube-api-access-zcnsv\" (UniqueName: \"kubernetes.io/projected/4f71b7db-e5b6-4fb6-b489-ea87e96450a7-kube-api-access-zcnsv\") pod \"4f71b7db-e5b6-4fb6-b489-ea87e96450a7\" (UID: \"4f71b7db-e5b6-4fb6-b489-ea87e96450a7\") "
Nov 02 13:22:41.466778 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:41.466708    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f71b7db-e5b6-4fb6-b489-ea87e96450a7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f71b7db-e5b6-4fb6-b489-ea87e96450a7-plugins-config-volume\") pod \"4f71b7db-e5b6-4fb6-b489-ea87e96450a7\" (UID: \"4f71b7db-e5b6-4fb6-b489-ea87e96450a7\") "
Nov 02 13:22:41.528812 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:41.528766    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f71b7db-e5b6-4fb6-b489-ea87e96450a7-plugins-config-volume" pod "4f71b7db-e5b6-4fb6-b489-ea87e96450a7" (UID: "4f71b7db-e5b6-4fb6-b489-ea87e96450a7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:22:41.533407 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:41.533035    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f71b7db-e5b6-4fb6-b489-ea87e96450a7-kube-api-access-zcnsv" pod "4f71b7db-e5b6-4fb6-b489-ea87e96450a7" (UID: "4f71b7db-e5b6-4fb6-b489-ea87e96450a7"). InnerVolumeSpecName "kube-api-access-zcnsv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:22:41.611791 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:41.611723    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e74c7087-2b2c-44c5-b480-8d692b91d134-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e74c7087-2b2c-44c5-b480-8d692b91d134-plugins-config-volume\") pod \"e74c7087-2b2c-44c5-b480-8d692b91d134\" (UID: \"e74c7087-2b2c-44c5-b480-8d692b91d134\") "
Nov 02 13:22:41.612025 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:41.611832    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e74c7087-2b2c-44c5-b480-8d692b91d134-kube-api-access-bw6tf\" (UniqueName: \"kubernetes.io/projected/e74c7087-2b2c-44c5-b480-8d692b91d134-kube-api-access-bw6tf\") pod \"e74c7087-2b2c-44c5-b480-8d692b91d134\" (UID: \"e74c7087-2b2c-44c5-b480-8d692b91d134\") "
Nov 02 13:22:41.612141 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:41.612114    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e74c7087-2b2c-44c5-b480-8d692b91d134-plugins-config-volume" pod "e74c7087-2b2c-44c5-b480-8d692b91d134" (UID: "e74c7087-2b2c-44c5-b480-8d692b91d134"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:22:41.612426 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:41.612406    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zcnsv\" (UniqueName: \"kubernetes.io/projected/4f71b7db-e5b6-4fb6-b489-ea87e96450a7-kube-api-access-zcnsv\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:22:41.612478 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:41.612429    4559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f71b7db-e5b6-4fb6-b489-ea87e96450a7-plugins-config-volume\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:22:41.612478 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:41.612440    4559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e74c7087-2b2c-44c5-b480-8d692b91d134-plugins-config-volume\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:22:41.613894 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:41.613874    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e74c7087-2b2c-44c5-b480-8d692b91d134-kube-api-access-bw6tf" pod "e74c7087-2b2c-44c5-b480-8d692b91d134" (UID: "e74c7087-2b2c-44c5-b480-8d692b91d134"). InnerVolumeSpecName "kube-api-access-bw6tf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:22:41.713883 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:41.713851    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bw6tf\" (UniqueName: \"kubernetes.io/projected/e74c7087-2b2c-44c5-b480-8d692b91d134-kube-api-access-bw6tf\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:22:42.327089 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:42.327062    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-48d74e-57f8487987-np4k8"
Nov 02 13:22:42.327089 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:42.327082    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-48d74e-57f8487987-np4k8" event={"ID":"4f71b7db-e5b6-4fb6-b489-ea87e96450a7","Type":"ContainerDied","Data":"da3d6702df157d50b23ef9e29dc1e5c9e20f40e64ee3d279168a631fded75447"}
Nov 02 13:22:42.327631 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:42.327141    4559 scope.go:122] "RemoveContainer" containerID="59932a3d203e08813bfa9712eb627a678d24f71a13a0a4db874e73375caf8b24"
Nov 02 13:22:42.328626 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:42.328607    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-48d74e-77f759984c-j6jsk"
Nov 02 13:22:42.328745 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:42.328601    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-48d74e-77f759984c-j6jsk" event={"ID":"e74c7087-2b2c-44c5-b480-8d692b91d134","Type":"ContainerDied","Data":"c150f9b8b237e6e9d2655944b584363a8d5f52c6ed94759ef2c16da1bb2dce3a"}
Nov 02 13:22:42.338186 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:42.338171    4559 scope.go:122] "RemoveContainer" containerID="2303c6e317d118f062da5202a64703c41c14837340f2f762686ee91c2bda8580"
Nov 02 13:22:42.383190 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:42.383167    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-48d74e-57f8487987-np4k8"]
Nov 02 13:22:42.396770 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:42.396747    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-48d74e-57f8487987-np4k8"]
Nov 02 13:22:42.433297 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:42.433275    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-48d74e-77f759984c-j6jsk"]
Nov 02 13:22:42.441894 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:42.441873    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-48d74e-77f759984c-j6jsk"]
Nov 02 13:22:43.464038 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:43.464004    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7kvjr"]
Nov 02 13:22:43.469141 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:43.469102    4559 cpu_manager.go:414] "containerMap: removing container" podUID="da6dd2b6-1e1c-4128-a988-cf86451095e1" containerName="payload-pre-processing"
Nov 02 13:22:43.469141 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:43.469132    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da6dd2b6-1e1c-4128-a988-cf86451095e1" containerName="payload-pre-processing"
Nov 02 13:22:43.469392 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:43.469165    4559 cpu_manager.go:414] "containerMap: removing container" podUID="e74c7087-2b2c-44c5-b480-8d692b91d134" containerName="payload-pre-processing"
Nov 02 13:22:43.469392 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:43.469174    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e74c7087-2b2c-44c5-b480-8d692b91d134" containerName="payload-pre-processing"
Nov 02 13:22:43.469392 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:43.469190    4559 cpu_manager.go:414] "containerMap: removing container" podUID="85ec8b4f-746f-4d0c-9560-7a79f0c64ef6" containerName="istio-proxy"
Nov 02 13:22:43.469392 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:43.469197    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85ec8b4f-746f-4d0c-9560-7a79f0c64ef6" containerName="istio-proxy"
Nov 02 13:22:43.469392 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:43.469208    4559 cpu_manager.go:414] "containerMap: removing container" podUID="4f71b7db-e5b6-4fb6-b489-ea87e96450a7" containerName="payload-processing"
Nov 02 13:22:43.469392 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:43.469215    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f71b7db-e5b6-4fb6-b489-ea87e96450a7" containerName="payload-processing"
Nov 02 13:22:43.469392 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:43.469225    4559 cpu_manager.go:414] "containerMap: removing container" podUID="004721a0-56e6-42d6-bca7-c9cb11cf0e5f" containerName="maas-api"
Nov 02 13:22:43.469392 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:43.469232    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="004721a0-56e6-42d6-bca7-c9cb11cf0e5f" containerName="maas-api"
Nov 02 13:22:43.469392 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:43.469258    4559 cpu_manager.go:414] "containerMap: removing container" podUID="4e103054-2a54-43b1-b73d-a50a44c6492a" containerName="istio-proxy"
Nov 02 13:22:43.469392 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:43.469266    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e103054-2a54-43b1-b73d-a50a44c6492a" containerName="istio-proxy"
Nov 02 13:22:43.469392 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:43.469276    4559 cpu_manager.go:414] "containerMap: removing container" podUID="27dd21d8-102d-45c3-8e54-3c1e846829de" containerName="istio-proxy"
Nov 02 13:22:43.469392 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:43.469283    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27dd21d8-102d-45c3-8e54-3c1e846829de" containerName="istio-proxy"
Nov 02 13:22:43.469392 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:43.469294    4559 cpu_manager.go:414] "containerMap: removing container" podUID="2f43f207-7f92-4aab-a554-903c5d18c8f3" containerName="main"
Nov 02 13:22:43.469392 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:43.469300    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f43f207-7f92-4aab-a554-903c5d18c8f3" containerName="main"
Nov 02 13:22:43.469392 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:43.469312    4559 cpu_manager.go:414] "containerMap: removing container" podUID="279e09ef-b780-457c-a0b0-66948ee0f699" containerName="istio-proxy"
Nov 02 13:22:43.469392 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:43.469319    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="279e09ef-b780-457c-a0b0-66948ee0f699" containerName="istio-proxy"
Nov 02 13:22:43.469392 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:43.469331    4559 cpu_manager.go:414] "containerMap: removing container" podUID="d60e24f5-ef82-485d-a867-b3e3718b0e65" containerName="istio-proxy"
Nov 02 13:22:43.469392 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:43.469339    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d60e24f5-ef82-485d-a867-b3e3718b0e65" containerName="istio-proxy"
Nov 02 13:22:43.469392 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:43.469354    4559 cpu_manager.go:414] "containerMap: removing container" podUID="1b3239ce-770b-4f22-9461-89f8c62f850a" containerName="payload-processing"
Nov 02 13:22:43.469392 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:43.469361    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b3239ce-770b-4f22-9461-89f8c62f850a" containerName="payload-processing"
Nov 02 13:22:43.469392 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:43.469383    4559 cpu_manager.go:414] "containerMap: removing container" podUID="185fcba5-401f-4a43-bad1-f82f845221b8" containerName="istio-proxy"
Nov 02 13:22:43.469392 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:43.469390    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="185fcba5-401f-4a43-bad1-f82f845221b8" containerName="istio-proxy"
Nov 02 13:22:43.469392 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:43.469401    4559 cpu_manager.go:414] "containerMap: removing container" podUID="ada43a30-3953-47ab-9bb7-d623b8025e13" containerName="maas-api"
Nov 02 13:22:43.470630 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:43.469409    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ada43a30-3953-47ab-9bb7-d623b8025e13" containerName="maas-api"
Nov 02 13:22:43.470630 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:43.469422    4559 cpu_manager.go:414] "containerMap: removing container" podUID="3799b3ab-6927-4a3c-a610-2d7eb7fe507c" containerName="main"
Nov 02 13:22:43.470630 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:43.469430    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3799b3ab-6927-4a3c-a610-2d7eb7fe507c" containerName="main"
Nov 02 13:22:43.470630 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:43.469450    4559 cpu_manager.go:414] "containerMap: removing container" podUID="4436e381-1572-416e-ad03-2805362dde64" containerName="maas-api"
Nov 02 13:22:43.470630 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:43.469457    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4436e381-1572-416e-ad03-2805362dde64" containerName="maas-api"
Nov 02 13:22:43.470630 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:43.469473    4559 cpu_manager.go:414] "containerMap: removing container" podUID="f7a083ea-c6dc-4402-aac2-7842d1578ed2" containerName="istio-proxy"
Nov 02 13:22:43.470630 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:43.469482    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7a083ea-c6dc-4402-aac2-7842d1578ed2" containerName="istio-proxy"
Nov 02 13:22:43.470630 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:43.469507    4559 cpu_manager.go:414] "containerMap: removing container" podUID="affbe914-1335-4829-b1d5-c31bf0bd44ea" containerName="payload-pre-processing"
Nov 02 13:22:43.470630 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:43.469515    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="affbe914-1335-4829-b1d5-c31bf0bd44ea" containerName="payload-pre-processing"
Nov 02 13:22:43.470630 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:43.469705    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="27dd21d8-102d-45c3-8e54-3c1e846829de" containerName="istio-proxy"
Nov 02 13:22:43.470630 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:43.469719    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="4436e381-1572-416e-ad03-2805362dde64" containerName="maas-api"
Nov 02 13:22:43.470630 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:43.469731    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7a083ea-c6dc-4402-aac2-7842d1578ed2" containerName="istio-proxy"
Nov 02 13:22:43.470630 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:43.469743    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="85ec8b4f-746f-4d0c-9560-7a79f0c64ef6" containerName="istio-proxy"
Nov 02 13:22:43.470630 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:43.469752    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="affbe914-1335-4829-b1d5-c31bf0bd44ea" containerName="payload-pre-processing"
Nov 02 13:22:43.470630 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:43.469760    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="da6dd2b6-1e1c-4128-a988-cf86451095e1" containerName="payload-pre-processing"
Nov 02 13:22:43.470630 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:43.469774    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="e74c7087-2b2c-44c5-b480-8d692b91d134" containerName="payload-pre-processing"
Nov 02 13:22:43.470630 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:43.469786    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f71b7db-e5b6-4fb6-b489-ea87e96450a7" containerName="payload-processing"
Nov 02 13:22:43.470630 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:43.469797    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="004721a0-56e6-42d6-bca7-c9cb11cf0e5f" containerName="maas-api"
Nov 02 13:22:43.470630 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:43.469811    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e103054-2a54-43b1-b73d-a50a44c6492a" containerName="istio-proxy"
Nov 02 13:22:43.470630 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:43.469820    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="27dd21d8-102d-45c3-8e54-3c1e846829de" containerName="istio-proxy"
Nov 02 13:22:43.470630 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:43.469831    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f43f207-7f92-4aab-a554-903c5d18c8f3" containerName="main"
Nov 02 13:22:43.470630 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:43.469838    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="279e09ef-b780-457c-a0b0-66948ee0f699" containerName="istio-proxy"
Nov 02 13:22:43.470630 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:43.469847    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="d60e24f5-ef82-485d-a867-b3e3718b0e65" containerName="istio-proxy"
Nov 02 13:22:43.470630 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:43.469857    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b3239ce-770b-4f22-9461-89f8c62f850a" containerName="payload-processing"
Nov 02 13:22:43.470630 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:43.469867    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="185fcba5-401f-4a43-bad1-f82f845221b8" containerName="istio-proxy"
Nov 02 13:22:43.470630 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:43.469875    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="ada43a30-3953-47ab-9bb7-d623b8025e13" containerName="maas-api"
Nov 02 13:22:43.470630 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:43.469883    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="3799b3ab-6927-4a3c-a610-2d7eb7fe507c" containerName="main"
Nov 02 13:22:43.475845 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:43.475825    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7kvjr"]
Nov 02 13:22:43.476179 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:43.475948    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7kvjr"
Nov 02 13:22:43.546532 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:43.546481    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b736e009-d002-4177-8f79-8f50cd5325f3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-7kvjr\" (UID: \"b736e009-d002-4177-8f79-8f50cd5325f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7kvjr"
Nov 02 13:22:43.546871 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:43.546801    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b736e009-d002-4177-8f79-8f50cd5325f3-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-7kvjr\" (UID: \"b736e009-d002-4177-8f79-8f50cd5325f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7kvjr"
Nov 02 13:22:43.547000 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:43.546872    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v46qb\" (UniqueName: \"kubernetes.io/projected/b736e009-d002-4177-8f79-8f50cd5325f3-kube-api-access-v46qb\") pod \"maas-default-gateway-openshift-default-5bc884dd87-7kvjr\" (UID: \"b736e009-d002-4177-8f79-8f50cd5325f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7kvjr"
Nov 02 13:22:43.547000 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:43.546920    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b736e009-d002-4177-8f79-8f50cd5325f3-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-7kvjr\" (UID: \"b736e009-d002-4177-8f79-8f50cd5325f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7kvjr"
Nov 02 13:22:43.547000 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:43.546983    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b736e009-d002-4177-8f79-8f50cd5325f3-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-7kvjr\" (UID: \"b736e009-d002-4177-8f79-8f50cd5325f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7kvjr"
Nov 02 13:22:43.547247 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:43.547040    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b736e009-d002-4177-8f79-8f50cd5325f3-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-7kvjr\" (UID: \"b736e009-d002-4177-8f79-8f50cd5325f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7kvjr"
Nov 02 13:22:43.547247 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:43.547114    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b736e009-d002-4177-8f79-8f50cd5325f3-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-7kvjr\" (UID: \"b736e009-d002-4177-8f79-8f50cd5325f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7kvjr"
Nov 02 13:22:43.547371 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:43.547242    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b736e009-d002-4177-8f79-8f50cd5325f3-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-7kvjr\" (UID: \"b736e009-d002-4177-8f79-8f50cd5325f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7kvjr"
Nov 02 13:22:43.547427 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:43.547385    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b736e009-d002-4177-8f79-8f50cd5325f3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-7kvjr\" (UID: \"b736e009-d002-4177-8f79-8f50cd5325f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7kvjr"
Nov 02 13:22:43.648375 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:43.648340    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b736e009-d002-4177-8f79-8f50cd5325f3-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-7kvjr\" (UID: \"b736e009-d002-4177-8f79-8f50cd5325f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7kvjr"
Nov 02 13:22:43.648547 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:43.648423    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b736e009-d002-4177-8f79-8f50cd5325f3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-7kvjr\" (UID: \"b736e009-d002-4177-8f79-8f50cd5325f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7kvjr"
Nov 02 13:22:43.648547 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:43.648476    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b736e009-d002-4177-8f79-8f50cd5325f3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-7kvjr\" (UID: \"b736e009-d002-4177-8f79-8f50cd5325f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7kvjr"
Nov 02 13:22:43.648547 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:43.648538    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b736e009-d002-4177-8f79-8f50cd5325f3-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-7kvjr\" (UID: \"b736e009-d002-4177-8f79-8f50cd5325f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7kvjr"
Nov 02 13:22:43.648754 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:43.648565    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v46qb\" (UniqueName: \"kubernetes.io/projected/b736e009-d002-4177-8f79-8f50cd5325f3-kube-api-access-v46qb\") pod \"maas-default-gateway-openshift-default-5bc884dd87-7kvjr\" (UID: \"b736e009-d002-4177-8f79-8f50cd5325f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7kvjr"
Nov 02 13:22:43.648754 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:43.648592    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b736e009-d002-4177-8f79-8f50cd5325f3-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-7kvjr\" (UID: \"b736e009-d002-4177-8f79-8f50cd5325f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7kvjr"
Nov 02 13:22:43.648754 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:43.648656    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b736e009-d002-4177-8f79-8f50cd5325f3-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-7kvjr\" (UID: \"b736e009-d002-4177-8f79-8f50cd5325f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7kvjr"
Nov 02 13:22:43.648939 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:43.648792    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b736e009-d002-4177-8f79-8f50cd5325f3-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-7kvjr\" (UID: \"b736e009-d002-4177-8f79-8f50cd5325f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7kvjr"
Nov 02 13:22:43.648939 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:43.648791    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b736e009-d002-4177-8f79-8f50cd5325f3-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-7kvjr\" (UID: \"b736e009-d002-4177-8f79-8f50cd5325f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7kvjr"
Nov 02 13:22:43.648939 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:43.648929    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b736e009-d002-4177-8f79-8f50cd5325f3-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-7kvjr\" (UID: \"b736e009-d002-4177-8f79-8f50cd5325f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7kvjr"
Nov 02 13:22:43.649120 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:43.648940    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b736e009-d002-4177-8f79-8f50cd5325f3-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-7kvjr\" (UID: \"b736e009-d002-4177-8f79-8f50cd5325f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7kvjr"
Nov 02 13:22:43.649120 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:43.649050    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b736e009-d002-4177-8f79-8f50cd5325f3-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-7kvjr\" (UID: \"b736e009-d002-4177-8f79-8f50cd5325f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7kvjr"
Nov 02 13:22:43.649120 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:43.649081    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b736e009-d002-4177-8f79-8f50cd5325f3-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-7kvjr\" (UID: \"b736e009-d002-4177-8f79-8f50cd5325f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7kvjr"
Nov 02 13:22:43.649417 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:43.649396    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b736e009-d002-4177-8f79-8f50cd5325f3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-7kvjr\" (UID: \"b736e009-d002-4177-8f79-8f50cd5325f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7kvjr"
Nov 02 13:22:43.650957 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:43.650936    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b736e009-d002-4177-8f79-8f50cd5325f3-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-7kvjr\" (UID: \"b736e009-d002-4177-8f79-8f50cd5325f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7kvjr"
Nov 02 13:22:43.651092 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:43.651072    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b736e009-d002-4177-8f79-8f50cd5325f3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-7kvjr\" (UID: \"b736e009-d002-4177-8f79-8f50cd5325f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7kvjr"
Nov 02 13:22:43.673129 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:43.673104    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b736e009-d002-4177-8f79-8f50cd5325f3-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-7kvjr\" (UID: \"b736e009-d002-4177-8f79-8f50cd5325f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7kvjr"
Nov 02 13:22:43.673926 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:43.673903    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v46qb\" (UniqueName: \"kubernetes.io/projected/b736e009-d002-4177-8f79-8f50cd5325f3-kube-api-access-v46qb\") pod \"maas-default-gateway-openshift-default-5bc884dd87-7kvjr\" (UID: \"b736e009-d002-4177-8f79-8f50cd5325f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7kvjr"
Nov 02 13:22:43.824451 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:43.824384    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7kvjr"
Nov 02 13:22:43.905205 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:43.905167    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f71b7db-e5b6-4fb6-b489-ea87e96450a7" path="/var/lib/kubelet/pods/4f71b7db-e5b6-4fb6-b489-ea87e96450a7/volumes"
Nov 02 13:22:43.905906 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:43.905881    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e74c7087-2b2c-44c5-b480-8d692b91d134" path="/var/lib/kubelet/pods/e74c7087-2b2c-44c5-b480-8d692b91d134/volumes"
Nov 02 13:22:44.154680 ip-10-0-128-132 kubenswrapper[4559]: W1102 13:22:44.154614    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb736e009_d002_4177_8f79_8f50cd5325f3.slice/crio-220f5d89aed037e4349a2e1d847cd44191f43aa7e990587b689babf07244c8fa WatchSource:0}: Error finding container 220f5d89aed037e4349a2e1d847cd44191f43aa7e990587b689babf07244c8fa: Status 404 returned error can't find the container with id 220f5d89aed037e4349a2e1d847cd44191f43aa7e990587b689babf07244c8fa
Nov 02 13:22:44.155348 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:44.155322    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7kvjr"]
Nov 02 13:22:44.156958 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:44.156925    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 13:22:44.157069 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:44.156992    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 13:22:44.157069 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:44.157022    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 13:22:44.470712 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:44.470669    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7kvjr" event={"ID":"b736e009-d002-4177-8f79-8f50cd5325f3","Type":"ContainerStarted","Data":"cd4b240992256d01908e7156df44d1661f5a7926377344fdd68067abb3c3f87e"}
Nov 02 13:22:44.471326 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:44.470732    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7kvjr" event={"ID":"b736e009-d002-4177-8f79-8f50cd5325f3","Type":"ContainerStarted","Data":"220f5d89aed037e4349a2e1d847cd44191f43aa7e990587b689babf07244c8fa"}
Nov 02 13:22:44.824916 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:44.824843    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7kvjr"
Nov 02 13:22:45.086833 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:22:45.086757    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqcmj" containerName="extractor"
Nov 02 13:22:45.827480 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:45.827442    4559 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-7kvjr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.250:15021/healthz/ready\": dial tcp 10.133.0.250:15021: connect: connection refused" start-of-body=
Nov 02 13:22:45.828006 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:45.827500    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7kvjr" podUID="b736e009-d002-4177-8f79-8f50cd5325f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.250:15021/healthz/ready\": dial tcp 10.133.0.250:15021: connect: connection refused"
Nov 02 13:22:46.825602 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:46.825564    4559 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-7kvjr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.250:15021/healthz/ready\": dial tcp 10.133.0.250:15021: connect: connection refused" start-of-body=
Nov 02 13:22:46.825834 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:46.825628    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7kvjr" podUID="b736e009-d002-4177-8f79-8f50cd5325f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.250:15021/healthz/ready\": dial tcp 10.133.0.250:15021: connect: connection refused"
Nov 02 13:22:47.825336 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:47.825296    4559 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-7kvjr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.250:15021/healthz/ready\": dial tcp 10.133.0.250:15021: connect: connection refused" start-of-body=
Nov 02 13:22:47.825824 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:47.825379    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7kvjr" podUID="b736e009-d002-4177-8f79-8f50cd5325f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.250:15021/healthz/ready\": dial tcp 10.133.0.250:15021: connect: connection refused"
Nov 02 13:22:48.138796 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:48.138340    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7kvjr" podStartSLOduration=5.13831918 podStartE2EDuration="5.13831918s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:22:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:22:44.544253315 +0000 UTC m=+3360.799652411" watchObservedRunningTime="2026-11-02 13:22:48.13831918 +0000 UTC m=+3364.393718276"
Nov 02 13:22:48.147656 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:48.145723    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-361c3a-openshift-default-6f67d958cf-zhhhp"]
Nov 02 13:22:48.147656 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:48.145992    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-361c3a-openshift-default-6f67d958cf-zhhhp" podUID="5ae3a83d-c3a4-4305-8824-1cde1a6172c0" containerName="istio-proxy" containerID="cri-o://3c67b55c02942fa1365fef1d300ee087b9eaba29eafe6c699b7d1fa9b220e051" gracePeriod=30
Nov 02 13:22:48.825262 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:48.825222    4559 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-7kvjr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.250:15021/healthz/ready\": dial tcp 10.133.0.250:15021: connect: connection refused" start-of-body=
Nov 02 13:22:48.825475 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:48.825291    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7kvjr" podUID="b736e009-d002-4177-8f79-8f50cd5325f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.250:15021/healthz/ready\": dial tcp 10.133.0.250:15021: connect: connection refused"
Nov 02 13:22:49.824839 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:49.824804    4559 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-7kvjr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.250:15021/healthz/ready\": dial tcp 10.133.0.250:15021: connect: connection refused" start-of-body=
Nov 02 13:22:49.825027 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:49.824854    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7kvjr" podUID="b736e009-d002-4177-8f79-8f50cd5325f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.250:15021/healthz/ready\": dial tcp 10.133.0.250:15021: connect: connection refused"
Nov 02 13:22:49.971190 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:49.971152    4559 patch_prober.go:29] interesting pod/e2e-inf-b-361c3a-openshift-default-6f67d958cf-zhhhp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 13:22:49.971568 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:49.971204    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-361c3a-openshift-default-6f67d958cf-zhhhp" podUID="5ae3a83d-c3a4-4305-8824-1cde1a6172c0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 13:22:50.825180 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:50.825144    4559 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-7kvjr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.250:15021/healthz/ready\": dial tcp 10.133.0.250:15021: connect: connection refused" start-of-body=
Nov 02 13:22:50.825361 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:50.825201    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7kvjr" podUID="b736e009-d002-4177-8f79-8f50cd5325f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.250:15021/healthz/ready\": dial tcp 10.133.0.250:15021: connect: connection refused"
Nov 02 13:22:51.824955 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:51.824923    4559 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-7kvjr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.250:15021/healthz/ready\": dial tcp 10.133.0.250:15021: connect: connection refused" start-of-body=
Nov 02 13:22:51.825351 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:51.824977    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7kvjr" podUID="b736e009-d002-4177-8f79-8f50cd5325f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.250:15021/healthz/ready\": dial tcp 10.133.0.250:15021: connect: connection refused"
Nov 02 13:22:53.420580 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:53.419207    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7kvjr"
Nov 02 13:22:53.420580 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:53.420304    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7kvjr"
Nov 02 13:22:53.422363 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:53.422337    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7kvjr"
Nov 02 13:22:53.442469 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:53.442419    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-361c3a-openshift-default-6f67d958cf-zhhhp"
Nov 02 13:22:53.579464 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:53.578794    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ae3a83d-c3a4-4305-8824-1cde1a6172c0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ae3a83d-c3a4-4305-8824-1cde1a6172c0-istio-envoy\") pod \"5ae3a83d-c3a4-4305-8824-1cde1a6172c0\" (UID: \"5ae3a83d-c3a4-4305-8824-1cde1a6172c0\") "
Nov 02 13:22:53.579464 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:53.578875    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ae3a83d-c3a4-4305-8824-1cde1a6172c0-kube-api-access-bvxzc\" (UniqueName: \"kubernetes.io/projected/5ae3a83d-c3a4-4305-8824-1cde1a6172c0-kube-api-access-bvxzc\") pod \"5ae3a83d-c3a4-4305-8824-1cde1a6172c0\" (UID: \"5ae3a83d-c3a4-4305-8824-1cde1a6172c0\") "
Nov 02 13:22:53.579464 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:53.579319    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5ae3a83d-c3a4-4305-8824-1cde1a6172c0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ae3a83d-c3a4-4305-8824-1cde1a6172c0-istio-podinfo\") pod \"5ae3a83d-c3a4-4305-8824-1cde1a6172c0\" (UID: \"5ae3a83d-c3a4-4305-8824-1cde1a6172c0\") "
Nov 02 13:22:53.581480 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:53.580806    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ae3a83d-c3a4-4305-8824-1cde1a6172c0-istio-token\" (UniqueName: \"kubernetes.io/projected/5ae3a83d-c3a4-4305-8824-1cde1a6172c0-istio-token\") pod \"5ae3a83d-c3a4-4305-8824-1cde1a6172c0\" (UID: \"5ae3a83d-c3a4-4305-8824-1cde1a6172c0\") "
Nov 02 13:22:53.581958 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:53.581626    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ae3a83d-c3a4-4305-8824-1cde1a6172c0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ae3a83d-c3a4-4305-8824-1cde1a6172c0-workload-socket\") pod \"5ae3a83d-c3a4-4305-8824-1cde1a6172c0\" (UID: \"5ae3a83d-c3a4-4305-8824-1cde1a6172c0\") "
Nov 02 13:22:53.582084 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:53.581910    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ae3a83d-c3a4-4305-8824-1cde1a6172c0-workload-socket" pod "5ae3a83d-c3a4-4305-8824-1cde1a6172c0" (UID: "5ae3a83d-c3a4-4305-8824-1cde1a6172c0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:22:53.582214 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:53.582134    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ae3a83d-c3a4-4305-8824-1cde1a6172c0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ae3a83d-c3a4-4305-8824-1cde1a6172c0-credential-socket\") pod \"5ae3a83d-c3a4-4305-8824-1cde1a6172c0\" (UID: \"5ae3a83d-c3a4-4305-8824-1cde1a6172c0\") "
Nov 02 13:22:53.582452 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:53.582416    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ae3a83d-c3a4-4305-8824-1cde1a6172c0-credential-socket" pod "5ae3a83d-c3a4-4305-8824-1cde1a6172c0" (UID: "5ae3a83d-c3a4-4305-8824-1cde1a6172c0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:22:53.584002 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:53.583977    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ae3a83d-c3a4-4305-8824-1cde1a6172c0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ae3a83d-c3a4-4305-8824-1cde1a6172c0-istio-data\") pod \"5ae3a83d-c3a4-4305-8824-1cde1a6172c0\" (UID: \"5ae3a83d-c3a4-4305-8824-1cde1a6172c0\") "
Nov 02 13:22:53.584002 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:53.584090    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ae3a83d-c3a4-4305-8824-1cde1a6172c0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ae3a83d-c3a4-4305-8824-1cde1a6172c0-istiod-ca-cert\") pod \"5ae3a83d-c3a4-4305-8824-1cde1a6172c0\" (UID: \"5ae3a83d-c3a4-4305-8824-1cde1a6172c0\") "
Nov 02 13:22:53.584002 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:53.584139    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ae3a83d-c3a4-4305-8824-1cde1a6172c0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ae3a83d-c3a4-4305-8824-1cde1a6172c0-workload-certs\") pod \"5ae3a83d-c3a4-4305-8824-1cde1a6172c0\" (UID: \"5ae3a83d-c3a4-4305-8824-1cde1a6172c0\") "
Nov 02 13:22:53.584002 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:53.584582    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ae3a83d-c3a4-4305-8824-1cde1a6172c0-istiod-ca-cert" pod "5ae3a83d-c3a4-4305-8824-1cde1a6172c0" (UID: "5ae3a83d-c3a4-4305-8824-1cde1a6172c0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:22:53.584002 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:53.584791    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ae3a83d-c3a4-4305-8824-1cde1a6172c0-istio-data" pod "5ae3a83d-c3a4-4305-8824-1cde1a6172c0" (UID: "5ae3a83d-c3a4-4305-8824-1cde1a6172c0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:22:53.584002 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:53.585085    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ae3a83d-c3a4-4305-8824-1cde1a6172c0-workload-certs" pod "5ae3a83d-c3a4-4305-8824-1cde1a6172c0" (UID: "5ae3a83d-c3a4-4305-8824-1cde1a6172c0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:22:53.584002 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:53.585959    4559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ae3a83d-c3a4-4305-8824-1cde1a6172c0-istio-data\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:22:53.584002 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:53.586190    4559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ae3a83d-c3a4-4305-8824-1cde1a6172c0-istiod-ca-cert\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:22:53.584002 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:53.586210    4559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ae3a83d-c3a4-4305-8824-1cde1a6172c0-workload-certs\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:22:53.584002 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:53.586223    4559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ae3a83d-c3a4-4305-8824-1cde1a6172c0-workload-socket\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:22:53.584002 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:53.586238    4559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ae3a83d-c3a4-4305-8824-1cde1a6172c0-credential-socket\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:22:53.590469 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:53.590447    4559 generic.go:356] "Generic (PLEG): container finished" podID="5ae3a83d-c3a4-4305-8824-1cde1a6172c0" containerID="3c67b55c02942fa1365fef1d300ee087b9eaba29eafe6c699b7d1fa9b220e051" exitCode=0
Nov 02 13:22:53.590535 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:53.590505    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-361c3a-openshift-default-6f67d958cf-zhhhp"
Nov 02 13:22:53.590887 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:53.590847    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ae3a83d-c3a4-4305-8824-1cde1a6172c0-istio-token" pod "5ae3a83d-c3a4-4305-8824-1cde1a6172c0" (UID: "5ae3a83d-c3a4-4305-8824-1cde1a6172c0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:22:53.590887 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:53.590867    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-361c3a-openshift-default-6f67d958cf-zhhhp" event={"ID":"5ae3a83d-c3a4-4305-8824-1cde1a6172c0","Type":"ContainerDied","Data":"3c67b55c02942fa1365fef1d300ee087b9eaba29eafe6c699b7d1fa9b220e051"}
Nov 02 13:22:53.591080 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:53.590953    4559 scope.go:122] "RemoveContainer" containerID="3c67b55c02942fa1365fef1d300ee087b9eaba29eafe6c699b7d1fa9b220e051"
Nov 02 13:22:53.591418 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:53.591373    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ae3a83d-c3a4-4305-8824-1cde1a6172c0-kube-api-access-bvxzc" pod "5ae3a83d-c3a4-4305-8824-1cde1a6172c0" (UID: "5ae3a83d-c3a4-4305-8824-1cde1a6172c0"). InnerVolumeSpecName "kube-api-access-bvxzc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:22:53.591750 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:53.591724    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-361c3a-openshift-default-6f67d958cf-zhhhp" event={"ID":"5ae3a83d-c3a4-4305-8824-1cde1a6172c0","Type":"ContainerDied","Data":"d228deb82f4000d7f02172ccac56e5992498b0fc56548f1b9ca188ac85fb555a"}
Nov 02 13:22:53.593160 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:53.592818    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5ae3a83d-c3a4-4305-8824-1cde1a6172c0-istio-podinfo" pod "5ae3a83d-c3a4-4305-8824-1cde1a6172c0" (UID: "5ae3a83d-c3a4-4305-8824-1cde1a6172c0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 13:22:53.600057 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:53.600032    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ae3a83d-c3a4-4305-8824-1cde1a6172c0-istio-envoy" pod "5ae3a83d-c3a4-4305-8824-1cde1a6172c0" (UID: "5ae3a83d-c3a4-4305-8824-1cde1a6172c0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:22:53.623784 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:53.623766    4559 scope.go:122] "RemoveContainer" containerID="3c67b55c02942fa1365fef1d300ee087b9eaba29eafe6c699b7d1fa9b220e051"
Nov 02 13:22:53.624179 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:22:53.624161    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c67b55c02942fa1365fef1d300ee087b9eaba29eafe6c699b7d1fa9b220e051\": container with ID starting with 3c67b55c02942fa1365fef1d300ee087b9eaba29eafe6c699b7d1fa9b220e051 not found: ID does not exist" containerID="3c67b55c02942fa1365fef1d300ee087b9eaba29eafe6c699b7d1fa9b220e051"
Nov 02 13:22:53.624291 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:53.624186    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c67b55c02942fa1365fef1d300ee087b9eaba29eafe6c699b7d1fa9b220e051"} err="failed to get container status \"3c67b55c02942fa1365fef1d300ee087b9eaba29eafe6c699b7d1fa9b220e051\": rpc error: code = NotFound desc = could not find container \"3c67b55c02942fa1365fef1d300ee087b9eaba29eafe6c699b7d1fa9b220e051\": container with ID starting with 3c67b55c02942fa1365fef1d300ee087b9eaba29eafe6c699b7d1fa9b220e051 not found: ID does not exist"
Nov 02 13:22:53.691656 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:53.691615    4559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ae3a83d-c3a4-4305-8824-1cde1a6172c0-istio-envoy\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:22:53.691656 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:53.691653    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bvxzc\" (UniqueName: \"kubernetes.io/projected/5ae3a83d-c3a4-4305-8824-1cde1a6172c0-kube-api-access-bvxzc\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:22:53.691656 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:53.691662    4559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ae3a83d-c3a4-4305-8824-1cde1a6172c0-istio-podinfo\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:22:53.691912 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:53.691673    4559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ae3a83d-c3a4-4305-8824-1cde1a6172c0-istio-token\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:22:53.944278 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:53.944246    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-361c3a-openshift-default-6f67d958cf-zhhhp"]
Nov 02 13:22:53.956382 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:53.956336    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-361c3a-openshift-default-6f67d958cf-zhhhp"]
Nov 02 13:22:55.894902 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:55.894831    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ae3a83d-c3a4-4305-8824-1cde1a6172c0" path="/var/lib/kubelet/pods/5ae3a83d-c3a4-4305-8824-1cde1a6172c0/volumes"
Nov 02 13:22:57.345317 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:57.345285    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-ffpz2"]
Nov 02 13:22:57.345757 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:22:57.345501    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-ffpz2" podUID="91b0660b-bd2c-4f62-ac8a-e09af747a766" containerName="istio-proxy" containerID="cri-o://626d04a633ea192250c35ed7413e8b11f2df3f058ee9954bdd020c92e11fc5d2" gracePeriod=30
Nov 02 13:23:00.113100 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:23:00.113063    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-48d74e-55dbcdf789-hznjw"]
Nov 02 13:23:00.113606 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:23:00.113282    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-48d74e-55dbcdf789-hznjw" podUID="d5c39b0a-8b09-4c62-87e4-223aad82d163" containerName="maas-api" containerID="cri-o://1895262d20402f67d415982b8f84a9be237de44013def41310b4f67b2bef063e" gracePeriod=30
Nov 02 13:23:00.513706 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:23:00.513322    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-48d74e-55dbcdf789-hznjw"
Nov 02 13:23:00.762255 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:23:00.762222    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d5c39b0a-8b09-4c62-87e4-223aad82d163-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5c39b0a-8b09-4c62-87e4-223aad82d163-metrics-certs\") pod \"d5c39b0a-8b09-4c62-87e4-223aad82d163\" (UID: \"d5c39b0a-8b09-4c62-87e4-223aad82d163\") "
Nov 02 13:23:00.781687 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:23:00.763515    4559 generic.go:356] "Generic (PLEG): container finished" podID="d5c39b0a-8b09-4c62-87e4-223aad82d163" containerID="1895262d20402f67d415982b8f84a9be237de44013def41310b4f67b2bef063e" exitCode=0
Nov 02 13:23:00.783825 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:23:00.763544    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-48d74e-55dbcdf789-hznjw" event={"ID":"d5c39b0a-8b09-4c62-87e4-223aad82d163","Type":"ContainerDied","Data":"1895262d20402f67d415982b8f84a9be237de44013def41310b4f67b2bef063e"}
Nov 02 13:23:00.783825 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:23:00.782811    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-48d74e-55dbcdf789-hznjw" event={"ID":"d5c39b0a-8b09-4c62-87e4-223aad82d163","Type":"ContainerDied","Data":"6be2bbac28b21c4b0e8ec91fe1b10be29e3ca0a7f8eef4aaeaea16d4169e699f"}
Nov 02 13:23:00.783825 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:23:00.778558    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5c39b0a-8b09-4c62-87e4-223aad82d163-metrics-certs" pod "d5c39b0a-8b09-4c62-87e4-223aad82d163" (UID: "d5c39b0a-8b09-4c62-87e4-223aad82d163"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:23:00.783825 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:23:00.781717    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d5c39b0a-8b09-4c62-87e4-223aad82d163-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5c39b0a-8b09-4c62-87e4-223aad82d163-maas-api-tls\") pod \"d5c39b0a-8b09-4c62-87e4-223aad82d163\" (UID: \"d5c39b0a-8b09-4c62-87e4-223aad82d163\") "
Nov 02 13:23:00.783825 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:23:00.783015    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5c39b0a-8b09-4c62-87e4-223aad82d163-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5c39b0a-8b09-4c62-87e4-223aad82d163-openshift-service-ca\") pod \"d5c39b0a-8b09-4c62-87e4-223aad82d163\" (UID: \"d5c39b0a-8b09-4c62-87e4-223aad82d163\") "
Nov 02 13:23:00.783825 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:23:00.783054    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5c39b0a-8b09-4c62-87e4-223aad82d163-kube-api-access-jfm7w\" (UniqueName: \"kubernetes.io/projected/d5c39b0a-8b09-4c62-87e4-223aad82d163-kube-api-access-jfm7w\") pod \"d5c39b0a-8b09-4c62-87e4-223aad82d163\" (UID: \"d5c39b0a-8b09-4c62-87e4-223aad82d163\") "
Nov 02 13:23:00.783825 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:23:00.783118    4559 scope.go:122] "RemoveContainer" containerID="1895262d20402f67d415982b8f84a9be237de44013def41310b4f67b2bef063e"
Nov 02 13:23:00.783825 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:23:00.763610    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-48d74e-55dbcdf789-hznjw"
Nov 02 13:23:00.784358 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:23:00.784001    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5c39b0a-8b09-4c62-87e4-223aad82d163-openshift-service-ca" pod "d5c39b0a-8b09-4c62-87e4-223aad82d163" (UID: "d5c39b0a-8b09-4c62-87e4-223aad82d163"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:23:00.784358 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:23:00.784097    4559 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5c39b0a-8b09-4c62-87e4-223aad82d163-metrics-certs\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:23:00.784608 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:23:00.784544    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5c39b0a-8b09-4c62-87e4-223aad82d163-maas-api-tls" pod "d5c39b0a-8b09-4c62-87e4-223aad82d163" (UID: "d5c39b0a-8b09-4c62-87e4-223aad82d163"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:23:00.786268 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:23:00.786240    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5c39b0a-8b09-4c62-87e4-223aad82d163-kube-api-access-jfm7w" pod "d5c39b0a-8b09-4c62-87e4-223aad82d163" (UID: "d5c39b0a-8b09-4c62-87e4-223aad82d163"). InnerVolumeSpecName "kube-api-access-jfm7w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:23:00.817728 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:23:00.817158    4559 scope.go:122] "RemoveContainer" containerID="1895262d20402f67d415982b8f84a9be237de44013def41310b4f67b2bef063e"
Nov 02 13:23:00.817868 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:23:00.817751    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1895262d20402f67d415982b8f84a9be237de44013def41310b4f67b2bef063e\": container with ID starting with 1895262d20402f67d415982b8f84a9be237de44013def41310b4f67b2bef063e not found: ID does not exist" containerID="1895262d20402f67d415982b8f84a9be237de44013def41310b4f67b2bef063e"
Nov 02 13:23:00.817868 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:23:00.817791    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1895262d20402f67d415982b8f84a9be237de44013def41310b4f67b2bef063e"} err="failed to get container status \"1895262d20402f67d415982b8f84a9be237de44013def41310b4f67b2bef063e\": rpc error: code = NotFound desc = could not find container \"1895262d20402f67d415982b8f84a9be237de44013def41310b4f67b2bef063e\": container with ID starting with 1895262d20402f67d415982b8f84a9be237de44013def41310b4f67b2bef063e not found: ID does not exist"
Nov 02 13:23:00.886503 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:23:00.886477    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5c39b0a-8b09-4c62-87e4-223aad82d163-maas-api-tls\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:23:00.886503 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:23:00.886500    4559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5c39b0a-8b09-4c62-87e4-223aad82d163-openshift-service-ca\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:23:00.886690 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:23:00.886510    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jfm7w\" (UniqueName: \"kubernetes.io/projected/d5c39b0a-8b09-4c62-87e4-223aad82d163-kube-api-access-jfm7w\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:23:01.134959 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:23:01.134927    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-48d74e-55dbcdf789-hznjw"]
Nov 02 13:23:01.142468 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:23:01.142437    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-48d74e-55dbcdf789-hznjw"]
Nov 02 13:23:01.898633 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:23:01.898598    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5c39b0a-8b09-4c62-87e4-223aad82d163" path="/var/lib/kubelet/pods/d5c39b0a-8b09-4c62-87e4-223aad82d163/volumes"
Nov 02 13:23:02.631097 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:23:02.631075    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-ffpz2"
Nov 02 13:23:02.730249 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:23:02.725225    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91b0660b-bd2c-4f62-ac8a-e09af747a766-kube-api-access-tsgbv\" (UniqueName: \"kubernetes.io/projected/91b0660b-bd2c-4f62-ac8a-e09af747a766-kube-api-access-tsgbv\") pod \"91b0660b-bd2c-4f62-ac8a-e09af747a766\" (UID: \"91b0660b-bd2c-4f62-ac8a-e09af747a766\") "
Nov 02 13:23:02.730249 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:23:02.725346    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91b0660b-bd2c-4f62-ac8a-e09af747a766-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91b0660b-bd2c-4f62-ac8a-e09af747a766-workload-certs\") pod \"91b0660b-bd2c-4f62-ac8a-e09af747a766\" (UID: \"91b0660b-bd2c-4f62-ac8a-e09af747a766\") "
Nov 02 13:23:02.730249 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:23:02.725434    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91b0660b-bd2c-4f62-ac8a-e09af747a766-istio-token\" (UniqueName: \"kubernetes.io/projected/91b0660b-bd2c-4f62-ac8a-e09af747a766-istio-token\") pod \"91b0660b-bd2c-4f62-ac8a-e09af747a766\" (UID: \"91b0660b-bd2c-4f62-ac8a-e09af747a766\") "
Nov 02 13:23:02.730249 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:23:02.725464    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91b0660b-bd2c-4f62-ac8a-e09af747a766-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91b0660b-bd2c-4f62-ac8a-e09af747a766-istiod-ca-cert\") pod \"91b0660b-bd2c-4f62-ac8a-e09af747a766\" (UID: \"91b0660b-bd2c-4f62-ac8a-e09af747a766\") "
Nov 02 13:23:02.730249 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:23:02.725504    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/91b0660b-bd2c-4f62-ac8a-e09af747a766-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91b0660b-bd2c-4f62-ac8a-e09af747a766-istio-podinfo\") pod \"91b0660b-bd2c-4f62-ac8a-e09af747a766\" (UID: \"91b0660b-bd2c-4f62-ac8a-e09af747a766\") "
Nov 02 13:23:02.730249 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:23:02.725532    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91b0660b-bd2c-4f62-ac8a-e09af747a766-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91b0660b-bd2c-4f62-ac8a-e09af747a766-istio-envoy\") pod \"91b0660b-bd2c-4f62-ac8a-e09af747a766\" (UID: \"91b0660b-bd2c-4f62-ac8a-e09af747a766\") "
Nov 02 13:23:02.730249 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:23:02.725858    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91b0660b-bd2c-4f62-ac8a-e09af747a766-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91b0660b-bd2c-4f62-ac8a-e09af747a766-credential-socket\") pod \"91b0660b-bd2c-4f62-ac8a-e09af747a766\" (UID: \"91b0660b-bd2c-4f62-ac8a-e09af747a766\") "
Nov 02 13:23:02.730249 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:23:02.725891    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91b0660b-bd2c-4f62-ac8a-e09af747a766-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91b0660b-bd2c-4f62-ac8a-e09af747a766-istio-data\") pod \"91b0660b-bd2c-4f62-ac8a-e09af747a766\" (UID: \"91b0660b-bd2c-4f62-ac8a-e09af747a766\") "
Nov 02 13:23:02.735346 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:23:02.735306    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91b0660b-bd2c-4f62-ac8a-e09af747a766-istio-data" pod "91b0660b-bd2c-4f62-ac8a-e09af747a766" (UID: "91b0660b-bd2c-4f62-ac8a-e09af747a766"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:23:02.735517 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:23:02.735401    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91b0660b-bd2c-4f62-ac8a-e09af747a766-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91b0660b-bd2c-4f62-ac8a-e09af747a766-workload-socket\") pod \"91b0660b-bd2c-4f62-ac8a-e09af747a766\" (UID: \"91b0660b-bd2c-4f62-ac8a-e09af747a766\") "
Nov 02 13:23:02.736344 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:23:02.736315    4559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91b0660b-bd2c-4f62-ac8a-e09af747a766-istio-data\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:23:02.736595 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:23:02.736573    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91b0660b-bd2c-4f62-ac8a-e09af747a766-workload-socket" pod "91b0660b-bd2c-4f62-ac8a-e09af747a766" (UID: "91b0660b-bd2c-4f62-ac8a-e09af747a766"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:23:02.743680 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:23:02.743219    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91b0660b-bd2c-4f62-ac8a-e09af747a766-istiod-ca-cert" pod "91b0660b-bd2c-4f62-ac8a-e09af747a766" (UID: "91b0660b-bd2c-4f62-ac8a-e09af747a766"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:23:02.743680 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:23:02.743468    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91b0660b-bd2c-4f62-ac8a-e09af747a766-workload-certs" pod "91b0660b-bd2c-4f62-ac8a-e09af747a766" (UID: "91b0660b-bd2c-4f62-ac8a-e09af747a766"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:23:02.745903 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:23:02.745878    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91b0660b-bd2c-4f62-ac8a-e09af747a766-credential-socket" pod "91b0660b-bd2c-4f62-ac8a-e09af747a766" (UID: "91b0660b-bd2c-4f62-ac8a-e09af747a766"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:23:02.748296 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:23:02.748262    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/91b0660b-bd2c-4f62-ac8a-e09af747a766-istio-podinfo" pod "91b0660b-bd2c-4f62-ac8a-e09af747a766" (UID: "91b0660b-bd2c-4f62-ac8a-e09af747a766"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 13:23:02.769887 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:23:02.769853    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91b0660b-bd2c-4f62-ac8a-e09af747a766-istio-token" pod "91b0660b-bd2c-4f62-ac8a-e09af747a766" (UID: "91b0660b-bd2c-4f62-ac8a-e09af747a766"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:23:02.771617 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:23:02.771588    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91b0660b-bd2c-4f62-ac8a-e09af747a766-istio-envoy" pod "91b0660b-bd2c-4f62-ac8a-e09af747a766" (UID: "91b0660b-bd2c-4f62-ac8a-e09af747a766"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:23:02.773905 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:23:02.773870    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91b0660b-bd2c-4f62-ac8a-e09af747a766-kube-api-access-tsgbv" pod "91b0660b-bd2c-4f62-ac8a-e09af747a766" (UID: "91b0660b-bd2c-4f62-ac8a-e09af747a766"). InnerVolumeSpecName "kube-api-access-tsgbv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:23:02.812598 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:23:02.812563    4559 generic.go:356] "Generic (PLEG): container finished" podID="91b0660b-bd2c-4f62-ac8a-e09af747a766" containerID="626d04a633ea192250c35ed7413e8b11f2df3f058ee9954bdd020c92e11fc5d2" exitCode=0
Nov 02 13:23:02.812778 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:23:02.812754    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-ffpz2" event={"ID":"91b0660b-bd2c-4f62-ac8a-e09af747a766","Type":"ContainerDied","Data":"626d04a633ea192250c35ed7413e8b11f2df3f058ee9954bdd020c92e11fc5d2"}
Nov 02 13:23:02.812831 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:23:02.812818    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-ffpz2" event={"ID":"91b0660b-bd2c-4f62-ac8a-e09af747a766","Type":"ContainerDied","Data":"a211508dfbe1d54925de5494fe78efdb6c9079096bfc74398787ccd989515b2f"}
Nov 02 13:23:02.812876 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:23:02.812849    4559 scope.go:122] "RemoveContainer" containerID="626d04a633ea192250c35ed7413e8b11f2df3f058ee9954bdd020c92e11fc5d2"
Nov 02 13:23:02.813024 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:23:02.813011    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-ffpz2"
Nov 02 13:23:02.829753 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:23:02.829628    4559 scope.go:122] "RemoveContainer" containerID="f263118547ccecb716514445c970370884eac55bf80594ea71b34d72e20be424"
Nov 02 13:23:02.837794 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:23:02.837766    4559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91b0660b-bd2c-4f62-ac8a-e09af747a766-credential-socket\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:23:02.837794 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:23:02.837791    4559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91b0660b-bd2c-4f62-ac8a-e09af747a766-workload-socket\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:23:02.837946 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:23:02.837810    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tsgbv\" (UniqueName: \"kubernetes.io/projected/91b0660b-bd2c-4f62-ac8a-e09af747a766-kube-api-access-tsgbv\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:23:02.837946 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:23:02.837822    4559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91b0660b-bd2c-4f62-ac8a-e09af747a766-workload-certs\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:23:02.837946 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:23:02.837835    4559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91b0660b-bd2c-4f62-ac8a-e09af747a766-istio-token\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:23:02.837946 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:23:02.837847    4559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91b0660b-bd2c-4f62-ac8a-e09af747a766-istiod-ca-cert\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:23:02.837946 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:23:02.837859    4559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91b0660b-bd2c-4f62-ac8a-e09af747a766-istio-podinfo\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:23:02.837946 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:23:02.837873    4559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91b0660b-bd2c-4f62-ac8a-e09af747a766-istio-envoy\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:23:02.840809 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:23:02.840790    4559 scope.go:122] "RemoveContainer" containerID="626d04a633ea192250c35ed7413e8b11f2df3f058ee9954bdd020c92e11fc5d2"
Nov 02 13:23:02.841134 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:23:02.841108    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"626d04a633ea192250c35ed7413e8b11f2df3f058ee9954bdd020c92e11fc5d2\": container with ID starting with 626d04a633ea192250c35ed7413e8b11f2df3f058ee9954bdd020c92e11fc5d2 not found: ID does not exist" containerID="626d04a633ea192250c35ed7413e8b11f2df3f058ee9954bdd020c92e11fc5d2"
Nov 02 13:23:02.841244 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:23:02.841142    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"626d04a633ea192250c35ed7413e8b11f2df3f058ee9954bdd020c92e11fc5d2"} err="failed to get container status \"626d04a633ea192250c35ed7413e8b11f2df3f058ee9954bdd020c92e11fc5d2\": rpc error: code = NotFound desc = could not find container \"626d04a633ea192250c35ed7413e8b11f2df3f058ee9954bdd020c92e11fc5d2\": container with ID starting with 626d04a633ea192250c35ed7413e8b11f2df3f058ee9954bdd020c92e11fc5d2 not found: ID does not exist"
Nov 02 13:23:02.841244 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:23:02.841181    4559 scope.go:122] "RemoveContainer" containerID="f263118547ccecb716514445c970370884eac55bf80594ea71b34d72e20be424"
Nov 02 13:23:02.841481 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:23:02.841460    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f263118547ccecb716514445c970370884eac55bf80594ea71b34d72e20be424\": container with ID starting with f263118547ccecb716514445c970370884eac55bf80594ea71b34d72e20be424 not found: ID does not exist" containerID="f263118547ccecb716514445c970370884eac55bf80594ea71b34d72e20be424"
Nov 02 13:23:02.841530 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:23:02.841487    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f263118547ccecb716514445c970370884eac55bf80594ea71b34d72e20be424"} err="failed to get container status \"f263118547ccecb716514445c970370884eac55bf80594ea71b34d72e20be424\": rpc error: code = NotFound desc = could not find container \"f263118547ccecb716514445c970370884eac55bf80594ea71b34d72e20be424\": container with ID starting with f263118547ccecb716514445c970370884eac55bf80594ea71b34d72e20be424 not found: ID does not exist"
Nov 02 13:23:03.007661 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:23:03.006971    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-ffpz2"]
Nov 02 13:23:03.061672 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:23:03.061017    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-ffpz2"]
Nov 02 13:23:03.894585 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:23:03.894556    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91b0660b-bd2c-4f62-ac8a-e09af747a766" path="/var/lib/kubelet/pods/91b0660b-bd2c-4f62-ac8a-e09af747a766/volumes"
Nov 02 13:23:08.535445 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:23:08.535413    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-bh68g"]
Nov 02 13:23:08.535916 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:23:08.535655    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-bh68g" podUID="76dc13d1-a3a7-4f47-85a2-f596cafbcb94" containerName="istio-proxy" containerID="cri-o://50bad0ef48c7d48a03435a979ec8c43e27270fbdc3a8ecf343ef12ba2da24774" gracePeriod=30
Nov 02 13:23:13.821287 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:23:13.821225    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-bh68g"
Nov 02 13:23:13.897865 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:23:13.897836    4559 generic.go:356] "Generic (PLEG): container finished" podID="76dc13d1-a3a7-4f47-85a2-f596cafbcb94" containerID="50bad0ef48c7d48a03435a979ec8c43e27270fbdc3a8ecf343ef12ba2da24774" exitCode=0
Nov 02 13:23:13.898023 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:23:13.897953    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-bh68g"
Nov 02 13:23:13.905238 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:23:13.905206    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-bh68g" event={"ID":"76dc13d1-a3a7-4f47-85a2-f596cafbcb94","Type":"ContainerDied","Data":"50bad0ef48c7d48a03435a979ec8c43e27270fbdc3a8ecf343ef12ba2da24774"}
Nov 02 13:23:13.905403 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:23:13.905277    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-bh68g" event={"ID":"76dc13d1-a3a7-4f47-85a2-f596cafbcb94","Type":"ContainerDied","Data":"b4f5e2c2fbf6452a916beb60e9c9295ccef3a361952abd380cb2ff42e0bef271"}
Nov 02 13:23:13.905403 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:23:13.905312    4559 scope.go:122] "RemoveContainer" containerID="50bad0ef48c7d48a03435a979ec8c43e27270fbdc3a8ecf343ef12ba2da24774"
Nov 02 13:23:13.918271 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:23:13.918252    4559 scope.go:122] "RemoveContainer" containerID="2f30022edc685d1ad8c8dc1e219cd3e77b94ca787c2b9e95f24e108586b01ed7"
Nov 02 13:23:13.928934 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:23:13.928855    4559 scope.go:122] "RemoveContainer" containerID="50bad0ef48c7d48a03435a979ec8c43e27270fbdc3a8ecf343ef12ba2da24774"
Nov 02 13:23:13.929177 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:23:13.929156    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50bad0ef48c7d48a03435a979ec8c43e27270fbdc3a8ecf343ef12ba2da24774\": container with ID starting with 50bad0ef48c7d48a03435a979ec8c43e27270fbdc3a8ecf343ef12ba2da24774 not found: ID does not exist" containerID="50bad0ef48c7d48a03435a979ec8c43e27270fbdc3a8ecf343ef12ba2da24774"
Nov 02 13:23:13.929267 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:23:13.929188    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50bad0ef48c7d48a03435a979ec8c43e27270fbdc3a8ecf343ef12ba2da24774"} err="failed to get container status \"50bad0ef48c7d48a03435a979ec8c43e27270fbdc3a8ecf343ef12ba2da24774\": rpc error: code = NotFound desc = could not find container \"50bad0ef48c7d48a03435a979ec8c43e27270fbdc3a8ecf343ef12ba2da24774\": container with ID starting with 50bad0ef48c7d48a03435a979ec8c43e27270fbdc3a8ecf343ef12ba2da24774 not found: ID does not exist"
Nov 02 13:23:13.929267 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:23:13.929210    4559 scope.go:122] "RemoveContainer" containerID="2f30022edc685d1ad8c8dc1e219cd3e77b94ca787c2b9e95f24e108586b01ed7"
Nov 02 13:23:13.929552 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:23:13.929513    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f30022edc685d1ad8c8dc1e219cd3e77b94ca787c2b9e95f24e108586b01ed7\": container with ID starting with 2f30022edc685d1ad8c8dc1e219cd3e77b94ca787c2b9e95f24e108586b01ed7 not found: ID does not exist" containerID="2f30022edc685d1ad8c8dc1e219cd3e77b94ca787c2b9e95f24e108586b01ed7"
Nov 02 13:23:13.929624 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:23:13.929562    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f30022edc685d1ad8c8dc1e219cd3e77b94ca787c2b9e95f24e108586b01ed7"} err="failed to get container status \"2f30022edc685d1ad8c8dc1e219cd3e77b94ca787c2b9e95f24e108586b01ed7\": rpc error: code = NotFound desc = could not find container \"2f30022edc685d1ad8c8dc1e219cd3e77b94ca787c2b9e95f24e108586b01ed7\": container with ID starting with 2f30022edc685d1ad8c8dc1e219cd3e77b94ca787c2b9e95f24e108586b01ed7 not found: ID does not exist"
Nov 02 13:23:13.949744 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:23:13.949720    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76dc13d1-a3a7-4f47-85a2-f596cafbcb94-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76dc13d1-a3a7-4f47-85a2-f596cafbcb94-workload-certs\") pod \"76dc13d1-a3a7-4f47-85a2-f596cafbcb94\" (UID: \"76dc13d1-a3a7-4f47-85a2-f596cafbcb94\") "
Nov 02 13:23:13.949932 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:23:13.949914    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76dc13d1-a3a7-4f47-85a2-f596cafbcb94-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76dc13d1-a3a7-4f47-85a2-f596cafbcb94-istio-data\") pod \"76dc13d1-a3a7-4f47-85a2-f596cafbcb94\" (UID: \"76dc13d1-a3a7-4f47-85a2-f596cafbcb94\") "
Nov 02 13:23:13.950040 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:23:13.949956    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76dc13d1-a3a7-4f47-85a2-f596cafbcb94-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76dc13d1-a3a7-4f47-85a2-f596cafbcb94-istio-envoy\") pod \"76dc13d1-a3a7-4f47-85a2-f596cafbcb94\" (UID: \"76dc13d1-a3a7-4f47-85a2-f596cafbcb94\") "
Nov 02 13:23:13.950040 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:23:13.949963    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76dc13d1-a3a7-4f47-85a2-f596cafbcb94-workload-certs" pod "76dc13d1-a3a7-4f47-85a2-f596cafbcb94" (UID: "76dc13d1-a3a7-4f47-85a2-f596cafbcb94"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:23:13.950040 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:23:13.950036    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76dc13d1-a3a7-4f47-85a2-f596cafbcb94-istio-token\" (UniqueName: \"kubernetes.io/projected/76dc13d1-a3a7-4f47-85a2-f596cafbcb94-istio-token\") pod \"76dc13d1-a3a7-4f47-85a2-f596cafbcb94\" (UID: \"76dc13d1-a3a7-4f47-85a2-f596cafbcb94\") "
Nov 02 13:23:13.950223 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:23:13.950065    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76dc13d1-a3a7-4f47-85a2-f596cafbcb94-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76dc13d1-a3a7-4f47-85a2-f596cafbcb94-istiod-ca-cert\") pod \"76dc13d1-a3a7-4f47-85a2-f596cafbcb94\" (UID: \"76dc13d1-a3a7-4f47-85a2-f596cafbcb94\") "
Nov 02 13:23:13.950223 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:23:13.950148    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76dc13d1-a3a7-4f47-85a2-f596cafbcb94-kube-api-access-hfps2\" (UniqueName: \"kubernetes.io/projected/76dc13d1-a3a7-4f47-85a2-f596cafbcb94-kube-api-access-hfps2\") pod \"76dc13d1-a3a7-4f47-85a2-f596cafbcb94\" (UID: \"76dc13d1-a3a7-4f47-85a2-f596cafbcb94\") "
Nov 02 13:23:13.950223 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:23:13.950166    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76dc13d1-a3a7-4f47-85a2-f596cafbcb94-istio-data" pod "76dc13d1-a3a7-4f47-85a2-f596cafbcb94" (UID: "76dc13d1-a3a7-4f47-85a2-f596cafbcb94"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:23:13.950393 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:23:13.950291    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76dc13d1-a3a7-4f47-85a2-f596cafbcb94-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76dc13d1-a3a7-4f47-85a2-f596cafbcb94-credential-socket\") pod \"76dc13d1-a3a7-4f47-85a2-f596cafbcb94\" (UID: \"76dc13d1-a3a7-4f47-85a2-f596cafbcb94\") "
Nov 02 13:23:13.950462 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:23:13.950442    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76dc13d1-a3a7-4f47-85a2-f596cafbcb94-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76dc13d1-a3a7-4f47-85a2-f596cafbcb94-workload-socket\") pod \"76dc13d1-a3a7-4f47-85a2-f596cafbcb94\" (UID: \"76dc13d1-a3a7-4f47-85a2-f596cafbcb94\") "
Nov 02 13:23:13.950520 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:23:13.950482    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/76dc13d1-a3a7-4f47-85a2-f596cafbcb94-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76dc13d1-a3a7-4f47-85a2-f596cafbcb94-istio-podinfo\") pod \"76dc13d1-a3a7-4f47-85a2-f596cafbcb94\" (UID: \"76dc13d1-a3a7-4f47-85a2-f596cafbcb94\") "
Nov 02 13:23:13.950577 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:23:13.950495    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76dc13d1-a3a7-4f47-85a2-f596cafbcb94-istiod-ca-cert" pod "76dc13d1-a3a7-4f47-85a2-f596cafbcb94" (UID: "76dc13d1-a3a7-4f47-85a2-f596cafbcb94"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:23:13.950651 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:23:13.950573    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76dc13d1-a3a7-4f47-85a2-f596cafbcb94-credential-socket" pod "76dc13d1-a3a7-4f47-85a2-f596cafbcb94" (UID: "76dc13d1-a3a7-4f47-85a2-f596cafbcb94"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:23:13.950741 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:23:13.950718    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76dc13d1-a3a7-4f47-85a2-f596cafbcb94-workload-socket" pod "76dc13d1-a3a7-4f47-85a2-f596cafbcb94" (UID: "76dc13d1-a3a7-4f47-85a2-f596cafbcb94"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:23:13.951280 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:23:13.951192    4559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76dc13d1-a3a7-4f47-85a2-f596cafbcb94-credential-socket\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:23:13.951280 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:23:13.951218    4559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76dc13d1-a3a7-4f47-85a2-f596cafbcb94-workload-socket\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:23:13.951280 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:23:13.951233    4559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76dc13d1-a3a7-4f47-85a2-f596cafbcb94-workload-certs\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:23:13.951280 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:23:13.951246    4559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76dc13d1-a3a7-4f47-85a2-f596cafbcb94-istio-data\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:23:13.951280 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:23:13.951259    4559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76dc13d1-a3a7-4f47-85a2-f596cafbcb94-istiod-ca-cert\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:23:13.953060 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:23:13.952995    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/76dc13d1-a3a7-4f47-85a2-f596cafbcb94-istio-podinfo" pod "76dc13d1-a3a7-4f47-85a2-f596cafbcb94" (UID: "76dc13d1-a3a7-4f47-85a2-f596cafbcb94"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 13:23:13.953060 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:23:13.952995    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76dc13d1-a3a7-4f47-85a2-f596cafbcb94-istio-envoy" pod "76dc13d1-a3a7-4f47-85a2-f596cafbcb94" (UID: "76dc13d1-a3a7-4f47-85a2-f596cafbcb94"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:23:13.953060 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:23:13.953028    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76dc13d1-a3a7-4f47-85a2-f596cafbcb94-istio-token" pod "76dc13d1-a3a7-4f47-85a2-f596cafbcb94" (UID: "76dc13d1-a3a7-4f47-85a2-f596cafbcb94"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:23:13.953060 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:23:13.953046    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76dc13d1-a3a7-4f47-85a2-f596cafbcb94-kube-api-access-hfps2" pod "76dc13d1-a3a7-4f47-85a2-f596cafbcb94" (UID: "76dc13d1-a3a7-4f47-85a2-f596cafbcb94"). InnerVolumeSpecName "kube-api-access-hfps2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:23:14.052629 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:23:14.052598    4559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76dc13d1-a3a7-4f47-85a2-f596cafbcb94-istio-envoy\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:23:14.052847 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:23:14.052696    4559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76dc13d1-a3a7-4f47-85a2-f596cafbcb94-istio-token\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:23:14.052847 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:23:14.052713    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hfps2\" (UniqueName: \"kubernetes.io/projected/76dc13d1-a3a7-4f47-85a2-f596cafbcb94-kube-api-access-hfps2\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:23:14.052847 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:23:14.052725    4559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76dc13d1-a3a7-4f47-85a2-f596cafbcb94-istio-podinfo\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:23:14.250619 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:23:14.250588    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-bh68g"]
Nov 02 13:23:14.256149 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:23:14.256123    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-bh68g"]
Nov 02 13:23:15.088852 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:23:15.088816    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqcmj" containerName="extractor"
Nov 02 13:23:15.894707 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:23:15.894671    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76dc13d1-a3a7-4f47-85a2-f596cafbcb94" path="/var/lib/kubelet/pods/76dc13d1-a3a7-4f47-85a2-f596cafbcb94/volumes"
Nov 02 13:23:19.589060 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:23:19.589025    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"]
Nov 02 13:23:45.087414 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:23:45.087379    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqcmj" containerName="extractor"
Nov 02 13:23:55.250226 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:23:55.250164    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"]
Nov 02 13:24:15.087192 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:24:15.087153    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqcmj" containerName="extractor"
Nov 02 13:24:45.086623 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:24:45.086584    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqcmj" containerName="extractor"
Nov 02 13:24:47.627647 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:24:47.627582    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"]
Nov 02 13:24:50.766150 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:24:50.766114    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-b8a8dc4c-openshift-default-699557545-57rbb"]
Nov 02 13:24:50.767550 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:24:50.767520    4559 cpu_manager.go:414] "containerMap: removing container" podUID="91b0660b-bd2c-4f62-ac8a-e09af747a766" containerName="istio-proxy"
Nov 02 13:24:50.767550 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:24:50.767546    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91b0660b-bd2c-4f62-ac8a-e09af747a766" containerName="istio-proxy"
Nov 02 13:24:50.767742 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:24:50.767570    4559 cpu_manager.go:414] "containerMap: removing container" podUID="d5c39b0a-8b09-4c62-87e4-223aad82d163" containerName="maas-api"
Nov 02 13:24:50.767742 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:24:50.767578    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5c39b0a-8b09-4c62-87e4-223aad82d163" containerName="maas-api"
Nov 02 13:24:50.767742 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:24:50.767591    4559 cpu_manager.go:414] "containerMap: removing container" podUID="76dc13d1-a3a7-4f47-85a2-f596cafbcb94" containerName="istio-proxy"
Nov 02 13:24:50.767742 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:24:50.767598    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76dc13d1-a3a7-4f47-85a2-f596cafbcb94" containerName="istio-proxy"
Nov 02 13:24:50.767742 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:24:50.767625    4559 cpu_manager.go:414] "containerMap: removing container" podUID="85ec8b4f-746f-4d0c-9560-7a79f0c64ef6" containerName="istio-proxy"
Nov 02 13:24:50.767742 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:24:50.767632    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85ec8b4f-746f-4d0c-9560-7a79f0c64ef6" containerName="istio-proxy"
Nov 02 13:24:50.767742 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:24:50.767672    4559 cpu_manager.go:414] "containerMap: removing container" podUID="5ae3a83d-c3a4-4305-8824-1cde1a6172c0" containerName="istio-proxy"
Nov 02 13:24:50.767742 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:24:50.767679    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ae3a83d-c3a4-4305-8824-1cde1a6172c0" containerName="istio-proxy"
Nov 02 13:24:50.767742 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:24:50.767689    4559 cpu_manager.go:414] "containerMap: removing container" podUID="27dd21d8-102d-45c3-8e54-3c1e846829de" containerName="istio-proxy"
Nov 02 13:24:50.767742 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:24:50.767696    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27dd21d8-102d-45c3-8e54-3c1e846829de" containerName="istio-proxy"
Nov 02 13:24:50.768073 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:24:50.767834    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="91b0660b-bd2c-4f62-ac8a-e09af747a766" containerName="istio-proxy"
Nov 02 13:24:50.768073 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:24:50.767846    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="85ec8b4f-746f-4d0c-9560-7a79f0c64ef6" containerName="istio-proxy"
Nov 02 13:24:50.768073 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:24:50.767858    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5c39b0a-8b09-4c62-87e4-223aad82d163" containerName="maas-api"
Nov 02 13:24:50.768073 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:24:50.767869    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="76dc13d1-a3a7-4f47-85a2-f596cafbcb94" containerName="istio-proxy"
Nov 02 13:24:50.768073 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:24:50.767881    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="76dc13d1-a3a7-4f47-85a2-f596cafbcb94" containerName="istio-proxy"
Nov 02 13:24:50.768073 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:24:50.767890    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ae3a83d-c3a4-4305-8824-1cde1a6172c0" containerName="istio-proxy"
Nov 02 13:24:50.773186 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:24:50.772898    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b8a8dc4c-openshift-default-699557545-57rbb"
Nov 02 13:24:50.801564 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:24:50.801535    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-b8a8dc4c-openshift-default-dockercfg-9s8k7\""
Nov 02 13:24:50.811579 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:24:50.811544    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-b8a8dc4c-openshift-default-699557545-57rbb"]
Nov 02 13:24:50.861677 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:24:50.861617    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fff161f9-6bfb-4c48-bc75-ade5639c076b-istiod-ca-cert\") pod \"e2e-mt-b8a8dc4c-openshift-default-699557545-57rbb\" (UID: \"fff161f9-6bfb-4c48-bc75-ade5639c076b\") " pod="openshift-ingress/e2e-mt-b8a8dc4c-openshift-default-699557545-57rbb"
Nov 02 13:24:50.861908 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:24:50.861888    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fff161f9-6bfb-4c48-bc75-ade5639c076b-istio-envoy\") pod \"e2e-mt-b8a8dc4c-openshift-default-699557545-57rbb\" (UID: \"fff161f9-6bfb-4c48-bc75-ade5639c076b\") " pod="openshift-ingress/e2e-mt-b8a8dc4c-openshift-default-699557545-57rbb"
Nov 02 13:24:50.862032 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:24:50.862012    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fff161f9-6bfb-4c48-bc75-ade5639c076b-workload-socket\") pod \"e2e-mt-b8a8dc4c-openshift-default-699557545-57rbb\" (UID: \"fff161f9-6bfb-4c48-bc75-ade5639c076b\") " pod="openshift-ingress/e2e-mt-b8a8dc4c-openshift-default-699557545-57rbb"
Nov 02 13:24:50.862180 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:24:50.862058    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fff161f9-6bfb-4c48-bc75-ade5639c076b-istio-token\") pod \"e2e-mt-b8a8dc4c-openshift-default-699557545-57rbb\" (UID: \"fff161f9-6bfb-4c48-bc75-ade5639c076b\") " pod="openshift-ingress/e2e-mt-b8a8dc4c-openshift-default-699557545-57rbb"
Nov 02 13:24:50.862180 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:24:50.862091    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fff161f9-6bfb-4c48-bc75-ade5639c076b-istio-podinfo\") pod \"e2e-mt-b8a8dc4c-openshift-default-699557545-57rbb\" (UID: \"fff161f9-6bfb-4c48-bc75-ade5639c076b\") " pod="openshift-ingress/e2e-mt-b8a8dc4c-openshift-default-699557545-57rbb"
Nov 02 13:24:50.862315 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:24:50.862216    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fff161f9-6bfb-4c48-bc75-ade5639c076b-workload-certs\") pod \"e2e-mt-b8a8dc4c-openshift-default-699557545-57rbb\" (UID: \"fff161f9-6bfb-4c48-bc75-ade5639c076b\") " pod="openshift-ingress/e2e-mt-b8a8dc4c-openshift-default-699557545-57rbb"
Nov 02 13:24:50.862315 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:24:50.862298    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fff161f9-6bfb-4c48-bc75-ade5639c076b-credential-socket\") pod \"e2e-mt-b8a8dc4c-openshift-default-699557545-57rbb\" (UID: \"fff161f9-6bfb-4c48-bc75-ade5639c076b\") " pod="openshift-ingress/e2e-mt-b8a8dc4c-openshift-default-699557545-57rbb"
Nov 02 13:24:50.862426 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:24:50.862362    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fff161f9-6bfb-4c48-bc75-ade5639c076b-istio-data\") pod \"e2e-mt-b8a8dc4c-openshift-default-699557545-57rbb\" (UID: \"fff161f9-6bfb-4c48-bc75-ade5639c076b\") " pod="openshift-ingress/e2e-mt-b8a8dc4c-openshift-default-699557545-57rbb"
Nov 02 13:24:50.863242 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:24:50.863215    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qw8j\" (UniqueName: \"kubernetes.io/projected/fff161f9-6bfb-4c48-bc75-ade5639c076b-kube-api-access-5qw8j\") pod \"e2e-mt-b8a8dc4c-openshift-default-699557545-57rbb\" (UID: \"fff161f9-6bfb-4c48-bc75-ade5639c076b\") " pod="openshift-ingress/e2e-mt-b8a8dc4c-openshift-default-699557545-57rbb"
Nov 02 13:24:50.964467 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:24:50.964427    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fff161f9-6bfb-4c48-bc75-ade5639c076b-istio-data\") pod \"e2e-mt-b8a8dc4c-openshift-default-699557545-57rbb\" (UID: \"fff161f9-6bfb-4c48-bc75-ade5639c076b\") " pod="openshift-ingress/e2e-mt-b8a8dc4c-openshift-default-699557545-57rbb"
Nov 02 13:24:50.965328 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:24:50.965304    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qw8j\" (UniqueName: \"kubernetes.io/projected/fff161f9-6bfb-4c48-bc75-ade5639c076b-kube-api-access-5qw8j\") pod \"e2e-mt-b8a8dc4c-openshift-default-699557545-57rbb\" (UID: \"fff161f9-6bfb-4c48-bc75-ade5639c076b\") " pod="openshift-ingress/e2e-mt-b8a8dc4c-openshift-default-699557545-57rbb"
Nov 02 13:24:50.965615 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:24:50.965597    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fff161f9-6bfb-4c48-bc75-ade5639c076b-istiod-ca-cert\") pod \"e2e-mt-b8a8dc4c-openshift-default-699557545-57rbb\" (UID: \"fff161f9-6bfb-4c48-bc75-ade5639c076b\") " pod="openshift-ingress/e2e-mt-b8a8dc4c-openshift-default-699557545-57rbb"
Nov 02 13:24:50.967014 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:24:50.966994    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fff161f9-6bfb-4c48-bc75-ade5639c076b-istio-envoy\") pod \"e2e-mt-b8a8dc4c-openshift-default-699557545-57rbb\" (UID: \"fff161f9-6bfb-4c48-bc75-ade5639c076b\") " pod="openshift-ingress/e2e-mt-b8a8dc4c-openshift-default-699557545-57rbb"
Nov 02 13:24:50.967700 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:24:50.967680    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fff161f9-6bfb-4c48-bc75-ade5639c076b-workload-socket\") pod \"e2e-mt-b8a8dc4c-openshift-default-699557545-57rbb\" (UID: \"fff161f9-6bfb-4c48-bc75-ade5639c076b\") " pod="openshift-ingress/e2e-mt-b8a8dc4c-openshift-default-699557545-57rbb"
Nov 02 13:24:50.967879 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:24:50.967860    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fff161f9-6bfb-4c48-bc75-ade5639c076b-istio-token\") pod \"e2e-mt-b8a8dc4c-openshift-default-699557545-57rbb\" (UID: \"fff161f9-6bfb-4c48-bc75-ade5639c076b\") " pod="openshift-ingress/e2e-mt-b8a8dc4c-openshift-default-699557545-57rbb"
Nov 02 13:24:50.985194 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:24:50.985169    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fff161f9-6bfb-4c48-bc75-ade5639c076b-istio-podinfo\") pod \"e2e-mt-b8a8dc4c-openshift-default-699557545-57rbb\" (UID: \"fff161f9-6bfb-4c48-bc75-ade5639c076b\") " pod="openshift-ingress/e2e-mt-b8a8dc4c-openshift-default-699557545-57rbb"
Nov 02 13:24:50.985419 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:24:50.985402    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fff161f9-6bfb-4c48-bc75-ade5639c076b-workload-certs\") pod \"e2e-mt-b8a8dc4c-openshift-default-699557545-57rbb\" (UID: \"fff161f9-6bfb-4c48-bc75-ade5639c076b\") " pod="openshift-ingress/e2e-mt-b8a8dc4c-openshift-default-699557545-57rbb"
Nov 02 13:24:50.985619 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:24:50.985603    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fff161f9-6bfb-4c48-bc75-ade5639c076b-credential-socket\") pod \"e2e-mt-b8a8dc4c-openshift-default-699557545-57rbb\" (UID: \"fff161f9-6bfb-4c48-bc75-ade5639c076b\") " pod="openshift-ingress/e2e-mt-b8a8dc4c-openshift-default-699557545-57rbb"
Nov 02 13:24:51.004710 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:24:50.966900    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fff161f9-6bfb-4c48-bc75-ade5639c076b-istiod-ca-cert\") pod \"e2e-mt-b8a8dc4c-openshift-default-699557545-57rbb\" (UID: \"fff161f9-6bfb-4c48-bc75-ade5639c076b\") " pod="openshift-ingress/e2e-mt-b8a8dc4c-openshift-default-699557545-57rbb"
Nov 02 13:24:51.004710 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:24:50.995027    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fff161f9-6bfb-4c48-bc75-ade5639c076b-workload-certs\") pod \"e2e-mt-b8a8dc4c-openshift-default-699557545-57rbb\" (UID: \"fff161f9-6bfb-4c48-bc75-ade5639c076b\") " pod="openshift-ingress/e2e-mt-b8a8dc4c-openshift-default-699557545-57rbb"
Nov 02 13:24:51.004710 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:24:50.965092    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fff161f9-6bfb-4c48-bc75-ade5639c076b-istio-data\") pod \"e2e-mt-b8a8dc4c-openshift-default-699557545-57rbb\" (UID: \"fff161f9-6bfb-4c48-bc75-ade5639c076b\") " pod="openshift-ingress/e2e-mt-b8a8dc4c-openshift-default-699557545-57rbb"
Nov 02 13:24:51.004710 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:24:50.968913    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fff161f9-6bfb-4c48-bc75-ade5639c076b-workload-socket\") pod \"e2e-mt-b8a8dc4c-openshift-default-699557545-57rbb\" (UID: \"fff161f9-6bfb-4c48-bc75-ade5639c076b\") " pod="openshift-ingress/e2e-mt-b8a8dc4c-openshift-default-699557545-57rbb"
Nov 02 13:24:51.004710 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:24:50.995681    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fff161f9-6bfb-4c48-bc75-ade5639c076b-credential-socket\") pod \"e2e-mt-b8a8dc4c-openshift-default-699557545-57rbb\" (UID: \"fff161f9-6bfb-4c48-bc75-ade5639c076b\") " pod="openshift-ingress/e2e-mt-b8a8dc4c-openshift-default-699557545-57rbb"
Nov 02 13:24:51.006370 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:24:51.006347    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fff161f9-6bfb-4c48-bc75-ade5639c076b-istio-envoy\") pod \"e2e-mt-b8a8dc4c-openshift-default-699557545-57rbb\" (UID: \"fff161f9-6bfb-4c48-bc75-ade5639c076b\") " pod="openshift-ingress/e2e-mt-b8a8dc4c-openshift-default-699557545-57rbb"
Nov 02 13:24:51.013321 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:24:51.013295    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fff161f9-6bfb-4c48-bc75-ade5639c076b-istio-podinfo\") pod \"e2e-mt-b8a8dc4c-openshift-default-699557545-57rbb\" (UID: \"fff161f9-6bfb-4c48-bc75-ade5639c076b\") " pod="openshift-ingress/e2e-mt-b8a8dc4c-openshift-default-699557545-57rbb"
Nov 02 13:24:51.023453 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:24:51.023368    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qw8j\" (UniqueName: \"kubernetes.io/projected/fff161f9-6bfb-4c48-bc75-ade5639c076b-kube-api-access-5qw8j\") pod \"e2e-mt-b8a8dc4c-openshift-default-699557545-57rbb\" (UID: \"fff161f9-6bfb-4c48-bc75-ade5639c076b\") " pod="openshift-ingress/e2e-mt-b8a8dc4c-openshift-default-699557545-57rbb"
Nov 02 13:24:51.035130 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:24:51.035102    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fff161f9-6bfb-4c48-bc75-ade5639c076b-istio-token\") pod \"e2e-mt-b8a8dc4c-openshift-default-699557545-57rbb\" (UID: \"fff161f9-6bfb-4c48-bc75-ade5639c076b\") " pod="openshift-ingress/e2e-mt-b8a8dc4c-openshift-default-699557545-57rbb"
Nov 02 13:24:51.105211 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:24:51.105162    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b8a8dc4c-openshift-default-699557545-57rbb"
Nov 02 13:24:51.455129 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:24:51.455095    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-b8a8dc4c-openshift-default-699557545-57rbb"]
Nov 02 13:24:51.460755 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:24:51.460718    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 13:24:51.460853 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:24:51.460792    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 13:24:51.460853 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:24:51.460822    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 13:24:51.705885 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:24:51.705777    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b8a8dc4c-openshift-default-699557545-57rbb" event={"ID":"fff161f9-6bfb-4c48-bc75-ade5639c076b","Type":"ContainerStarted","Data":"1e2e7dacc8266b31ed051a83120561e72f02173086999afda21e17815be14972"}
Nov 02 13:24:51.705885 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:24:51.705847    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b8a8dc4c-openshift-default-699557545-57rbb" event={"ID":"fff161f9-6bfb-4c48-bc75-ade5639c076b","Type":"ContainerStarted","Data":"24bc3e16b6c32198e09db76b6db1827988baa4784ea7c222d71078aaa3c06979"}
Nov 02 13:24:51.756633 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:24:51.756587    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-b8a8dc4c-openshift-default-699557545-57rbb" podStartSLOduration=1.756572362 podStartE2EDuration="1.756572362s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:24:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:24:51.752699071 +0000 UTC m=+3488.008098169" watchObservedRunningTime="2026-11-02 13:24:51.756572362 +0000 UTC m=+3488.011971457"
Nov 02 13:24:52.106400 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:24:52.106319    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-b8a8dc4c-openshift-default-699557545-57rbb"
Nov 02 13:24:52.112519 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:24:52.112495    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-b8a8dc4c-openshift-default-699557545-57rbb"
Nov 02 13:24:52.712946 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:24:52.712913    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-b8a8dc4c-openshift-default-699557545-57rbb"
Nov 02 13:24:52.713855 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:24:52.713833    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-b8a8dc4c-openshift-default-699557545-57rbb"
Nov 02 13:25:15.087539 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:25:15.087500    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqcmj" containerName="extractor"
Nov 02 13:25:24.335665 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:25:24.324784    4559 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-ckslt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.223:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 13:25:24.335665 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:25:24.329658    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ckslt" podUID="826b92f5-161c-4f6a-9055-d0c838c9ce09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.223:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:25:24.371664 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:25:24.360133    4559 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-xj4xk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.222:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 13:25:24.371664 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:25:24.360204    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-xj4xk" podUID="d2c5f43a-3a0c-444b-afc1-e2bf090dc05c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.222:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:25:24.483674 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:25:24.482001    4559 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-7kvjr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.250:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 13:25:24.483674 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:25:24.482088    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7kvjr" podUID="b736e009-d002-4177-8f79-8f50cd5325f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.250:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:25:40.494666 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:25:40.494618    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b8a8dc4c-6fdcf4f98f-944jd"]
Nov 02 13:25:40.495316 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:25:40.495302    4559 cpu_manager.go:414] "containerMap: removing container" podUID="91b0660b-bd2c-4f62-ac8a-e09af747a766" containerName="istio-proxy"
Nov 02 13:25:40.495362 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:25:40.495317    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91b0660b-bd2c-4f62-ac8a-e09af747a766" containerName="istio-proxy"
Nov 02 13:25:40.495362 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:25:40.495332    4559 cpu_manager.go:414] "containerMap: removing container" podUID="76dc13d1-a3a7-4f47-85a2-f596cafbcb94" containerName="istio-proxy"
Nov 02 13:25:40.495362 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:25:40.495337    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76dc13d1-a3a7-4f47-85a2-f596cafbcb94" containerName="istio-proxy"
Nov 02 13:25:40.495467 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:25:40.495443    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="91b0660b-bd2c-4f62-ac8a-e09af747a766" containerName="istio-proxy"
Nov 02 13:25:40.499451 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:25:40.499435    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b8a8dc4c-6fdcf4f98f-944jd"
Nov 02 13:25:40.525460 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:25:40.525431    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-b8a8dc4c-serving-cert\""
Nov 02 13:25:40.533908 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:25:40.533875    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b8a8dc4c-6fdcf4f98f-944jd"]
Nov 02 13:25:40.615483 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:25:40.615456    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/41be38ac-3172-4e6b-af34-c83c9179b69f-metrics-certs\") pod \"maas-api-e2e-mt-b8a8dc4c-6fdcf4f98f-944jd\" (UID: \"41be38ac-3172-4e6b-af34-c83c9179b69f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b8a8dc4c-6fdcf4f98f-944jd"
Nov 02 13:25:40.615664 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:25:40.615598    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/41be38ac-3172-4e6b-af34-c83c9179b69f-openshift-service-ca\") pod \"maas-api-e2e-mt-b8a8dc4c-6fdcf4f98f-944jd\" (UID: \"41be38ac-3172-4e6b-af34-c83c9179b69f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b8a8dc4c-6fdcf4f98f-944jd"
Nov 02 13:25:40.615664 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:25:40.615651    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g67sm\" (UniqueName: \"kubernetes.io/projected/41be38ac-3172-4e6b-af34-c83c9179b69f-kube-api-access-g67sm\") pod \"maas-api-e2e-mt-b8a8dc4c-6fdcf4f98f-944jd\" (UID: \"41be38ac-3172-4e6b-af34-c83c9179b69f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b8a8dc4c-6fdcf4f98f-944jd"
Nov 02 13:25:40.615869 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:25:40.615850    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/41be38ac-3172-4e6b-af34-c83c9179b69f-maas-api-tls\") pod \"maas-api-e2e-mt-b8a8dc4c-6fdcf4f98f-944jd\" (UID: \"41be38ac-3172-4e6b-af34-c83c9179b69f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b8a8dc4c-6fdcf4f98f-944jd"
Nov 02 13:25:40.721665 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:25:40.718697    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/41be38ac-3172-4e6b-af34-c83c9179b69f-metrics-certs\") pod \"maas-api-e2e-mt-b8a8dc4c-6fdcf4f98f-944jd\" (UID: \"41be38ac-3172-4e6b-af34-c83c9179b69f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b8a8dc4c-6fdcf4f98f-944jd"
Nov 02 13:25:40.730676 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:25:40.727846    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/41be38ac-3172-4e6b-af34-c83c9179b69f-openshift-service-ca\") pod \"maas-api-e2e-mt-b8a8dc4c-6fdcf4f98f-944jd\" (UID: \"41be38ac-3172-4e6b-af34-c83c9179b69f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b8a8dc4c-6fdcf4f98f-944jd"
Nov 02 13:25:40.730676 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:25:40.727928    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g67sm\" (UniqueName: \"kubernetes.io/projected/41be38ac-3172-4e6b-af34-c83c9179b69f-kube-api-access-g67sm\") pod \"maas-api-e2e-mt-b8a8dc4c-6fdcf4f98f-944jd\" (UID: \"41be38ac-3172-4e6b-af34-c83c9179b69f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b8a8dc4c-6fdcf4f98f-944jd"
Nov 02 13:25:40.730676 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:25:40.728179    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/41be38ac-3172-4e6b-af34-c83c9179b69f-maas-api-tls\") pod \"maas-api-e2e-mt-b8a8dc4c-6fdcf4f98f-944jd\" (UID: \"41be38ac-3172-4e6b-af34-c83c9179b69f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b8a8dc4c-6fdcf4f98f-944jd"
Nov 02 13:25:40.730676 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:25:40.728482    4559 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-b8a8dc4c-serving-cert: secret "maas-api-e2e-mt-b8a8dc4c-serving-cert" not found
Nov 02 13:25:40.730676 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:25:40.728553    4559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/41be38ac-3172-4e6b-af34-c83c9179b69f-maas-api-tls podName:41be38ac-3172-4e6b-af34-c83c9179b69f nodeName:}" failed. No retries permitted until 2026-11-02 13:25:41.22853143 +0000 UTC m=+3537.483930507 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/41be38ac-3172-4e6b-af34-c83c9179b69f-maas-api-tls") pod "maas-api-e2e-mt-b8a8dc4c-6fdcf4f98f-944jd" (UID: "41be38ac-3172-4e6b-af34-c83c9179b69f") : secret "maas-api-e2e-mt-b8a8dc4c-serving-cert" not found
Nov 02 13:25:40.733688 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:25:40.733521    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/41be38ac-3172-4e6b-af34-c83c9179b69f-openshift-service-ca\") pod \"maas-api-e2e-mt-b8a8dc4c-6fdcf4f98f-944jd\" (UID: \"41be38ac-3172-4e6b-af34-c83c9179b69f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b8a8dc4c-6fdcf4f98f-944jd"
Nov 02 13:25:40.740938 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:25:40.740909    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/41be38ac-3172-4e6b-af34-c83c9179b69f-metrics-certs\") pod \"maas-api-e2e-mt-b8a8dc4c-6fdcf4f98f-944jd\" (UID: \"41be38ac-3172-4e6b-af34-c83c9179b69f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b8a8dc4c-6fdcf4f98f-944jd"
Nov 02 13:25:40.758037 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:25:40.757977    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g67sm\" (UniqueName: \"kubernetes.io/projected/41be38ac-3172-4e6b-af34-c83c9179b69f-kube-api-access-g67sm\") pod \"maas-api-e2e-mt-b8a8dc4c-6fdcf4f98f-944jd\" (UID: \"41be38ac-3172-4e6b-af34-c83c9179b69f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b8a8dc4c-6fdcf4f98f-944jd"
Nov 02 13:25:41.234349 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:25:41.234314    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/41be38ac-3172-4e6b-af34-c83c9179b69f-maas-api-tls\") pod \"maas-api-e2e-mt-b8a8dc4c-6fdcf4f98f-944jd\" (UID: \"41be38ac-3172-4e6b-af34-c83c9179b69f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b8a8dc4c-6fdcf4f98f-944jd"
Nov 02 13:25:41.236811 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:25:41.236789    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/41be38ac-3172-4e6b-af34-c83c9179b69f-maas-api-tls\") pod \"maas-api-e2e-mt-b8a8dc4c-6fdcf4f98f-944jd\" (UID: \"41be38ac-3172-4e6b-af34-c83c9179b69f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b8a8dc4c-6fdcf4f98f-944jd"
Nov 02 13:25:41.377004 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:25:41.376968    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"]
Nov 02 13:25:41.413935 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:25:41.413910    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b8a8dc4c-6fdcf4f98f-944jd"
Nov 02 13:25:41.736259 ip-10-0-128-132 kubenswrapper[4559]: W1102 13:25:41.736214    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41be38ac_3172_4e6b_af34_c83c9179b69f.slice/crio-7389e097912b3dc734e36a8da70554e34b72258911bbf72ab06d0f88f85c731a WatchSource:0}: Error finding container 7389e097912b3dc734e36a8da70554e34b72258911bbf72ab06d0f88f85c731a: Status 404 returned error can't find the container with id 7389e097912b3dc734e36a8da70554e34b72258911bbf72ab06d0f88f85c731a
Nov 02 13:25:41.737243 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:25:41.737194    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b8a8dc4c-6fdcf4f98f-944jd"]
Nov 02 13:25:41.738470 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:25:41.738449    4559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 13:25:42.245384 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:25:42.245345    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b8a8dc4c-6fdcf4f98f-944jd" event={"ID":"41be38ac-3172-4e6b-af34-c83c9179b69f","Type":"ContainerStarted","Data":"7389e097912b3dc734e36a8da70554e34b72258911bbf72ab06d0f88f85c731a"}
Nov 02 13:25:43.253516 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:25:43.253477    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b8a8dc4c-6fdcf4f98f-944jd" event={"ID":"41be38ac-3172-4e6b-af34-c83c9179b69f","Type":"ContainerStarted","Data":"525298d5b763371a3965062582601020325a91965aea6280245d38b5d88c0458"}
Nov 02 13:25:43.253933 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:25:43.253580    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b8a8dc4c-6fdcf4f98f-944jd"
Nov 02 13:25:43.297299 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:25:43.297249    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b8a8dc4c-6fdcf4f98f-944jd" podStartSLOduration=2.772176951 podStartE2EDuration="3.297234193s" totalImagesPullingTime="525.057242ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:25:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:25:43.296865283 +0000 UTC m=+3539.552264375" watchObservedRunningTime="2026-11-02 13:25:43.297234193 +0000 UTC m=+3539.552633288"
Nov 02 13:25:45.086600 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:25:45.086562    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqcmj" containerName="extractor"
Nov 02 13:25:49.267249 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:25:49.267221    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b8a8dc4c-6fdcf4f98f-944jd"
Nov 02 13:25:50.988714 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:25:50.988680    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-bb4qf"]
Nov 02 13:25:50.989117 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:25:50.988895    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-bb4qf" podUID="dae5c89b-c8f9-46e6-9ca9-5444d1cb9269" containerName="istio-proxy" containerID="cri-o://4a13ec4d340bcfa618bd48ffa445df3e5a11ce3a54b3e020d57c6e445e7aeb03" gracePeriod=30
Nov 02 13:25:54.050777 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:25:54.050732    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-ef5efe-openshift-default-679746c7d9-x5qtc"]
Nov 02 13:25:54.051176 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:25:54.050950    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-ef5efe-openshift-default-679746c7d9-x5qtc" podUID="2fd94eba-ab06-4d75-a63b-e31bca0326d6" containerName="istio-proxy" containerID="cri-o://712549f074112340d65eb3ef1b661beb6e934332b2dbc804bc4c71d14cd447ca" gracePeriod=30
Nov 02 13:25:56.264660 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:25:56.264584    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-bb4qf"
Nov 02 13:25:56.346093 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:25:56.346055    4559 generic.go:356] "Generic (PLEG): container finished" podID="dae5c89b-c8f9-46e6-9ca9-5444d1cb9269" containerID="4a13ec4d340bcfa618bd48ffa445df3e5a11ce3a54b3e020d57c6e445e7aeb03" exitCode=0
Nov 02 13:25:56.346301 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:25:56.346122    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-bb4qf" event={"ID":"dae5c89b-c8f9-46e6-9ca9-5444d1cb9269","Type":"ContainerDied","Data":"4a13ec4d340bcfa618bd48ffa445df3e5a11ce3a54b3e020d57c6e445e7aeb03"}
Nov 02 13:25:56.346301 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:25:56.346156    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-bb4qf"
Nov 02 13:25:56.346301 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:25:56.346174    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-bb4qf" event={"ID":"dae5c89b-c8f9-46e6-9ca9-5444d1cb9269","Type":"ContainerDied","Data":"518819b8e9c9eefd171e0325b16b1141fc50864b786febe2471ace99739ab374"}
Nov 02 13:25:56.346301 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:25:56.346197    4559 scope.go:122] "RemoveContainer" containerID="4a13ec4d340bcfa618bd48ffa445df3e5a11ce3a54b3e020d57c6e445e7aeb03"
Nov 02 13:25:56.360584 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:25:56.360557    4559 scope.go:122] "RemoveContainer" containerID="4a13ec4d340bcfa618bd48ffa445df3e5a11ce3a54b3e020d57c6e445e7aeb03"
Nov 02 13:25:56.360922 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:25:56.360902    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a13ec4d340bcfa618bd48ffa445df3e5a11ce3a54b3e020d57c6e445e7aeb03\": container with ID starting with 4a13ec4d340bcfa618bd48ffa445df3e5a11ce3a54b3e020d57c6e445e7aeb03 not found: ID does not exist" containerID="4a13ec4d340bcfa618bd48ffa445df3e5a11ce3a54b3e020d57c6e445e7aeb03"
Nov 02 13:25:56.360997 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:25:56.360931    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a13ec4d340bcfa618bd48ffa445df3e5a11ce3a54b3e020d57c6e445e7aeb03"} err="failed to get container status \"4a13ec4d340bcfa618bd48ffa445df3e5a11ce3a54b3e020d57c6e445e7aeb03\": rpc error: code = NotFound desc = could not find container \"4a13ec4d340bcfa618bd48ffa445df3e5a11ce3a54b3e020d57c6e445e7aeb03\": container with ID starting with 4a13ec4d340bcfa618bd48ffa445df3e5a11ce3a54b3e020d57c6e445e7aeb03 not found: ID does not exist"
Nov 02 13:25:56.364605 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:25:56.364578    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dae5c89b-c8f9-46e6-9ca9-5444d1cb9269-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dae5c89b-c8f9-46e6-9ca9-5444d1cb9269-istio-envoy\") pod \"dae5c89b-c8f9-46e6-9ca9-5444d1cb9269\" (UID: \"dae5c89b-c8f9-46e6-9ca9-5444d1cb9269\") "
Nov 02 13:25:56.364773 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:25:56.364675    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dae5c89b-c8f9-46e6-9ca9-5444d1cb9269-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dae5c89b-c8f9-46e6-9ca9-5444d1cb9269-istio-podinfo\") pod \"dae5c89b-c8f9-46e6-9ca9-5444d1cb9269\" (UID: \"dae5c89b-c8f9-46e6-9ca9-5444d1cb9269\") "
Nov 02 13:25:56.364855 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:25:56.364799    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dae5c89b-c8f9-46e6-9ca9-5444d1cb9269-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dae5c89b-c8f9-46e6-9ca9-5444d1cb9269-istio-data\") pod \"dae5c89b-c8f9-46e6-9ca9-5444d1cb9269\" (UID: \"dae5c89b-c8f9-46e6-9ca9-5444d1cb9269\") "
Nov 02 13:25:56.364855 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:25:56.364848    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dae5c89b-c8f9-46e6-9ca9-5444d1cb9269-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dae5c89b-c8f9-46e6-9ca9-5444d1cb9269-workload-certs\") pod \"dae5c89b-c8f9-46e6-9ca9-5444d1cb9269\" (UID: \"dae5c89b-c8f9-46e6-9ca9-5444d1cb9269\") "
Nov 02 13:25:56.365321 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:25:56.364910    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dae5c89b-c8f9-46e6-9ca9-5444d1cb9269-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dae5c89b-c8f9-46e6-9ca9-5444d1cb9269-workload-socket\") pod \"dae5c89b-c8f9-46e6-9ca9-5444d1cb9269\" (UID: \"dae5c89b-c8f9-46e6-9ca9-5444d1cb9269\") "
Nov 02 13:25:56.365321 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:25:56.365036    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dae5c89b-c8f9-46e6-9ca9-5444d1cb9269-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dae5c89b-c8f9-46e6-9ca9-5444d1cb9269-istiod-ca-cert\") pod \"dae5c89b-c8f9-46e6-9ca9-5444d1cb9269\" (UID: \"dae5c89b-c8f9-46e6-9ca9-5444d1cb9269\") "
Nov 02 13:25:56.365321 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:25:56.365065    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dae5c89b-c8f9-46e6-9ca9-5444d1cb9269-istio-token\" (UniqueName: \"kubernetes.io/projected/dae5c89b-c8f9-46e6-9ca9-5444d1cb9269-istio-token\") pod \"dae5c89b-c8f9-46e6-9ca9-5444d1cb9269\" (UID: \"dae5c89b-c8f9-46e6-9ca9-5444d1cb9269\") "
Nov 02 13:25:56.365321 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:25:56.365102    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dae5c89b-c8f9-46e6-9ca9-5444d1cb9269-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dae5c89b-c8f9-46e6-9ca9-5444d1cb9269-credential-socket\") pod \"dae5c89b-c8f9-46e6-9ca9-5444d1cb9269\" (UID: \"dae5c89b-c8f9-46e6-9ca9-5444d1cb9269\") "
Nov 02 13:25:56.365321 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:25:56.365241    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dae5c89b-c8f9-46e6-9ca9-5444d1cb9269-kube-api-access-cd8gw\" (UniqueName: \"kubernetes.io/projected/dae5c89b-c8f9-46e6-9ca9-5444d1cb9269-kube-api-access-cd8gw\") pod \"dae5c89b-c8f9-46e6-9ca9-5444d1cb9269\" (UID: \"dae5c89b-c8f9-46e6-9ca9-5444d1cb9269\") "
Nov 02 13:25:56.365321 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:25:56.365280    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dae5c89b-c8f9-46e6-9ca9-5444d1cb9269-istio-data" pod "dae5c89b-c8f9-46e6-9ca9-5444d1cb9269" (UID: "dae5c89b-c8f9-46e6-9ca9-5444d1cb9269"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:25:56.365998 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:25:56.365973    4559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dae5c89b-c8f9-46e6-9ca9-5444d1cb9269-istio-data\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:25:56.366090 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:25:56.366010    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dae5c89b-c8f9-46e6-9ca9-5444d1cb9269-workload-socket" pod "dae5c89b-c8f9-46e6-9ca9-5444d1cb9269" (UID: "dae5c89b-c8f9-46e6-9ca9-5444d1cb9269"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:25:56.366308 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:25:56.366284    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dae5c89b-c8f9-46e6-9ca9-5444d1cb9269-credential-socket" pod "dae5c89b-c8f9-46e6-9ca9-5444d1cb9269" (UID: "dae5c89b-c8f9-46e6-9ca9-5444d1cb9269"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:25:56.366912 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:25:56.366714    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dae5c89b-c8f9-46e6-9ca9-5444d1cb9269-workload-certs" pod "dae5c89b-c8f9-46e6-9ca9-5444d1cb9269" (UID: "dae5c89b-c8f9-46e6-9ca9-5444d1cb9269"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:25:56.366912 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:25:56.366881    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dae5c89b-c8f9-46e6-9ca9-5444d1cb9269-istiod-ca-cert" pod "dae5c89b-c8f9-46e6-9ca9-5444d1cb9269" (UID: "dae5c89b-c8f9-46e6-9ca9-5444d1cb9269"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:25:56.368092 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:25:56.368071    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dae5c89b-c8f9-46e6-9ca9-5444d1cb9269-istio-envoy" pod "dae5c89b-c8f9-46e6-9ca9-5444d1cb9269" (UID: "dae5c89b-c8f9-46e6-9ca9-5444d1cb9269"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:25:56.368627 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:25:56.368596    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dae5c89b-c8f9-46e6-9ca9-5444d1cb9269-istio-podinfo" pod "dae5c89b-c8f9-46e6-9ca9-5444d1cb9269" (UID: "dae5c89b-c8f9-46e6-9ca9-5444d1cb9269"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 13:25:56.368866 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:25:56.368840    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dae5c89b-c8f9-46e6-9ca9-5444d1cb9269-kube-api-access-cd8gw" pod "dae5c89b-c8f9-46e6-9ca9-5444d1cb9269" (UID: "dae5c89b-c8f9-46e6-9ca9-5444d1cb9269"). InnerVolumeSpecName "kube-api-access-cd8gw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:25:56.368959 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:25:56.368917    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dae5c89b-c8f9-46e6-9ca9-5444d1cb9269-istio-token" pod "dae5c89b-c8f9-46e6-9ca9-5444d1cb9269" (UID: "dae5c89b-c8f9-46e6-9ca9-5444d1cb9269"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:25:56.467427 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:25:56.467390    4559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dae5c89b-c8f9-46e6-9ca9-5444d1cb9269-istiod-ca-cert\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:25:56.467427 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:25:56.467425    4559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dae5c89b-c8f9-46e6-9ca9-5444d1cb9269-istio-token\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:25:56.467629 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:25:56.467434    4559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dae5c89b-c8f9-46e6-9ca9-5444d1cb9269-credential-socket\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:25:56.467629 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:25:56.467445    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cd8gw\" (UniqueName: \"kubernetes.io/projected/dae5c89b-c8f9-46e6-9ca9-5444d1cb9269-kube-api-access-cd8gw\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:25:56.467629 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:25:56.467455    4559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dae5c89b-c8f9-46e6-9ca9-5444d1cb9269-istio-envoy\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:25:56.467629 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:25:56.467463    4559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dae5c89b-c8f9-46e6-9ca9-5444d1cb9269-istio-podinfo\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:25:56.467629 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:25:56.467471    4559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dae5c89b-c8f9-46e6-9ca9-5444d1cb9269-workload-certs\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:25:56.467629 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:25:56.467479    4559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dae5c89b-c8f9-46e6-9ca9-5444d1cb9269-workload-socket\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:25:56.696496 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:25:56.696461    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-bb4qf"]
Nov 02 13:25:56.700283 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:25:56.700259    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-bb4qf"]
Nov 02 13:25:57.909567 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:25:57.909532    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dae5c89b-c8f9-46e6-9ca9-5444d1cb9269" path="/var/lib/kubelet/pods/dae5c89b-c8f9-46e6-9ca9-5444d1cb9269/volumes"
Nov 02 13:25:58.580720 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:25:58.580682    4559 patch_prober.go:29] interesting pod/e2e-extmod-a-ef5efe-openshift-default-679746c7d9-x5qtc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 13:25:58.580894 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:25:58.580757    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-ef5efe-openshift-default-679746c7d9-x5qtc" podUID="2fd94eba-ab06-4d75-a63b-e31bca0326d6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 13:25:59.337567 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:25:59.337505    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-ef5efe-openshift-default-679746c7d9-x5qtc"
Nov 02 13:25:59.371678 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:25:59.371653    4559 generic.go:356] "Generic (PLEG): container finished" podID="2fd94eba-ab06-4d75-a63b-e31bca0326d6" containerID="712549f074112340d65eb3ef1b661beb6e934332b2dbc804bc4c71d14cd447ca" exitCode=0
Nov 02 13:25:59.371835 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:25:59.371708    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-ef5efe-openshift-default-679746c7d9-x5qtc"
Nov 02 13:25:59.371835 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:25:59.371734    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-ef5efe-openshift-default-679746c7d9-x5qtc" event={"ID":"2fd94eba-ab06-4d75-a63b-e31bca0326d6","Type":"ContainerDied","Data":"712549f074112340d65eb3ef1b661beb6e934332b2dbc804bc4c71d14cd447ca"}
Nov 02 13:25:59.371835 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:25:59.371794    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-ef5efe-openshift-default-679746c7d9-x5qtc" event={"ID":"2fd94eba-ab06-4d75-a63b-e31bca0326d6","Type":"ContainerDied","Data":"4968e28cecd962a397484e1e22fc9607aa9437e56f9ac96e46e1e0587e378ba1"}
Nov 02 13:25:59.371835 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:25:59.371825    4559 scope.go:122] "RemoveContainer" containerID="712549f074112340d65eb3ef1b661beb6e934332b2dbc804bc4c71d14cd447ca"
Nov 02 13:25:59.385918 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:25:59.385895    4559 scope.go:122] "RemoveContainer" containerID="712549f074112340d65eb3ef1b661beb6e934332b2dbc804bc4c71d14cd447ca"
Nov 02 13:25:59.386176 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:25:59.386158    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"712549f074112340d65eb3ef1b661beb6e934332b2dbc804bc4c71d14cd447ca\": container with ID starting with 712549f074112340d65eb3ef1b661beb6e934332b2dbc804bc4c71d14cd447ca not found: ID does not exist" containerID="712549f074112340d65eb3ef1b661beb6e934332b2dbc804bc4c71d14cd447ca"
Nov 02 13:25:59.386246 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:25:59.386183    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"712549f074112340d65eb3ef1b661beb6e934332b2dbc804bc4c71d14cd447ca"} err="failed to get container status \"712549f074112340d65eb3ef1b661beb6e934332b2dbc804bc4c71d14cd447ca\": rpc error: code = NotFound desc = could not find container \"712549f074112340d65eb3ef1b661beb6e934332b2dbc804bc4c71d14cd447ca\": container with ID starting with 712549f074112340d65eb3ef1b661beb6e934332b2dbc804bc4c71d14cd447ca not found: ID does not exist"
Nov 02 13:25:59.516386 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:25:59.516340    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2fd94eba-ab06-4d75-a63b-e31bca0326d6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2fd94eba-ab06-4d75-a63b-e31bca0326d6-istio-podinfo\") pod \"2fd94eba-ab06-4d75-a63b-e31bca0326d6\" (UID: \"2fd94eba-ab06-4d75-a63b-e31bca0326d6\") "
Nov 02 13:25:59.516612 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:25:59.516405    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2fd94eba-ab06-4d75-a63b-e31bca0326d6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2fd94eba-ab06-4d75-a63b-e31bca0326d6-istiod-ca-cert\") pod \"2fd94eba-ab06-4d75-a63b-e31bca0326d6\" (UID: \"2fd94eba-ab06-4d75-a63b-e31bca0326d6\") "
Nov 02 13:25:59.516612 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:25:59.516454    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fd94eba-ab06-4d75-a63b-e31bca0326d6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2fd94eba-ab06-4d75-a63b-e31bca0326d6-workload-certs\") pod \"2fd94eba-ab06-4d75-a63b-e31bca0326d6\" (UID: \"2fd94eba-ab06-4d75-a63b-e31bca0326d6\") "
Nov 02 13:25:59.516612 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:25:59.516508    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2fd94eba-ab06-4d75-a63b-e31bca0326d6-istio-token\" (UniqueName: \"kubernetes.io/projected/2fd94eba-ab06-4d75-a63b-e31bca0326d6-istio-token\") pod \"2fd94eba-ab06-4d75-a63b-e31bca0326d6\" (UID: \"2fd94eba-ab06-4d75-a63b-e31bca0326d6\") "
Nov 02 13:25:59.516612 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:25:59.516595    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2fd94eba-ab06-4d75-a63b-e31bca0326d6-kube-api-access-n5gtj\" (UniqueName: \"kubernetes.io/projected/2fd94eba-ab06-4d75-a63b-e31bca0326d6-kube-api-access-n5gtj\") pod \"2fd94eba-ab06-4d75-a63b-e31bca0326d6\" (UID: \"2fd94eba-ab06-4d75-a63b-e31bca0326d6\") "
Nov 02 13:25:59.516921 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:25:59.516674    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fd94eba-ab06-4d75-a63b-e31bca0326d6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2fd94eba-ab06-4d75-a63b-e31bca0326d6-istio-envoy\") pod \"2fd94eba-ab06-4d75-a63b-e31bca0326d6\" (UID: \"2fd94eba-ab06-4d75-a63b-e31bca0326d6\") "
Nov 02 13:25:59.516921 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:25:59.516734    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fd94eba-ab06-4d75-a63b-e31bca0326d6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fd94eba-ab06-4d75-a63b-e31bca0326d6-workload-socket\") pod \"2fd94eba-ab06-4d75-a63b-e31bca0326d6\" (UID: \"2fd94eba-ab06-4d75-a63b-e31bca0326d6\") "
Nov 02 13:25:59.516921 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:25:59.516775    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fd94eba-ab06-4d75-a63b-e31bca0326d6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fd94eba-ab06-4d75-a63b-e31bca0326d6-credential-socket\") pod \"2fd94eba-ab06-4d75-a63b-e31bca0326d6\" (UID: \"2fd94eba-ab06-4d75-a63b-e31bca0326d6\") "
Nov 02 13:25:59.516921 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:25:59.516821    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fd94eba-ab06-4d75-a63b-e31bca0326d6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2fd94eba-ab06-4d75-a63b-e31bca0326d6-istio-data\") pod \"2fd94eba-ab06-4d75-a63b-e31bca0326d6\" (UID: \"2fd94eba-ab06-4d75-a63b-e31bca0326d6\") "
Nov 02 13:25:59.517677 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:25:59.517652    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fd94eba-ab06-4d75-a63b-e31bca0326d6-istio-data" pod "2fd94eba-ab06-4d75-a63b-e31bca0326d6" (UID: "2fd94eba-ab06-4d75-a63b-e31bca0326d6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:25:59.518078 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:25:59.518045    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fd94eba-ab06-4d75-a63b-e31bca0326d6-workload-certs" pod "2fd94eba-ab06-4d75-a63b-e31bca0326d6" (UID: "2fd94eba-ab06-4d75-a63b-e31bca0326d6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:25:59.518212 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:25:59.518153    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2fd94eba-ab06-4d75-a63b-e31bca0326d6-istiod-ca-cert" pod "2fd94eba-ab06-4d75-a63b-e31bca0326d6" (UID: "2fd94eba-ab06-4d75-a63b-e31bca0326d6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:25:59.518292 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:25:59.518243    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fd94eba-ab06-4d75-a63b-e31bca0326d6-credential-socket" pod "2fd94eba-ab06-4d75-a63b-e31bca0326d6" (UID: "2fd94eba-ab06-4d75-a63b-e31bca0326d6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:25:59.518354 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:25:59.518327    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fd94eba-ab06-4d75-a63b-e31bca0326d6-workload-socket" pod "2fd94eba-ab06-4d75-a63b-e31bca0326d6" (UID: "2fd94eba-ab06-4d75-a63b-e31bca0326d6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:25:59.519984 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:25:59.519951    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fd94eba-ab06-4d75-a63b-e31bca0326d6-istio-token" pod "2fd94eba-ab06-4d75-a63b-e31bca0326d6" (UID: "2fd94eba-ab06-4d75-a63b-e31bca0326d6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:25:59.520061 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:25:59.520049    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fd94eba-ab06-4d75-a63b-e31bca0326d6-istio-envoy" pod "2fd94eba-ab06-4d75-a63b-e31bca0326d6" (UID: "2fd94eba-ab06-4d75-a63b-e31bca0326d6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:25:59.520333 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:25:59.520310    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2fd94eba-ab06-4d75-a63b-e31bca0326d6-istio-podinfo" pod "2fd94eba-ab06-4d75-a63b-e31bca0326d6" (UID: "2fd94eba-ab06-4d75-a63b-e31bca0326d6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 13:25:59.520417 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:25:59.520382    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fd94eba-ab06-4d75-a63b-e31bca0326d6-kube-api-access-n5gtj" pod "2fd94eba-ab06-4d75-a63b-e31bca0326d6" (UID: "2fd94eba-ab06-4d75-a63b-e31bca0326d6"). InnerVolumeSpecName "kube-api-access-n5gtj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:25:59.619266 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:25:59.619205    4559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2fd94eba-ab06-4d75-a63b-e31bca0326d6-istio-token\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:25:59.619266 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:25:59.619233    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n5gtj\" (UniqueName: \"kubernetes.io/projected/2fd94eba-ab06-4d75-a63b-e31bca0326d6-kube-api-access-n5gtj\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:25:59.619266 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:25:59.619244    4559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2fd94eba-ab06-4d75-a63b-e31bca0326d6-istio-envoy\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:25:59.619266 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:25:59.619252    4559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fd94eba-ab06-4d75-a63b-e31bca0326d6-workload-socket\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:25:59.619266 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:25:59.619261    4559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fd94eba-ab06-4d75-a63b-e31bca0326d6-credential-socket\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:25:59.619509 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:25:59.619271    4559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2fd94eba-ab06-4d75-a63b-e31bca0326d6-istio-data\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:25:59.619509 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:25:59.619280    4559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2fd94eba-ab06-4d75-a63b-e31bca0326d6-istio-podinfo\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:25:59.619509 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:25:59.619289    4559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2fd94eba-ab06-4d75-a63b-e31bca0326d6-istiod-ca-cert\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:25:59.619509 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:25:59.619297    4559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2fd94eba-ab06-4d75-a63b-e31bca0326d6-workload-certs\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:25:59.721820 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:25:59.721789    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-ef5efe-openshift-default-679746c7d9-x5qtc"]
Nov 02 13:25:59.725308 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:25:59.725283    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-ef5efe-openshift-default-679746c7d9-x5qtc"]
Nov 02 13:25:59.893859 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:25:59.893786    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2fd94eba-ab06-4d75-a63b-e31bca0326d6" path="/var/lib/kubelet/pods/2fd94eba-ab06-4d75-a63b-e31bca0326d6/volumes"
Nov 02 13:26:15.091421 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:26:15.091387    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqcmj" containerName="extractor"
Nov 02 13:26:15.496858 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:26:15.496833    4559 generic.go:356] "Generic (PLEG): container finished" podID="fff161f9-6bfb-4c48-bc75-ade5639c076b" containerID="1e2e7dacc8266b31ed051a83120561e72f02173086999afda21e17815be14972" exitCode=0
Nov 02 13:26:15.497013 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:26:15.496907    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b8a8dc4c-openshift-default-699557545-57rbb" event={"ID":"fff161f9-6bfb-4c48-bc75-ade5639c076b","Type":"ContainerDied","Data":"1e2e7dacc8266b31ed051a83120561e72f02173086999afda21e17815be14972"}
Nov 02 13:26:15.497313 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:26:15.497297    4559 scope.go:122] "RemoveContainer" containerID="1e2e7dacc8266b31ed051a83120561e72f02173086999afda21e17815be14972"
Nov 02 13:26:15.499028 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:26:15.498997    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 13:26:15.499118 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:26:15.499074    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 13:26:15.499118 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:26:15.499107    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 13:26:16.514664 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:26:16.513944    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b8a8dc4c-openshift-default-699557545-57rbb" event={"ID":"fff161f9-6bfb-4c48-bc75-ade5639c076b","Type":"ContainerStarted","Data":"528f01ff77310124bbd9ae6fc46db8b3a73688035e354d78e588466c759a730e"}
Nov 02 13:26:17.107039 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:26:17.106997    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-b8a8dc4c-openshift-default-699557545-57rbb"
Nov 02 13:26:17.107430 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:26:17.107403    4559 patch_prober.go:29] interesting pod/e2e-mt-b8a8dc4c-openshift-default-699557545-57rbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.251:15021/healthz/ready\": dial tcp 10.133.0.251:15021: connect: connection refused" start-of-body=
Nov 02 13:26:17.107602 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:26:17.107579    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b8a8dc4c-openshift-default-699557545-57rbb" podUID="fff161f9-6bfb-4c48-bc75-ade5639c076b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.251:15021/healthz/ready\": dial tcp 10.133.0.251:15021: connect: connection refused"
Nov 02 13:26:18.109017 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:26:18.108979    4559 patch_prober.go:29] interesting pod/e2e-mt-b8a8dc4c-openshift-default-699557545-57rbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.251:15021/healthz/ready\": dial tcp 10.133.0.251:15021: connect: connection refused" start-of-body=
Nov 02 13:26:18.109536 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:26:18.109044    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b8a8dc4c-openshift-default-699557545-57rbb" podUID="fff161f9-6bfb-4c48-bc75-ade5639c076b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.251:15021/healthz/ready\": dial tcp 10.133.0.251:15021: connect: connection refused"
Nov 02 13:26:19.108115 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:26:19.108072    4559 patch_prober.go:29] interesting pod/e2e-mt-b8a8dc4c-openshift-default-699557545-57rbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.251:15021/healthz/ready\": dial tcp 10.133.0.251:15021: connect: connection refused" start-of-body=
Nov 02 13:26:19.108311 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:26:19.108141    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b8a8dc4c-openshift-default-699557545-57rbb" podUID="fff161f9-6bfb-4c48-bc75-ade5639c076b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.251:15021/healthz/ready\": dial tcp 10.133.0.251:15021: connect: connection refused"
Nov 02 13:26:20.107666 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:26:20.107161    4559 patch_prober.go:29] interesting pod/e2e-mt-b8a8dc4c-openshift-default-699557545-57rbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.251:15021/healthz/ready\": dial tcp 10.133.0.251:15021: connect: connection refused" start-of-body=
Nov 02 13:26:20.107666 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:26:20.107239    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b8a8dc4c-openshift-default-699557545-57rbb" podUID="fff161f9-6bfb-4c48-bc75-ade5639c076b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.251:15021/healthz/ready\": dial tcp 10.133.0.251:15021: connect: connection refused"
Nov 02 13:26:21.107971 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:26:21.107924    4559 patch_prober.go:29] interesting pod/e2e-mt-b8a8dc4c-openshift-default-699557545-57rbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.251:15021/healthz/ready\": dial tcp 10.133.0.251:15021: connect: connection refused" start-of-body=
Nov 02 13:26:21.108466 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:26:21.108029    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b8a8dc4c-openshift-default-699557545-57rbb" podUID="fff161f9-6bfb-4c48-bc75-ade5639c076b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.251:15021/healthz/ready\": dial tcp 10.133.0.251:15021: connect: connection refused"
Nov 02 13:26:22.106466 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:26:22.106429    4559 patch_prober.go:29] interesting pod/e2e-mt-b8a8dc4c-openshift-default-699557545-57rbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.251:15021/healthz/ready\": dial tcp 10.133.0.251:15021: connect: connection refused" start-of-body=
Nov 02 13:26:22.106661 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:26:22.106494    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b8a8dc4c-openshift-default-699557545-57rbb" podUID="fff161f9-6bfb-4c48-bc75-ade5639c076b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.251:15021/healthz/ready\": dial tcp 10.133.0.251:15021: connect: connection refused"
Nov 02 13:26:22.713062 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:26:22.713029    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-b8a8dc4c-openshift-default-699557545-57rbb"
Nov 02 13:26:23.110571 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:26:23.110499    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-b8a8dc4c-openshift-default-699557545-57rbb"
Nov 02 13:26:23.566205 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:26:23.566177    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-b8a8dc4c-openshift-default-699557545-57rbb"
Nov 02 13:26:37.580524 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:26:37.580488    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"]
Nov 02 13:26:45.088922 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:26:45.088872    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqcmj" containerName="extractor"
Nov 02 13:27:02.732660 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:27:02.731674    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-23f720-75984ff486-7cn9p"]
Nov 02 13:27:02.732660 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:27:02.731956    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-23f720-75984ff486-7cn9p" podUID="5a9eb160-1c8d-456c-8e37-f2d2e8e234e4" containerName="maas-api" containerID="cri-o://4ddb4722c84cf14a6fce18487755734ed8820a0daf412c73d18508f2759dafcb" gracePeriod=30
Nov 02 13:27:03.501766 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:27:03.500496    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-23f720-75984ff486-7cn9p"
Nov 02 13:27:03.564665 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:27:03.563602    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5a9eb160-1c8d-456c-8e37-f2d2e8e234e4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5a9eb160-1c8d-456c-8e37-f2d2e8e234e4-metrics-certs\") pod \"5a9eb160-1c8d-456c-8e37-f2d2e8e234e4\" (UID: \"5a9eb160-1c8d-456c-8e37-f2d2e8e234e4\") "
Nov 02 13:27:03.564665 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:27:03.563802    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5a9eb160-1c8d-456c-8e37-f2d2e8e234e4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5a9eb160-1c8d-456c-8e37-f2d2e8e234e4-maas-api-tls\") pod \"5a9eb160-1c8d-456c-8e37-f2d2e8e234e4\" (UID: \"5a9eb160-1c8d-456c-8e37-f2d2e8e234e4\") "
Nov 02 13:27:03.564665 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:27:03.563869    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a9eb160-1c8d-456c-8e37-f2d2e8e234e4-kube-api-access-mwcvw\" (UniqueName: \"kubernetes.io/projected/5a9eb160-1c8d-456c-8e37-f2d2e8e234e4-kube-api-access-mwcvw\") pod \"5a9eb160-1c8d-456c-8e37-f2d2e8e234e4\" (UID: \"5a9eb160-1c8d-456c-8e37-f2d2e8e234e4\") "
Nov 02 13:27:03.564665 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:27:03.563996    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a9eb160-1c8d-456c-8e37-f2d2e8e234e4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a9eb160-1c8d-456c-8e37-f2d2e8e234e4-openshift-service-ca\") pod \"5a9eb160-1c8d-456c-8e37-f2d2e8e234e4\" (UID: \"5a9eb160-1c8d-456c-8e37-f2d2e8e234e4\") "
Nov 02 13:27:03.565052 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:27:03.564899    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a9eb160-1c8d-456c-8e37-f2d2e8e234e4-openshift-service-ca" pod "5a9eb160-1c8d-456c-8e37-f2d2e8e234e4" (UID: "5a9eb160-1c8d-456c-8e37-f2d2e8e234e4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:27:03.579711 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:27:03.577905    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a9eb160-1c8d-456c-8e37-f2d2e8e234e4-kube-api-access-mwcvw" pod "5a9eb160-1c8d-456c-8e37-f2d2e8e234e4" (UID: "5a9eb160-1c8d-456c-8e37-f2d2e8e234e4"). InnerVolumeSpecName "kube-api-access-mwcvw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:27:03.579711 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:27:03.578460    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a9eb160-1c8d-456c-8e37-f2d2e8e234e4-metrics-certs" pod "5a9eb160-1c8d-456c-8e37-f2d2e8e234e4" (UID: "5a9eb160-1c8d-456c-8e37-f2d2e8e234e4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:27:03.579711 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:27:03.578527    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a9eb160-1c8d-456c-8e37-f2d2e8e234e4-maas-api-tls" pod "5a9eb160-1c8d-456c-8e37-f2d2e8e234e4" (UID: "5a9eb160-1c8d-456c-8e37-f2d2e8e234e4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:27:03.676671 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:27:03.675578    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5a9eb160-1c8d-456c-8e37-f2d2e8e234e4-maas-api-tls\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:27:03.676671 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:27:03.675616    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mwcvw\" (UniqueName: \"kubernetes.io/projected/5a9eb160-1c8d-456c-8e37-f2d2e8e234e4-kube-api-access-mwcvw\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:27:03.676671 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:27:03.675652    4559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a9eb160-1c8d-456c-8e37-f2d2e8e234e4-openshift-service-ca\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:27:03.676671 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:27:03.675668    4559 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5a9eb160-1c8d-456c-8e37-f2d2e8e234e4-metrics-certs\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:27:04.004486 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:27:04.004447    4559 generic.go:356] "Generic (PLEG): container finished" podID="5a9eb160-1c8d-456c-8e37-f2d2e8e234e4" containerID="4ddb4722c84cf14a6fce18487755734ed8820a0daf412c73d18508f2759dafcb" exitCode=0
Nov 02 13:27:04.005101 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:27:04.005077    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-23f720-75984ff486-7cn9p" event={"ID":"5a9eb160-1c8d-456c-8e37-f2d2e8e234e4","Type":"ContainerDied","Data":"4ddb4722c84cf14a6fce18487755734ed8820a0daf412c73d18508f2759dafcb"}
Nov 02 13:27:04.005265 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:27:04.005247    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-23f720-75984ff486-7cn9p" event={"ID":"5a9eb160-1c8d-456c-8e37-f2d2e8e234e4","Type":"ContainerDied","Data":"87f823483cdda29e232cf1c7a57b10bfd1048a13bc97b92a6dfeea694c3b17d1"}
Nov 02 13:27:04.005394 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:27:04.005381    4559 scope.go:122] "RemoveContainer" containerID="4ddb4722c84cf14a6fce18487755734ed8820a0daf412c73d18508f2759dafcb"
Nov 02 13:27:04.005678 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:27:04.005658    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-23f720-75984ff486-7cn9p"
Nov 02 13:27:04.124189 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:27:04.120791    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-23f720-75984ff486-7cn9p"]
Nov 02 13:27:04.133935 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:27:04.133910    4559 scope.go:122] "RemoveContainer" containerID="4ddb4722c84cf14a6fce18487755734ed8820a0daf412c73d18508f2759dafcb"
Nov 02 13:27:04.134615 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:27:04.134535    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ddb4722c84cf14a6fce18487755734ed8820a0daf412c73d18508f2759dafcb\": container with ID starting with 4ddb4722c84cf14a6fce18487755734ed8820a0daf412c73d18508f2759dafcb not found: ID does not exist" containerID="4ddb4722c84cf14a6fce18487755734ed8820a0daf412c73d18508f2759dafcb"
Nov 02 13:27:04.134615 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:27:04.134580    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ddb4722c84cf14a6fce18487755734ed8820a0daf412c73d18508f2759dafcb"} err="failed to get container status \"4ddb4722c84cf14a6fce18487755734ed8820a0daf412c73d18508f2759dafcb\": rpc error: code = NotFound desc = could not find container \"4ddb4722c84cf14a6fce18487755734ed8820a0daf412c73d18508f2759dafcb\": container with ID starting with 4ddb4722c84cf14a6fce18487755734ed8820a0daf412c73d18508f2759dafcb not found: ID does not exist"
Nov 02 13:27:04.195664 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:27:04.175782    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-23f720-75984ff486-7cn9p"]
Nov 02 13:27:05.894684 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:27:05.894652    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a9eb160-1c8d-456c-8e37-f2d2e8e234e4" path="/var/lib/kubelet/pods/5a9eb160-1c8d-456c-8e37-f2d2e8e234e4/volumes"
Nov 02 13:27:07.069952 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:27:07.069918    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-23f720-openshift-default-6c8d4f9ff5-vpdqd"]
Nov 02 13:27:07.071987 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:27:07.071947    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-23f720-openshift-default-6c8d4f9ff5-vpdqd" podUID="2d3ed32f-021a-456e-85eb-864596d28c3f" containerName="istio-proxy" containerID="cri-o://fa17bc1362dc89348a5ceaadb90ebc414eafa81682f2b06cf0eb0f16f3854aa4" gracePeriod=30
Nov 02 13:27:08.083427 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:27:08.083390    4559 patch_prober.go:29] interesting pod/e2e-inf-a-23f720-openshift-default-6c8d4f9ff5-vpdqd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 13:27:08.083978 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:27:08.083448    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-23f720-openshift-default-6c8d4f9ff5-vpdqd" podUID="2d3ed32f-021a-456e-85eb-864596d28c3f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 13:27:12.349392 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:27:12.349338    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-23f720-openshift-default-6c8d4f9ff5-vpdqd"
Nov 02 13:27:12.396745 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:27:12.396715    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2d3ed32f-021a-456e-85eb-864596d28c3f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d3ed32f-021a-456e-85eb-864596d28c3f-istio-podinfo\") pod \"2d3ed32f-021a-456e-85eb-864596d28c3f\" (UID: \"2d3ed32f-021a-456e-85eb-864596d28c3f\") "
Nov 02 13:27:12.396886 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:27:12.396780    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d3ed32f-021a-456e-85eb-864596d28c3f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d3ed32f-021a-456e-85eb-864596d28c3f-workload-socket\") pod \"2d3ed32f-021a-456e-85eb-864596d28c3f\" (UID: \"2d3ed32f-021a-456e-85eb-864596d28c3f\") "
Nov 02 13:27:12.396886 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:27:12.396802    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d3ed32f-021a-456e-85eb-864596d28c3f-istio-token\" (UniqueName: \"kubernetes.io/projected/2d3ed32f-021a-456e-85eb-864596d28c3f-istio-token\") pod \"2d3ed32f-021a-456e-85eb-864596d28c3f\" (UID: \"2d3ed32f-021a-456e-85eb-864596d28c3f\") "
Nov 02 13:27:12.396886 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:27:12.396826    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d3ed32f-021a-456e-85eb-864596d28c3f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d3ed32f-021a-456e-85eb-864596d28c3f-istiod-ca-cert\") pod \"2d3ed32f-021a-456e-85eb-864596d28c3f\" (UID: \"2d3ed32f-021a-456e-85eb-864596d28c3f\") "
Nov 02 13:27:12.396886 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:27:12.396885    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d3ed32f-021a-456e-85eb-864596d28c3f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d3ed32f-021a-456e-85eb-864596d28c3f-credential-socket\") pod \"2d3ed32f-021a-456e-85eb-864596d28c3f\" (UID: \"2d3ed32f-021a-456e-85eb-864596d28c3f\") "
Nov 02 13:27:12.397118 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:27:12.396912    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d3ed32f-021a-456e-85eb-864596d28c3f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d3ed32f-021a-456e-85eb-864596d28c3f-workload-certs\") pod \"2d3ed32f-021a-456e-85eb-864596d28c3f\" (UID: \"2d3ed32f-021a-456e-85eb-864596d28c3f\") "
Nov 02 13:27:12.397118 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:27:12.396996    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d3ed32f-021a-456e-85eb-864596d28c3f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d3ed32f-021a-456e-85eb-864596d28c3f-istio-envoy\") pod \"2d3ed32f-021a-456e-85eb-864596d28c3f\" (UID: \"2d3ed32f-021a-456e-85eb-864596d28c3f\") "
Nov 02 13:27:12.397118 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:27:12.397059    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d3ed32f-021a-456e-85eb-864596d28c3f-kube-api-access-mc6d9\" (UniqueName: \"kubernetes.io/projected/2d3ed32f-021a-456e-85eb-864596d28c3f-kube-api-access-mc6d9\") pod \"2d3ed32f-021a-456e-85eb-864596d28c3f\" (UID: \"2d3ed32f-021a-456e-85eb-864596d28c3f\") "
Nov 02 13:27:12.397304 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:27:12.397107    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d3ed32f-021a-456e-85eb-864596d28c3f-workload-socket" pod "2d3ed32f-021a-456e-85eb-864596d28c3f" (UID: "2d3ed32f-021a-456e-85eb-864596d28c3f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:27:12.397304 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:27:12.397186    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d3ed32f-021a-456e-85eb-864596d28c3f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d3ed32f-021a-456e-85eb-864596d28c3f-istio-data\") pod \"2d3ed32f-021a-456e-85eb-864596d28c3f\" (UID: \"2d3ed32f-021a-456e-85eb-864596d28c3f\") "
Nov 02 13:27:12.397304 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:27:12.397189    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d3ed32f-021a-456e-85eb-864596d28c3f-credential-socket" pod "2d3ed32f-021a-456e-85eb-864596d28c3f" (UID: "2d3ed32f-021a-456e-85eb-864596d28c3f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:27:12.397304 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:27:12.397208    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d3ed32f-021a-456e-85eb-864596d28c3f-workload-certs" pod "2d3ed32f-021a-456e-85eb-864596d28c3f" (UID: "2d3ed32f-021a-456e-85eb-864596d28c3f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:27:12.397304 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:27:12.397280    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d3ed32f-021a-456e-85eb-864596d28c3f-istiod-ca-cert" pod "2d3ed32f-021a-456e-85eb-864596d28c3f" (UID: "2d3ed32f-021a-456e-85eb-864596d28c3f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:27:12.397601 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:27:12.397484    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d3ed32f-021a-456e-85eb-864596d28c3f-istio-data" pod "2d3ed32f-021a-456e-85eb-864596d28c3f" (UID: "2d3ed32f-021a-456e-85eb-864596d28c3f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:27:12.397873 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:27:12.397851    4559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d3ed32f-021a-456e-85eb-864596d28c3f-istio-data\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:27:12.397982 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:27:12.397879    4559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d3ed32f-021a-456e-85eb-864596d28c3f-workload-socket\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:27:12.397982 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:27:12.397895    4559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d3ed32f-021a-456e-85eb-864596d28c3f-istiod-ca-cert\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:27:12.397982 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:27:12.397909    4559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d3ed32f-021a-456e-85eb-864596d28c3f-credential-socket\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:27:12.397982 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:27:12.397924    4559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d3ed32f-021a-456e-85eb-864596d28c3f-workload-certs\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:27:12.399065 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:27:12.399037    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d3ed32f-021a-456e-85eb-864596d28c3f-istio-envoy" pod "2d3ed32f-021a-456e-85eb-864596d28c3f" (UID: "2d3ed32f-021a-456e-85eb-864596d28c3f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:27:12.399181 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:27:12.399117    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2d3ed32f-021a-456e-85eb-864596d28c3f-istio-podinfo" pod "2d3ed32f-021a-456e-85eb-864596d28c3f" (UID: "2d3ed32f-021a-456e-85eb-864596d28c3f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 13:27:12.399271 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:27:12.399209    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d3ed32f-021a-456e-85eb-864596d28c3f-istio-token" pod "2d3ed32f-021a-456e-85eb-864596d28c3f" (UID: "2d3ed32f-021a-456e-85eb-864596d28c3f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:27:12.399271 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:27:12.399248    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d3ed32f-021a-456e-85eb-864596d28c3f-kube-api-access-mc6d9" pod "2d3ed32f-021a-456e-85eb-864596d28c3f" (UID: "2d3ed32f-021a-456e-85eb-864596d28c3f"). InnerVolumeSpecName "kube-api-access-mc6d9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:27:12.499201 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:27:12.499166    4559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d3ed32f-021a-456e-85eb-864596d28c3f-istio-envoy\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:27:12.499201 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:27:12.499195    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mc6d9\" (UniqueName: \"kubernetes.io/projected/2d3ed32f-021a-456e-85eb-864596d28c3f-kube-api-access-mc6d9\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:27:12.499394 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:27:12.499208    4559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d3ed32f-021a-456e-85eb-864596d28c3f-istio-podinfo\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:27:12.499394 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:27:12.499222    4559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d3ed32f-021a-456e-85eb-864596d28c3f-istio-token\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:27:13.121120 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:27:13.121075    4559 generic.go:356] "Generic (PLEG): container finished" podID="2d3ed32f-021a-456e-85eb-864596d28c3f" containerID="fa17bc1362dc89348a5ceaadb90ebc414eafa81682f2b06cf0eb0f16f3854aa4" exitCode=0
Nov 02 13:27:13.121306 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:27:13.121146    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-23f720-openshift-default-6c8d4f9ff5-vpdqd"
Nov 02 13:27:13.121306 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:27:13.121163    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-23f720-openshift-default-6c8d4f9ff5-vpdqd" event={"ID":"2d3ed32f-021a-456e-85eb-864596d28c3f","Type":"ContainerDied","Data":"fa17bc1362dc89348a5ceaadb90ebc414eafa81682f2b06cf0eb0f16f3854aa4"}
Nov 02 13:27:13.121306 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:27:13.121222    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-23f720-openshift-default-6c8d4f9ff5-vpdqd" event={"ID":"2d3ed32f-021a-456e-85eb-864596d28c3f","Type":"ContainerDied","Data":"fd74f1a89641aa0f8e697bc6ca22af0d20607295c05b5cda3a4e5672ef93e1af"}
Nov 02 13:27:13.121306 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:27:13.121256    4559 scope.go:122] "RemoveContainer" containerID="fa17bc1362dc89348a5ceaadb90ebc414eafa81682f2b06cf0eb0f16f3854aa4"
Nov 02 13:27:13.133700 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:27:13.133685    4559 scope.go:122] "RemoveContainer" containerID="20602e8e5be6c5b18679fc49a63b7b5d46ba260dd464be42f7448b7123a1a4cd"
Nov 02 13:27:13.142987 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:27:13.142961    4559 scope.go:122] "RemoveContainer" containerID="fa17bc1362dc89348a5ceaadb90ebc414eafa81682f2b06cf0eb0f16f3854aa4"
Nov 02 13:27:13.143230 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:27:13.143212    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa17bc1362dc89348a5ceaadb90ebc414eafa81682f2b06cf0eb0f16f3854aa4\": container with ID starting with fa17bc1362dc89348a5ceaadb90ebc414eafa81682f2b06cf0eb0f16f3854aa4 not found: ID does not exist" containerID="fa17bc1362dc89348a5ceaadb90ebc414eafa81682f2b06cf0eb0f16f3854aa4"
Nov 02 13:27:13.143301 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:27:13.143239    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa17bc1362dc89348a5ceaadb90ebc414eafa81682f2b06cf0eb0f16f3854aa4"} err="failed to get container status \"fa17bc1362dc89348a5ceaadb90ebc414eafa81682f2b06cf0eb0f16f3854aa4\": rpc error: code = NotFound desc = could not find container \"fa17bc1362dc89348a5ceaadb90ebc414eafa81682f2b06cf0eb0f16f3854aa4\": container with ID starting with fa17bc1362dc89348a5ceaadb90ebc414eafa81682f2b06cf0eb0f16f3854aa4 not found: ID does not exist"
Nov 02 13:27:13.143301 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:27:13.143256    4559 scope.go:122] "RemoveContainer" containerID="20602e8e5be6c5b18679fc49a63b7b5d46ba260dd464be42f7448b7123a1a4cd"
Nov 02 13:27:13.143486 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:27:13.143469    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20602e8e5be6c5b18679fc49a63b7b5d46ba260dd464be42f7448b7123a1a4cd\": container with ID starting with 20602e8e5be6c5b18679fc49a63b7b5d46ba260dd464be42f7448b7123a1a4cd not found: ID does not exist" containerID="20602e8e5be6c5b18679fc49a63b7b5d46ba260dd464be42f7448b7123a1a4cd"
Nov 02 13:27:13.143546 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:27:13.143499    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20602e8e5be6c5b18679fc49a63b7b5d46ba260dd464be42f7448b7123a1a4cd"} err="failed to get container status \"20602e8e5be6c5b18679fc49a63b7b5d46ba260dd464be42f7448b7123a1a4cd\": rpc error: code = NotFound desc = could not find container \"20602e8e5be6c5b18679fc49a63b7b5d46ba260dd464be42f7448b7123a1a4cd\": container with ID starting with 20602e8e5be6c5b18679fc49a63b7b5d46ba260dd464be42f7448b7123a1a4cd not found: ID does not exist"
Nov 02 13:27:13.192179 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:27:13.192150    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-23f720-openshift-default-6c8d4f9ff5-vpdqd"]
Nov 02 13:27:13.205578 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:27:13.205551    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-23f720-openshift-default-6c8d4f9ff5-vpdqd"]
Nov 02 13:27:13.894672 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:27:13.894600    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d3ed32f-021a-456e-85eb-864596d28c3f" path="/var/lib/kubelet/pods/2d3ed32f-021a-456e-85eb-864596d28c3f/volumes"
Nov 02 13:27:15.088633 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:27:15.088601    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqcmj" containerName="extractor"
Nov 02 13:27:34.161263 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:27:34.161185    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"]
Nov 02 13:27:45.086281 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:27:45.086246    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqcmj" containerName="extractor"
Nov 02 13:27:50.885760 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:27:50.885722    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-5b6gt"]
Nov 02 13:27:50.886217 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:27:50.885920    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-5b6gt" podUID="63746d03-ec60-494a-96f8-39436d2e6cb8" containerName="istio-proxy" containerID="cri-o://7e5d2e39bab8890591db07e72702275305d7634ef536bb27272992d1f8026061" gracePeriod=30
Nov 02 13:27:56.185145 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:27:56.185112    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-5b6gt"
Nov 02 13:27:56.284017 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:27:56.283976    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63746d03-ec60-494a-96f8-39436d2e6cb8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63746d03-ec60-494a-96f8-39436d2e6cb8-istio-data\") pod \"63746d03-ec60-494a-96f8-39436d2e6cb8\" (UID: \"63746d03-ec60-494a-96f8-39436d2e6cb8\") "
Nov 02 13:27:56.284215 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:27:56.284057    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63746d03-ec60-494a-96f8-39436d2e6cb8-kube-api-access-f5lpm\" (UniqueName: \"kubernetes.io/projected/63746d03-ec60-494a-96f8-39436d2e6cb8-kube-api-access-f5lpm\") pod \"63746d03-ec60-494a-96f8-39436d2e6cb8\" (UID: \"63746d03-ec60-494a-96f8-39436d2e6cb8\") "
Nov 02 13:27:56.284215 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:27:56.284094    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63746d03-ec60-494a-96f8-39436d2e6cb8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63746d03-ec60-494a-96f8-39436d2e6cb8-credential-socket\") pod \"63746d03-ec60-494a-96f8-39436d2e6cb8\" (UID: \"63746d03-ec60-494a-96f8-39436d2e6cb8\") "
Nov 02 13:27:56.284215 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:27:56.284141    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63746d03-ec60-494a-96f8-39436d2e6cb8-istio-token\" (UniqueName: \"kubernetes.io/projected/63746d03-ec60-494a-96f8-39436d2e6cb8-istio-token\") pod \"63746d03-ec60-494a-96f8-39436d2e6cb8\" (UID: \"63746d03-ec60-494a-96f8-39436d2e6cb8\") "
Nov 02 13:27:56.284215 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:27:56.284186    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63746d03-ec60-494a-96f8-39436d2e6cb8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63746d03-ec60-494a-96f8-39436d2e6cb8-workload-certs\") pod \"63746d03-ec60-494a-96f8-39436d2e6cb8\" (UID: \"63746d03-ec60-494a-96f8-39436d2e6cb8\") "
Nov 02 13:27:56.284480 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:27:56.284269    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63746d03-ec60-494a-96f8-39436d2e6cb8-istio-data" pod "63746d03-ec60-494a-96f8-39436d2e6cb8" (UID: "63746d03-ec60-494a-96f8-39436d2e6cb8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:27:56.284480 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:27:56.284423    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63746d03-ec60-494a-96f8-39436d2e6cb8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63746d03-ec60-494a-96f8-39436d2e6cb8-istio-envoy\") pod \"63746d03-ec60-494a-96f8-39436d2e6cb8\" (UID: \"63746d03-ec60-494a-96f8-39436d2e6cb8\") "
Nov 02 13:27:56.284480 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:27:56.284455    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63746d03-ec60-494a-96f8-39436d2e6cb8-credential-socket" pod "63746d03-ec60-494a-96f8-39436d2e6cb8" (UID: "63746d03-ec60-494a-96f8-39436d2e6cb8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:27:56.284686 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:27:56.284466    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/63746d03-ec60-494a-96f8-39436d2e6cb8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63746d03-ec60-494a-96f8-39436d2e6cb8-istio-podinfo\") pod \"63746d03-ec60-494a-96f8-39436d2e6cb8\" (UID: \"63746d03-ec60-494a-96f8-39436d2e6cb8\") "
Nov 02 13:27:56.284686 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:27:56.284589    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63746d03-ec60-494a-96f8-39436d2e6cb8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63746d03-ec60-494a-96f8-39436d2e6cb8-workload-socket\") pod \"63746d03-ec60-494a-96f8-39436d2e6cb8\" (UID: \"63746d03-ec60-494a-96f8-39436d2e6cb8\") "
Nov 02 13:27:56.284686 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:27:56.284626    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63746d03-ec60-494a-96f8-39436d2e6cb8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63746d03-ec60-494a-96f8-39436d2e6cb8-istiod-ca-cert\") pod \"63746d03-ec60-494a-96f8-39436d2e6cb8\" (UID: \"63746d03-ec60-494a-96f8-39436d2e6cb8\") "
Nov 02 13:27:56.284868 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:27:56.284847    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63746d03-ec60-494a-96f8-39436d2e6cb8-workload-certs" pod "63746d03-ec60-494a-96f8-39436d2e6cb8" (UID: "63746d03-ec60-494a-96f8-39436d2e6cb8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:27:56.286164 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:27:56.285459    4559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63746d03-ec60-494a-96f8-39436d2e6cb8-istio-data\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:27:56.286164 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:27:56.285503    4559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63746d03-ec60-494a-96f8-39436d2e6cb8-credential-socket\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:27:56.286164 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:27:56.285520    4559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63746d03-ec60-494a-96f8-39436d2e6cb8-workload-certs\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:27:56.286164 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:27:56.285851    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63746d03-ec60-494a-96f8-39436d2e6cb8-istiod-ca-cert" pod "63746d03-ec60-494a-96f8-39436d2e6cb8" (UID: "63746d03-ec60-494a-96f8-39436d2e6cb8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:27:56.286164 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:27:56.286113    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63746d03-ec60-494a-96f8-39436d2e6cb8-workload-socket" pod "63746d03-ec60-494a-96f8-39436d2e6cb8" (UID: "63746d03-ec60-494a-96f8-39436d2e6cb8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:27:56.286913 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:27:56.286891    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/63746d03-ec60-494a-96f8-39436d2e6cb8-istio-podinfo" pod "63746d03-ec60-494a-96f8-39436d2e6cb8" (UID: "63746d03-ec60-494a-96f8-39436d2e6cb8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 13:27:56.287214 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:27:56.287195    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63746d03-ec60-494a-96f8-39436d2e6cb8-kube-api-access-f5lpm" pod "63746d03-ec60-494a-96f8-39436d2e6cb8" (UID: "63746d03-ec60-494a-96f8-39436d2e6cb8"). InnerVolumeSpecName "kube-api-access-f5lpm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:27:56.288074 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:27:56.288044    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63746d03-ec60-494a-96f8-39436d2e6cb8-istio-envoy" pod "63746d03-ec60-494a-96f8-39436d2e6cb8" (UID: "63746d03-ec60-494a-96f8-39436d2e6cb8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:27:56.288451 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:27:56.288431    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63746d03-ec60-494a-96f8-39436d2e6cb8-istio-token" pod "63746d03-ec60-494a-96f8-39436d2e6cb8" (UID: "63746d03-ec60-494a-96f8-39436d2e6cb8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:27:56.386726 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:27:56.386690    4559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63746d03-ec60-494a-96f8-39436d2e6cb8-istio-envoy\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:27:56.386726 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:27:56.386727    4559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63746d03-ec60-494a-96f8-39436d2e6cb8-istio-podinfo\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:27:56.386933 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:27:56.386741    4559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63746d03-ec60-494a-96f8-39436d2e6cb8-workload-socket\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:27:56.386933 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:27:56.386757    4559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63746d03-ec60-494a-96f8-39436d2e6cb8-istiod-ca-cert\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:27:56.386933 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:27:56.386771    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f5lpm\" (UniqueName: \"kubernetes.io/projected/63746d03-ec60-494a-96f8-39436d2e6cb8-kube-api-access-f5lpm\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:27:56.386933 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:27:56.386785    4559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63746d03-ec60-494a-96f8-39436d2e6cb8-istio-token\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:27:56.446385 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:27:56.446352    4559 generic.go:356] "Generic (PLEG): container finished" podID="63746d03-ec60-494a-96f8-39436d2e6cb8" containerID="7e5d2e39bab8890591db07e72702275305d7634ef536bb27272992d1f8026061" exitCode=0
Nov 02 13:27:56.446599 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:27:56.446408    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-5b6gt"
Nov 02 13:27:56.446599 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:27:56.446432    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-5b6gt" event={"ID":"63746d03-ec60-494a-96f8-39436d2e6cb8","Type":"ContainerDied","Data":"7e5d2e39bab8890591db07e72702275305d7634ef536bb27272992d1f8026061"}
Nov 02 13:27:56.446599 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:27:56.446484    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-5b6gt" event={"ID":"63746d03-ec60-494a-96f8-39436d2e6cb8","Type":"ContainerDied","Data":"169ce93027b9a3faf245e5d9fc3740c5147e05493ad20598003fe47241c4931a"}
Nov 02 13:27:56.446599 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:27:56.446505    4559 scope.go:122] "RemoveContainer" containerID="7e5d2e39bab8890591db07e72702275305d7634ef536bb27272992d1f8026061"
Nov 02 13:27:56.461543 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:27:56.461429    4559 scope.go:122] "RemoveContainer" containerID="7e5d2e39bab8890591db07e72702275305d7634ef536bb27272992d1f8026061"
Nov 02 13:27:56.461818 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:27:56.461789    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e5d2e39bab8890591db07e72702275305d7634ef536bb27272992d1f8026061\": container with ID starting with 7e5d2e39bab8890591db07e72702275305d7634ef536bb27272992d1f8026061 not found: ID does not exist" containerID="7e5d2e39bab8890591db07e72702275305d7634ef536bb27272992d1f8026061"
Nov 02 13:27:56.461943 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:27:56.461830    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e5d2e39bab8890591db07e72702275305d7634ef536bb27272992d1f8026061"} err="failed to get container status \"7e5d2e39bab8890591db07e72702275305d7634ef536bb27272992d1f8026061\": rpc error: code = NotFound desc = could not find container \"7e5d2e39bab8890591db07e72702275305d7634ef536bb27272992d1f8026061\": container with ID starting with 7e5d2e39bab8890591db07e72702275305d7634ef536bb27272992d1f8026061 not found: ID does not exist"
Nov 02 13:27:56.517267 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:27:56.517238    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-5b6gt"]
Nov 02 13:27:56.530251 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:27:56.530223    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-5b6gt"]
Nov 02 13:27:57.893713 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:27:57.893679    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63746d03-ec60-494a-96f8-39436d2e6cb8" path="/var/lib/kubelet/pods/63746d03-ec60-494a-96f8-39436d2e6cb8/volumes"
Nov 02 13:28:15.086512 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:28:15.086473    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqcmj" containerName="extractor"
Nov 02 13:28:18.071944 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:28:18.071909    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"]
Nov 02 13:28:45.087392 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:28:45.087356    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqcmj" containerName="extractor"
Nov 02 13:28:53.388409 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:28:53.388375    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b8a8dc4c-6fdcf4f98f-944jd"]
Nov 02 13:28:53.389147 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:28:53.388571    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b8a8dc4c-6fdcf4f98f-944jd" podUID="41be38ac-3172-4e6b-af34-c83c9179b69f" containerName="maas-api" containerID="cri-o://525298d5b763371a3965062582601020325a91965aea6280245d38b5d88c0458" gracePeriod=30
Nov 02 13:28:53.776029 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:28:53.776004    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b8a8dc4c-6fdcf4f98f-944jd"
Nov 02 13:28:53.978860 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:28:53.978814    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41be38ac-3172-4e6b-af34-c83c9179b69f-kube-api-access-g67sm\" (UniqueName: \"kubernetes.io/projected/41be38ac-3172-4e6b-af34-c83c9179b69f-kube-api-access-g67sm\") pod \"41be38ac-3172-4e6b-af34-c83c9179b69f\" (UID: \"41be38ac-3172-4e6b-af34-c83c9179b69f\") "
Nov 02 13:28:53.979022 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:28:53.978890    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/41be38ac-3172-4e6b-af34-c83c9179b69f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/41be38ac-3172-4e6b-af34-c83c9179b69f-maas-api-tls\") pod \"41be38ac-3172-4e6b-af34-c83c9179b69f\" (UID: \"41be38ac-3172-4e6b-af34-c83c9179b69f\") "
Nov 02 13:28:53.979022 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:28:53.979019    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/41be38ac-3172-4e6b-af34-c83c9179b69f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/41be38ac-3172-4e6b-af34-c83c9179b69f-metrics-certs\") pod \"41be38ac-3172-4e6b-af34-c83c9179b69f\" (UID: \"41be38ac-3172-4e6b-af34-c83c9179b69f\") "
Nov 02 13:28:53.979110 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:28:53.979049    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41be38ac-3172-4e6b-af34-c83c9179b69f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/41be38ac-3172-4e6b-af34-c83c9179b69f-openshift-service-ca\") pod \"41be38ac-3172-4e6b-af34-c83c9179b69f\" (UID: \"41be38ac-3172-4e6b-af34-c83c9179b69f\") "
Nov 02 13:28:53.979800 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:28:53.979776    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41be38ac-3172-4e6b-af34-c83c9179b69f-openshift-service-ca" pod "41be38ac-3172-4e6b-af34-c83c9179b69f" (UID: "41be38ac-3172-4e6b-af34-c83c9179b69f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:28:53.992422 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:28:53.992157    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41be38ac-3172-4e6b-af34-c83c9179b69f-kube-api-access-g67sm" pod "41be38ac-3172-4e6b-af34-c83c9179b69f" (UID: "41be38ac-3172-4e6b-af34-c83c9179b69f"). InnerVolumeSpecName "kube-api-access-g67sm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:28:54.005010 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:28:54.004924    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41be38ac-3172-4e6b-af34-c83c9179b69f-maas-api-tls" pod "41be38ac-3172-4e6b-af34-c83c9179b69f" (UID: "41be38ac-3172-4e6b-af34-c83c9179b69f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:28:54.007970 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:28:54.007927    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41be38ac-3172-4e6b-af34-c83c9179b69f-metrics-certs" pod "41be38ac-3172-4e6b-af34-c83c9179b69f" (UID: "41be38ac-3172-4e6b-af34-c83c9179b69f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:28:54.013524 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:28:54.013498    4559 generic.go:356] "Generic (PLEG): container finished" podID="41be38ac-3172-4e6b-af34-c83c9179b69f" containerID="525298d5b763371a3965062582601020325a91965aea6280245d38b5d88c0458" exitCode=0
Nov 02 13:28:54.013734 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:28:54.013714    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b8a8dc4c-6fdcf4f98f-944jd" event={"ID":"41be38ac-3172-4e6b-af34-c83c9179b69f","Type":"ContainerDied","Data":"525298d5b763371a3965062582601020325a91965aea6280245d38b5d88c0458"}
Nov 02 13:28:54.013867 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:28:54.013849    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b8a8dc4c-6fdcf4f98f-944jd" event={"ID":"41be38ac-3172-4e6b-af34-c83c9179b69f","Type":"ContainerDied","Data":"7389e097912b3dc734e36a8da70554e34b72258911bbf72ab06d0f88f85c731a"}
Nov 02 13:28:54.013948 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:28:54.013887    4559 scope.go:122] "RemoveContainer" containerID="525298d5b763371a3965062582601020325a91965aea6280245d38b5d88c0458"
Nov 02 13:28:54.014062 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:28:54.014047    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b8a8dc4c-6fdcf4f98f-944jd"
Nov 02 13:28:54.028364 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:28:54.028345    4559 scope.go:122] "RemoveContainer" containerID="525298d5b763371a3965062582601020325a91965aea6280245d38b5d88c0458"
Nov 02 13:28:54.028610 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:28:54.028593    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"525298d5b763371a3965062582601020325a91965aea6280245d38b5d88c0458\": container with ID starting with 525298d5b763371a3965062582601020325a91965aea6280245d38b5d88c0458 not found: ID does not exist" containerID="525298d5b763371a3965062582601020325a91965aea6280245d38b5d88c0458"
Nov 02 13:28:54.028725 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:28:54.028614    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"525298d5b763371a3965062582601020325a91965aea6280245d38b5d88c0458"} err="failed to get container status \"525298d5b763371a3965062582601020325a91965aea6280245d38b5d88c0458\": rpc error: code = NotFound desc = could not find container \"525298d5b763371a3965062582601020325a91965aea6280245d38b5d88c0458\": container with ID starting with 525298d5b763371a3965062582601020325a91965aea6280245d38b5d88c0458 not found: ID does not exist"
Nov 02 13:28:54.072924 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:28:54.072897    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b8a8dc4c-6fdcf4f98f-944jd"]
Nov 02 13:28:54.076707 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:28:54.076684    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b8a8dc4c-6fdcf4f98f-944jd"]
Nov 02 13:28:54.081774 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:28:54.081381    4559 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/41be38ac-3172-4e6b-af34-c83c9179b69f-metrics-certs\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:28:54.081774 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:28:54.081413    4559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/41be38ac-3172-4e6b-af34-c83c9179b69f-openshift-service-ca\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:28:54.081774 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:28:54.081428    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g67sm\" (UniqueName: \"kubernetes.io/projected/41be38ac-3172-4e6b-af34-c83c9179b69f-kube-api-access-g67sm\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:28:54.081774 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:28:54.081442    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/41be38ac-3172-4e6b-af34-c83c9179b69f-maas-api-tls\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:28:55.893939 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:28:55.893849    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41be38ac-3172-4e6b-af34-c83c9179b69f" path="/var/lib/kubelet/pods/41be38ac-3172-4e6b-af34-c83c9179b69f/volumes"
Nov 02 13:28:58.097087 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:28:58.097053    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-b8a8dc4c-openshift-default-699557545-57rbb"]
Nov 02 13:28:58.097918 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:28:58.097864    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-b8a8dc4c-openshift-default-699557545-57rbb" podUID="fff161f9-6bfb-4c48-bc75-ade5639c076b" containerName="istio-proxy" containerID="cri-o://528f01ff77310124bbd9ae6fc46db8b3a73688035e354d78e588466c759a730e" gracePeriod=30
Nov 02 13:29:03.380967 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:29:03.380908    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b8a8dc4c-openshift-default-699557545-57rbb"
Nov 02 13:29:03.443266 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:29:03.443234    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fff161f9-6bfb-4c48-bc75-ade5639c076b-istio-token\" (UniqueName: \"kubernetes.io/projected/fff161f9-6bfb-4c48-bc75-ade5639c076b-istio-token\") pod \"fff161f9-6bfb-4c48-bc75-ade5639c076b\" (UID: \"fff161f9-6bfb-4c48-bc75-ade5639c076b\") "
Nov 02 13:29:03.443430 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:29:03.443272    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fff161f9-6bfb-4c48-bc75-ade5639c076b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fff161f9-6bfb-4c48-bc75-ade5639c076b-istio-podinfo\") pod \"fff161f9-6bfb-4c48-bc75-ade5639c076b\" (UID: \"fff161f9-6bfb-4c48-bc75-ade5639c076b\") "
Nov 02 13:29:03.443430 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:29:03.443317    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fff161f9-6bfb-4c48-bc75-ade5639c076b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fff161f9-6bfb-4c48-bc75-ade5639c076b-workload-socket\") pod \"fff161f9-6bfb-4c48-bc75-ade5639c076b\" (UID: \"fff161f9-6bfb-4c48-bc75-ade5639c076b\") "
Nov 02 13:29:03.443430 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:29:03.443381    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fff161f9-6bfb-4c48-bc75-ade5639c076b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fff161f9-6bfb-4c48-bc75-ade5639c076b-credential-socket\") pod \"fff161f9-6bfb-4c48-bc75-ade5639c076b\" (UID: \"fff161f9-6bfb-4c48-bc75-ade5639c076b\") "
Nov 02 13:29:03.443430 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:29:03.443414    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fff161f9-6bfb-4c48-bc75-ade5639c076b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fff161f9-6bfb-4c48-bc75-ade5639c076b-istiod-ca-cert\") pod \"fff161f9-6bfb-4c48-bc75-ade5639c076b\" (UID: \"fff161f9-6bfb-4c48-bc75-ade5639c076b\") "
Nov 02 13:29:03.443703 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:29:03.443505    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fff161f9-6bfb-4c48-bc75-ade5639c076b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fff161f9-6bfb-4c48-bc75-ade5639c076b-istio-data\") pod \"fff161f9-6bfb-4c48-bc75-ade5639c076b\" (UID: \"fff161f9-6bfb-4c48-bc75-ade5639c076b\") "
Nov 02 13:29:03.443703 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:29:03.443542    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fff161f9-6bfb-4c48-bc75-ade5639c076b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fff161f9-6bfb-4c48-bc75-ade5639c076b-workload-certs\") pod \"fff161f9-6bfb-4c48-bc75-ade5639c076b\" (UID: \"fff161f9-6bfb-4c48-bc75-ade5639c076b\") "
Nov 02 13:29:03.443834 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:29:03.443706    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fff161f9-6bfb-4c48-bc75-ade5639c076b-workload-socket" pod "fff161f9-6bfb-4c48-bc75-ade5639c076b" (UID: "fff161f9-6bfb-4c48-bc75-ade5639c076b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:29:03.443834 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:29:03.443755    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fff161f9-6bfb-4c48-bc75-ade5639c076b-credential-socket" pod "fff161f9-6bfb-4c48-bc75-ade5639c076b" (UID: "fff161f9-6bfb-4c48-bc75-ade5639c076b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:29:03.443834 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:29:03.443801    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fff161f9-6bfb-4c48-bc75-ade5639c076b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fff161f9-6bfb-4c48-bc75-ade5639c076b-istio-envoy\") pod \"fff161f9-6bfb-4c48-bc75-ade5639c076b\" (UID: \"fff161f9-6bfb-4c48-bc75-ade5639c076b\") "
Nov 02 13:29:03.444009 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:29:03.443836    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fff161f9-6bfb-4c48-bc75-ade5639c076b-kube-api-access-5qw8j\" (UniqueName: \"kubernetes.io/projected/fff161f9-6bfb-4c48-bc75-ade5639c076b-kube-api-access-5qw8j\") pod \"fff161f9-6bfb-4c48-bc75-ade5639c076b\" (UID: \"fff161f9-6bfb-4c48-bc75-ade5639c076b\") "
Nov 02 13:29:03.444079 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:29:03.444003    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fff161f9-6bfb-4c48-bc75-ade5639c076b-istio-data" pod "fff161f9-6bfb-4c48-bc75-ade5639c076b" (UID: "fff161f9-6bfb-4c48-bc75-ade5639c076b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:29:03.444079 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:29:03.444044    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fff161f9-6bfb-4c48-bc75-ade5639c076b-workload-certs" pod "fff161f9-6bfb-4c48-bc75-ade5639c076b" (UID: "fff161f9-6bfb-4c48-bc75-ade5639c076b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:29:03.444450 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:29:03.444420    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fff161f9-6bfb-4c48-bc75-ade5639c076b-istiod-ca-cert" pod "fff161f9-6bfb-4c48-bc75-ade5639c076b" (UID: "fff161f9-6bfb-4c48-bc75-ade5639c076b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:29:03.444570 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:29:03.444480    4559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fff161f9-6bfb-4c48-bc75-ade5639c076b-workload-socket\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:29:03.444570 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:29:03.444506    4559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fff161f9-6bfb-4c48-bc75-ade5639c076b-credential-socket\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:29:03.444570 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:29:03.444520    4559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fff161f9-6bfb-4c48-bc75-ade5639c076b-istio-data\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:29:03.444570 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:29:03.444533    4559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fff161f9-6bfb-4c48-bc75-ade5639c076b-workload-certs\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:29:03.446578 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:29:03.446542    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fff161f9-6bfb-4c48-bc75-ade5639c076b-istio-envoy" pod "fff161f9-6bfb-4c48-bc75-ade5639c076b" (UID: "fff161f9-6bfb-4c48-bc75-ade5639c076b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:29:03.446710 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:29:03.446576    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fff161f9-6bfb-4c48-bc75-ade5639c076b-istio-token" pod "fff161f9-6bfb-4c48-bc75-ade5639c076b" (UID: "fff161f9-6bfb-4c48-bc75-ade5639c076b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:29:03.446710 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:29:03.446629    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fff161f9-6bfb-4c48-bc75-ade5639c076b-kube-api-access-5qw8j" pod "fff161f9-6bfb-4c48-bc75-ade5639c076b" (UID: "fff161f9-6bfb-4c48-bc75-ade5639c076b"). InnerVolumeSpecName "kube-api-access-5qw8j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:29:03.446710 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:29:03.446695    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fff161f9-6bfb-4c48-bc75-ade5639c076b-istio-podinfo" pod "fff161f9-6bfb-4c48-bc75-ade5639c076b" (UID: "fff161f9-6bfb-4c48-bc75-ade5639c076b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 13:29:03.546239 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:29:03.546196    4559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fff161f9-6bfb-4c48-bc75-ade5639c076b-istio-token\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:29:03.546239 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:29:03.546242    4559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fff161f9-6bfb-4c48-bc75-ade5639c076b-istio-podinfo\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:29:03.546523 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:29:03.546257    4559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fff161f9-6bfb-4c48-bc75-ade5639c076b-istiod-ca-cert\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:29:03.546523 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:29:03.546271    4559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fff161f9-6bfb-4c48-bc75-ade5639c076b-istio-envoy\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:29:03.546523 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:29:03.546285    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5qw8j\" (UniqueName: \"kubernetes.io/projected/fff161f9-6bfb-4c48-bc75-ade5639c076b-kube-api-access-5qw8j\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:29:04.112829 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:29:04.112795    4559 generic.go:356] "Generic (PLEG): container finished" podID="fff161f9-6bfb-4c48-bc75-ade5639c076b" containerID="528f01ff77310124bbd9ae6fc46db8b3a73688035e354d78e588466c759a730e" exitCode=0
Nov 02 13:29:04.112998 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:29:04.112857    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b8a8dc4c-openshift-default-699557545-57rbb"
Nov 02 13:29:04.112998 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:29:04.112877    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b8a8dc4c-openshift-default-699557545-57rbb" event={"ID":"fff161f9-6bfb-4c48-bc75-ade5639c076b","Type":"ContainerDied","Data":"528f01ff77310124bbd9ae6fc46db8b3a73688035e354d78e588466c759a730e"}
Nov 02 13:29:04.112998 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:29:04.112936    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b8a8dc4c-openshift-default-699557545-57rbb" event={"ID":"fff161f9-6bfb-4c48-bc75-ade5639c076b","Type":"ContainerDied","Data":"24bc3e16b6c32198e09db76b6db1827988baa4784ea7c222d71078aaa3c06979"}
Nov 02 13:29:04.112998 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:29:04.112958    4559 scope.go:122] "RemoveContainer" containerID="528f01ff77310124bbd9ae6fc46db8b3a73688035e354d78e588466c759a730e"
Nov 02 13:29:04.123302 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:29:04.123276    4559 scope.go:122] "RemoveContainer" containerID="1e2e7dacc8266b31ed051a83120561e72f02173086999afda21e17815be14972"
Nov 02 13:29:04.132393 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:29:04.132373    4559 scope.go:122] "RemoveContainer" containerID="528f01ff77310124bbd9ae6fc46db8b3a73688035e354d78e588466c759a730e"
Nov 02 13:29:04.132709 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:29:04.132688    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"528f01ff77310124bbd9ae6fc46db8b3a73688035e354d78e588466c759a730e\": container with ID starting with 528f01ff77310124bbd9ae6fc46db8b3a73688035e354d78e588466c759a730e not found: ID does not exist" containerID="528f01ff77310124bbd9ae6fc46db8b3a73688035e354d78e588466c759a730e"
Nov 02 13:29:04.132773 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:29:04.132721    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"528f01ff77310124bbd9ae6fc46db8b3a73688035e354d78e588466c759a730e"} err="failed to get container status \"528f01ff77310124bbd9ae6fc46db8b3a73688035e354d78e588466c759a730e\": rpc error: code = NotFound desc = could not find container \"528f01ff77310124bbd9ae6fc46db8b3a73688035e354d78e588466c759a730e\": container with ID starting with 528f01ff77310124bbd9ae6fc46db8b3a73688035e354d78e588466c759a730e not found: ID does not exist"
Nov 02 13:29:04.132773 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:29:04.132745    4559 scope.go:122] "RemoveContainer" containerID="1e2e7dacc8266b31ed051a83120561e72f02173086999afda21e17815be14972"
Nov 02 13:29:04.133010 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:29:04.132989    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e2e7dacc8266b31ed051a83120561e72f02173086999afda21e17815be14972\": container with ID starting with 1e2e7dacc8266b31ed051a83120561e72f02173086999afda21e17815be14972 not found: ID does not exist" containerID="1e2e7dacc8266b31ed051a83120561e72f02173086999afda21e17815be14972"
Nov 02 13:29:04.133072 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:29:04.133020    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e2e7dacc8266b31ed051a83120561e72f02173086999afda21e17815be14972"} err="failed to get container status \"1e2e7dacc8266b31ed051a83120561e72f02173086999afda21e17815be14972\": rpc error: code = NotFound desc = could not find container \"1e2e7dacc8266b31ed051a83120561e72f02173086999afda21e17815be14972\": container with ID starting with 1e2e7dacc8266b31ed051a83120561e72f02173086999afda21e17815be14972 not found: ID does not exist"
Nov 02 13:29:04.185322 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:29:04.185293    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-b8a8dc4c-openshift-default-699557545-57rbb"]
Nov 02 13:29:04.198423 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:29:04.198395    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-b8a8dc4c-openshift-default-699557545-57rbb"]
Nov 02 13:29:05.893977 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:29:05.893946    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fff161f9-6bfb-4c48-bc75-ade5639c076b" path="/var/lib/kubelet/pods/fff161f9-6bfb-4c48-bc75-ade5639c076b/volumes"
Nov 02 13:29:10.804149 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:29:10.804109    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-035dac-7bf5549cc6-hd2vb"]
Nov 02 13:29:10.804670 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:29:10.804289    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-035dac-7bf5549cc6-hd2vb" podUID="cc6f8750-45f1-4dfe-b2a0-f381d0dde490" containerName="maas-api" containerID="cri-o://f816912a6917249d7fbfe5103d545f57c66a0a443764fa6d03da2d7666b4f066" gracePeriod=30
Nov 02 13:29:11.073374 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:29:11.073315    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-035dac-7bf5549cc6-hd2vb"
Nov 02 13:29:11.148464 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:29:11.148435    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc6f8750-45f1-4dfe-b2a0-f381d0dde490-kube-api-access-79z7q\" (UniqueName: \"kubernetes.io/projected/cc6f8750-45f1-4dfe-b2a0-f381d0dde490-kube-api-access-79z7q\") pod \"cc6f8750-45f1-4dfe-b2a0-f381d0dde490\" (UID: \"cc6f8750-45f1-4dfe-b2a0-f381d0dde490\") "
Nov 02 13:29:11.148653 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:29:11.148478    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cc6f8750-45f1-4dfe-b2a0-f381d0dde490-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc6f8750-45f1-4dfe-b2a0-f381d0dde490-maas-api-tls\") pod \"cc6f8750-45f1-4dfe-b2a0-f381d0dde490\" (UID: \"cc6f8750-45f1-4dfe-b2a0-f381d0dde490\") "
Nov 02 13:29:11.148653 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:29:11.148627    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cc6f8750-45f1-4dfe-b2a0-f381d0dde490-metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc6f8750-45f1-4dfe-b2a0-f381d0dde490-metrics-certs\") pod \"cc6f8750-45f1-4dfe-b2a0-f381d0dde490\" (UID: \"cc6f8750-45f1-4dfe-b2a0-f381d0dde490\") "
Nov 02 13:29:11.148773 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:29:11.148755    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc6f8750-45f1-4dfe-b2a0-f381d0dde490-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc6f8750-45f1-4dfe-b2a0-f381d0dde490-openshift-service-ca\") pod \"cc6f8750-45f1-4dfe-b2a0-f381d0dde490\" (UID: \"cc6f8750-45f1-4dfe-b2a0-f381d0dde490\") "
Nov 02 13:29:11.149756 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:29:11.149724    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc6f8750-45f1-4dfe-b2a0-f381d0dde490-openshift-service-ca" pod "cc6f8750-45f1-4dfe-b2a0-f381d0dde490" (UID: "cc6f8750-45f1-4dfe-b2a0-f381d0dde490"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:29:11.150921 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:29:11.150891    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc6f8750-45f1-4dfe-b2a0-f381d0dde490-maas-api-tls" pod "cc6f8750-45f1-4dfe-b2a0-f381d0dde490" (UID: "cc6f8750-45f1-4dfe-b2a0-f381d0dde490"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:29:11.151188 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:29:11.151149    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc6f8750-45f1-4dfe-b2a0-f381d0dde490-kube-api-access-79z7q" pod "cc6f8750-45f1-4dfe-b2a0-f381d0dde490" (UID: "cc6f8750-45f1-4dfe-b2a0-f381d0dde490"). InnerVolumeSpecName "kube-api-access-79z7q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:29:11.151553 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:29:11.151532    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc6f8750-45f1-4dfe-b2a0-f381d0dde490-metrics-certs" pod "cc6f8750-45f1-4dfe-b2a0-f381d0dde490" (UID: "cc6f8750-45f1-4dfe-b2a0-f381d0dde490"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:29:11.160599 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:29:11.160572    4559 generic.go:356] "Generic (PLEG): container finished" podID="cc6f8750-45f1-4dfe-b2a0-f381d0dde490" containerID="f816912a6917249d7fbfe5103d545f57c66a0a443764fa6d03da2d7666b4f066" exitCode=0
Nov 02 13:29:11.160739 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:29:11.160614    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-035dac-7bf5549cc6-hd2vb"
Nov 02 13:29:11.160739 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:29:11.160676    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-035dac-7bf5549cc6-hd2vb" event={"ID":"cc6f8750-45f1-4dfe-b2a0-f381d0dde490","Type":"ContainerDied","Data":"f816912a6917249d7fbfe5103d545f57c66a0a443764fa6d03da2d7666b4f066"}
Nov 02 13:29:11.160739 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:29:11.160733    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-035dac-7bf5549cc6-hd2vb" event={"ID":"cc6f8750-45f1-4dfe-b2a0-f381d0dde490","Type":"ContainerDied","Data":"bdcbf9f676bdeab5d7274e8e1c2d1c3deefdb53ddc940219fa4ca256d59e7f79"}
Nov 02 13:29:11.160920 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:29:11.160764    4559 scope.go:122] "RemoveContainer" containerID="f816912a6917249d7fbfe5103d545f57c66a0a443764fa6d03da2d7666b4f066"
Nov 02 13:29:11.186705 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:29:11.186684    4559 scope.go:122] "RemoveContainer" containerID="f816912a6917249d7fbfe5103d545f57c66a0a443764fa6d03da2d7666b4f066"
Nov 02 13:29:11.187030 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:29:11.187008    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f816912a6917249d7fbfe5103d545f57c66a0a443764fa6d03da2d7666b4f066\": container with ID starting with f816912a6917249d7fbfe5103d545f57c66a0a443764fa6d03da2d7666b4f066 not found: ID does not exist" containerID="f816912a6917249d7fbfe5103d545f57c66a0a443764fa6d03da2d7666b4f066"
Nov 02 13:29:11.187087 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:29:11.187043    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f816912a6917249d7fbfe5103d545f57c66a0a443764fa6d03da2d7666b4f066"} err="failed to get container status \"f816912a6917249d7fbfe5103d545f57c66a0a443764fa6d03da2d7666b4f066\": rpc error: code = NotFound desc = could not find container \"f816912a6917249d7fbfe5103d545f57c66a0a443764fa6d03da2d7666b4f066\": container with ID starting with f816912a6917249d7fbfe5103d545f57c66a0a443764fa6d03da2d7666b4f066 not found: ID does not exist"
Nov 02 13:29:11.224253 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:29:11.224224    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-035dac-7bf5549cc6-hd2vb"]
Nov 02 13:29:11.237477 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:29:11.237453    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-035dac-7bf5549cc6-hd2vb"]
Nov 02 13:29:11.250917 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:29:11.250893    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79z7q\" (UniqueName: \"kubernetes.io/projected/cc6f8750-45f1-4dfe-b2a0-f381d0dde490-kube-api-access-79z7q\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:29:11.250917 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:29:11.250915    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc6f8750-45f1-4dfe-b2a0-f381d0dde490-maas-api-tls\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:29:11.251059 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:29:11.250925    4559 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc6f8750-45f1-4dfe-b2a0-f381d0dde490-metrics-certs\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:29:11.251059 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:29:11.250934    4559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc6f8750-45f1-4dfe-b2a0-f381d0dde490-openshift-service-ca\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:29:11.893686 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:29:11.893619    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc6f8750-45f1-4dfe-b2a0-f381d0dde490" path="/var/lib/kubelet/pods/cc6f8750-45f1-4dfe-b2a0-f381d0dde490/volumes"
Nov 02 13:29:15.087097 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:29:15.087052    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqcmj" containerName="extractor"
Nov 02 13:29:21.457391 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:29:21.457352    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-035dac-openshift-default-6d5cb5658f-8kmlt"]
Nov 02 13:29:21.457849 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:29:21.457689    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-035dac-openshift-default-6d5cb5658f-8kmlt" podUID="a47a44e4-6226-4ce7-9760-58ee5eaf65bf" containerName="istio-proxy" containerID="cri-o://1e8ff1f9ec03ea2de2cb7c3549205a4866938f527053c6caf9b6492e9438362e" gracePeriod=30
Nov 02 13:29:26.090508 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:29:26.090473    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"]
Nov 02 13:29:26.757935 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:29:26.757902    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-035dac-openshift-default-6d5cb5658f-8kmlt"
Nov 02 13:29:26.808892 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:29:26.807195    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a47a44e4-6226-4ce7-9760-58ee5eaf65bf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a47a44e4-6226-4ce7-9760-58ee5eaf65bf-workload-socket\") pod \"a47a44e4-6226-4ce7-9760-58ee5eaf65bf\" (UID: \"a47a44e4-6226-4ce7-9760-58ee5eaf65bf\") "
Nov 02 13:29:26.808892 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:29:26.807260    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a47a44e4-6226-4ce7-9760-58ee5eaf65bf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a47a44e4-6226-4ce7-9760-58ee5eaf65bf-istio-envoy\") pod \"a47a44e4-6226-4ce7-9760-58ee5eaf65bf\" (UID: \"a47a44e4-6226-4ce7-9760-58ee5eaf65bf\") "
Nov 02 13:29:26.808892 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:29:26.807314    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a47a44e4-6226-4ce7-9760-58ee5eaf65bf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a47a44e4-6226-4ce7-9760-58ee5eaf65bf-istio-podinfo\") pod \"a47a44e4-6226-4ce7-9760-58ee5eaf65bf\" (UID: \"a47a44e4-6226-4ce7-9760-58ee5eaf65bf\") "
Nov 02 13:29:26.808892 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:29:26.807365    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a47a44e4-6226-4ce7-9760-58ee5eaf65bf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a47a44e4-6226-4ce7-9760-58ee5eaf65bf-credential-socket\") pod \"a47a44e4-6226-4ce7-9760-58ee5eaf65bf\" (UID: \"a47a44e4-6226-4ce7-9760-58ee5eaf65bf\") "
Nov 02 13:29:26.808892 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:29:26.807407    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a47a44e4-6226-4ce7-9760-58ee5eaf65bf-istio-token\" (UniqueName: \"kubernetes.io/projected/a47a44e4-6226-4ce7-9760-58ee5eaf65bf-istio-token\") pod \"a47a44e4-6226-4ce7-9760-58ee5eaf65bf\" (UID: \"a47a44e4-6226-4ce7-9760-58ee5eaf65bf\") "
Nov 02 13:29:26.808892 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:29:26.807540    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a47a44e4-6226-4ce7-9760-58ee5eaf65bf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a47a44e4-6226-4ce7-9760-58ee5eaf65bf-istiod-ca-cert\") pod \"a47a44e4-6226-4ce7-9760-58ee5eaf65bf\" (UID: \"a47a44e4-6226-4ce7-9760-58ee5eaf65bf\") "
Nov 02 13:29:26.808892 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:29:26.807567    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a47a44e4-6226-4ce7-9760-58ee5eaf65bf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a47a44e4-6226-4ce7-9760-58ee5eaf65bf-istio-data\") pod \"a47a44e4-6226-4ce7-9760-58ee5eaf65bf\" (UID: \"a47a44e4-6226-4ce7-9760-58ee5eaf65bf\") "
Nov 02 13:29:26.808892 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:29:26.807600    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a47a44e4-6226-4ce7-9760-58ee5eaf65bf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a47a44e4-6226-4ce7-9760-58ee5eaf65bf-workload-certs\") pod \"a47a44e4-6226-4ce7-9760-58ee5eaf65bf\" (UID: \"a47a44e4-6226-4ce7-9760-58ee5eaf65bf\") "
Nov 02 13:29:26.808892 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:29:26.807766    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a47a44e4-6226-4ce7-9760-58ee5eaf65bf-kube-api-access-ghmvm\" (UniqueName: \"kubernetes.io/projected/a47a44e4-6226-4ce7-9760-58ee5eaf65bf-kube-api-access-ghmvm\") pod \"a47a44e4-6226-4ce7-9760-58ee5eaf65bf\" (UID: \"a47a44e4-6226-4ce7-9760-58ee5eaf65bf\") "
Nov 02 13:29:26.812131 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:29:26.809306    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a47a44e4-6226-4ce7-9760-58ee5eaf65bf-workload-socket" pod "a47a44e4-6226-4ce7-9760-58ee5eaf65bf" (UID: "a47a44e4-6226-4ce7-9760-58ee5eaf65bf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:29:26.850688 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:29:26.812820    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a47a44e4-6226-4ce7-9760-58ee5eaf65bf-credential-socket" pod "a47a44e4-6226-4ce7-9760-58ee5eaf65bf" (UID: "a47a44e4-6226-4ce7-9760-58ee5eaf65bf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:29:26.850688 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:29:26.813028    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a47a44e4-6226-4ce7-9760-58ee5eaf65bf-workload-certs" pod "a47a44e4-6226-4ce7-9760-58ee5eaf65bf" (UID: "a47a44e4-6226-4ce7-9760-58ee5eaf65bf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:29:26.850688 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:29:26.813372    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a47a44e4-6226-4ce7-9760-58ee5eaf65bf-istiod-ca-cert" pod "a47a44e4-6226-4ce7-9760-58ee5eaf65bf" (UID: "a47a44e4-6226-4ce7-9760-58ee5eaf65bf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:29:26.850688 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:29:26.813712    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a47a44e4-6226-4ce7-9760-58ee5eaf65bf-istio-data" pod "a47a44e4-6226-4ce7-9760-58ee5eaf65bf" (UID: "a47a44e4-6226-4ce7-9760-58ee5eaf65bf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:29:26.850688 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:29:26.816268    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a47a44e4-6226-4ce7-9760-58ee5eaf65bf-kube-api-access-ghmvm" pod "a47a44e4-6226-4ce7-9760-58ee5eaf65bf" (UID: "a47a44e4-6226-4ce7-9760-58ee5eaf65bf"). InnerVolumeSpecName "kube-api-access-ghmvm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:29:26.857056 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:29:26.857008    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a47a44e4-6226-4ce7-9760-58ee5eaf65bf-istio-envoy" pod "a47a44e4-6226-4ce7-9760-58ee5eaf65bf" (UID: "a47a44e4-6226-4ce7-9760-58ee5eaf65bf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:29:26.857235 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:29:26.857161    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a47a44e4-6226-4ce7-9760-58ee5eaf65bf-istio-token" pod "a47a44e4-6226-4ce7-9760-58ee5eaf65bf" (UID: "a47a44e4-6226-4ce7-9760-58ee5eaf65bf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:29:26.858026 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:29:26.857995    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a47a44e4-6226-4ce7-9760-58ee5eaf65bf-istio-podinfo" pod "a47a44e4-6226-4ce7-9760-58ee5eaf65bf" (UID: "a47a44e4-6226-4ce7-9760-58ee5eaf65bf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 13:29:26.909659 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:29:26.909611    4559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a47a44e4-6226-4ce7-9760-58ee5eaf65bf-workload-socket\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:29:26.909881 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:29:26.909851    4559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a47a44e4-6226-4ce7-9760-58ee5eaf65bf-istio-envoy\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:29:26.909881 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:29:26.909881    4559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a47a44e4-6226-4ce7-9760-58ee5eaf65bf-istio-podinfo\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:29:26.910041 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:29:26.909896    4559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a47a44e4-6226-4ce7-9760-58ee5eaf65bf-credential-socket\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:29:26.910041 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:29:26.909911    4559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a47a44e4-6226-4ce7-9760-58ee5eaf65bf-istio-token\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:29:26.910041 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:29:26.909925    4559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a47a44e4-6226-4ce7-9760-58ee5eaf65bf-istiod-ca-cert\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:29:26.910041 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:29:26.909939    4559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a47a44e4-6226-4ce7-9760-58ee5eaf65bf-istio-data\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:29:26.910041 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:29:26.909951    4559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a47a44e4-6226-4ce7-9760-58ee5eaf65bf-workload-certs\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:29:26.910041 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:29:26.909964    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ghmvm\" (UniqueName: \"kubernetes.io/projected/a47a44e4-6226-4ce7-9760-58ee5eaf65bf-kube-api-access-ghmvm\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:29:27.273175 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:29:27.273142    4559 generic.go:356] "Generic (PLEG): container finished" podID="a47a44e4-6226-4ce7-9760-58ee5eaf65bf" containerID="1e8ff1f9ec03ea2de2cb7c3549205a4866938f527053c6caf9b6492e9438362e" exitCode=0
Nov 02 13:29:27.273733 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:29:27.273193    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-035dac-openshift-default-6d5cb5658f-8kmlt"
Nov 02 13:29:27.273733 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:29:27.273205    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-035dac-openshift-default-6d5cb5658f-8kmlt" event={"ID":"a47a44e4-6226-4ce7-9760-58ee5eaf65bf","Type":"ContainerDied","Data":"1e8ff1f9ec03ea2de2cb7c3549205a4866938f527053c6caf9b6492e9438362e"}
Nov 02 13:29:27.273733 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:29:27.273246    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-035dac-openshift-default-6d5cb5658f-8kmlt" event={"ID":"a47a44e4-6226-4ce7-9760-58ee5eaf65bf","Type":"ContainerDied","Data":"88e9ca2c9abc2d5a75a7978f6f13377320be40b38745dda93599a990ec042371"}
Nov 02 13:29:27.273733 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:29:27.273266    4559 scope.go:122] "RemoveContainer" containerID="1e8ff1f9ec03ea2de2cb7c3549205a4866938f527053c6caf9b6492e9438362e"
Nov 02 13:29:27.286710 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:29:27.286689    4559 scope.go:122] "RemoveContainer" containerID="215bfa85b115960b75bd02c0ea3213a462ea4cb0494f640b969ee693e9c7d93c"
Nov 02 13:29:27.300077 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:29:27.299998    4559 scope.go:122] "RemoveContainer" containerID="1e8ff1f9ec03ea2de2cb7c3549205a4866938f527053c6caf9b6492e9438362e"
Nov 02 13:29:27.304678 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:29:27.304495    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e8ff1f9ec03ea2de2cb7c3549205a4866938f527053c6caf9b6492e9438362e\": container with ID starting with 1e8ff1f9ec03ea2de2cb7c3549205a4866938f527053c6caf9b6492e9438362e not found: ID does not exist" containerID="1e8ff1f9ec03ea2de2cb7c3549205a4866938f527053c6caf9b6492e9438362e"
Nov 02 13:29:27.304678 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:29:27.304536    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e8ff1f9ec03ea2de2cb7c3549205a4866938f527053c6caf9b6492e9438362e"} err="failed to get container status \"1e8ff1f9ec03ea2de2cb7c3549205a4866938f527053c6caf9b6492e9438362e\": rpc error: code = NotFound desc = could not find container \"1e8ff1f9ec03ea2de2cb7c3549205a4866938f527053c6caf9b6492e9438362e\": container with ID starting with 1e8ff1f9ec03ea2de2cb7c3549205a4866938f527053c6caf9b6492e9438362e not found: ID does not exist"
Nov 02 13:29:27.304678 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:29:27.304561    4559 scope.go:122] "RemoveContainer" containerID="215bfa85b115960b75bd02c0ea3213a462ea4cb0494f640b969ee693e9c7d93c"
Nov 02 13:29:27.305420 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:29:27.305400    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"215bfa85b115960b75bd02c0ea3213a462ea4cb0494f640b969ee693e9c7d93c\": container with ID starting with 215bfa85b115960b75bd02c0ea3213a462ea4cb0494f640b969ee693e9c7d93c not found: ID does not exist" containerID="215bfa85b115960b75bd02c0ea3213a462ea4cb0494f640b969ee693e9c7d93c"
Nov 02 13:29:27.305504 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:29:27.305430    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"215bfa85b115960b75bd02c0ea3213a462ea4cb0494f640b969ee693e9c7d93c"} err="failed to get container status \"215bfa85b115960b75bd02c0ea3213a462ea4cb0494f640b969ee693e9c7d93c\": rpc error: code = NotFound desc = could not find container \"215bfa85b115960b75bd02c0ea3213a462ea4cb0494f640b969ee693e9c7d93c\": container with ID starting with 215bfa85b115960b75bd02c0ea3213a462ea4cb0494f640b969ee693e9c7d93c not found: ID does not exist"
Nov 02 13:29:27.341475 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:29:27.341441    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-035dac-openshift-default-6d5cb5658f-8kmlt"]
Nov 02 13:29:27.354095 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:29:27.354067    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-035dac-openshift-default-6d5cb5658f-8kmlt"]
Nov 02 13:29:27.894452 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:29:27.894415    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a47a44e4-6226-4ce7-9760-58ee5eaf65bf" path="/var/lib/kubelet/pods/a47a44e4-6226-4ce7-9760-58ee5eaf65bf/volumes"
Nov 02 13:29:34.747426 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:29:34.747383    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-936529-d8d95fdc9-62p9f"]
Nov 02 13:29:34.747948 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:29:34.747662    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-936529-d8d95fdc9-62p9f" podUID="70323c42-ec34-40c6-a906-d646b3025523" containerName="maas-api" containerID="cri-o://ecab3a9a4db6830de1e1e398be6871fde6add381f0857429ee57626c40979cfc" gracePeriod=30
Nov 02 13:29:35.063714 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:29:35.062867    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-936529-d8d95fdc9-62p9f"
Nov 02 13:29:35.127041 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:29:35.126995    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/70323c42-ec34-40c6-a906-d646b3025523-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70323c42-ec34-40c6-a906-d646b3025523-maas-api-tls\") pod \"70323c42-ec34-40c6-a906-d646b3025523\" (UID: \"70323c42-ec34-40c6-a906-d646b3025523\") "
Nov 02 13:29:35.127438 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:29:35.127407    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70323c42-ec34-40c6-a906-d646b3025523-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/70323c42-ec34-40c6-a906-d646b3025523-openshift-service-ca\") pod \"70323c42-ec34-40c6-a906-d646b3025523\" (UID: \"70323c42-ec34-40c6-a906-d646b3025523\") "
Nov 02 13:29:35.127571 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:29:35.127480    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/70323c42-ec34-40c6-a906-d646b3025523-metrics-certs\" (UniqueName: \"kubernetes.io/secret/70323c42-ec34-40c6-a906-d646b3025523-metrics-certs\") pod \"70323c42-ec34-40c6-a906-d646b3025523\" (UID: \"70323c42-ec34-40c6-a906-d646b3025523\") "
Nov 02 13:29:35.127571 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:29:35.127559    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70323c42-ec34-40c6-a906-d646b3025523-kube-api-access-fg9f8\" (UniqueName: \"kubernetes.io/projected/70323c42-ec34-40c6-a906-d646b3025523-kube-api-access-fg9f8\") pod \"70323c42-ec34-40c6-a906-d646b3025523\" (UID: \"70323c42-ec34-40c6-a906-d646b3025523\") "
Nov 02 13:29:35.127818 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:29:35.127778    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70323c42-ec34-40c6-a906-d646b3025523-openshift-service-ca" pod "70323c42-ec34-40c6-a906-d646b3025523" (UID: "70323c42-ec34-40c6-a906-d646b3025523"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:29:35.128241 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:29:35.128215    4559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/70323c42-ec34-40c6-a906-d646b3025523-openshift-service-ca\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:29:35.132077 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:29:35.132053    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70323c42-ec34-40c6-a906-d646b3025523-metrics-certs" pod "70323c42-ec34-40c6-a906-d646b3025523" (UID: "70323c42-ec34-40c6-a906-d646b3025523"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:29:35.132166 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:29:35.132075    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70323c42-ec34-40c6-a906-d646b3025523-kube-api-access-fg9f8" pod "70323c42-ec34-40c6-a906-d646b3025523" (UID: "70323c42-ec34-40c6-a906-d646b3025523"). InnerVolumeSpecName "kube-api-access-fg9f8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:29:35.133631 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:29:35.133607    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70323c42-ec34-40c6-a906-d646b3025523-maas-api-tls" pod "70323c42-ec34-40c6-a906-d646b3025523" (UID: "70323c42-ec34-40c6-a906-d646b3025523"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:29:35.258756 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:29:35.258711    4559 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70323c42-ec34-40c6-a906-d646b3025523-metrics-certs\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:29:35.258756 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:29:35.258756    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fg9f8\" (UniqueName: \"kubernetes.io/projected/70323c42-ec34-40c6-a906-d646b3025523-kube-api-access-fg9f8\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:29:35.259120 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:29:35.258771    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70323c42-ec34-40c6-a906-d646b3025523-maas-api-tls\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:29:35.366523 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:29:35.366437    4559 generic.go:356] "Generic (PLEG): container finished" podID="70323c42-ec34-40c6-a906-d646b3025523" containerID="ecab3a9a4db6830de1e1e398be6871fde6add381f0857429ee57626c40979cfc" exitCode=0
Nov 02 13:29:35.366523 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:29:35.366498    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-936529-d8d95fdc9-62p9f"
Nov 02 13:29:35.366747 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:29:35.366516    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-936529-d8d95fdc9-62p9f" event={"ID":"70323c42-ec34-40c6-a906-d646b3025523","Type":"ContainerDied","Data":"ecab3a9a4db6830de1e1e398be6871fde6add381f0857429ee57626c40979cfc"}
Nov 02 13:29:35.366747 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:29:35.366570    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-936529-d8d95fdc9-62p9f" event={"ID":"70323c42-ec34-40c6-a906-d646b3025523","Type":"ContainerDied","Data":"e7b67b328b1771abbe7bf3a745ad4cebdb057923bb87e99b68137b7ba8dff7d9"}
Nov 02 13:29:35.366747 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:29:35.366591    4559 scope.go:122] "RemoveContainer" containerID="ecab3a9a4db6830de1e1e398be6871fde6add381f0857429ee57626c40979cfc"
Nov 02 13:29:35.378754 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:29:35.378670    4559 scope.go:122] "RemoveContainer" containerID="ecab3a9a4db6830de1e1e398be6871fde6add381f0857429ee57626c40979cfc"
Nov 02 13:29:35.379011 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:29:35.378985    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ecab3a9a4db6830de1e1e398be6871fde6add381f0857429ee57626c40979cfc\": container with ID starting with ecab3a9a4db6830de1e1e398be6871fde6add381f0857429ee57626c40979cfc not found: ID does not exist" containerID="ecab3a9a4db6830de1e1e398be6871fde6add381f0857429ee57626c40979cfc"
Nov 02 13:29:35.379114 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:29:35.379021    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ecab3a9a4db6830de1e1e398be6871fde6add381f0857429ee57626c40979cfc"} err="failed to get container status \"ecab3a9a4db6830de1e1e398be6871fde6add381f0857429ee57626c40979cfc\": rpc error: code = NotFound desc = could not find container \"ecab3a9a4db6830de1e1e398be6871fde6add381f0857429ee57626c40979cfc\": container with ID starting with ecab3a9a4db6830de1e1e398be6871fde6add381f0857429ee57626c40979cfc not found: ID does not exist"
Nov 02 13:29:35.427183 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:29:35.427159    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-936529-d8d95fdc9-62p9f"]
Nov 02 13:29:35.439598 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:29:35.439576    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-936529-d8d95fdc9-62p9f"]
Nov 02 13:29:35.894877 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:29:35.894844    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70323c42-ec34-40c6-a906-d646b3025523" path="/var/lib/kubelet/pods/70323c42-ec34-40c6-a906-d646b3025523/volumes"
Nov 02 13:29:45.085872 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:29:45.085832    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqcmj" containerName="extractor"
Nov 02 13:29:46.371652 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:29:46.371616    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-936529-openshift-default-5578bccc97-9cw4k"]
Nov 02 13:29:46.372099 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:29:46.371851    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-936529-openshift-default-5578bccc97-9cw4k" podUID="6ae3594b-0d9b-431e-b165-bb290002858a" containerName="istio-proxy" containerID="cri-o://ce6a7a7afb91e946a884853ec8f96119491db4ef21f11aeb621a8d0189d32f66" gracePeriod=30
Nov 02 13:29:51.652956 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:29:51.652923    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-936529-openshift-default-5578bccc97-9cw4k"
Nov 02 13:29:51.785936 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:29:51.785902    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ae3594b-0d9b-431e-b165-bb290002858a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ae3594b-0d9b-431e-b165-bb290002858a-istio-data\") pod \"6ae3594b-0d9b-431e-b165-bb290002858a\" (UID: \"6ae3594b-0d9b-431e-b165-bb290002858a\") "
Nov 02 13:29:51.786117 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:29:51.785953    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ae3594b-0d9b-431e-b165-bb290002858a-istio-token\" (UniqueName: \"kubernetes.io/projected/6ae3594b-0d9b-431e-b165-bb290002858a-istio-token\") pod \"6ae3594b-0d9b-431e-b165-bb290002858a\" (UID: \"6ae3594b-0d9b-431e-b165-bb290002858a\") "
Nov 02 13:29:51.786117 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:29:51.786025    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6ae3594b-0d9b-431e-b165-bb290002858a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ae3594b-0d9b-431e-b165-bb290002858a-istio-podinfo\") pod \"6ae3594b-0d9b-431e-b165-bb290002858a\" (UID: \"6ae3594b-0d9b-431e-b165-bb290002858a\") "
Nov 02 13:29:51.786117 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:29:51.786058    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ae3594b-0d9b-431e-b165-bb290002858a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ae3594b-0d9b-431e-b165-bb290002858a-istiod-ca-cert\") pod \"6ae3594b-0d9b-431e-b165-bb290002858a\" (UID: \"6ae3594b-0d9b-431e-b165-bb290002858a\") "
Nov 02 13:29:51.786117 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:29:51.786085    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ae3594b-0d9b-431e-b165-bb290002858a-kube-api-access-kck68\" (UniqueName: \"kubernetes.io/projected/6ae3594b-0d9b-431e-b165-bb290002858a-kube-api-access-kck68\") pod \"6ae3594b-0d9b-431e-b165-bb290002858a\" (UID: \"6ae3594b-0d9b-431e-b165-bb290002858a\") "
Nov 02 13:29:51.786419 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:29:51.786126    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ae3594b-0d9b-431e-b165-bb290002858a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ae3594b-0d9b-431e-b165-bb290002858a-workload-socket\") pod \"6ae3594b-0d9b-431e-b165-bb290002858a\" (UID: \"6ae3594b-0d9b-431e-b165-bb290002858a\") "
Nov 02 13:29:51.786419 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:29:51.786156    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ae3594b-0d9b-431e-b165-bb290002858a-istio-data" pod "6ae3594b-0d9b-431e-b165-bb290002858a" (UID: "6ae3594b-0d9b-431e-b165-bb290002858a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:29:51.786419 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:29:51.786199    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ae3594b-0d9b-431e-b165-bb290002858a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ae3594b-0d9b-431e-b165-bb290002858a-workload-certs\") pod \"6ae3594b-0d9b-431e-b165-bb290002858a\" (UID: \"6ae3594b-0d9b-431e-b165-bb290002858a\") "
Nov 02 13:29:51.786419 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:29:51.786297    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ae3594b-0d9b-431e-b165-bb290002858a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ae3594b-0d9b-431e-b165-bb290002858a-credential-socket\") pod \"6ae3594b-0d9b-431e-b165-bb290002858a\" (UID: \"6ae3594b-0d9b-431e-b165-bb290002858a\") "
Nov 02 13:29:51.786419 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:29:51.786356    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ae3594b-0d9b-431e-b165-bb290002858a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ae3594b-0d9b-431e-b165-bb290002858a-istio-envoy\") pod \"6ae3594b-0d9b-431e-b165-bb290002858a\" (UID: \"6ae3594b-0d9b-431e-b165-bb290002858a\") "
Nov 02 13:29:51.786752 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:29:51.786578    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ae3594b-0d9b-431e-b165-bb290002858a-workload-socket" pod "6ae3594b-0d9b-431e-b165-bb290002858a" (UID: "6ae3594b-0d9b-431e-b165-bb290002858a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:29:51.786752 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:29:51.786615    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ae3594b-0d9b-431e-b165-bb290002858a-istiod-ca-cert" pod "6ae3594b-0d9b-431e-b165-bb290002858a" (UID: "6ae3594b-0d9b-431e-b165-bb290002858a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:29:51.786846 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:29:51.786822    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ae3594b-0d9b-431e-b165-bb290002858a-credential-socket" pod "6ae3594b-0d9b-431e-b165-bb290002858a" (UID: "6ae3594b-0d9b-431e-b165-bb290002858a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:29:51.787070 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:29:51.787046    4559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ae3594b-0d9b-431e-b165-bb290002858a-istio-data\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:29:51.787143 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:29:51.787071    4559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ae3594b-0d9b-431e-b165-bb290002858a-istiod-ca-cert\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:29:51.787143 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:29:51.787086    4559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ae3594b-0d9b-431e-b165-bb290002858a-workload-socket\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:29:51.787143 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:29:51.787101    4559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ae3594b-0d9b-431e-b165-bb290002858a-credential-socket\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:29:51.787273 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:29:51.787153    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ae3594b-0d9b-431e-b165-bb290002858a-workload-certs" pod "6ae3594b-0d9b-431e-b165-bb290002858a" (UID: "6ae3594b-0d9b-431e-b165-bb290002858a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:29:51.788568 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:29:51.788542    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6ae3594b-0d9b-431e-b165-bb290002858a-istio-podinfo" pod "6ae3594b-0d9b-431e-b165-bb290002858a" (UID: "6ae3594b-0d9b-431e-b165-bb290002858a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 13:29:51.788862 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:29:51.788837    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ae3594b-0d9b-431e-b165-bb290002858a-kube-api-access-kck68" pod "6ae3594b-0d9b-431e-b165-bb290002858a" (UID: "6ae3594b-0d9b-431e-b165-bb290002858a"). InnerVolumeSpecName "kube-api-access-kck68". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:29:51.788964 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:29:51.788876    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ae3594b-0d9b-431e-b165-bb290002858a-istio-envoy" pod "6ae3594b-0d9b-431e-b165-bb290002858a" (UID: "6ae3594b-0d9b-431e-b165-bb290002858a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:29:51.789461 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:29:51.789440    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ae3594b-0d9b-431e-b165-bb290002858a-istio-token" pod "6ae3594b-0d9b-431e-b165-bb290002858a" (UID: "6ae3594b-0d9b-431e-b165-bb290002858a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:29:51.825700 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:29:51.825661    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-843978/distinct-w4-843978-kserve-6fcf588858-4nrwx"]
Nov 02 13:29:51.888311 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:29:51.888225    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kck68\" (UniqueName: \"kubernetes.io/projected/6ae3594b-0d9b-431e-b165-bb290002858a-kube-api-access-kck68\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:29:51.888311 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:29:51.888263    4559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ae3594b-0d9b-431e-b165-bb290002858a-workload-certs\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:29:51.888311 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:29:51.888280    4559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ae3594b-0d9b-431e-b165-bb290002858a-istio-envoy\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:29:51.888311 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:29:51.888292    4559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ae3594b-0d9b-431e-b165-bb290002858a-istio-token\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:29:51.888311 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:29:51.888304    4559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ae3594b-0d9b-431e-b165-bb290002858a-istio-podinfo\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:29:52.482295 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:29:52.482256    4559 generic.go:356] "Generic (PLEG): container finished" podID="6ae3594b-0d9b-431e-b165-bb290002858a" containerID="ce6a7a7afb91e946a884853ec8f96119491db4ef21f11aeb621a8d0189d32f66" exitCode=0
Nov 02 13:29:52.482712 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:29:52.482686    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-936529-openshift-default-5578bccc97-9cw4k" event={"ID":"6ae3594b-0d9b-431e-b165-bb290002858a","Type":"ContainerDied","Data":"ce6a7a7afb91e946a884853ec8f96119491db4ef21f11aeb621a8d0189d32f66"}
Nov 02 13:29:52.482943 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:29:52.482927    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-936529-openshift-default-5578bccc97-9cw4k" event={"ID":"6ae3594b-0d9b-431e-b165-bb290002858a","Type":"ContainerDied","Data":"35975e05c621db548df1a1b13766762be48d4beb4e97d764c2ba617843304e94"}
Nov 02 13:29:52.483071 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:29:52.483058    4559 scope.go:122] "RemoveContainer" containerID="ce6a7a7afb91e946a884853ec8f96119491db4ef21f11aeb621a8d0189d32f66"
Nov 02 13:29:52.483615 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:29:52.483592    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-936529-openshift-default-5578bccc97-9cw4k"
Nov 02 13:29:52.496079 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:29:52.495820    4559 scope.go:122] "RemoveContainer" containerID="9558dea98056df34d056ae3419f34b5c7a53a5c076914de1f956a03cf8c76cf4"
Nov 02 13:29:52.506281 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:29:52.506259    4559 scope.go:122] "RemoveContainer" containerID="ce6a7a7afb91e946a884853ec8f96119491db4ef21f11aeb621a8d0189d32f66"
Nov 02 13:29:52.506605 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:29:52.506575    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce6a7a7afb91e946a884853ec8f96119491db4ef21f11aeb621a8d0189d32f66\": container with ID starting with ce6a7a7afb91e946a884853ec8f96119491db4ef21f11aeb621a8d0189d32f66 not found: ID does not exist" containerID="ce6a7a7afb91e946a884853ec8f96119491db4ef21f11aeb621a8d0189d32f66"
Nov 02 13:29:52.506739 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:29:52.506615    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce6a7a7afb91e946a884853ec8f96119491db4ef21f11aeb621a8d0189d32f66"} err="failed to get container status \"ce6a7a7afb91e946a884853ec8f96119491db4ef21f11aeb621a8d0189d32f66\": rpc error: code = NotFound desc = could not find container \"ce6a7a7afb91e946a884853ec8f96119491db4ef21f11aeb621a8d0189d32f66\": container with ID starting with ce6a7a7afb91e946a884853ec8f96119491db4ef21f11aeb621a8d0189d32f66 not found: ID does not exist"
Nov 02 13:29:52.506739 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:29:52.506635    4559 scope.go:122] "RemoveContainer" containerID="9558dea98056df34d056ae3419f34b5c7a53a5c076914de1f956a03cf8c76cf4"
Nov 02 13:29:52.506927 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:29:52.506905    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9558dea98056df34d056ae3419f34b5c7a53a5c076914de1f956a03cf8c76cf4\": container with ID starting with 9558dea98056df34d056ae3419f34b5c7a53a5c076914de1f956a03cf8c76cf4 not found: ID does not exist" containerID="9558dea98056df34d056ae3419f34b5c7a53a5c076914de1f956a03cf8c76cf4"
Nov 02 13:29:52.506987 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:29:52.506936    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9558dea98056df34d056ae3419f34b5c7a53a5c076914de1f956a03cf8c76cf4"} err="failed to get container status \"9558dea98056df34d056ae3419f34b5c7a53a5c076914de1f956a03cf8c76cf4\": rpc error: code = NotFound desc = could not find container \"9558dea98056df34d056ae3419f34b5c7a53a5c076914de1f956a03cf8c76cf4\": container with ID starting with 9558dea98056df34d056ae3419f34b5c7a53a5c076914de1f956a03cf8c76cf4 not found: ID does not exist"
Nov 02 13:29:52.591287 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:29:52.591252    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-936529-openshift-default-5578bccc97-9cw4k"]
Nov 02 13:29:52.597493 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:29:52.597466    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-936529-openshift-default-5578bccc97-9cw4k"]
Nov 02 13:29:53.893393 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:29:53.893360    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ae3594b-0d9b-431e-b165-bb290002858a" path="/var/lib/kubelet/pods/6ae3594b-0d9b-431e-b165-bb290002858a/volumes"
Nov 02 13:30:00.261746 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:00.261701    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893770-xzkcq"]
Nov 02 13:30:00.262425 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:00.262407    4559 cpu_manager.go:414] "containerMap: removing container" podUID="dae5c89b-c8f9-46e6-9ca9-5444d1cb9269" containerName="istio-proxy"
Nov 02 13:30:00.262505 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:00.262426    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dae5c89b-c8f9-46e6-9ca9-5444d1cb9269" containerName="istio-proxy"
Nov 02 13:30:00.262505 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:00.262440    4559 cpu_manager.go:414] "containerMap: removing container" podUID="fff161f9-6bfb-4c48-bc75-ade5639c076b" containerName="istio-proxy"
Nov 02 13:30:00.262505 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:00.262445    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fff161f9-6bfb-4c48-bc75-ade5639c076b" containerName="istio-proxy"
Nov 02 13:30:00.262505 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:00.262459    4559 cpu_manager.go:414] "containerMap: removing container" podUID="2d3ed32f-021a-456e-85eb-864596d28c3f" containerName="istio-proxy"
Nov 02 13:30:00.262505 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:00.262465    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d3ed32f-021a-456e-85eb-864596d28c3f" containerName="istio-proxy"
Nov 02 13:30:00.262505 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:00.262475    4559 cpu_manager.go:414] "containerMap: removing container" podUID="41be38ac-3172-4e6b-af34-c83c9179b69f" containerName="maas-api"
Nov 02 13:30:00.262505 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:00.262479    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41be38ac-3172-4e6b-af34-c83c9179b69f" containerName="maas-api"
Nov 02 13:30:00.262505 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:00.262490    4559 cpu_manager.go:414] "containerMap: removing container" podUID="63746d03-ec60-494a-96f8-39436d2e6cb8" containerName="istio-proxy"
Nov 02 13:30:00.262505 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:00.262494    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63746d03-ec60-494a-96f8-39436d2e6cb8" containerName="istio-proxy"
Nov 02 13:30:00.262505 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:00.262502    4559 cpu_manager.go:414] "containerMap: removing container" podUID="2fd94eba-ab06-4d75-a63b-e31bca0326d6" containerName="istio-proxy"
Nov 02 13:30:00.262505 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:00.262508    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2fd94eba-ab06-4d75-a63b-e31bca0326d6" containerName="istio-proxy"
Nov 02 13:30:00.263090 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:00.262519    4559 cpu_manager.go:414] "containerMap: removing container" podUID="cc6f8750-45f1-4dfe-b2a0-f381d0dde490" containerName="maas-api"
Nov 02 13:30:00.263090 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:00.262524    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc6f8750-45f1-4dfe-b2a0-f381d0dde490" containerName="maas-api"
Nov 02 13:30:00.263090 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:00.262536    4559 cpu_manager.go:414] "containerMap: removing container" podUID="6ae3594b-0d9b-431e-b165-bb290002858a" containerName="istio-proxy"
Nov 02 13:30:00.263090 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:00.262540    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ae3594b-0d9b-431e-b165-bb290002858a" containerName="istio-proxy"
Nov 02 13:30:00.263090 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:00.262548    4559 cpu_manager.go:414] "containerMap: removing container" podUID="70323c42-ec34-40c6-a906-d646b3025523" containerName="maas-api"
Nov 02 13:30:00.263090 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:00.262553    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70323c42-ec34-40c6-a906-d646b3025523" containerName="maas-api"
Nov 02 13:30:00.263090 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:00.262561    4559 cpu_manager.go:414] "containerMap: removing container" podUID="5a9eb160-1c8d-456c-8e37-f2d2e8e234e4" containerName="maas-api"
Nov 02 13:30:00.263090 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:00.262565    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a9eb160-1c8d-456c-8e37-f2d2e8e234e4" containerName="maas-api"
Nov 02 13:30:00.263090 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:00.262572    4559 cpu_manager.go:414] "containerMap: removing container" podUID="6ae3594b-0d9b-431e-b165-bb290002858a" containerName="istio-proxy"
Nov 02 13:30:00.263090 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:00.262577    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ae3594b-0d9b-431e-b165-bb290002858a" containerName="istio-proxy"
Nov 02 13:30:00.263090 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:00.262584    4559 cpu_manager.go:414] "containerMap: removing container" podUID="a47a44e4-6226-4ce7-9760-58ee5eaf65bf" containerName="istio-proxy"
Nov 02 13:30:00.263090 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:00.262589    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a47a44e4-6226-4ce7-9760-58ee5eaf65bf" containerName="istio-proxy"
Nov 02 13:30:00.263090 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:00.262605    4559 cpu_manager.go:414] "containerMap: removing container" podUID="2d3ed32f-021a-456e-85eb-864596d28c3f" containerName="istio-proxy"
Nov 02 13:30:00.263090 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:00.262609    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d3ed32f-021a-456e-85eb-864596d28c3f" containerName="istio-proxy"
Nov 02 13:30:00.263090 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:00.262619    4559 cpu_manager.go:414] "containerMap: removing container" podUID="a47a44e4-6226-4ce7-9760-58ee5eaf65bf" containerName="istio-proxy"
Nov 02 13:30:00.263090 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:00.262624    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a47a44e4-6226-4ce7-9760-58ee5eaf65bf" containerName="istio-proxy"
Nov 02 13:30:00.263090 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:00.262733    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d3ed32f-021a-456e-85eb-864596d28c3f" containerName="istio-proxy"
Nov 02 13:30:00.263090 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:00.262744    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="a47a44e4-6226-4ce7-9760-58ee5eaf65bf" containerName="istio-proxy"
Nov 02 13:30:00.263090 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:00.262751    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="fff161f9-6bfb-4c48-bc75-ade5639c076b" containerName="istio-proxy"
Nov 02 13:30:00.263090 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:00.262758    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="dae5c89b-c8f9-46e6-9ca9-5444d1cb9269" containerName="istio-proxy"
Nov 02 13:30:00.263090 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:00.262766    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="41be38ac-3172-4e6b-af34-c83c9179b69f" containerName="maas-api"
Nov 02 13:30:00.263090 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:00.262773    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="63746d03-ec60-494a-96f8-39436d2e6cb8" containerName="istio-proxy"
Nov 02 13:30:00.263090 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:00.262780    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="2fd94eba-ab06-4d75-a63b-e31bca0326d6" containerName="istio-proxy"
Nov 02 13:30:00.263090 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:00.262786    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc6f8750-45f1-4dfe-b2a0-f381d0dde490" containerName="maas-api"
Nov 02 13:30:00.263090 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:00.262793    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ae3594b-0d9b-431e-b165-bb290002858a" containerName="istio-proxy"
Nov 02 13:30:00.263090 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:00.262801    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="70323c42-ec34-40c6-a906-d646b3025523" containerName="maas-api"
Nov 02 13:30:00.263090 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:00.262810    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a9eb160-1c8d-456c-8e37-f2d2e8e234e4" containerName="maas-api"
Nov 02 13:30:00.263090 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:00.262819    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ae3594b-0d9b-431e-b165-bb290002858a" containerName="istio-proxy"
Nov 02 13:30:00.268287 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:00.268261    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-1aa18c-29893770-gcn5r"]
Nov 02 13:30:00.268438 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:00.268403    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893770-xzkcq"
Nov 02 13:30:00.269250 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:00.269228    4559 cpu_manager.go:414] "containerMap: removing container" podUID="fff161f9-6bfb-4c48-bc75-ade5639c076b" containerName="istio-proxy"
Nov 02 13:30:00.269350 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:00.269251    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fff161f9-6bfb-4c48-bc75-ade5639c076b" containerName="istio-proxy"
Nov 02 13:30:00.269410 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:00.269396    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="a47a44e4-6226-4ce7-9760-58ee5eaf65bf" containerName="istio-proxy"
Nov 02 13:30:00.269455 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:00.269418    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="fff161f9-6bfb-4c48-bc75-ade5639c076b" containerName="istio-proxy"
Nov 02 13:30:00.269455 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:00.269444    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d3ed32f-021a-456e-85eb-864596d28c3f" containerName="istio-proxy"
Nov 02 13:30:00.273295 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:00.273275    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-1aa18c-29893770-gcn5r"
Nov 02 13:30:00.312018 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:00.311978    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-1aa18c-29893770-gcn5r"]
Nov 02 13:30:00.318710 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:00.318684    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893770-xzkcq"]
Nov 02 13:30:00.323284 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:00.323248    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-740f09-29893770-vs8m7"]
Nov 02 13:30:00.328323 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:00.328305    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a6f2c0-29893770-kfh8s"]
Nov 02 13:30:00.328441 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:00.328422    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-740f09-29893770-vs8m7"
Nov 02 13:30:00.332570 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:00.332550    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a6f2c0-29893770-kfh8s"
Nov 02 13:30:00.342162 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:00.342139    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-740f09-29893770-vs8m7"]
Nov 02 13:30:00.345764 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:00.345454    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a6f2c0-29893770-kfh8s"]
Nov 02 13:30:00.352003 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:00.351897    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w9cwp\""
Nov 02 13:30:00.392719 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:00.392686    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eede656a-d78e-4cf2-b98d-14d172fd36c6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-740f09-29893770-vs8m7\" (UID: \"eede656a-d78e-4cf2-b98d-14d172fd36c6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-740f09-29893770-vs8m7"
Nov 02 13:30:00.392867 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:00.392760    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5sw7\" (UniqueName: \"kubernetes.io/projected/d7505d42-8673-4ed0-8a6e-b6923d8ca25e-kube-api-access-j5sw7\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-a6f2c0-29893770-kfh8s\" (UID: \"d7505d42-8673-4ed0-8a6e-b6923d8ca25e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a6f2c0-29893770-kfh8s"
Nov 02 13:30:00.392932 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:00.392880    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvkkj\" (UniqueName: \"kubernetes.io/projected/46747055-1220-4a7e-8177-915d463a9f10-kube-api-access-hvkkj\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-1aa18c-29893770-gcn5r\" (UID: \"46747055-1220-4a7e-8177-915d463a9f10\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-1aa18c-29893770-gcn5r"
Nov 02 13:30:00.393018 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:00.392994    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7505d42-8673-4ed0-8a6e-b6923d8ca25e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-a6f2c0-29893770-kfh8s\" (UID: \"d7505d42-8673-4ed0-8a6e-b6923d8ca25e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a6f2c0-29893770-kfh8s"
Nov 02 13:30:00.393073 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:00.393041    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h48dc\" (UniqueName: \"kubernetes.io/projected/4aa05b9d-f18c-4c47-82e9-bfe95d1afe48-kube-api-access-h48dc\") pod \"maas-api-key-cleanup-29893770-xzkcq\" (UID: \"4aa05b9d-f18c-4c47-82e9-bfe95d1afe48\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893770-xzkcq"
Nov 02 13:30:00.393173 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:00.393152    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/46747055-1220-4a7e-8177-915d463a9f10-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-1aa18c-29893770-gcn5r\" (UID: \"46747055-1220-4a7e-8177-915d463a9f10\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-1aa18c-29893770-gcn5r"
Nov 02 13:30:00.393239 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:00.393220    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2jmf\" (UniqueName: \"kubernetes.io/projected/eede656a-d78e-4cf2-b98d-14d172fd36c6-kube-api-access-t2jmf\") pod \"maas-api-key-cleanup-e2e-extmod-a-740f09-29893770-vs8m7\" (UID: \"eede656a-d78e-4cf2-b98d-14d172fd36c6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-740f09-29893770-vs8m7"
Nov 02 13:30:00.393324 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:00.393307    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4aa05b9d-f18c-4c47-82e9-bfe95d1afe48-maas-api-service-ca\") pod \"maas-api-key-cleanup-29893770-xzkcq\" (UID: \"4aa05b9d-f18c-4c47-82e9-bfe95d1afe48\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893770-xzkcq"
Nov 02 13:30:00.494848 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:00.494819    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvkkj\" (UniqueName: \"kubernetes.io/projected/46747055-1220-4a7e-8177-915d463a9f10-kube-api-access-hvkkj\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-1aa18c-29893770-gcn5r\" (UID: \"46747055-1220-4a7e-8177-915d463a9f10\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-1aa18c-29893770-gcn5r"
Nov 02 13:30:00.495006 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:00.494871    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7505d42-8673-4ed0-8a6e-b6923d8ca25e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-a6f2c0-29893770-kfh8s\" (UID: \"d7505d42-8673-4ed0-8a6e-b6923d8ca25e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a6f2c0-29893770-kfh8s"
Nov 02 13:30:00.495006 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:00.494894    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h48dc\" (UniqueName: \"kubernetes.io/projected/4aa05b9d-f18c-4c47-82e9-bfe95d1afe48-kube-api-access-h48dc\") pod \"maas-api-key-cleanup-29893770-xzkcq\" (UID: \"4aa05b9d-f18c-4c47-82e9-bfe95d1afe48\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893770-xzkcq"
Nov 02 13:30:00.495006 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:00.494926    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/46747055-1220-4a7e-8177-915d463a9f10-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-1aa18c-29893770-gcn5r\" (UID: \"46747055-1220-4a7e-8177-915d463a9f10\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-1aa18c-29893770-gcn5r"
Nov 02 13:30:00.495006 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:00.494961    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2jmf\" (UniqueName: \"kubernetes.io/projected/eede656a-d78e-4cf2-b98d-14d172fd36c6-kube-api-access-t2jmf\") pod \"maas-api-key-cleanup-e2e-extmod-a-740f09-29893770-vs8m7\" (UID: \"eede656a-d78e-4cf2-b98d-14d172fd36c6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-740f09-29893770-vs8m7"
Nov 02 13:30:00.495006 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:00.495002    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4aa05b9d-f18c-4c47-82e9-bfe95d1afe48-maas-api-service-ca\") pod \"maas-api-key-cleanup-29893770-xzkcq\" (UID: \"4aa05b9d-f18c-4c47-82e9-bfe95d1afe48\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893770-xzkcq"
Nov 02 13:30:00.495283 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:00.495030    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eede656a-d78e-4cf2-b98d-14d172fd36c6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-740f09-29893770-vs8m7\" (UID: \"eede656a-d78e-4cf2-b98d-14d172fd36c6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-740f09-29893770-vs8m7"
Nov 02 13:30:00.495283 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:00.495062    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5sw7\" (UniqueName: \"kubernetes.io/projected/d7505d42-8673-4ed0-8a6e-b6923d8ca25e-kube-api-access-j5sw7\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-a6f2c0-29893770-kfh8s\" (UID: \"d7505d42-8673-4ed0-8a6e-b6923d8ca25e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a6f2c0-29893770-kfh8s"
Nov 02 13:30:00.495718 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:00.495698    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7505d42-8673-4ed0-8a6e-b6923d8ca25e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-a6f2c0-29893770-kfh8s\" (UID: \"d7505d42-8673-4ed0-8a6e-b6923d8ca25e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a6f2c0-29893770-kfh8s"
Nov 02 13:30:00.495782 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:00.495703    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4aa05b9d-f18c-4c47-82e9-bfe95d1afe48-maas-api-service-ca\") pod \"maas-api-key-cleanup-29893770-xzkcq\" (UID: \"4aa05b9d-f18c-4c47-82e9-bfe95d1afe48\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893770-xzkcq"
Nov 02 13:30:00.495782 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:00.495767    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/46747055-1220-4a7e-8177-915d463a9f10-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-1aa18c-29893770-gcn5r\" (UID: \"46747055-1220-4a7e-8177-915d463a9f10\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-1aa18c-29893770-gcn5r"
Nov 02 13:30:00.495851 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:00.495772    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eede656a-d78e-4cf2-b98d-14d172fd36c6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-740f09-29893770-vs8m7\" (UID: \"eede656a-d78e-4cf2-b98d-14d172fd36c6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-740f09-29893770-vs8m7"
Nov 02 13:30:00.520154 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:00.520082    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5sw7\" (UniqueName: \"kubernetes.io/projected/d7505d42-8673-4ed0-8a6e-b6923d8ca25e-kube-api-access-j5sw7\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-a6f2c0-29893770-kfh8s\" (UID: \"d7505d42-8673-4ed0-8a6e-b6923d8ca25e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a6f2c0-29893770-kfh8s"
Nov 02 13:30:00.520915 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:00.520891    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2jmf\" (UniqueName: \"kubernetes.io/projected/eede656a-d78e-4cf2-b98d-14d172fd36c6-kube-api-access-t2jmf\") pod \"maas-api-key-cleanup-e2e-extmod-a-740f09-29893770-vs8m7\" (UID: \"eede656a-d78e-4cf2-b98d-14d172fd36c6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-740f09-29893770-vs8m7"
Nov 02 13:30:00.522118 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:00.522096    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvkkj\" (UniqueName: \"kubernetes.io/projected/46747055-1220-4a7e-8177-915d463a9f10-kube-api-access-hvkkj\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-1aa18c-29893770-gcn5r\" (UID: \"46747055-1220-4a7e-8177-915d463a9f10\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-1aa18c-29893770-gcn5r"
Nov 02 13:30:00.525511 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:00.525485    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h48dc\" (UniqueName: \"kubernetes.io/projected/4aa05b9d-f18c-4c47-82e9-bfe95d1afe48-kube-api-access-h48dc\") pod \"maas-api-key-cleanup-29893770-xzkcq\" (UID: \"4aa05b9d-f18c-4c47-82e9-bfe95d1afe48\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893770-xzkcq"
Nov 02 13:30:00.588538 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:00.588515    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893770-xzkcq"
Nov 02 13:30:00.595869 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:00.595849    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-1aa18c-29893770-gcn5r"
Nov 02 13:30:00.660661 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:00.658884    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-740f09-29893770-vs8m7"
Nov 02 13:30:00.675044 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:00.670143    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a6f2c0-29893770-kfh8s"
Nov 02 13:30:00.962250 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:00.962220    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893770-xzkcq"]
Nov 02 13:30:00.964442 ip-10-0-128-132 kubenswrapper[4559]: W1102 13:30:00.964413    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4aa05b9d_f18c_4c47_82e9_bfe95d1afe48.slice/crio-b4eddb2510f229ab798425f8ab88350593c75cbe21c5283a6881c798401d9210 WatchSource:0}: Error finding container b4eddb2510f229ab798425f8ab88350593c75cbe21c5283a6881c798401d9210: Status 404 returned error can't find the container with id b4eddb2510f229ab798425f8ab88350593c75cbe21c5283a6881c798401d9210
Nov 02 13:30:01.069709 ip-10-0-128-132 kubenswrapper[4559]: W1102 13:30:01.069674    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46747055_1220_4a7e_8177_915d463a9f10.slice/crio-803e496cf9f5d374e69dc52ea665a4274dc5ef25bf194334ed4c4df1d43c1a75 WatchSource:0}: Error finding container 803e496cf9f5d374e69dc52ea665a4274dc5ef25bf194334ed4c4df1d43c1a75: Status 404 returned error can't find the container with id 803e496cf9f5d374e69dc52ea665a4274dc5ef25bf194334ed4c4df1d43c1a75
Nov 02 13:30:01.069845 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:01.069747    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-1aa18c-29893770-gcn5r"]
Nov 02 13:30:01.128961 ip-10-0-128-132 kubenswrapper[4559]: W1102 13:30:01.128928    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7505d42_8673_4ed0_8a6e_b6923d8ca25e.slice/crio-7e10d2c12ff41970979fff9d74de628b903d3e74cfb03744021c440e63e12c42 WatchSource:0}: Error finding container 7e10d2c12ff41970979fff9d74de628b903d3e74cfb03744021c440e63e12c42: Status 404 returned error can't find the container with id 7e10d2c12ff41970979fff9d74de628b903d3e74cfb03744021c440e63e12c42
Nov 02 13:30:01.129943 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:01.129919    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a6f2c0-29893770-kfh8s"]
Nov 02 13:30:01.144873 ip-10-0-128-132 kubenswrapper[4559]: W1102 13:30:01.144846    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeede656a_d78e_4cf2_b98d_14d172fd36c6.slice/crio-2ad558e60c68316c0eb3e441fa84454295f8e1e4772b38bed578f5aa714fd4d7 WatchSource:0}: Error finding container 2ad558e60c68316c0eb3e441fa84454295f8e1e4772b38bed578f5aa714fd4d7: Status 404 returned error can't find the container with id 2ad558e60c68316c0eb3e441fa84454295f8e1e4772b38bed578f5aa714fd4d7
Nov 02 13:30:01.146078 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:01.146058    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-740f09-29893770-vs8m7"]
Nov 02 13:30:01.544757 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:01.544726    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a6f2c0-29893770-kfh8s" event={"ID":"d7505d42-8673-4ed0-8a6e-b6923d8ca25e","Type":"ContainerStarted","Data":"cf96cfc9603314751b4d8179039275c1a6bc2a2fe6692282762289b56572fdd9"}
Nov 02 13:30:01.545144 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:01.544816    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a6f2c0-29893770-kfh8s" event={"ID":"d7505d42-8673-4ed0-8a6e-b6923d8ca25e","Type":"ContainerStarted","Data":"7e10d2c12ff41970979fff9d74de628b903d3e74cfb03744021c440e63e12c42"}
Nov 02 13:30:01.546320 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:01.546296    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893770-xzkcq" event={"ID":"4aa05b9d-f18c-4c47-82e9-bfe95d1afe48","Type":"ContainerStarted","Data":"81d78f2a20cf5690d070a58fc57ab0dc8336381887e5b70d43553b03052fb44d"}
Nov 02 13:30:01.546449 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:01.546342    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893770-xzkcq" event={"ID":"4aa05b9d-f18c-4c47-82e9-bfe95d1afe48","Type":"ContainerStarted","Data":"b4eddb2510f229ab798425f8ab88350593c75cbe21c5283a6881c798401d9210"}
Nov 02 13:30:01.547787 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:01.547765    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-740f09-29893770-vs8m7" event={"ID":"eede656a-d78e-4cf2-b98d-14d172fd36c6","Type":"ContainerStarted","Data":"0cb3263a6d1a87ce87edeaa9e3defd65ab4595374d60e5f7eacddaaaa224452e"}
Nov 02 13:30:01.547882 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:01.547810    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-740f09-29893770-vs8m7" event={"ID":"eede656a-d78e-4cf2-b98d-14d172fd36c6","Type":"ContainerStarted","Data":"2ad558e60c68316c0eb3e441fa84454295f8e1e4772b38bed578f5aa714fd4d7"}
Nov 02 13:30:01.549299 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:01.549268    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-1aa18c-29893770-gcn5r" event={"ID":"46747055-1220-4a7e-8177-915d463a9f10","Type":"ContainerStarted","Data":"b4c6a522b8a953fe529b1b83acd8e08a605b9df5928bbd23141c1bbbdfea494f"}
Nov 02 13:30:01.549380 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:01.549308    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-1aa18c-29893770-gcn5r" event={"ID":"46747055-1220-4a7e-8177-915d463a9f10","Type":"ContainerStarted","Data":"803e496cf9f5d374e69dc52ea665a4274dc5ef25bf194334ed4c4df1d43c1a75"}
Nov 02 13:30:01.590287 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:01.590240    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a6f2c0-29893770-kfh8s" podStartSLOduration=1.59022759 podStartE2EDuration="1.59022759s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:30:01.588622382 +0000 UTC m=+3797.844021477" watchObservedRunningTime="2026-11-02 13:30:01.59022759 +0000 UTC m=+3797.845626684"
Nov 02 13:30:01.635739 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:01.635700    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-1aa18c-29893770-gcn5r" podStartSLOduration=1.6356892379999999 podStartE2EDuration="1.635689238s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:30:01.632614826 +0000 UTC m=+3797.888013920" watchObservedRunningTime="2026-11-02 13:30:01.635689238 +0000 UTC m=+3797.891088333"
Nov 02 13:30:01.677678 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:01.677610    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893770-xzkcq" podStartSLOduration=1.677595325 podStartE2EDuration="1.677595325s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:30:01.676482276 +0000 UTC m=+3797.931881372" watchObservedRunningTime="2026-11-02 13:30:01.677595325 +0000 UTC m=+3797.932994420"
Nov 02 13:30:01.718840 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:01.718799    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-740f09-29893770-vs8m7" podStartSLOduration=1.718786913 podStartE2EDuration="1.718786913s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:30:01.71774115 +0000 UTC m=+3797.973140246" watchObservedRunningTime="2026-11-02 13:30:01.718786913 +0000 UTC m=+3797.974186007"
Nov 02 13:30:02.557619 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:02.557587    4559 generic.go:356] "Generic (PLEG): container finished" podID="d7505d42-8673-4ed0-8a6e-b6923d8ca25e" containerID="cf96cfc9603314751b4d8179039275c1a6bc2a2fe6692282762289b56572fdd9" exitCode=60
Nov 02 13:30:02.558073 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:02.557666    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a6f2c0-29893770-kfh8s" event={"ID":"d7505d42-8673-4ed0-8a6e-b6923d8ca25e","Type":"ContainerDied","Data":"cf96cfc9603314751b4d8179039275c1a6bc2a2fe6692282762289b56572fdd9"}
Nov 02 13:30:02.558073 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:02.557973    4559 scope.go:122] "RemoveContainer" containerID="cf96cfc9603314751b4d8179039275c1a6bc2a2fe6692282762289b56572fdd9"
Nov 02 13:30:02.559284 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:02.559252    4559 generic.go:356] "Generic (PLEG): container finished" podID="4aa05b9d-f18c-4c47-82e9-bfe95d1afe48" containerID="81d78f2a20cf5690d070a58fc57ab0dc8336381887e5b70d43553b03052fb44d" exitCode=60
Nov 02 13:30:02.559390 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:02.559301    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893770-xzkcq" event={"ID":"4aa05b9d-f18c-4c47-82e9-bfe95d1afe48","Type":"ContainerDied","Data":"81d78f2a20cf5690d070a58fc57ab0dc8336381887e5b70d43553b03052fb44d"}
Nov 02 13:30:02.559672 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:02.559571    4559 scope.go:122] "RemoveContainer" containerID="81d78f2a20cf5690d070a58fc57ab0dc8336381887e5b70d43553b03052fb44d"
Nov 02 13:30:02.561187 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:02.561117    4559 generic.go:356] "Generic (PLEG): container finished" podID="eede656a-d78e-4cf2-b98d-14d172fd36c6" containerID="0cb3263a6d1a87ce87edeaa9e3defd65ab4595374d60e5f7eacddaaaa224452e" exitCode=60
Nov 02 13:30:02.561280 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:02.561257    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-740f09-29893770-vs8m7" event={"ID":"eede656a-d78e-4cf2-b98d-14d172fd36c6","Type":"ContainerDied","Data":"0cb3263a6d1a87ce87edeaa9e3defd65ab4595374d60e5f7eacddaaaa224452e"}
Nov 02 13:30:02.561521 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:02.561488    4559 scope.go:122] "RemoveContainer" containerID="0cb3263a6d1a87ce87edeaa9e3defd65ab4595374d60e5f7eacddaaaa224452e"
Nov 02 13:30:02.562841 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:02.562825    4559 generic.go:356] "Generic (PLEG): container finished" podID="46747055-1220-4a7e-8177-915d463a9f10" containerID="b4c6a522b8a953fe529b1b83acd8e08a605b9df5928bbd23141c1bbbdfea494f" exitCode=60
Nov 02 13:30:02.562890 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:02.562859    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-1aa18c-29893770-gcn5r" event={"ID":"46747055-1220-4a7e-8177-915d463a9f10","Type":"ContainerDied","Data":"b4c6a522b8a953fe529b1b83acd8e08a605b9df5928bbd23141c1bbbdfea494f"}
Nov 02 13:30:02.563139 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:02.563122    4559 scope.go:122] "RemoveContainer" containerID="b4c6a522b8a953fe529b1b83acd8e08a605b9df5928bbd23141c1bbbdfea494f"
Nov 02 13:30:03.493895 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:03.493861    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"]
Nov 02 13:30:03.572330 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:03.572293    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-1aa18c-29893770-gcn5r" event={"ID":"46747055-1220-4a7e-8177-915d463a9f10","Type":"ContainerStarted","Data":"7208966fbb20e8b1377153a7bef1cff2e20f4bf0d38df34791ffc7a6a927f32c"}
Nov 02 13:30:03.574572 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:03.574546    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a6f2c0-29893770-kfh8s" event={"ID":"d7505d42-8673-4ed0-8a6e-b6923d8ca25e","Type":"ContainerStarted","Data":"8a4e274604bc262224d387381b2eed3ee4b600bff033e55f8ab49beedd6ceb4c"}
Nov 02 13:30:03.576515 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:03.576487    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893770-xzkcq" event={"ID":"4aa05b9d-f18c-4c47-82e9-bfe95d1afe48","Type":"ContainerStarted","Data":"3875c112a3dd06df1e57841b0501dbc03611e8cfefd6be98f75cfc58aa05235f"}
Nov 02 13:30:03.578480 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:03.578460    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-740f09-29893770-vs8m7" event={"ID":"eede656a-d78e-4cf2-b98d-14d172fd36c6","Type":"ContainerStarted","Data":"3611fe2e2dca2423220ab401cf16e07cbe0deb73a2ef356bb89404f70348a933"}
Nov 02 13:30:04.589048 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:04.588998    4559 generic.go:356] "Generic (PLEG): container finished" podID="46747055-1220-4a7e-8177-915d463a9f10" containerID="7208966fbb20e8b1377153a7bef1cff2e20f4bf0d38df34791ffc7a6a927f32c" exitCode=60
Nov 02 13:30:04.589541 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:04.589056    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-1aa18c-29893770-gcn5r" event={"ID":"46747055-1220-4a7e-8177-915d463a9f10","Type":"ContainerDied","Data":"7208966fbb20e8b1377153a7bef1cff2e20f4bf0d38df34791ffc7a6a927f32c"}
Nov 02 13:30:04.589541 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:04.589137    4559 scope.go:122] "RemoveContainer" containerID="b4c6a522b8a953fe529b1b83acd8e08a605b9df5928bbd23141c1bbbdfea494f"
Nov 02 13:30:04.589721 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:04.589560    4559 scope.go:122] "RemoveContainer" containerID="7208966fbb20e8b1377153a7bef1cff2e20f4bf0d38df34791ffc7a6a927f32c"
Nov 02 13:30:04.589962 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:30:04.589938    4559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-1aa18c-29893770-gcn5r_odh-ai-gateway-infra(46747055-1220-4a7e-8177-915d463a9f10)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-1aa18c-29893770-gcn5r" podUID="46747055-1220-4a7e-8177-915d463a9f10"
Nov 02 13:30:04.591561 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:04.591531    4559 generic.go:356] "Generic (PLEG): container finished" podID="d7505d42-8673-4ed0-8a6e-b6923d8ca25e" containerID="8a4e274604bc262224d387381b2eed3ee4b600bff033e55f8ab49beedd6ceb4c" exitCode=60
Nov 02 13:30:04.591692 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:04.591614    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a6f2c0-29893770-kfh8s" event={"ID":"d7505d42-8673-4ed0-8a6e-b6923d8ca25e","Type":"ContainerDied","Data":"8a4e274604bc262224d387381b2eed3ee4b600bff033e55f8ab49beedd6ceb4c"}
Nov 02 13:30:04.591907 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:04.591892    4559 scope.go:122] "RemoveContainer" containerID="8a4e274604bc262224d387381b2eed3ee4b600bff033e55f8ab49beedd6ceb4c"
Nov 02 13:30:04.592253 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:30:04.592232    4559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-a6f2c0-29893770-kfh8s_odh-ai-gateway-infra(d7505d42-8673-4ed0-8a6e-b6923d8ca25e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a6f2c0-29893770-kfh8s" podUID="d7505d42-8673-4ed0-8a6e-b6923d8ca25e"
Nov 02 13:30:04.593714 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:04.593679    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893770-xzkcq" event={"ID":"4aa05b9d-f18c-4c47-82e9-bfe95d1afe48","Type":"ContainerDied","Data":"3875c112a3dd06df1e57841b0501dbc03611e8cfefd6be98f75cfc58aa05235f"}
Nov 02 13:30:04.593714 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:04.593627    4559 generic.go:356] "Generic (PLEG): container finished" podID="4aa05b9d-f18c-4c47-82e9-bfe95d1afe48" containerID="3875c112a3dd06df1e57841b0501dbc03611e8cfefd6be98f75cfc58aa05235f" exitCode=60
Nov 02 13:30:04.593995 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:04.593977    4559 scope.go:122] "RemoveContainer" containerID="3875c112a3dd06df1e57841b0501dbc03611e8cfefd6be98f75cfc58aa05235f"
Nov 02 13:30:04.594305 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:30:04.594283    4559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29893770-xzkcq_odh-ai-gateway-infra(4aa05b9d-f18c-4c47-82e9-bfe95d1afe48)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893770-xzkcq" podUID="4aa05b9d-f18c-4c47-82e9-bfe95d1afe48"
Nov 02 13:30:04.596012 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:04.595979    4559 generic.go:356] "Generic (PLEG): container finished" podID="eede656a-d78e-4cf2-b98d-14d172fd36c6" containerID="3611fe2e2dca2423220ab401cf16e07cbe0deb73a2ef356bb89404f70348a933" exitCode=60
Nov 02 13:30:04.596105 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:04.596007    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-740f09-29893770-vs8m7" event={"ID":"eede656a-d78e-4cf2-b98d-14d172fd36c6","Type":"ContainerDied","Data":"3611fe2e2dca2423220ab401cf16e07cbe0deb73a2ef356bb89404f70348a933"}
Nov 02 13:30:04.596279 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:04.596264    4559 scope.go:122] "RemoveContainer" containerID="3611fe2e2dca2423220ab401cf16e07cbe0deb73a2ef356bb89404f70348a933"
Nov 02 13:30:04.596531 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:30:04.596510    4559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-740f09-29893770-vs8m7_odh-ai-gateway-infra(eede656a-d78e-4cf2-b98d-14d172fd36c6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-740f09-29893770-vs8m7" podUID="eede656a-d78e-4cf2-b98d-14d172fd36c6"
Nov 02 13:30:04.603969 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:04.603947    4559 scope.go:122] "RemoveContainer" containerID="cf96cfc9603314751b4d8179039275c1a6bc2a2fe6692282762289b56572fdd9"
Nov 02 13:30:04.618241 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:04.618222    4559 scope.go:122] "RemoveContainer" containerID="81d78f2a20cf5690d070a58fc57ab0dc8336381887e5b70d43553b03052fb44d"
Nov 02 13:30:04.631429 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:04.631410    4559 scope.go:122] "RemoveContainer" containerID="0cb3263a6d1a87ce87edeaa9e3defd65ab4595374d60e5f7eacddaaaa224452e"
Nov 02 13:30:05.604112 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:05.604084    4559 scope.go:122] "RemoveContainer" containerID="8a4e274604bc262224d387381b2eed3ee4b600bff033e55f8ab49beedd6ceb4c"
Nov 02 13:30:05.604531 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:30:05.604393    4559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-a6f2c0-29893770-kfh8s_odh-ai-gateway-infra(d7505d42-8673-4ed0-8a6e-b6923d8ca25e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a6f2c0-29893770-kfh8s" podUID="d7505d42-8673-4ed0-8a6e-b6923d8ca25e"
Nov 02 13:30:05.605660 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:05.605612    4559 scope.go:122] "RemoveContainer" containerID="3875c112a3dd06df1e57841b0501dbc03611e8cfefd6be98f75cfc58aa05235f"
Nov 02 13:30:05.605946 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:30:05.605921    4559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29893770-xzkcq_odh-ai-gateway-infra(4aa05b9d-f18c-4c47-82e9-bfe95d1afe48)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893770-xzkcq" podUID="4aa05b9d-f18c-4c47-82e9-bfe95d1afe48"
Nov 02 13:30:05.607352 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:05.607332    4559 scope.go:122] "RemoveContainer" containerID="3611fe2e2dca2423220ab401cf16e07cbe0deb73a2ef356bb89404f70348a933"
Nov 02 13:30:05.607663 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:30:05.607614    4559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-740f09-29893770-vs8m7_odh-ai-gateway-infra(eede656a-d78e-4cf2-b98d-14d172fd36c6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-740f09-29893770-vs8m7" podUID="eede656a-d78e-4cf2-b98d-14d172fd36c6"
Nov 02 13:30:05.609086 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:05.609065    4559 scope.go:122] "RemoveContainer" containerID="7208966fbb20e8b1377153a7bef1cff2e20f4bf0d38df34791ffc7a6a927f32c"
Nov 02 13:30:05.609348 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:30:05.609328    4559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-1aa18c-29893770-gcn5r_odh-ai-gateway-infra(46747055-1220-4a7e-8177-915d463a9f10)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-1aa18c-29893770-gcn5r" podUID="46747055-1220-4a7e-8177-915d463a9f10"
Nov 02 13:30:06.847338 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:06.847302    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-843978/distinct-w4-843978-kserve-6fcf588858-4nrwx" podUID="82e5d483-e95b-43e6-9135-af960823427a" containerName="main" containerID="cri-o://d56829801a880a1e94e54324247067e9cf52a5f8e6fd688a4f4c161c95ca6f8e" gracePeriod=45
Nov 02 13:30:07.111603 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:07.111544    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-843978/distinct-w4-843978-kserve-6fcf588858-4nrwx"
Nov 02 13:30:07.230474 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:07.230444    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82e5d483-e95b-43e6-9135-af960823427a-kube-api-access-s6f8l\" (UniqueName: \"kubernetes.io/projected/82e5d483-e95b-43e6-9135-af960823427a-kube-api-access-s6f8l\") pod \"82e5d483-e95b-43e6-9135-af960823427a\" (UID: \"82e5d483-e95b-43e6-9135-af960823427a\") "
Nov 02 13:30:07.230697 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:07.230505    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82e5d483-e95b-43e6-9135-af960823427a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/82e5d483-e95b-43e6-9135-af960823427a-dshm\") pod \"82e5d483-e95b-43e6-9135-af960823427a\" (UID: \"82e5d483-e95b-43e6-9135-af960823427a\") "
Nov 02 13:30:07.230697 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:07.230585    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82e5d483-e95b-43e6-9135-af960823427a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82e5d483-e95b-43e6-9135-af960823427a-model-cache\") pod \"82e5d483-e95b-43e6-9135-af960823427a\" (UID: \"82e5d483-e95b-43e6-9135-af960823427a\") "
Nov 02 13:30:07.230855 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:07.230706    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82e5d483-e95b-43e6-9135-af960823427a-home\" (UniqueName: \"kubernetes.io/empty-dir/82e5d483-e95b-43e6-9135-af960823427a-home\") pod \"82e5d483-e95b-43e6-9135-af960823427a\" (UID: \"82e5d483-e95b-43e6-9135-af960823427a\") "
Nov 02 13:30:07.230855 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:07.230832    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82e5d483-e95b-43e6-9135-af960823427a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82e5d483-e95b-43e6-9135-af960823427a-tmp-dir\") pod \"82e5d483-e95b-43e6-9135-af960823427a\" (UID: \"82e5d483-e95b-43e6-9135-af960823427a\") "
Nov 02 13:30:07.230989 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:07.230924    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82e5d483-e95b-43e6-9135-af960823427a-model-cache" pod "82e5d483-e95b-43e6-9135-af960823427a" (UID: "82e5d483-e95b-43e6-9135-af960823427a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:30:07.230989 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:07.230937    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/82e5d483-e95b-43e6-9135-af960823427a-tls-certs\" (UniqueName: \"kubernetes.io/secret/82e5d483-e95b-43e6-9135-af960823427a-tls-certs\") pod \"82e5d483-e95b-43e6-9135-af960823427a\" (UID: \"82e5d483-e95b-43e6-9135-af960823427a\") "
Nov 02 13:30:07.231202 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:07.231179    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82e5d483-e95b-43e6-9135-af960823427a-home" pod "82e5d483-e95b-43e6-9135-af960823427a" (UID: "82e5d483-e95b-43e6-9135-af960823427a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:30:07.231358 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:07.231333    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82e5d483-e95b-43e6-9135-af960823427a-tmp-dir" pod "82e5d483-e95b-43e6-9135-af960823427a" (UID: "82e5d483-e95b-43e6-9135-af960823427a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:30:07.231619 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:07.231598    4559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82e5d483-e95b-43e6-9135-af960823427a-model-cache\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:30:07.231742 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:07.231622    4559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/82e5d483-e95b-43e6-9135-af960823427a-home\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:30:07.231742 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:07.231631    4559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82e5d483-e95b-43e6-9135-af960823427a-tmp-dir\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:30:07.233379 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:07.233355    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82e5d483-e95b-43e6-9135-af960823427a-tls-certs" pod "82e5d483-e95b-43e6-9135-af960823427a" (UID: "82e5d483-e95b-43e6-9135-af960823427a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:30:07.233470 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:07.233429    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82e5d483-e95b-43e6-9135-af960823427a-kube-api-access-s6f8l" pod "82e5d483-e95b-43e6-9135-af960823427a" (UID: "82e5d483-e95b-43e6-9135-af960823427a"). InnerVolumeSpecName "kube-api-access-s6f8l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:30:07.233470 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:07.233455    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82e5d483-e95b-43e6-9135-af960823427a-dshm" pod "82e5d483-e95b-43e6-9135-af960823427a" (UID: "82e5d483-e95b-43e6-9135-af960823427a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:30:07.336752 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:07.336662    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s6f8l\" (UniqueName: \"kubernetes.io/projected/82e5d483-e95b-43e6-9135-af960823427a-kube-api-access-s6f8l\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:30:07.336752 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:07.336709    4559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/82e5d483-e95b-43e6-9135-af960823427a-dshm\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:30:07.336752 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:07.336724    4559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82e5d483-e95b-43e6-9135-af960823427a-tls-certs\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:30:07.623370 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:07.623339    4559 generic.go:356] "Generic (PLEG): container finished" podID="82e5d483-e95b-43e6-9135-af960823427a" containerID="d56829801a880a1e94e54324247067e9cf52a5f8e6fd688a4f4c161c95ca6f8e" exitCode=0
Nov 02 13:30:07.623532 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:07.623384    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-843978/distinct-w4-843978-kserve-6fcf588858-4nrwx"
Nov 02 13:30:07.623532 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:07.623424    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-843978/distinct-w4-843978-kserve-6fcf588858-4nrwx" event={"ID":"82e5d483-e95b-43e6-9135-af960823427a","Type":"ContainerDied","Data":"d56829801a880a1e94e54324247067e9cf52a5f8e6fd688a4f4c161c95ca6f8e"}
Nov 02 13:30:07.623532 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:07.623467    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-843978/distinct-w4-843978-kserve-6fcf588858-4nrwx" event={"ID":"82e5d483-e95b-43e6-9135-af960823427a","Type":"ContainerDied","Data":"536c5ed1a3cd5c8bf9dd14f3dbb51dc59dbb5280a187549345e1cec39a8f750c"}
Nov 02 13:30:07.623532 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:07.623490    4559 scope.go:122] "RemoveContainer" containerID="d56829801a880a1e94e54324247067e9cf52a5f8e6fd688a4f4c161c95ca6f8e"
Nov 02 13:30:07.634528 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:07.634507    4559 scope.go:122] "RemoveContainer" containerID="d56829801a880a1e94e54324247067e9cf52a5f8e6fd688a4f4c161c95ca6f8e"
Nov 02 13:30:07.634831 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:30:07.634810    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d56829801a880a1e94e54324247067e9cf52a5f8e6fd688a4f4c161c95ca6f8e\": container with ID starting with d56829801a880a1e94e54324247067e9cf52a5f8e6fd688a4f4c161c95ca6f8e not found: ID does not exist" containerID="d56829801a880a1e94e54324247067e9cf52a5f8e6fd688a4f4c161c95ca6f8e"
Nov 02 13:30:07.634911 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:07.634843    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d56829801a880a1e94e54324247067e9cf52a5f8e6fd688a4f4c161c95ca6f8e"} err="failed to get container status \"d56829801a880a1e94e54324247067e9cf52a5f8e6fd688a4f4c161c95ca6f8e\": rpc error: code = NotFound desc = could not find container \"d56829801a880a1e94e54324247067e9cf52a5f8e6fd688a4f4c161c95ca6f8e\": container with ID starting with d56829801a880a1e94e54324247067e9cf52a5f8e6fd688a4f4c161c95ca6f8e not found: ID does not exist"
Nov 02 13:30:07.677341 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:07.677318    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-843978/distinct-w4-843978-kserve-6fcf588858-4nrwx"]
Nov 02 13:30:07.690034 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:07.690012    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-843978/distinct-w4-843978-kserve-6fcf588858-4nrwx"]
Nov 02 13:30:07.893161 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:07.893091    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82e5d483-e95b-43e6-9135-af960823427a" path="/var/lib/kubelet/pods/82e5d483-e95b-43e6-9135-af960823427a/volumes"
Nov 02 13:30:13.889866 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:13.889831    4559 scope.go:122] "RemoveContainer" containerID="7208966fbb20e8b1377153a7bef1cff2e20f4bf0d38df34791ffc7a6a927f32c"
Nov 02 13:30:14.679387 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:14.679344    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-1aa18c-29893770-gcn5r" event={"ID":"46747055-1220-4a7e-8177-915d463a9f10","Type":"ContainerStarted","Data":"d586779398f9f35bad4377e9786a93adc8c5fc60fe8c2b5e27697ced0bd66fc2"}
Nov 02 13:30:14.766544 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:14.766512    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-843978-679c455f86-rj4mr"]
Nov 02 13:30:14.766959 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:14.766930    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-843978-679c455f86-rj4mr" podUID="c68bfb23-6abb-4a96-979b-6b00ea4d5f7e" containerName="maas-api" containerID="cri-o://37b390df297c04f9616c77c5257b05c3ee9928391c2e5fdde5fbf3a7f7edc728" gracePeriod=30
Nov 02 13:30:14.886414 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:14.886392    4559 scope.go:122] "RemoveContainer" containerID="3611fe2e2dca2423220ab401cf16e07cbe0deb73a2ef356bb89404f70348a933"
Nov 02 13:30:14.886477 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:14.886458    4559 scope.go:122] "RemoveContainer" containerID="8a4e274604bc262224d387381b2eed3ee4b600bff033e55f8ab49beedd6ceb4c"
Nov 02 13:30:15.280265 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:30:15.280203    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqcmj" containerName="extractor"
Nov 02 13:30:15.292032 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:15.292003    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-843978-679c455f86-rj4mr"
Nov 02 13:30:15.393264 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:15.393235    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c68bfb23-6abb-4a96-979b-6b00ea4d5f7e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c68bfb23-6abb-4a96-979b-6b00ea4d5f7e-metrics-certs\") pod \"c68bfb23-6abb-4a96-979b-6b00ea4d5f7e\" (UID: \"c68bfb23-6abb-4a96-979b-6b00ea4d5f7e\") "
Nov 02 13:30:15.393388 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:15.393334    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c68bfb23-6abb-4a96-979b-6b00ea4d5f7e-kube-api-access-58hhj\" (UniqueName: \"kubernetes.io/projected/c68bfb23-6abb-4a96-979b-6b00ea4d5f7e-kube-api-access-58hhj\") pod \"c68bfb23-6abb-4a96-979b-6b00ea4d5f7e\" (UID: \"c68bfb23-6abb-4a96-979b-6b00ea4d5f7e\") "
Nov 02 13:30:15.393388 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:15.393365    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c68bfb23-6abb-4a96-979b-6b00ea4d5f7e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c68bfb23-6abb-4a96-979b-6b00ea4d5f7e-maas-api-tls\") pod \"c68bfb23-6abb-4a96-979b-6b00ea4d5f7e\" (UID: \"c68bfb23-6abb-4a96-979b-6b00ea4d5f7e\") "
Nov 02 13:30:15.393912 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:15.393476    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c68bfb23-6abb-4a96-979b-6b00ea4d5f7e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c68bfb23-6abb-4a96-979b-6b00ea4d5f7e-openshift-service-ca\") pod \"c68bfb23-6abb-4a96-979b-6b00ea4d5f7e\" (UID: \"c68bfb23-6abb-4a96-979b-6b00ea4d5f7e\") "
Nov 02 13:30:15.393980 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:15.393929    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c68bfb23-6abb-4a96-979b-6b00ea4d5f7e-openshift-service-ca" pod "c68bfb23-6abb-4a96-979b-6b00ea4d5f7e" (UID: "c68bfb23-6abb-4a96-979b-6b00ea4d5f7e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:30:15.394120 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:15.394101    4559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c68bfb23-6abb-4a96-979b-6b00ea4d5f7e-openshift-service-ca\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:30:15.395763 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:15.395737    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c68bfb23-6abb-4a96-979b-6b00ea4d5f7e-metrics-certs" pod "c68bfb23-6abb-4a96-979b-6b00ea4d5f7e" (UID: "c68bfb23-6abb-4a96-979b-6b00ea4d5f7e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:30:15.395763 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:15.395746    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c68bfb23-6abb-4a96-979b-6b00ea4d5f7e-kube-api-access-58hhj" pod "c68bfb23-6abb-4a96-979b-6b00ea4d5f7e" (UID: "c68bfb23-6abb-4a96-979b-6b00ea4d5f7e"). InnerVolumeSpecName "kube-api-access-58hhj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:30:15.400039 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:15.400013    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c68bfb23-6abb-4a96-979b-6b00ea4d5f7e-maas-api-tls" pod "c68bfb23-6abb-4a96-979b-6b00ea4d5f7e" (UID: "c68bfb23-6abb-4a96-979b-6b00ea4d5f7e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:30:15.496132 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:15.495903    4559 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c68bfb23-6abb-4a96-979b-6b00ea4d5f7e-metrics-certs\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:30:15.496132 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:15.495932    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-58hhj\" (UniqueName: \"kubernetes.io/projected/c68bfb23-6abb-4a96-979b-6b00ea4d5f7e-kube-api-access-58hhj\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:30:15.496132 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:15.495942    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c68bfb23-6abb-4a96-979b-6b00ea4d5f7e-maas-api-tls\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:30:15.696727 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:15.694423    4559 generic.go:356] "Generic (PLEG): container finished" podID="c68bfb23-6abb-4a96-979b-6b00ea4d5f7e" containerID="37b390df297c04f9616c77c5257b05c3ee9928391c2e5fdde5fbf3a7f7edc728" exitCode=0
Nov 02 13:30:15.696727 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:15.694512    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-843978-679c455f86-rj4mr" event={"ID":"c68bfb23-6abb-4a96-979b-6b00ea4d5f7e","Type":"ContainerDied","Data":"37b390df297c04f9616c77c5257b05c3ee9928391c2e5fdde5fbf3a7f7edc728"}
Nov 02 13:30:15.696727 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:15.694563    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-843978-679c455f86-rj4mr" event={"ID":"c68bfb23-6abb-4a96-979b-6b00ea4d5f7e","Type":"ContainerDied","Data":"8275bc7069571f59f93c8cbe87de3340764726e7b7481ae3257dd70d2cdb6c23"}
Nov 02 13:30:15.696727 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:15.694596    4559 scope.go:122] "RemoveContainer" containerID="37b390df297c04f9616c77c5257b05c3ee9928391c2e5fdde5fbf3a7f7edc728"
Nov 02 13:30:15.696727 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:15.694792    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-843978-679c455f86-rj4mr"
Nov 02 13:30:15.727176 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:15.727145    4559 scope.go:122] "RemoveContainer" containerID="37b390df297c04f9616c77c5257b05c3ee9928391c2e5fdde5fbf3a7f7edc728"
Nov 02 13:30:15.727875 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:30:15.727845    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37b390df297c04f9616c77c5257b05c3ee9928391c2e5fdde5fbf3a7f7edc728\": container with ID starting with 37b390df297c04f9616c77c5257b05c3ee9928391c2e5fdde5fbf3a7f7edc728 not found: ID does not exist" containerID="37b390df297c04f9616c77c5257b05c3ee9928391c2e5fdde5fbf3a7f7edc728"
Nov 02 13:30:15.728031 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:15.727883    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37b390df297c04f9616c77c5257b05c3ee9928391c2e5fdde5fbf3a7f7edc728"} err="failed to get container status \"37b390df297c04f9616c77c5257b05c3ee9928391c2e5fdde5fbf3a7f7edc728\": rpc error: code = NotFound desc = could not find container \"37b390df297c04f9616c77c5257b05c3ee9928391c2e5fdde5fbf3a7f7edc728\": container with ID starting with 37b390df297c04f9616c77c5257b05c3ee9928391c2e5fdde5fbf3a7f7edc728 not found: ID does not exist"
Nov 02 13:30:15.729406 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:15.728375    4559 generic.go:356] "Generic (PLEG): container finished" podID="46747055-1220-4a7e-8177-915d463a9f10" containerID="d586779398f9f35bad4377e9786a93adc8c5fc60fe8c2b5e27697ced0bd66fc2" exitCode=60
Nov 02 13:30:15.729406 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:15.728530    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-1aa18c-29893770-gcn5r" event={"ID":"46747055-1220-4a7e-8177-915d463a9f10","Type":"ContainerDied","Data":"d586779398f9f35bad4377e9786a93adc8c5fc60fe8c2b5e27697ced0bd66fc2"}
Nov 02 13:30:15.729406 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:15.728581    4559 scope.go:122] "RemoveContainer" containerID="7208966fbb20e8b1377153a7bef1cff2e20f4bf0d38df34791ffc7a6a927f32c"
Nov 02 13:30:15.730149 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:15.730125    4559 scope.go:122] "RemoveContainer" containerID="d586779398f9f35bad4377e9786a93adc8c5fc60fe8c2b5e27697ced0bd66fc2"
Nov 02 13:30:15.730664 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:30:15.730532    4559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-1aa18c-29893770-gcn5r_odh-ai-gateway-infra(46747055-1220-4a7e-8177-915d463a9f10)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-1aa18c-29893770-gcn5r" podUID="46747055-1220-4a7e-8177-915d463a9f10"
Nov 02 13:30:15.736451 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:15.736425    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a6f2c0-29893770-kfh8s" event={"ID":"d7505d42-8673-4ed0-8a6e-b6923d8ca25e","Type":"ContainerStarted","Data":"4ef0db3dd40ff5211d9d6d1841537033c0cd8f8e5f6cf27d1d5454e6a81c8621"}
Nov 02 13:30:15.740230 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:15.740206    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-740f09-29893770-vs8m7" event={"ID":"eede656a-d78e-4cf2-b98d-14d172fd36c6","Type":"ContainerStarted","Data":"3a7551094b17e40fffbf7e73df77e05b50a5c47432a7d7830063f4fb37a586dd"}
Nov 02 13:30:15.740484 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:15.740458    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-1aa18c-29893770-gcn5r"]
Nov 02 13:30:15.924159 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:15.924122    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-843978-679c455f86-rj4mr"]
Nov 02 13:30:15.927077 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:15.927056    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-843978-679c455f86-rj4mr"]
Nov 02 13:30:16.748730 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:16.748698    4559 generic.go:356] "Generic (PLEG): container finished" podID="d7505d42-8673-4ed0-8a6e-b6923d8ca25e" containerID="4ef0db3dd40ff5211d9d6d1841537033c0cd8f8e5f6cf27d1d5454e6a81c8621" exitCode=60
Nov 02 13:30:16.749169 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:16.748767    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a6f2c0-29893770-kfh8s" event={"ID":"d7505d42-8673-4ed0-8a6e-b6923d8ca25e","Type":"ContainerDied","Data":"4ef0db3dd40ff5211d9d6d1841537033c0cd8f8e5f6cf27d1d5454e6a81c8621"}
Nov 02 13:30:16.749169 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:16.748823    4559 scope.go:122] "RemoveContainer" containerID="8a4e274604bc262224d387381b2eed3ee4b600bff033e55f8ab49beedd6ceb4c"
Nov 02 13:30:16.749298 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:16.749239    4559 scope.go:122] "RemoveContainer" containerID="4ef0db3dd40ff5211d9d6d1841537033c0cd8f8e5f6cf27d1d5454e6a81c8621"
Nov 02 13:30:16.749653 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:30:16.749618    4559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-a6f2c0-29893770-kfh8s_odh-ai-gateway-infra(d7505d42-8673-4ed0-8a6e-b6923d8ca25e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a6f2c0-29893770-kfh8s" podUID="d7505d42-8673-4ed0-8a6e-b6923d8ca25e"
Nov 02 13:30:16.750886 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:16.750869    4559 generic.go:356] "Generic (PLEG): container finished" podID="eede656a-d78e-4cf2-b98d-14d172fd36c6" containerID="3a7551094b17e40fffbf7e73df77e05b50a5c47432a7d7830063f4fb37a586dd" exitCode=60
Nov 02 13:30:16.750964 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:16.750941    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-740f09-29893770-vs8m7" event={"ID":"eede656a-d78e-4cf2-b98d-14d172fd36c6","Type":"ContainerDied","Data":"3a7551094b17e40fffbf7e73df77e05b50a5c47432a7d7830063f4fb37a586dd"}
Nov 02 13:30:16.751303 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:16.751284    4559 scope.go:122] "RemoveContainer" containerID="3a7551094b17e40fffbf7e73df77e05b50a5c47432a7d7830063f4fb37a586dd"
Nov 02 13:30:16.751577 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:30:16.751559    4559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-740f09-29893770-vs8m7_odh-ai-gateway-infra(eede656a-d78e-4cf2-b98d-14d172fd36c6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-740f09-29893770-vs8m7" podUID="eede656a-d78e-4cf2-b98d-14d172fd36c6"
Nov 02 13:30:16.762405 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:16.762373    4559 scope.go:122] "RemoveContainer" containerID="3611fe2e2dca2423220ab401cf16e07cbe0deb73a2ef356bb89404f70348a933"
Nov 02 13:30:16.846542 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:16.846508    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-740f09-29893770-vs8m7"]
Nov 02 13:30:16.888431 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:16.888404    4559 scope.go:122] "RemoveContainer" containerID="3875c112a3dd06df1e57841b0501dbc03611e8cfefd6be98f75cfc58aa05235f"
Nov 02 13:30:16.897352 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:16.897328    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a6f2c0-29893770-kfh8s"]
Nov 02 13:30:16.949571 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:16.949546    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-1aa18c-29893770-gcn5r"
Nov 02 13:30:17.119038 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:17.119005    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46747055-1220-4a7e-8177-915d463a9f10-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/46747055-1220-4a7e-8177-915d463a9f10-maas-api-service-ca\") pod \"46747055-1220-4a7e-8177-915d463a9f10\" (UID: \"46747055-1220-4a7e-8177-915d463a9f10\") "
Nov 02 13:30:17.119179 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:17.119131    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46747055-1220-4a7e-8177-915d463a9f10-kube-api-access-hvkkj\" (UniqueName: \"kubernetes.io/projected/46747055-1220-4a7e-8177-915d463a9f10-kube-api-access-hvkkj\") pod \"46747055-1220-4a7e-8177-915d463a9f10\" (UID: \"46747055-1220-4a7e-8177-915d463a9f10\") "
Nov 02 13:30:17.119359 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:17.119340    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46747055-1220-4a7e-8177-915d463a9f10-maas-api-service-ca" pod "46747055-1220-4a7e-8177-915d463a9f10" (UID: "46747055-1220-4a7e-8177-915d463a9f10"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:30:17.119582 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:17.119561    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/46747055-1220-4a7e-8177-915d463a9f10-maas-api-service-ca\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:30:17.121402 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:17.121381    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46747055-1220-4a7e-8177-915d463a9f10-kube-api-access-hvkkj" pod "46747055-1220-4a7e-8177-915d463a9f10" (UID: "46747055-1220-4a7e-8177-915d463a9f10"). InnerVolumeSpecName "kube-api-access-hvkkj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:30:17.220613 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:17.220585    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hvkkj\" (UniqueName: \"kubernetes.io/projected/46747055-1220-4a7e-8177-915d463a9f10-kube-api-access-hvkkj\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:30:17.766013 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:17.765985    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-1aa18c-29893770-gcn5r"
Nov 02 13:30:17.766465 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:17.766024    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-1aa18c-29893770-gcn5r" event={"ID":"46747055-1220-4a7e-8177-915d463a9f10","Type":"ContainerDied","Data":"803e496cf9f5d374e69dc52ea665a4274dc5ef25bf194334ed4c4df1d43c1a75"}
Nov 02 13:30:17.766465 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:17.766095    4559 scope.go:122] "RemoveContainer" containerID="d586779398f9f35bad4377e9786a93adc8c5fc60fe8c2b5e27697ced0bd66fc2"
Nov 02 13:30:17.789840 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:17.789813    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893770-xzkcq" event={"ID":"4aa05b9d-f18c-4c47-82e9-bfe95d1afe48","Type":"ContainerStarted","Data":"6ff55f92c339b769a3c9285d18331db6cfa824691425d08f695aa0ba26dd8c81"}
Nov 02 13:30:17.866095 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:17.866064    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-1aa18c-29893770-gcn5r"]
Nov 02 13:30:17.870494 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:17.870470    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-1aa18c-29893770-gcn5r"]
Nov 02 13:30:17.895501 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:17.895471    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46747055-1220-4a7e-8177-915d463a9f10" path="/var/lib/kubelet/pods/46747055-1220-4a7e-8177-915d463a9f10/volumes"
Nov 02 13:30:17.896066 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:17.896049    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c68bfb23-6abb-4a96-979b-6b00ea4d5f7e" path="/var/lib/kubelet/pods/c68bfb23-6abb-4a96-979b-6b00ea4d5f7e/volumes"
Nov 02 13:30:17.979403 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:17.979316    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-740f09-29893770-vs8m7"
Nov 02 13:30:18.079345 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:18.079317    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a6f2c0-29893770-kfh8s"
Nov 02 13:30:18.139065 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:18.136796    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eede656a-d78e-4cf2-b98d-14d172fd36c6-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eede656a-d78e-4cf2-b98d-14d172fd36c6-maas-api-service-ca\") pod \"eede656a-d78e-4cf2-b98d-14d172fd36c6\" (UID: \"eede656a-d78e-4cf2-b98d-14d172fd36c6\") "
Nov 02 13:30:18.139065 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:18.137081    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eede656a-d78e-4cf2-b98d-14d172fd36c6-kube-api-access-t2jmf\" (UniqueName: \"kubernetes.io/projected/eede656a-d78e-4cf2-b98d-14d172fd36c6-kube-api-access-t2jmf\") pod \"eede656a-d78e-4cf2-b98d-14d172fd36c6\" (UID: \"eede656a-d78e-4cf2-b98d-14d172fd36c6\") "
Nov 02 13:30:18.139065 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:18.138550    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eede656a-d78e-4cf2-b98d-14d172fd36c6-maas-api-service-ca" pod "eede656a-d78e-4cf2-b98d-14d172fd36c6" (UID: "eede656a-d78e-4cf2-b98d-14d172fd36c6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:30:18.151096 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:18.151065    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eede656a-d78e-4cf2-b98d-14d172fd36c6-kube-api-access-t2jmf" pod "eede656a-d78e-4cf2-b98d-14d172fd36c6" (UID: "eede656a-d78e-4cf2-b98d-14d172fd36c6"). InnerVolumeSpecName "kube-api-access-t2jmf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:30:18.238323 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:18.238284    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7505d42-8673-4ed0-8a6e-b6923d8ca25e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7505d42-8673-4ed0-8a6e-b6923d8ca25e-maas-api-service-ca\") pod \"d7505d42-8673-4ed0-8a6e-b6923d8ca25e\" (UID: \"d7505d42-8673-4ed0-8a6e-b6923d8ca25e\") "
Nov 02 13:30:18.238549 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:18.238461    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7505d42-8673-4ed0-8a6e-b6923d8ca25e-kube-api-access-j5sw7\" (UniqueName: \"kubernetes.io/projected/d7505d42-8673-4ed0-8a6e-b6923d8ca25e-kube-api-access-j5sw7\") pod \"d7505d42-8673-4ed0-8a6e-b6923d8ca25e\" (UID: \"d7505d42-8673-4ed0-8a6e-b6923d8ca25e\") "
Nov 02 13:30:18.238781 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:18.238751    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7505d42-8673-4ed0-8a6e-b6923d8ca25e-maas-api-service-ca" pod "d7505d42-8673-4ed0-8a6e-b6923d8ca25e" (UID: "d7505d42-8673-4ed0-8a6e-b6923d8ca25e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:30:18.239143 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:18.239120    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7505d42-8673-4ed0-8a6e-b6923d8ca25e-maas-api-service-ca\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:30:18.239251 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:18.239147    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t2jmf\" (UniqueName: \"kubernetes.io/projected/eede656a-d78e-4cf2-b98d-14d172fd36c6-kube-api-access-t2jmf\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:30:18.239251 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:18.239163    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eede656a-d78e-4cf2-b98d-14d172fd36c6-maas-api-service-ca\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:30:18.240571 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:18.240546    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7505d42-8673-4ed0-8a6e-b6923d8ca25e-kube-api-access-j5sw7" pod "d7505d42-8673-4ed0-8a6e-b6923d8ca25e" (UID: "d7505d42-8673-4ed0-8a6e-b6923d8ca25e"). InnerVolumeSpecName "kube-api-access-j5sw7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:30:18.340069 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:18.339979    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j5sw7\" (UniqueName: \"kubernetes.io/projected/d7505d42-8673-4ed0-8a6e-b6923d8ca25e-kube-api-access-j5sw7\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:30:18.798892 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:18.798862    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a6f2c0-29893770-kfh8s"
Nov 02 13:30:18.799356 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:18.798865    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a6f2c0-29893770-kfh8s" event={"ID":"d7505d42-8673-4ed0-8a6e-b6923d8ca25e","Type":"ContainerDied","Data":"7e10d2c12ff41970979fff9d74de628b903d3e74cfb03744021c440e63e12c42"}
Nov 02 13:30:18.799356 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:18.799006    4559 scope.go:122] "RemoveContainer" containerID="4ef0db3dd40ff5211d9d6d1841537033c0cd8f8e5f6cf27d1d5454e6a81c8621"
Nov 02 13:30:18.800915 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:18.800888    4559 generic.go:356] "Generic (PLEG): container finished" podID="4aa05b9d-f18c-4c47-82e9-bfe95d1afe48" containerID="6ff55f92c339b769a3c9285d18331db6cfa824691425d08f695aa0ba26dd8c81" exitCode=60
Nov 02 13:30:18.801032 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:18.801016    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893770-xzkcq" event={"ID":"4aa05b9d-f18c-4c47-82e9-bfe95d1afe48","Type":"ContainerDied","Data":"6ff55f92c339b769a3c9285d18331db6cfa824691425d08f695aa0ba26dd8c81"}
Nov 02 13:30:18.801451 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:18.801433    4559 scope.go:122] "RemoveContainer" containerID="6ff55f92c339b769a3c9285d18331db6cfa824691425d08f695aa0ba26dd8c81"
Nov 02 13:30:18.801810 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:30:18.801785    4559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29893770-xzkcq_odh-ai-gateway-infra(4aa05b9d-f18c-4c47-82e9-bfe95d1afe48)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893770-xzkcq" podUID="4aa05b9d-f18c-4c47-82e9-bfe95d1afe48"
Nov 02 13:30:18.802525 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:18.802504    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-740f09-29893770-vs8m7" event={"ID":"eede656a-d78e-4cf2-b98d-14d172fd36c6","Type":"ContainerDied","Data":"2ad558e60c68316c0eb3e441fa84454295f8e1e4772b38bed578f5aa714fd4d7"}
Nov 02 13:30:18.802615 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:18.802527    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-740f09-29893770-vs8m7"
Nov 02 13:30:18.808843 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:18.808824    4559 scope.go:122] "RemoveContainer" containerID="3875c112a3dd06df1e57841b0501dbc03611e8cfefd6be98f75cfc58aa05235f"
Nov 02 13:30:18.821466 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:18.821444    4559 scope.go:122] "RemoveContainer" containerID="3a7551094b17e40fffbf7e73df77e05b50a5c47432a7d7830063f4fb37a586dd"
Nov 02 13:30:18.853344 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:18.853307    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-740f09-29893770-vs8m7"]
Nov 02 13:30:18.858947 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:18.858924    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-740f09-29893770-vs8m7"]
Nov 02 13:30:18.897150 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:18.897120    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a6f2c0-29893770-kfh8s"]
Nov 02 13:30:18.900808 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:18.900784    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a6f2c0-29893770-kfh8s"]
Nov 02 13:30:18.946786 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:18.946752    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893770-xzkcq"]
Nov 02 13:30:19.900267 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:19.900237    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7505d42-8673-4ed0-8a6e-b6923d8ca25e" path="/var/lib/kubelet/pods/d7505d42-8673-4ed0-8a6e-b6923d8ca25e/volumes"
Nov 02 13:30:19.900895 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:19.900874    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eede656a-d78e-4cf2-b98d-14d172fd36c6" path="/var/lib/kubelet/pods/eede656a-d78e-4cf2-b98d-14d172fd36c6/volumes"
Nov 02 13:30:19.976693 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:19.976670    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893770-xzkcq"
Nov 02 13:30:20.055699 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:20.055664    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4aa05b9d-f18c-4c47-82e9-bfe95d1afe48-kube-api-access-h48dc\" (UniqueName: \"kubernetes.io/projected/4aa05b9d-f18c-4c47-82e9-bfe95d1afe48-kube-api-access-h48dc\") pod \"4aa05b9d-f18c-4c47-82e9-bfe95d1afe48\" (UID: \"4aa05b9d-f18c-4c47-82e9-bfe95d1afe48\") "
Nov 02 13:30:20.055909 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:20.055804    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4aa05b9d-f18c-4c47-82e9-bfe95d1afe48-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4aa05b9d-f18c-4c47-82e9-bfe95d1afe48-maas-api-service-ca\") pod \"4aa05b9d-f18c-4c47-82e9-bfe95d1afe48\" (UID: \"4aa05b9d-f18c-4c47-82e9-bfe95d1afe48\") "
Nov 02 13:30:20.056450 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:20.056419    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4aa05b9d-f18c-4c47-82e9-bfe95d1afe48-maas-api-service-ca" pod "4aa05b9d-f18c-4c47-82e9-bfe95d1afe48" (UID: "4aa05b9d-f18c-4c47-82e9-bfe95d1afe48"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:30:20.058316 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:20.058289    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4aa05b9d-f18c-4c47-82e9-bfe95d1afe48-kube-api-access-h48dc" pod "4aa05b9d-f18c-4c47-82e9-bfe95d1afe48" (UID: "4aa05b9d-f18c-4c47-82e9-bfe95d1afe48"). InnerVolumeSpecName "kube-api-access-h48dc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:30:20.157424 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:20.157392    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h48dc\" (UniqueName: \"kubernetes.io/projected/4aa05b9d-f18c-4c47-82e9-bfe95d1afe48-kube-api-access-h48dc\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:30:20.157424 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:20.157419    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4aa05b9d-f18c-4c47-82e9-bfe95d1afe48-maas-api-service-ca\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:30:20.826832 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:20.826801    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893770-xzkcq"
Nov 02 13:30:20.826832 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:20.826826    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893770-xzkcq" event={"ID":"4aa05b9d-f18c-4c47-82e9-bfe95d1afe48","Type":"ContainerDied","Data":"b4eddb2510f229ab798425f8ab88350593c75cbe21c5283a6881c798401d9210"}
Nov 02 13:30:20.827080 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:20.826914    4559 scope.go:122] "RemoveContainer" containerID="6ff55f92c339b769a3c9285d18331db6cfa824691425d08f695aa0ba26dd8c81"
Nov 02 13:30:20.880680 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:20.880634    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893770-xzkcq"]
Nov 02 13:30:20.894933 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:20.894907    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893770-xzkcq"]
Nov 02 13:30:21.894103 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:21.894068    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4aa05b9d-f18c-4c47-82e9-bfe95d1afe48" path="/var/lib/kubelet/pods/4aa05b9d-f18c-4c47-82e9-bfe95d1afe48/volumes"
Nov 02 13:30:31.677518 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:31.677432    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-qk2dp"]
Nov 02 13:30:31.678047 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:31.677744    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-qk2dp" podUID="ee4f0fd1-9b89-4c29-ba5f-3cf47a18d2be" containerName="istio-proxy" containerID="cri-o://a62e6950cf74c7cf03426b860986d491f12b196efe29b3024eb0966bc2e7d227" gracePeriod=30
Nov 02 13:30:33.305443 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:33.305405    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-e7561a-openshift-default-7598f89b79-m5rrm"]
Nov 02 13:30:33.305973 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:33.305690    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-e7561a-openshift-default-7598f89b79-m5rrm" podUID="4293ef7c-ce26-475f-98e1-ebf007c35f00" containerName="istio-proxy" containerID="cri-o://19b2b5345b9ea13a7075f8ba886ab0f6af8f2e3a48dc9e42611fd54bd9467bf2" gracePeriod=30
Nov 02 13:30:36.947244 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:36.947169    4559 generic.go:356] "Generic (PLEG): container finished" podID="ee4f0fd1-9b89-4c29-ba5f-3cf47a18d2be" containerID="a62e6950cf74c7cf03426b860986d491f12b196efe29b3024eb0966bc2e7d227" exitCode=0
Nov 02 13:30:36.947244 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:36.947200    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-qk2dp" event={"ID":"ee4f0fd1-9b89-4c29-ba5f-3cf47a18d2be","Type":"ContainerDied","Data":"a62e6950cf74c7cf03426b860986d491f12b196efe29b3024eb0966bc2e7d227"}
Nov 02 13:30:36.947691 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:36.947286    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-qk2dp" event={"ID":"ee4f0fd1-9b89-4c29-ba5f-3cf47a18d2be","Type":"ContainerDied","Data":"31a33445ab51a02222149901ec417661ca4c8e3d8c0a1317d41564c876dd6853"}
Nov 02 13:30:36.947691 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:36.947314    4559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="31a33445ab51a02222149901ec417661ca4c8e3d8c0a1317d41564c876dd6853"
Nov 02 13:30:36.949857 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:36.949835    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-qk2dp"
Nov 02 13:30:37.018868 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:37.018837    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ee4f0fd1-9b89-4c29-ba5f-3cf47a18d2be-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee4f0fd1-9b89-4c29-ba5f-3cf47a18d2be-istio-podinfo\") pod \"ee4f0fd1-9b89-4c29-ba5f-3cf47a18d2be\" (UID: \"ee4f0fd1-9b89-4c29-ba5f-3cf47a18d2be\") "
Nov 02 13:30:37.019036 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:37.018942    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee4f0fd1-9b89-4c29-ba5f-3cf47a18d2be-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee4f0fd1-9b89-4c29-ba5f-3cf47a18d2be-credential-socket\") pod \"ee4f0fd1-9b89-4c29-ba5f-3cf47a18d2be\" (UID: \"ee4f0fd1-9b89-4c29-ba5f-3cf47a18d2be\") "
Nov 02 13:30:37.019036 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:37.019017    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee4f0fd1-9b89-4c29-ba5f-3cf47a18d2be-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee4f0fd1-9b89-4c29-ba5f-3cf47a18d2be-workload-certs\") pod \"ee4f0fd1-9b89-4c29-ba5f-3cf47a18d2be\" (UID: \"ee4f0fd1-9b89-4c29-ba5f-3cf47a18d2be\") "
Nov 02 13:30:37.019165 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:37.019046    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee4f0fd1-9b89-4c29-ba5f-3cf47a18d2be-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee4f0fd1-9b89-4c29-ba5f-3cf47a18d2be-istio-data\") pod \"ee4f0fd1-9b89-4c29-ba5f-3cf47a18d2be\" (UID: \"ee4f0fd1-9b89-4c29-ba5f-3cf47a18d2be\") "
Nov 02 13:30:37.019165 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:37.019100    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee4f0fd1-9b89-4c29-ba5f-3cf47a18d2be-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee4f0fd1-9b89-4c29-ba5f-3cf47a18d2be-istiod-ca-cert\") pod \"ee4f0fd1-9b89-4c29-ba5f-3cf47a18d2be\" (UID: \"ee4f0fd1-9b89-4c29-ba5f-3cf47a18d2be\") "
Nov 02 13:30:37.019165 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:37.019128    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee4f0fd1-9b89-4c29-ba5f-3cf47a18d2be-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee4f0fd1-9b89-4c29-ba5f-3cf47a18d2be-workload-socket\") pod \"ee4f0fd1-9b89-4c29-ba5f-3cf47a18d2be\" (UID: \"ee4f0fd1-9b89-4c29-ba5f-3cf47a18d2be\") "
Nov 02 13:30:37.019326 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:37.019184    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee4f0fd1-9b89-4c29-ba5f-3cf47a18d2be-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee4f0fd1-9b89-4c29-ba5f-3cf47a18d2be-istio-envoy\") pod \"ee4f0fd1-9b89-4c29-ba5f-3cf47a18d2be\" (UID: \"ee4f0fd1-9b89-4c29-ba5f-3cf47a18d2be\") "
Nov 02 13:30:37.019326 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:37.019270    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee4f0fd1-9b89-4c29-ba5f-3cf47a18d2be-istio-token\" (UniqueName: \"kubernetes.io/projected/ee4f0fd1-9b89-4c29-ba5f-3cf47a18d2be-istio-token\") pod \"ee4f0fd1-9b89-4c29-ba5f-3cf47a18d2be\" (UID: \"ee4f0fd1-9b89-4c29-ba5f-3cf47a18d2be\") "
Nov 02 13:30:37.019441 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:37.019330    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee4f0fd1-9b89-4c29-ba5f-3cf47a18d2be-kube-api-access-6rhm8\" (UniqueName: \"kubernetes.io/projected/ee4f0fd1-9b89-4c29-ba5f-3cf47a18d2be-kube-api-access-6rhm8\") pod \"ee4f0fd1-9b89-4c29-ba5f-3cf47a18d2be\" (UID: \"ee4f0fd1-9b89-4c29-ba5f-3cf47a18d2be\") "
Nov 02 13:30:37.019555 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:37.019523    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee4f0fd1-9b89-4c29-ba5f-3cf47a18d2be-istiod-ca-cert" pod "ee4f0fd1-9b89-4c29-ba5f-3cf47a18d2be" (UID: "ee4f0fd1-9b89-4c29-ba5f-3cf47a18d2be"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:30:37.020073 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:37.019740    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee4f0fd1-9b89-4c29-ba5f-3cf47a18d2be-workload-socket" pod "ee4f0fd1-9b89-4c29-ba5f-3cf47a18d2be" (UID: "ee4f0fd1-9b89-4c29-ba5f-3cf47a18d2be"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:30:37.020073 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:37.019873    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee4f0fd1-9b89-4c29-ba5f-3cf47a18d2be-istio-data" pod "ee4f0fd1-9b89-4c29-ba5f-3cf47a18d2be" (UID: "ee4f0fd1-9b89-4c29-ba5f-3cf47a18d2be"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:30:37.020073 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:37.020014    4559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee4f0fd1-9b89-4c29-ba5f-3cf47a18d2be-istio-data\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:30:37.020073 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:37.020021    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee4f0fd1-9b89-4c29-ba5f-3cf47a18d2be-workload-certs" pod "ee4f0fd1-9b89-4c29-ba5f-3cf47a18d2be" (UID: "ee4f0fd1-9b89-4c29-ba5f-3cf47a18d2be"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:30:37.020073 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:37.020036    4559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee4f0fd1-9b89-4c29-ba5f-3cf47a18d2be-istiod-ca-cert\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:30:37.020073 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:37.020052    4559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee4f0fd1-9b89-4c29-ba5f-3cf47a18d2be-workload-socket\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:30:37.020488 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:37.020467    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee4f0fd1-9b89-4c29-ba5f-3cf47a18d2be-credential-socket" pod "ee4f0fd1-9b89-4c29-ba5f-3cf47a18d2be" (UID: "ee4f0fd1-9b89-4c29-ba5f-3cf47a18d2be"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:30:37.021968 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:37.021944    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ee4f0fd1-9b89-4c29-ba5f-3cf47a18d2be-istio-podinfo" pod "ee4f0fd1-9b89-4c29-ba5f-3cf47a18d2be" (UID: "ee4f0fd1-9b89-4c29-ba5f-3cf47a18d2be"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 13:30:37.022072 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:37.022048    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee4f0fd1-9b89-4c29-ba5f-3cf47a18d2be-istio-token" pod "ee4f0fd1-9b89-4c29-ba5f-3cf47a18d2be" (UID: "ee4f0fd1-9b89-4c29-ba5f-3cf47a18d2be"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:30:37.022321 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:37.022289    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee4f0fd1-9b89-4c29-ba5f-3cf47a18d2be-istio-envoy" pod "ee4f0fd1-9b89-4c29-ba5f-3cf47a18d2be" (UID: "ee4f0fd1-9b89-4c29-ba5f-3cf47a18d2be"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:30:37.023075 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:37.023055    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee4f0fd1-9b89-4c29-ba5f-3cf47a18d2be-kube-api-access-6rhm8" pod "ee4f0fd1-9b89-4c29-ba5f-3cf47a18d2be" (UID: "ee4f0fd1-9b89-4c29-ba5f-3cf47a18d2be"). InnerVolumeSpecName "kube-api-access-6rhm8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:30:37.122064 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:37.122029    4559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee4f0fd1-9b89-4c29-ba5f-3cf47a18d2be-istio-podinfo\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:30:37.122064 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:37.122065    4559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee4f0fd1-9b89-4c29-ba5f-3cf47a18d2be-credential-socket\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:30:37.122276 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:37.122079    4559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee4f0fd1-9b89-4c29-ba5f-3cf47a18d2be-workload-certs\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:30:37.122276 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:37.122091    4559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee4f0fd1-9b89-4c29-ba5f-3cf47a18d2be-istio-envoy\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:30:37.122276 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:37.122104    4559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee4f0fd1-9b89-4c29-ba5f-3cf47a18d2be-istio-token\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:30:37.122276 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:37.122118    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6rhm8\" (UniqueName: \"kubernetes.io/projected/ee4f0fd1-9b89-4c29-ba5f-3cf47a18d2be-kube-api-access-6rhm8\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:30:37.954898 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:37.954869    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-qk2dp"
Nov 02 13:30:38.057826 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:38.057791    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-qk2dp"]
Nov 02 13:30:38.069592 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:38.069554    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-qk2dp"]
Nov 02 13:30:38.585544 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:38.585481    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-e7561a-openshift-default-7598f89b79-m5rrm"
Nov 02 13:30:38.642502 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:38.642470    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4293ef7c-ce26-475f-98e1-ebf007c35f00-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4293ef7c-ce26-475f-98e1-ebf007c35f00-istiod-ca-cert\") pod \"4293ef7c-ce26-475f-98e1-ebf007c35f00\" (UID: \"4293ef7c-ce26-475f-98e1-ebf007c35f00\") "
Nov 02 13:30:38.642720 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:38.642511    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4293ef7c-ce26-475f-98e1-ebf007c35f00-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4293ef7c-ce26-475f-98e1-ebf007c35f00-istio-data\") pod \"4293ef7c-ce26-475f-98e1-ebf007c35f00\" (UID: \"4293ef7c-ce26-475f-98e1-ebf007c35f00\") "
Nov 02 13:30:38.642720 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:38.642676    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4293ef7c-ce26-475f-98e1-ebf007c35f00-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4293ef7c-ce26-475f-98e1-ebf007c35f00-istio-envoy\") pod \"4293ef7c-ce26-475f-98e1-ebf007c35f00\" (UID: \"4293ef7c-ce26-475f-98e1-ebf007c35f00\") "
Nov 02 13:30:38.642865 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:38.642771    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4293ef7c-ce26-475f-98e1-ebf007c35f00-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4293ef7c-ce26-475f-98e1-ebf007c35f00-workload-certs\") pod \"4293ef7c-ce26-475f-98e1-ebf007c35f00\" (UID: \"4293ef7c-ce26-475f-98e1-ebf007c35f00\") "
Nov 02 13:30:38.642865 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:38.642820    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4293ef7c-ce26-475f-98e1-ebf007c35f00-istio-data" pod "4293ef7c-ce26-475f-98e1-ebf007c35f00" (UID: "4293ef7c-ce26-475f-98e1-ebf007c35f00"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:30:38.642865 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:38.642856    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4293ef7c-ce26-475f-98e1-ebf007c35f00-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4293ef7c-ce26-475f-98e1-ebf007c35f00-workload-socket\") pod \"4293ef7c-ce26-475f-98e1-ebf007c35f00\" (UID: \"4293ef7c-ce26-475f-98e1-ebf007c35f00\") "
Nov 02 13:30:38.643036 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:38.642892    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4293ef7c-ce26-475f-98e1-ebf007c35f00-kube-api-access-5ts9j\" (UniqueName: \"kubernetes.io/projected/4293ef7c-ce26-475f-98e1-ebf007c35f00-kube-api-access-5ts9j\") pod \"4293ef7c-ce26-475f-98e1-ebf007c35f00\" (UID: \"4293ef7c-ce26-475f-98e1-ebf007c35f00\") "
Nov 02 13:30:38.643036 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:38.642949    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4293ef7c-ce26-475f-98e1-ebf007c35f00-istiod-ca-cert" pod "4293ef7c-ce26-475f-98e1-ebf007c35f00" (UID: "4293ef7c-ce26-475f-98e1-ebf007c35f00"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:30:38.643036 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:38.643021    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4293ef7c-ce26-475f-98e1-ebf007c35f00-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4293ef7c-ce26-475f-98e1-ebf007c35f00-istio-podinfo\") pod \"4293ef7c-ce26-475f-98e1-ebf007c35f00\" (UID: \"4293ef7c-ce26-475f-98e1-ebf007c35f00\") "
Nov 02 13:30:38.643195 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:38.643058    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4293ef7c-ce26-475f-98e1-ebf007c35f00-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4293ef7c-ce26-475f-98e1-ebf007c35f00-credential-socket\") pod \"4293ef7c-ce26-475f-98e1-ebf007c35f00\" (UID: \"4293ef7c-ce26-475f-98e1-ebf007c35f00\") "
Nov 02 13:30:38.643195 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:38.643125    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4293ef7c-ce26-475f-98e1-ebf007c35f00-istio-token\" (UniqueName: \"kubernetes.io/projected/4293ef7c-ce26-475f-98e1-ebf007c35f00-istio-token\") pod \"4293ef7c-ce26-475f-98e1-ebf007c35f00\" (UID: \"4293ef7c-ce26-475f-98e1-ebf007c35f00\") "
Nov 02 13:30:38.643303 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:38.643191    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4293ef7c-ce26-475f-98e1-ebf007c35f00-workload-certs" pod "4293ef7c-ce26-475f-98e1-ebf007c35f00" (UID: "4293ef7c-ce26-475f-98e1-ebf007c35f00"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:30:38.643413 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:38.643391    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4293ef7c-ce26-475f-98e1-ebf007c35f00-workload-socket" pod "4293ef7c-ce26-475f-98e1-ebf007c35f00" (UID: "4293ef7c-ce26-475f-98e1-ebf007c35f00"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:30:38.643837 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:38.643813    4559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4293ef7c-ce26-475f-98e1-ebf007c35f00-workload-certs\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:30:38.644038 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:38.644022    4559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4293ef7c-ce26-475f-98e1-ebf007c35f00-workload-socket\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:30:38.644167 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:38.644142    4559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4293ef7c-ce26-475f-98e1-ebf007c35f00-istiod-ca-cert\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:30:38.644167 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:38.644168    4559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4293ef7c-ce26-475f-98e1-ebf007c35f00-istio-data\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:30:38.644348 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:38.643940    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4293ef7c-ce26-475f-98e1-ebf007c35f00-credential-socket" pod "4293ef7c-ce26-475f-98e1-ebf007c35f00" (UID: "4293ef7c-ce26-475f-98e1-ebf007c35f00"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:30:38.645723 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:38.645683    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4293ef7c-ce26-475f-98e1-ebf007c35f00-kube-api-access-5ts9j" pod "4293ef7c-ce26-475f-98e1-ebf007c35f00" (UID: "4293ef7c-ce26-475f-98e1-ebf007c35f00"). InnerVolumeSpecName "kube-api-access-5ts9j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:30:38.647093 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:38.647040    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4293ef7c-ce26-475f-98e1-ebf007c35f00-istio-envoy" pod "4293ef7c-ce26-475f-98e1-ebf007c35f00" (UID: "4293ef7c-ce26-475f-98e1-ebf007c35f00"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:30:38.647093 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:38.647062    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4293ef7c-ce26-475f-98e1-ebf007c35f00-istio-token" pod "4293ef7c-ce26-475f-98e1-ebf007c35f00" (UID: "4293ef7c-ce26-475f-98e1-ebf007c35f00"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:30:38.647299 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:38.647274    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4293ef7c-ce26-475f-98e1-ebf007c35f00-istio-podinfo" pod "4293ef7c-ce26-475f-98e1-ebf007c35f00" (UID: "4293ef7c-ce26-475f-98e1-ebf007c35f00"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 13:30:38.745376 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:38.745340    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5ts9j\" (UniqueName: \"kubernetes.io/projected/4293ef7c-ce26-475f-98e1-ebf007c35f00-kube-api-access-5ts9j\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:30:38.745376 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:38.745378    4559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4293ef7c-ce26-475f-98e1-ebf007c35f00-istio-podinfo\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:30:38.745568 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:38.745397    4559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4293ef7c-ce26-475f-98e1-ebf007c35f00-credential-socket\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:30:38.745568 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:38.745410    4559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4293ef7c-ce26-475f-98e1-ebf007c35f00-istio-token\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:30:38.745568 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:38.745424    4559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4293ef7c-ce26-475f-98e1-ebf007c35f00-istio-envoy\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:30:38.963020 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:38.962981    4559 generic.go:356] "Generic (PLEG): container finished" podID="4293ef7c-ce26-475f-98e1-ebf007c35f00" containerID="19b2b5345b9ea13a7075f8ba886ab0f6af8f2e3a48dc9e42611fd54bd9467bf2" exitCode=0
Nov 02 13:30:38.963527 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:38.963097    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-e7561a-openshift-default-7598f89b79-m5rrm"
Nov 02 13:30:38.963527 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:38.963100    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-e7561a-openshift-default-7598f89b79-m5rrm" event={"ID":"4293ef7c-ce26-475f-98e1-ebf007c35f00","Type":"ContainerDied","Data":"19b2b5345b9ea13a7075f8ba886ab0f6af8f2e3a48dc9e42611fd54bd9467bf2"}
Nov 02 13:30:38.963527 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:38.963234    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-e7561a-openshift-default-7598f89b79-m5rrm" event={"ID":"4293ef7c-ce26-475f-98e1-ebf007c35f00","Type":"ContainerDied","Data":"c10bb62b8e4efbbe9bffb82478bf968f2dfc4253acd122ba82acc1215d0a93ea"}
Nov 02 13:30:38.963527 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:38.963264    4559 scope.go:122] "RemoveContainer" containerID="19b2b5345b9ea13a7075f8ba886ab0f6af8f2e3a48dc9e42611fd54bd9467bf2"
Nov 02 13:30:38.974618 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:38.974604    4559 scope.go:122] "RemoveContainer" containerID="ccbfe98cbcb6cee4011b7e65b5332a6b8f935dc779efd6365f629dd39731a796"
Nov 02 13:30:38.985019 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:38.984996    4559 scope.go:122] "RemoveContainer" containerID="19b2b5345b9ea13a7075f8ba886ab0f6af8f2e3a48dc9e42611fd54bd9467bf2"
Nov 02 13:30:38.985337 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:30:38.985315    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19b2b5345b9ea13a7075f8ba886ab0f6af8f2e3a48dc9e42611fd54bd9467bf2\": container with ID starting with 19b2b5345b9ea13a7075f8ba886ab0f6af8f2e3a48dc9e42611fd54bd9467bf2 not found: ID does not exist" containerID="19b2b5345b9ea13a7075f8ba886ab0f6af8f2e3a48dc9e42611fd54bd9467bf2"
Nov 02 13:30:38.985454 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:38.985344    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19b2b5345b9ea13a7075f8ba886ab0f6af8f2e3a48dc9e42611fd54bd9467bf2"} err="failed to get container status \"19b2b5345b9ea13a7075f8ba886ab0f6af8f2e3a48dc9e42611fd54bd9467bf2\": rpc error: code = NotFound desc = could not find container \"19b2b5345b9ea13a7075f8ba886ab0f6af8f2e3a48dc9e42611fd54bd9467bf2\": container with ID starting with 19b2b5345b9ea13a7075f8ba886ab0f6af8f2e3a48dc9e42611fd54bd9467bf2 not found: ID does not exist"
Nov 02 13:30:38.985454 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:38.985361    4559 scope.go:122] "RemoveContainer" containerID="ccbfe98cbcb6cee4011b7e65b5332a6b8f935dc779efd6365f629dd39731a796"
Nov 02 13:30:38.985719 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:30:38.985690    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ccbfe98cbcb6cee4011b7e65b5332a6b8f935dc779efd6365f629dd39731a796\": container with ID starting with ccbfe98cbcb6cee4011b7e65b5332a6b8f935dc779efd6365f629dd39731a796 not found: ID does not exist" containerID="ccbfe98cbcb6cee4011b7e65b5332a6b8f935dc779efd6365f629dd39731a796"
Nov 02 13:30:38.985807 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:38.985728    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ccbfe98cbcb6cee4011b7e65b5332a6b8f935dc779efd6365f629dd39731a796"} err="failed to get container status \"ccbfe98cbcb6cee4011b7e65b5332a6b8f935dc779efd6365f629dd39731a796\": rpc error: code = NotFound desc = could not find container \"ccbfe98cbcb6cee4011b7e65b5332a6b8f935dc779efd6365f629dd39731a796\": container with ID starting with ccbfe98cbcb6cee4011b7e65b5332a6b8f935dc779efd6365f629dd39731a796 not found: ID does not exist"
Nov 02 13:30:39.022151 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:39.022118    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-e7561a-openshift-default-7598f89b79-m5rrm"]
Nov 02 13:30:39.036045 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:39.036011    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-e7561a-openshift-default-7598f89b79-m5rrm"]
Nov 02 13:30:39.896582 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:39.896547    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4293ef7c-ce26-475f-98e1-ebf007c35f00" path="/var/lib/kubelet/pods/4293ef7c-ce26-475f-98e1-ebf007c35f00/volumes"
Nov 02 13:30:39.897109 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:39.897094    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee4f0fd1-9b89-4c29-ba5f-3cf47a18d2be" path="/var/lib/kubelet/pods/ee4f0fd1-9b89-4c29-ba5f-3cf47a18d2be/volumes"
Nov 02 13:30:39.897448 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:39.897433    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-6416f0-7f56ccb547-q7k5v"]
Nov 02 13:30:44.898698 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:44.898631    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-6416f0-7f56ccb547-q7k5v" podUID="459db7e8-1294-4758-93a4-1fc9b9320595" containerName="payload-pre-processing" containerID="cri-o://8aee7ce49b53f0aed49cce17e60deb50fa3b7a9c09916c2e6a856ce8a5078d61" gracePeriod=25
Nov 02 13:30:45.086050 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:30:45.086015    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqcmj" containerName="extractor"
Nov 02 13:30:47.116222 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:47.116187    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-6416f0-7f56ccb547-q7k5v" podUID="459db7e8-1294-4758-93a4-1fc9b9320595" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 13:30:47.184633 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:47.184604    4559 scope.go:122] "RemoveContainer" containerID="a62e6950cf74c7cf03426b860986d491f12b196efe29b3024eb0966bc2e7d227"
Nov 02 13:30:56.982633 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:56.982597    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"]
Nov 02 13:30:57.116015 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:30:57.115981    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-6416f0-7f56ccb547-q7k5v" podUID="459db7e8-1294-4758-93a4-1fc9b9320595" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 13:31:02.142944 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:31:02.142911    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-068756-bdb7bcbbf-nnssk"]
Nov 02 13:31:02.178714 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:31:02.178683    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-068756-8565b4cd8c-fpqt6"]
Nov 02 13:31:05.164932 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:31:05.164867    4559 generic.go:356] "Generic (PLEG): container finished" podID="459db7e8-1294-4758-93a4-1fc9b9320595" containerID="8aee7ce49b53f0aed49cce17e60deb50fa3b7a9c09916c2e6a856ce8a5078d61" exitCode=0
Nov 02 13:31:05.165341 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:31:05.164935    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-6416f0-7f56ccb547-q7k5v" event={"ID":"459db7e8-1294-4758-93a4-1fc9b9320595","Type":"ContainerDied","Data":"8aee7ce49b53f0aed49cce17e60deb50fa3b7a9c09916c2e6a856ce8a5078d61"}
Nov 02 13:31:05.165341 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:31:05.164987    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-6416f0-7f56ccb547-q7k5v" event={"ID":"459db7e8-1294-4758-93a4-1fc9b9320595","Type":"ContainerDied","Data":"6a095b9ea54263fe206d5a135bee8f6c9afcf27d12f6d6ae10f501e117c210b1"}
Nov 02 13:31:05.165341 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:31:05.165006    4559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6a095b9ea54263fe206d5a135bee8f6c9afcf27d12f6d6ae10f501e117c210b1"
Nov 02 13:31:05.168122 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:31:05.168106    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-6416f0-7f56ccb547-q7k5v"
Nov 02 13:31:05.307494 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:31:05.307438    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/459db7e8-1294-4758-93a4-1fc9b9320595-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/459db7e8-1294-4758-93a4-1fc9b9320595-plugins-config-volume\") pod \"459db7e8-1294-4758-93a4-1fc9b9320595\" (UID: \"459db7e8-1294-4758-93a4-1fc9b9320595\") "
Nov 02 13:31:05.308425 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:31:05.307701    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/459db7e8-1294-4758-93a4-1fc9b9320595-kube-api-access-5bh4k\" (UniqueName: \"kubernetes.io/projected/459db7e8-1294-4758-93a4-1fc9b9320595-kube-api-access-5bh4k\") pod \"459db7e8-1294-4758-93a4-1fc9b9320595\" (UID: \"459db7e8-1294-4758-93a4-1fc9b9320595\") "
Nov 02 13:31:05.308545 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:31:05.308518    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/459db7e8-1294-4758-93a4-1fc9b9320595-plugins-config-volume" pod "459db7e8-1294-4758-93a4-1fc9b9320595" (UID: "459db7e8-1294-4758-93a4-1fc9b9320595"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:31:05.310374 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:31:05.310346    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/459db7e8-1294-4758-93a4-1fc9b9320595-kube-api-access-5bh4k" pod "459db7e8-1294-4758-93a4-1fc9b9320595" (UID: "459db7e8-1294-4758-93a4-1fc9b9320595"). InnerVolumeSpecName "kube-api-access-5bh4k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:31:05.409007 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:31:05.408976    4559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/459db7e8-1294-4758-93a4-1fc9b9320595-plugins-config-volume\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:31:05.409007 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:31:05.409003    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5bh4k\" (UniqueName: \"kubernetes.io/projected/459db7e8-1294-4758-93a4-1fc9b9320595-kube-api-access-5bh4k\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:31:06.170553 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:31:06.170524    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-6416f0-7f56ccb547-q7k5v"
Nov 02 13:31:06.226187 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:31:06.226147    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-6416f0-7f56ccb547-q7k5v"]
Nov 02 13:31:06.231245 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:31:06.231220    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-6416f0-7f56ccb547-q7k5v"]
Nov 02 13:31:07.144036 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:31:07.143970    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-068756-bdb7bcbbf-nnssk" podUID="226d3b62-6d8e-4f46-bf0e-870d7891314f" containerName="payload-processing" containerID="cri-o://ff131871a786e5661f10d591961f457330c5df9a3b85751fd502a222c2f76199" gracePeriod=25
Nov 02 13:31:07.179566 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:31:07.179526    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-068756-8565b4cd8c-fpqt6" podUID="1972d657-0f0b-40b8-85ba-ac651e09d31f" containerName="payload-pre-processing" containerID="cri-o://81a9d7a5c53155c5c46aff1d22d20825a7c3b50ccb5b34bb4d62778d8a038969" gracePeriod=25
Nov 02 13:31:07.896429 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:31:07.896396    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="459db7e8-1294-4758-93a4-1fc9b9320595" path="/var/lib/kubelet/pods/459db7e8-1294-4758-93a4-1fc9b9320595/volumes"
Nov 02 13:31:15.086217 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:31:15.086180    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqcmj" containerName="extractor"
Nov 02 13:31:15.432589 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:31:15.432552    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-068756-bdb7bcbbf-nnssk" podUID="226d3b62-6d8e-4f46-bf0e-870d7891314f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 13:31:15.434787 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:31:15.432565    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-068756-8565b4cd8c-fpqt6" podUID="1972d657-0f0b-40b8-85ba-ac651e09d31f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 13:31:25.432055 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:31:25.432010    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-068756-8565b4cd8c-fpqt6" podUID="1972d657-0f0b-40b8-85ba-ac651e09d31f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 13:31:25.432448 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:31:25.432113    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-068756-bdb7bcbbf-nnssk" podUID="226d3b62-6d8e-4f46-bf0e-870d7891314f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 13:31:27.240076 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:31:27.240048    4559 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1972d657_0f0b_40b8_85ba_ac651e09d31f.slice/crio-conmon-81a9d7a5c53155c5c46aff1d22d20825a7c3b50ccb5b34bb4d62778d8a038969.scope\": RecentStats: unable to find data in memory cache]"
Nov 02 13:31:27.334662 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:31:27.331350    4559 generic.go:356] "Generic (PLEG): container finished" podID="226d3b62-6d8e-4f46-bf0e-870d7891314f" containerID="ff131871a786e5661f10d591961f457330c5df9a3b85751fd502a222c2f76199" exitCode=0
Nov 02 13:31:27.334662 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:31:27.331529    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-068756-bdb7bcbbf-nnssk" event={"ID":"226d3b62-6d8e-4f46-bf0e-870d7891314f","Type":"ContainerDied","Data":"ff131871a786e5661f10d591961f457330c5df9a3b85751fd502a222c2f76199"}
Nov 02 13:31:27.334662 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:31:27.333663    4559 generic.go:356] "Generic (PLEG): container finished" podID="1972d657-0f0b-40b8-85ba-ac651e09d31f" containerID="81a9d7a5c53155c5c46aff1d22d20825a7c3b50ccb5b34bb4d62778d8a038969" exitCode=0
Nov 02 13:31:27.334662 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:31:27.333797    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-068756-8565b4cd8c-fpqt6" event={"ID":"1972d657-0f0b-40b8-85ba-ac651e09d31f","Type":"ContainerDied","Data":"81a9d7a5c53155c5c46aff1d22d20825a7c3b50ccb5b34bb4d62778d8a038969"}
Nov 02 13:31:27.492084 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:31:27.492056    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-068756-bdb7bcbbf-nnssk"
Nov 02 13:31:27.514277 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:31:27.514251    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-068756-8565b4cd8c-fpqt6"
Nov 02 13:31:27.630588 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:31:27.630555    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1972d657-0f0b-40b8-85ba-ac651e09d31f-kube-api-access-j7k87\" (UniqueName: \"kubernetes.io/projected/1972d657-0f0b-40b8-85ba-ac651e09d31f-kube-api-access-j7k87\") pod \"1972d657-0f0b-40b8-85ba-ac651e09d31f\" (UID: \"1972d657-0f0b-40b8-85ba-ac651e09d31f\") "
Nov 02 13:31:27.630830 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:31:27.630729    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/226d3b62-6d8e-4f46-bf0e-870d7891314f-kube-api-access-f4zdr\" (UniqueName: \"kubernetes.io/projected/226d3b62-6d8e-4f46-bf0e-870d7891314f-kube-api-access-f4zdr\") pod \"226d3b62-6d8e-4f46-bf0e-870d7891314f\" (UID: \"226d3b62-6d8e-4f46-bf0e-870d7891314f\") "
Nov 02 13:31:27.630830 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:31:27.630829    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/226d3b62-6d8e-4f46-bf0e-870d7891314f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/226d3b62-6d8e-4f46-bf0e-870d7891314f-plugins-config-volume\") pod \"226d3b62-6d8e-4f46-bf0e-870d7891314f\" (UID: \"226d3b62-6d8e-4f46-bf0e-870d7891314f\") "
Nov 02 13:31:27.630966 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:31:27.630887    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1972d657-0f0b-40b8-85ba-ac651e09d31f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1972d657-0f0b-40b8-85ba-ac651e09d31f-plugins-config-volume\") pod \"1972d657-0f0b-40b8-85ba-ac651e09d31f\" (UID: \"1972d657-0f0b-40b8-85ba-ac651e09d31f\") "
Nov 02 13:31:27.631581 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:31:27.631549    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1972d657-0f0b-40b8-85ba-ac651e09d31f-plugins-config-volume" pod "1972d657-0f0b-40b8-85ba-ac651e09d31f" (UID: "1972d657-0f0b-40b8-85ba-ac651e09d31f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:31:27.631729 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:31:27.631554    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/226d3b62-6d8e-4f46-bf0e-870d7891314f-plugins-config-volume" pod "226d3b62-6d8e-4f46-bf0e-870d7891314f" (UID: "226d3b62-6d8e-4f46-bf0e-870d7891314f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:31:27.633845 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:31:27.633804    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1972d657-0f0b-40b8-85ba-ac651e09d31f-kube-api-access-j7k87" pod "1972d657-0f0b-40b8-85ba-ac651e09d31f" (UID: "1972d657-0f0b-40b8-85ba-ac651e09d31f"). InnerVolumeSpecName "kube-api-access-j7k87". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:31:27.634342 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:31:27.634315    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/226d3b62-6d8e-4f46-bf0e-870d7891314f-kube-api-access-f4zdr" pod "226d3b62-6d8e-4f46-bf0e-870d7891314f" (UID: "226d3b62-6d8e-4f46-bf0e-870d7891314f"). InnerVolumeSpecName "kube-api-access-f4zdr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:31:27.732252 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:31:27.732170    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j7k87\" (UniqueName: \"kubernetes.io/projected/1972d657-0f0b-40b8-85ba-ac651e09d31f-kube-api-access-j7k87\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:31:27.732252 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:31:27.732208    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f4zdr\" (UniqueName: \"kubernetes.io/projected/226d3b62-6d8e-4f46-bf0e-870d7891314f-kube-api-access-f4zdr\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:31:27.732252 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:31:27.732225    4559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/226d3b62-6d8e-4f46-bf0e-870d7891314f-plugins-config-volume\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:31:27.732252 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:31:27.732238    4559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1972d657-0f0b-40b8-85ba-ac651e09d31f-plugins-config-volume\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:31:28.341775 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:31:28.341741    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-068756-8565b4cd8c-fpqt6"
Nov 02 13:31:28.341775 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:31:28.341769    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-068756-8565b4cd8c-fpqt6" event={"ID":"1972d657-0f0b-40b8-85ba-ac651e09d31f","Type":"ContainerDied","Data":"518455ef3971849c9c23df2b0ef07076549cc36bfe296d3b346800ea315b3c74"}
Nov 02 13:31:28.342349 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:31:28.341833    4559 scope.go:122] "RemoveContainer" containerID="81a9d7a5c53155c5c46aff1d22d20825a7c3b50ccb5b34bb4d62778d8a038969"
Nov 02 13:31:28.344234 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:31:28.344144    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-068756-bdb7bcbbf-nnssk"
Nov 02 13:31:28.344380 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:31:28.344138    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-068756-bdb7bcbbf-nnssk" event={"ID":"226d3b62-6d8e-4f46-bf0e-870d7891314f","Type":"ContainerDied","Data":"1370b4784999742a13921b864a158712ce60fc96b7a7675c678cdf2bea1d7526"}
Nov 02 13:31:28.352298 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:31:28.352279    4559 scope.go:122] "RemoveContainer" containerID="ff131871a786e5661f10d591961f457330c5df9a3b85751fd502a222c2f76199"
Nov 02 13:31:28.399529 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:31:28.399498    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-068756-8565b4cd8c-fpqt6"]
Nov 02 13:31:28.412163 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:31:28.412132    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-068756-8565b4cd8c-fpqt6"]
Nov 02 13:31:28.440320 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:31:28.440286    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-068756-bdb7bcbbf-nnssk"]
Nov 02 13:31:28.448164 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:31:28.448136    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-068756-bdb7bcbbf-nnssk"]
Nov 02 13:31:29.893239 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:31:29.893209    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1972d657-0f0b-40b8-85ba-ac651e09d31f" path="/var/lib/kubelet/pods/1972d657-0f0b-40b8-85ba-ac651e09d31f/volumes"
Nov 02 13:31:29.893661 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:31:29.893610    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="226d3b62-6d8e-4f46-bf0e-870d7891314f" path="/var/lib/kubelet/pods/226d3b62-6d8e-4f46-bf0e-870d7891314f/volumes"
Nov 02 13:31:45.086677 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:31:45.086622    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqcmj" containerName="extractor"
Nov 02 13:31:47.246058 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:31:47.246024    4559 scope.go:122] "RemoveContainer" containerID="8aee7ce49b53f0aed49cce17e60deb50fa3b7a9c09916c2e6a856ce8a5078d61"
Nov 02 13:31:51.271428 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:31:51.271398    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"]
Nov 02 13:32:11.078021 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:11.077987    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lwbxm"]
Nov 02 13:32:11.078465 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:11.078167    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lwbxm" podUID="10370825-cfb5-4493-83bd-a883abf2d789" containerName="discovery" containerID="cri-o://a9345030d784c103f676bd0f43a359131e40e62184e8ec939a1dcfd6ed400af3" gracePeriod=30
Nov 02 13:32:11.349294 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:11.349269    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lwbxm"
Nov 02 13:32:11.459539 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:11.459510    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/10370825-cfb5-4493-83bd-a883abf2d789-cacerts\" (UniqueName: \"kubernetes.io/secret/10370825-cfb5-4493-83bd-a883abf2d789-cacerts\") pod \"10370825-cfb5-4493-83bd-a883abf2d789\" (UID: \"10370825-cfb5-4493-83bd-a883abf2d789\") "
Nov 02 13:32:11.459732 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:11.459552    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10370825-cfb5-4493-83bd-a883abf2d789-istio-token\" (UniqueName: \"kubernetes.io/projected/10370825-cfb5-4493-83bd-a883abf2d789-istio-token\") pod \"10370825-cfb5-4493-83bd-a883abf2d789\" (UID: \"10370825-cfb5-4493-83bd-a883abf2d789\") "
Nov 02 13:32:11.459732 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:11.459593    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10370825-cfb5-4493-83bd-a883abf2d789-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/10370825-cfb5-4493-83bd-a883abf2d789-istio-csr-ca-configmap\") pod \"10370825-cfb5-4493-83bd-a883abf2d789\" (UID: \"10370825-cfb5-4493-83bd-a883abf2d789\") "
Nov 02 13:32:11.459732 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:11.459710    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10370825-cfb5-4493-83bd-a883abf2d789-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/10370825-cfb5-4493-83bd-a883abf2d789-local-certs\") pod \"10370825-cfb5-4493-83bd-a883abf2d789\" (UID: \"10370825-cfb5-4493-83bd-a883abf2d789\") "
Nov 02 13:32:11.459921 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:11.459862    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10370825-cfb5-4493-83bd-a883abf2d789-kube-api-access-kkvpr\" (UniqueName: \"kubernetes.io/projected/10370825-cfb5-4493-83bd-a883abf2d789-kube-api-access-kkvpr\") pod \"10370825-cfb5-4493-83bd-a883abf2d789\" (UID: \"10370825-cfb5-4493-83bd-a883abf2d789\") "
Nov 02 13:32:11.459981 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:11.459923    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/10370825-cfb5-4493-83bd-a883abf2d789-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/10370825-cfb5-4493-83bd-a883abf2d789-istio-csr-dns-cert\") pod \"10370825-cfb5-4493-83bd-a883abf2d789\" (UID: \"10370825-cfb5-4493-83bd-a883abf2d789\") "
Nov 02 13:32:11.460039 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:11.459983    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/10370825-cfb5-4493-83bd-a883abf2d789-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/10370825-cfb5-4493-83bd-a883abf2d789-istio-kubeconfig\") pod \"10370825-cfb5-4493-83bd-a883abf2d789\" (UID: \"10370825-cfb5-4493-83bd-a883abf2d789\") "
Nov 02 13:32:11.460431 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:11.460263    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10370825-cfb5-4493-83bd-a883abf2d789-istio-csr-ca-configmap" pod "10370825-cfb5-4493-83bd-a883abf2d789" (UID: "10370825-cfb5-4493-83bd-a883abf2d789"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:32:11.460577 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:11.460500    4559 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/10370825-cfb5-4493-83bd-a883abf2d789-istio-csr-ca-configmap\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:32:11.462346 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:11.462299    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10370825-cfb5-4493-83bd-a883abf2d789-local-certs" pod "10370825-cfb5-4493-83bd-a883abf2d789" (UID: "10370825-cfb5-4493-83bd-a883abf2d789"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:32:11.462463 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:11.462437    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10370825-cfb5-4493-83bd-a883abf2d789-istio-token" pod "10370825-cfb5-4493-83bd-a883abf2d789" (UID: "10370825-cfb5-4493-83bd-a883abf2d789"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:32:11.462863 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:11.462832    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10370825-cfb5-4493-83bd-a883abf2d789-istio-csr-dns-cert" pod "10370825-cfb5-4493-83bd-a883abf2d789" (UID: "10370825-cfb5-4493-83bd-a883abf2d789"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:32:11.463284 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:11.463256    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10370825-cfb5-4493-83bd-a883abf2d789-cacerts" pod "10370825-cfb5-4493-83bd-a883abf2d789" (UID: "10370825-cfb5-4493-83bd-a883abf2d789"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:32:11.463284 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:11.463277    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10370825-cfb5-4493-83bd-a883abf2d789-istio-kubeconfig" pod "10370825-cfb5-4493-83bd-a883abf2d789" (UID: "10370825-cfb5-4493-83bd-a883abf2d789"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:32:11.463680 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:11.463619    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10370825-cfb5-4493-83bd-a883abf2d789-kube-api-access-kkvpr" pod "10370825-cfb5-4493-83bd-a883abf2d789" (UID: "10370825-cfb5-4493-83bd-a883abf2d789"). InnerVolumeSpecName "kube-api-access-kkvpr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:32:11.561375 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:11.561345    4559 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/10370825-cfb5-4493-83bd-a883abf2d789-istio-csr-dns-cert\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:32:11.561375 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:11.561374    4559 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/10370825-cfb5-4493-83bd-a883abf2d789-istio-kubeconfig\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:32:11.561587 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:11.561384    4559 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/10370825-cfb5-4493-83bd-a883abf2d789-cacerts\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:32:11.561587 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:11.561392    4559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10370825-cfb5-4493-83bd-a883abf2d789-istio-token\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:32:11.561587 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:11.561402    4559 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/10370825-cfb5-4493-83bd-a883abf2d789-local-certs\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:32:11.561587 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:11.561410    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kkvpr\" (UniqueName: \"kubernetes.io/projected/10370825-cfb5-4493-83bd-a883abf2d789-kube-api-access-kkvpr\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:32:11.611189 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:11.611092    4559 generic.go:356] "Generic (PLEG): container finished" podID="10370825-cfb5-4493-83bd-a883abf2d789" containerID="a9345030d784c103f676bd0f43a359131e40e62184e8ec939a1dcfd6ed400af3" exitCode=0
Nov 02 13:32:11.611359 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:11.611208    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lwbxm"
Nov 02 13:32:11.611359 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:11.611231    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lwbxm" event={"ID":"10370825-cfb5-4493-83bd-a883abf2d789","Type":"ContainerDied","Data":"a9345030d784c103f676bd0f43a359131e40e62184e8ec939a1dcfd6ed400af3"}
Nov 02 13:32:11.611359 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:11.611284    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lwbxm" event={"ID":"10370825-cfb5-4493-83bd-a883abf2d789","Type":"ContainerDied","Data":"765561ce3bbb63effe943a648fd27bc751d73600fd64d5562459631f50f940e4"}
Nov 02 13:32:11.611359 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:11.611309    4559 scope.go:122] "RemoveContainer" containerID="a9345030d784c103f676bd0f43a359131e40e62184e8ec939a1dcfd6ed400af3"
Nov 02 13:32:11.625464 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:11.625444    4559 scope.go:122] "RemoveContainer" containerID="a9345030d784c103f676bd0f43a359131e40e62184e8ec939a1dcfd6ed400af3"
Nov 02 13:32:11.625775 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:32:11.625756    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9345030d784c103f676bd0f43a359131e40e62184e8ec939a1dcfd6ed400af3\": container with ID starting with a9345030d784c103f676bd0f43a359131e40e62184e8ec939a1dcfd6ed400af3 not found: ID does not exist" containerID="a9345030d784c103f676bd0f43a359131e40e62184e8ec939a1dcfd6ed400af3"
Nov 02 13:32:11.625856 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:11.625786    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9345030d784c103f676bd0f43a359131e40e62184e8ec939a1dcfd6ed400af3"} err="failed to get container status \"a9345030d784c103f676bd0f43a359131e40e62184e8ec939a1dcfd6ed400af3\": rpc error: code = NotFound desc = could not find container \"a9345030d784c103f676bd0f43a359131e40e62184e8ec939a1dcfd6ed400af3\": container with ID starting with a9345030d784c103f676bd0f43a359131e40e62184e8ec939a1dcfd6ed400af3 not found: ID does not exist"
Nov 02 13:32:11.716402 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:11.716323    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lwbxm"]
Nov 02 13:32:11.721133 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:11.720164    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lwbxm"]
Nov 02 13:32:11.893412 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:11.893336    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10370825-cfb5-4493-83bd-a883abf2d789" path="/var/lib/kubelet/pods/10370825-cfb5-4493-83bd-a883abf2d789/volumes"
Nov 02 13:32:15.087076 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:32:15.087033    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqcmj" containerName="extractor"
Nov 02 13:32:15.661027 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:15.660996    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"]
Nov 02 13:32:29.950170 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:29.950131    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-k5zn8"]
Nov 02 13:32:29.950776 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:29.950340    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-k5zn8" podUID="d3b2832d-2aad-406f-86b5-de405575f9b2" containerName="istio-proxy" containerID="cri-o://e11beb411d5d8cb9ad1ee54ee6367d6245a564a1de071e3b6a623192e83fa165" gracePeriod=30
Nov 02 13:32:29.962922 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:29.962893    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7kvjr"]
Nov 02 13:32:29.963132 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:29.963097    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7kvjr" podUID="b736e009-d002-4177-8f79-8f50cd5325f3" containerName="istio-proxy" containerID="cri-o://cd4b240992256d01908e7156df44d1661f5a7926377344fdd68067abb3c3f87e" gracePeriod=30
Nov 02 13:32:29.965475 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:29.965452    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-xj4xk"]
Nov 02 13:32:29.965699 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:29.965678    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-xj4xk" podUID="d2c5f43a-3a0c-444b-afc1-e2bf090dc05c" containerName="istio-proxy" containerID="cri-o://96ed21c4883f0d9a981d09f6cefccbd271ccb36cf3d4a1146353b01d1680067a" gracePeriod=30
Nov 02 13:32:29.975216 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:29.975191    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ckslt"]
Nov 02 13:32:29.975397 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:29.975373    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ckslt" podUID="826b92f5-161c-4f6a-9055-d0c838c9ce09" containerName="istio-proxy" containerID="cri-o://bd06e4fbfb52ae78f4d1d88b88740481e7fb5e29e35825a9512b1bec633649b4" gracePeriod=30
Nov 02 13:32:32.418678 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:32.418619    4559 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-k5zn8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 13:32:32.419074 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:32.418706    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-k5zn8" podUID="d3b2832d-2aad-406f-86b5-de405575f9b2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 13:32:35.284689 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:35.283024    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-k5zn8"
Nov 02 13:32:35.312612 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:35.312581    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-xj4xk"
Nov 02 13:32:35.353279 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:35.353251    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ckslt"
Nov 02 13:32:35.363261 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:35.363231    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7kvjr"
Nov 02 13:32:35.370213 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:35.370184    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2c5f43a-3a0c-444b-afc1-e2bf090dc05c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2c5f43a-3a0c-444b-afc1-e2bf090dc05c-istio-data\") pod \"d2c5f43a-3a0c-444b-afc1-e2bf090dc05c\" (UID: \"d2c5f43a-3a0c-444b-afc1-e2bf090dc05c\") "
Nov 02 13:32:35.370394 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:35.370257    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b736e009-d002-4177-8f79-8f50cd5325f3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b736e009-d002-4177-8f79-8f50cd5325f3-workload-socket\") pod \"b736e009-d002-4177-8f79-8f50cd5325f3\" (UID: \"b736e009-d002-4177-8f79-8f50cd5325f3\") "
Nov 02 13:32:35.370394 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:35.370300    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3b2832d-2aad-406f-86b5-de405575f9b2-kube-api-access-tzzhd\" (UniqueName: \"kubernetes.io/projected/d3b2832d-2aad-406f-86b5-de405575f9b2-kube-api-access-tzzhd\") pod \"d3b2832d-2aad-406f-86b5-de405575f9b2\" (UID: \"d3b2832d-2aad-406f-86b5-de405575f9b2\") "
Nov 02 13:32:35.370394 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:35.370350    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b736e009-d002-4177-8f79-8f50cd5325f3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b736e009-d002-4177-8f79-8f50cd5325f3-istio-envoy\") pod \"b736e009-d002-4177-8f79-8f50cd5325f3\" (UID: \"b736e009-d002-4177-8f79-8f50cd5325f3\") "
Nov 02 13:32:35.370506 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:35.370390    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b736e009-d002-4177-8f79-8f50cd5325f3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b736e009-d002-4177-8f79-8f50cd5325f3-istio-data\") pod \"b736e009-d002-4177-8f79-8f50cd5325f3\" (UID: \"b736e009-d002-4177-8f79-8f50cd5325f3\") "
Nov 02 13:32:35.370506 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:35.370424    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/826b92f5-161c-4f6a-9055-d0c838c9ce09-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/826b92f5-161c-4f6a-9055-d0c838c9ce09-workload-socket\") pod \"826b92f5-161c-4f6a-9055-d0c838c9ce09\" (UID: \"826b92f5-161c-4f6a-9055-d0c838c9ce09\") "
Nov 02 13:32:35.370506 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:35.370464    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b736e009-d002-4177-8f79-8f50cd5325f3-kube-api-access-v46qb\" (UniqueName: \"kubernetes.io/projected/b736e009-d002-4177-8f79-8f50cd5325f3-kube-api-access-v46qb\") pod \"b736e009-d002-4177-8f79-8f50cd5325f3\" (UID: \"b736e009-d002-4177-8f79-8f50cd5325f3\") "
Nov 02 13:32:35.370506 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:35.370479    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2c5f43a-3a0c-444b-afc1-e2bf090dc05c-istio-data" pod "d2c5f43a-3a0c-444b-afc1-e2bf090dc05c" (UID: "d2c5f43a-3a0c-444b-afc1-e2bf090dc05c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:32:35.370651 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:35.370499    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/826b92f5-161c-4f6a-9055-d0c838c9ce09-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/826b92f5-161c-4f6a-9055-d0c838c9ce09-workload-certs\") pod \"826b92f5-161c-4f6a-9055-d0c838c9ce09\" (UID: \"826b92f5-161c-4f6a-9055-d0c838c9ce09\") "
Nov 02 13:32:35.370651 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:35.370565    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b736e009-d002-4177-8f79-8f50cd5325f3-workload-socket" pod "b736e009-d002-4177-8f79-8f50cd5325f3" (UID: "b736e009-d002-4177-8f79-8f50cd5325f3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:32:35.370651 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:35.370593    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/826b92f5-161c-4f6a-9055-d0c838c9ce09-istio-token\" (UniqueName: \"kubernetes.io/projected/826b92f5-161c-4f6a-9055-d0c838c9ce09-istio-token\") pod \"826b92f5-161c-4f6a-9055-d0c838c9ce09\" (UID: \"826b92f5-161c-4f6a-9055-d0c838c9ce09\") "
Nov 02 13:32:35.370768 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:35.370634    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/826b92f5-161c-4f6a-9055-d0c838c9ce09-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/826b92f5-161c-4f6a-9055-d0c838c9ce09-credential-socket\") pod \"826b92f5-161c-4f6a-9055-d0c838c9ce09\" (UID: \"826b92f5-161c-4f6a-9055-d0c838c9ce09\") "
Nov 02 13:32:35.370768 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:35.370695    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/826b92f5-161c-4f6a-9055-d0c838c9ce09-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/826b92f5-161c-4f6a-9055-d0c838c9ce09-istio-podinfo\") pod \"826b92f5-161c-4f6a-9055-d0c838c9ce09\" (UID: \"826b92f5-161c-4f6a-9055-d0c838c9ce09\") "
Nov 02 13:32:35.370768 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:35.370750    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/826b92f5-161c-4f6a-9055-d0c838c9ce09-workload-certs" pod "826b92f5-161c-4f6a-9055-d0c838c9ce09" (UID: "826b92f5-161c-4f6a-9055-d0c838c9ce09"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:32:35.370893 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:35.370764    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d2c5f43a-3a0c-444b-afc1-e2bf090dc05c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2c5f43a-3a0c-444b-afc1-e2bf090dc05c-istio-podinfo\") pod \"d2c5f43a-3a0c-444b-afc1-e2bf090dc05c\" (UID: \"d2c5f43a-3a0c-444b-afc1-e2bf090dc05c\") "
Nov 02 13:32:35.370893 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:35.370795    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3b2832d-2aad-406f-86b5-de405575f9b2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3b2832d-2aad-406f-86b5-de405575f9b2-istiod-ca-cert\") pod \"d3b2832d-2aad-406f-86b5-de405575f9b2\" (UID: \"d3b2832d-2aad-406f-86b5-de405575f9b2\") "
Nov 02 13:32:35.370893 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:35.370826    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3b2832d-2aad-406f-86b5-de405575f9b2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3b2832d-2aad-406f-86b5-de405575f9b2-istio-data\") pod \"d3b2832d-2aad-406f-86b5-de405575f9b2\" (UID: \"d3b2832d-2aad-406f-86b5-de405575f9b2\") "
Nov 02 13:32:35.370893 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:35.370839    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b736e009-d002-4177-8f79-8f50cd5325f3-istio-data" pod "b736e009-d002-4177-8f79-8f50cd5325f3" (UID: "b736e009-d002-4177-8f79-8f50cd5325f3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:32:35.370893 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:35.370859    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2c5f43a-3a0c-444b-afc1-e2bf090dc05c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2c5f43a-3a0c-444b-afc1-e2bf090dc05c-workload-certs\") pod \"d2c5f43a-3a0c-444b-afc1-e2bf090dc05c\" (UID: \"d2c5f43a-3a0c-444b-afc1-e2bf090dc05c\") "
Nov 02 13:32:35.371124 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:35.370906    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/826b92f5-161c-4f6a-9055-d0c838c9ce09-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/826b92f5-161c-4f6a-9055-d0c838c9ce09-istiod-ca-cert\") pod \"826b92f5-161c-4f6a-9055-d0c838c9ce09\" (UID: \"826b92f5-161c-4f6a-9055-d0c838c9ce09\") "
Nov 02 13:32:35.371124 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:35.370937    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b736e009-d002-4177-8f79-8f50cd5325f3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b736e009-d002-4177-8f79-8f50cd5325f3-credential-socket\") pod \"b736e009-d002-4177-8f79-8f50cd5325f3\" (UID: \"b736e009-d002-4177-8f79-8f50cd5325f3\") "
Nov 02 13:32:35.371124 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:35.370972    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/826b92f5-161c-4f6a-9055-d0c838c9ce09-workload-socket" pod "826b92f5-161c-4f6a-9055-d0c838c9ce09" (UID: "826b92f5-161c-4f6a-9055-d0c838c9ce09"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:32:35.371124 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:35.370984    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2c5f43a-3a0c-444b-afc1-e2bf090dc05c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2c5f43a-3a0c-444b-afc1-e2bf090dc05c-istio-envoy\") pod \"d2c5f43a-3a0c-444b-afc1-e2bf090dc05c\" (UID: \"d2c5f43a-3a0c-444b-afc1-e2bf090dc05c\") "
Nov 02 13:32:35.380445 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:35.371594    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2c5f43a-3a0c-444b-afc1-e2bf090dc05c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2c5f43a-3a0c-444b-afc1-e2bf090dc05c-workload-socket\") pod \"d2c5f43a-3a0c-444b-afc1-e2bf090dc05c\" (UID: \"d2c5f43a-3a0c-444b-afc1-e2bf090dc05c\") "
Nov 02 13:32:35.380445 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:35.371681    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/826b92f5-161c-4f6a-9055-d0c838c9ce09-kube-api-access-dq79r\" (UniqueName: \"kubernetes.io/projected/826b92f5-161c-4f6a-9055-d0c838c9ce09-kube-api-access-dq79r\") pod \"826b92f5-161c-4f6a-9055-d0c838c9ce09\" (UID: \"826b92f5-161c-4f6a-9055-d0c838c9ce09\") "
Nov 02 13:32:35.380445 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:35.371718    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3b2832d-2aad-406f-86b5-de405575f9b2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3b2832d-2aad-406f-86b5-de405575f9b2-workload-socket\") pod \"d3b2832d-2aad-406f-86b5-de405575f9b2\" (UID: \"d3b2832d-2aad-406f-86b5-de405575f9b2\") "
Nov 02 13:32:35.380445 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:35.371747    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/826b92f5-161c-4f6a-9055-d0c838c9ce09-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/826b92f5-161c-4f6a-9055-d0c838c9ce09-istio-data\") pod \"826b92f5-161c-4f6a-9055-d0c838c9ce09\" (UID: \"826b92f5-161c-4f6a-9055-d0c838c9ce09\") "
Nov 02 13:32:35.380445 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:35.371788    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b736e009-d002-4177-8f79-8f50cd5325f3-istio-token\" (UniqueName: \"kubernetes.io/projected/b736e009-d002-4177-8f79-8f50cd5325f3-istio-token\") pod \"b736e009-d002-4177-8f79-8f50cd5325f3\" (UID: \"b736e009-d002-4177-8f79-8f50cd5325f3\") "
Nov 02 13:32:35.380445 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:35.371828    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2c5f43a-3a0c-444b-afc1-e2bf090dc05c-kube-api-access-znb9l\" (UniqueName: \"kubernetes.io/projected/d2c5f43a-3a0c-444b-afc1-e2bf090dc05c-kube-api-access-znb9l\") pod \"d2c5f43a-3a0c-444b-afc1-e2bf090dc05c\" (UID: \"d2c5f43a-3a0c-444b-afc1-e2bf090dc05c\") "
Nov 02 13:32:35.380445 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:35.371873    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/826b92f5-161c-4f6a-9055-d0c838c9ce09-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/826b92f5-161c-4f6a-9055-d0c838c9ce09-istio-envoy\") pod \"826b92f5-161c-4f6a-9055-d0c838c9ce09\" (UID: \"826b92f5-161c-4f6a-9055-d0c838c9ce09\") "
Nov 02 13:32:35.380445 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:35.371904    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3b2832d-2aad-406f-86b5-de405575f9b2-istio-token\" (UniqueName: \"kubernetes.io/projected/d3b2832d-2aad-406f-86b5-de405575f9b2-istio-token\") pod \"d3b2832d-2aad-406f-86b5-de405575f9b2\" (UID: \"d3b2832d-2aad-406f-86b5-de405575f9b2\") "
Nov 02 13:32:35.380445 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:35.371940    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b736e009-d002-4177-8f79-8f50cd5325f3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b736e009-d002-4177-8f79-8f50cd5325f3-istio-podinfo\") pod \"b736e009-d002-4177-8f79-8f50cd5325f3\" (UID: \"b736e009-d002-4177-8f79-8f50cd5325f3\") "
Nov 02 13:32:35.380445 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:35.371972    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2c5f43a-3a0c-444b-afc1-e2bf090dc05c-workload-certs" pod "d2c5f43a-3a0c-444b-afc1-e2bf090dc05c" (UID: "d2c5f43a-3a0c-444b-afc1-e2bf090dc05c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:32:35.380445 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:35.371992    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2c5f43a-3a0c-444b-afc1-e2bf090dc05c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2c5f43a-3a0c-444b-afc1-e2bf090dc05c-istiod-ca-cert\") pod \"d2c5f43a-3a0c-444b-afc1-e2bf090dc05c\" (UID: \"d2c5f43a-3a0c-444b-afc1-e2bf090dc05c\") "
Nov 02 13:32:35.380445 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:35.372021    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3b2832d-2aad-406f-86b5-de405575f9b2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3b2832d-2aad-406f-86b5-de405575f9b2-credential-socket\") pod \"d3b2832d-2aad-406f-86b5-de405575f9b2\" (UID: \"d3b2832d-2aad-406f-86b5-de405575f9b2\") "
Nov 02 13:32:35.380445 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:35.372050    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d3b2832d-2aad-406f-86b5-de405575f9b2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3b2832d-2aad-406f-86b5-de405575f9b2-istio-podinfo\") pod \"d3b2832d-2aad-406f-86b5-de405575f9b2\" (UID: \"d3b2832d-2aad-406f-86b5-de405575f9b2\") "
Nov 02 13:32:35.380445 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:35.372084    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2c5f43a-3a0c-444b-afc1-e2bf090dc05c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2c5f43a-3a0c-444b-afc1-e2bf090dc05c-credential-socket\") pod \"d2c5f43a-3a0c-444b-afc1-e2bf090dc05c\" (UID: \"d2c5f43a-3a0c-444b-afc1-e2bf090dc05c\") "
Nov 02 13:32:35.380445 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:35.372143    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2c5f43a-3a0c-444b-afc1-e2bf090dc05c-istio-token\" (UniqueName: \"kubernetes.io/projected/d2c5f43a-3a0c-444b-afc1-e2bf090dc05c-istio-token\") pod \"d2c5f43a-3a0c-444b-afc1-e2bf090dc05c\" (UID: \"d2c5f43a-3a0c-444b-afc1-e2bf090dc05c\") "
Nov 02 13:32:35.380445 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:35.372173    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3b2832d-2aad-406f-86b5-de405575f9b2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3b2832d-2aad-406f-86b5-de405575f9b2-workload-certs\") pod \"d3b2832d-2aad-406f-86b5-de405575f9b2\" (UID: \"d3b2832d-2aad-406f-86b5-de405575f9b2\") "
Nov 02 13:32:35.380445 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:35.372177    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3b2832d-2aad-406f-86b5-de405575f9b2-istio-data" pod "d3b2832d-2aad-406f-86b5-de405575f9b2" (UID: "d3b2832d-2aad-406f-86b5-de405575f9b2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:32:35.380445 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:35.372201    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b736e009-d002-4177-8f79-8f50cd5325f3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b736e009-d002-4177-8f79-8f50cd5325f3-istiod-ca-cert\") pod \"b736e009-d002-4177-8f79-8f50cd5325f3\" (UID: \"b736e009-d002-4177-8f79-8f50cd5325f3\") "
Nov 02 13:32:35.380445 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:35.372222    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/826b92f5-161c-4f6a-9055-d0c838c9ce09-credential-socket" pod "826b92f5-161c-4f6a-9055-d0c838c9ce09" (UID: "826b92f5-161c-4f6a-9055-d0c838c9ce09"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:32:35.380445 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:35.376417    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/826b92f5-161c-4f6a-9055-d0c838c9ce09-istiod-ca-cert" pod "826b92f5-161c-4f6a-9055-d0c838c9ce09" (UID: "826b92f5-161c-4f6a-9055-d0c838c9ce09"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:32:35.380445 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:35.377786    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3b2832d-2aad-406f-86b5-de405575f9b2-istiod-ca-cert" pod "d3b2832d-2aad-406f-86b5-de405575f9b2" (UID: "d3b2832d-2aad-406f-86b5-de405575f9b2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:32:35.380445 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:35.378957    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b736e009-d002-4177-8f79-8f50cd5325f3-istiod-ca-cert" pod "b736e009-d002-4177-8f79-8f50cd5325f3" (UID: "b736e009-d002-4177-8f79-8f50cd5325f3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:32:35.380445 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:35.379243    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b736e009-d002-4177-8f79-8f50cd5325f3-kube-api-access-v46qb" pod "b736e009-d002-4177-8f79-8f50cd5325f3" (UID: "b736e009-d002-4177-8f79-8f50cd5325f3"). InnerVolumeSpecName "kube-api-access-v46qb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:32:35.380445 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:35.379328    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/826b92f5-161c-4f6a-9055-d0c838c9ce09-istio-token" pod "826b92f5-161c-4f6a-9055-d0c838c9ce09" (UID: "826b92f5-161c-4f6a-9055-d0c838c9ce09"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:32:35.380445 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:35.379376    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b736e009-d002-4177-8f79-8f50cd5325f3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b736e009-d002-4177-8f79-8f50cd5325f3-workload-certs\") pod \"b736e009-d002-4177-8f79-8f50cd5325f3\" (UID: \"b736e009-d002-4177-8f79-8f50cd5325f3\") "
Nov 02 13:32:35.380445 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:35.379447    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3b2832d-2aad-406f-86b5-de405575f9b2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3b2832d-2aad-406f-86b5-de405575f9b2-istio-envoy\") pod \"d3b2832d-2aad-406f-86b5-de405575f9b2\" (UID: \"d3b2832d-2aad-406f-86b5-de405575f9b2\") "
Nov 02 13:32:35.380445 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:35.379534    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3b2832d-2aad-406f-86b5-de405575f9b2-kube-api-access-tzzhd" pod "d3b2832d-2aad-406f-86b5-de405575f9b2" (UID: "d3b2832d-2aad-406f-86b5-de405575f9b2"). InnerVolumeSpecName "kube-api-access-tzzhd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:32:35.380445 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:35.379655    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2c5f43a-3a0c-444b-afc1-e2bf090dc05c-credential-socket" pod "d2c5f43a-3a0c-444b-afc1-e2bf090dc05c" (UID: "d2c5f43a-3a0c-444b-afc1-e2bf090dc05c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:32:35.380445 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:35.379956    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2c5f43a-3a0c-444b-afc1-e2bf090dc05c-istiod-ca-cert" pod "d2c5f43a-3a0c-444b-afc1-e2bf090dc05c" (UID: "d2c5f43a-3a0c-444b-afc1-e2bf090dc05c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:32:35.380445 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:35.380192    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3b2832d-2aad-406f-86b5-de405575f9b2-credential-socket" pod "d3b2832d-2aad-406f-86b5-de405575f9b2" (UID: "d3b2832d-2aad-406f-86b5-de405575f9b2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:32:35.380445 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:35.380206    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b736e009-d002-4177-8f79-8f50cd5325f3-credential-socket" pod "b736e009-d002-4177-8f79-8f50cd5325f3" (UID: "b736e009-d002-4177-8f79-8f50cd5325f3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:32:35.380445 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:35.380273    4559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b736e009-d002-4177-8f79-8f50cd5325f3-istiod-ca-cert\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:32:35.380445 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:35.380295    4559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2c5f43a-3a0c-444b-afc1-e2bf090dc05c-istio-data\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:32:35.380445 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:35.380309    4559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b736e009-d002-4177-8f79-8f50cd5325f3-workload-socket\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:32:35.380445 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:35.380325    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tzzhd\" (UniqueName: \"kubernetes.io/projected/d3b2832d-2aad-406f-86b5-de405575f9b2-kube-api-access-tzzhd\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:32:35.380445 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:35.380339    4559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b736e009-d002-4177-8f79-8f50cd5325f3-istio-data\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:32:35.380445 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:35.380353    4559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/826b92f5-161c-4f6a-9055-d0c838c9ce09-workload-socket\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:32:35.380445 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:35.380366    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v46qb\" (UniqueName: \"kubernetes.io/projected/b736e009-d002-4177-8f79-8f50cd5325f3-kube-api-access-v46qb\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:32:35.380445 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:35.380382    4559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/826b92f5-161c-4f6a-9055-d0c838c9ce09-workload-certs\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:32:35.380445 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:35.380395    4559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/826b92f5-161c-4f6a-9055-d0c838c9ce09-istio-token\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:32:35.380445 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:35.380399    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3b2832d-2aad-406f-86b5-de405575f9b2-workload-certs" pod "d3b2832d-2aad-406f-86b5-de405575f9b2" (UID: "d3b2832d-2aad-406f-86b5-de405575f9b2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:32:35.380445 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:35.380408    4559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/826b92f5-161c-4f6a-9055-d0c838c9ce09-credential-socket\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:32:35.380445 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:35.380421    4559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3b2832d-2aad-406f-86b5-de405575f9b2-istiod-ca-cert\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:32:35.380445 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:35.380434    4559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3b2832d-2aad-406f-86b5-de405575f9b2-istio-data\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:32:35.380445 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:35.380447    4559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2c5f43a-3a0c-444b-afc1-e2bf090dc05c-workload-certs\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:32:35.380445 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:35.380460    4559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/826b92f5-161c-4f6a-9055-d0c838c9ce09-istiod-ca-cert\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:32:35.380445 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:35.380475    4559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2c5f43a-3a0c-444b-afc1-e2bf090dc05c-istiod-ca-cert\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:32:35.382952 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:35.380490    4559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2c5f43a-3a0c-444b-afc1-e2bf090dc05c-credential-socket\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:32:35.382952 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:35.380490    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d2c5f43a-3a0c-444b-afc1-e2bf090dc05c-istio-podinfo" pod "d2c5f43a-3a0c-444b-afc1-e2bf090dc05c" (UID: "d2c5f43a-3a0c-444b-afc1-e2bf090dc05c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 13:32:35.382952 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:35.380711    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/826b92f5-161c-4f6a-9055-d0c838c9ce09-istio-data" pod "826b92f5-161c-4f6a-9055-d0c838c9ce09" (UID: "826b92f5-161c-4f6a-9055-d0c838c9ce09"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:32:35.382952 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:35.380735    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b736e009-d002-4177-8f79-8f50cd5325f3-workload-certs" pod "b736e009-d002-4177-8f79-8f50cd5325f3" (UID: "b736e009-d002-4177-8f79-8f50cd5325f3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:32:35.382952 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:35.380946    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2c5f43a-3a0c-444b-afc1-e2bf090dc05c-workload-socket" pod "d2c5f43a-3a0c-444b-afc1-e2bf090dc05c" (UID: "d2c5f43a-3a0c-444b-afc1-e2bf090dc05c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:32:35.382952 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:35.381143    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3b2832d-2aad-406f-86b5-de405575f9b2-workload-socket" pod "d3b2832d-2aad-406f-86b5-de405575f9b2" (UID: "d3b2832d-2aad-406f-86b5-de405575f9b2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:32:35.382952 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:35.381602    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2c5f43a-3a0c-444b-afc1-e2bf090dc05c-istio-envoy" pod "d2c5f43a-3a0c-444b-afc1-e2bf090dc05c" (UID: "d2c5f43a-3a0c-444b-afc1-e2bf090dc05c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:32:35.383337 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:35.383241    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/826b92f5-161c-4f6a-9055-d0c838c9ce09-istio-podinfo" pod "826b92f5-161c-4f6a-9055-d0c838c9ce09" (UID: "826b92f5-161c-4f6a-9055-d0c838c9ce09"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 13:32:35.385978 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:35.385913    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3b2832d-2aad-406f-86b5-de405575f9b2-istio-token" pod "d3b2832d-2aad-406f-86b5-de405575f9b2" (UID: "d3b2832d-2aad-406f-86b5-de405575f9b2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:32:35.385978 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:35.385915    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d3b2832d-2aad-406f-86b5-de405575f9b2-istio-podinfo" pod "d3b2832d-2aad-406f-86b5-de405575f9b2" (UID: "d3b2832d-2aad-406f-86b5-de405575f9b2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 13:32:35.385978 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:35.385941    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2c5f43a-3a0c-444b-afc1-e2bf090dc05c-kube-api-access-znb9l" pod "d2c5f43a-3a0c-444b-afc1-e2bf090dc05c" (UID: "d2c5f43a-3a0c-444b-afc1-e2bf090dc05c"). InnerVolumeSpecName "kube-api-access-znb9l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:32:35.385978 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:35.385946    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b736e009-d002-4177-8f79-8f50cd5325f3-istio-podinfo" pod "b736e009-d002-4177-8f79-8f50cd5325f3" (UID: "b736e009-d002-4177-8f79-8f50cd5325f3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 13:32:35.385978 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:35.385964    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b736e009-d002-4177-8f79-8f50cd5325f3-istio-token" pod "b736e009-d002-4177-8f79-8f50cd5325f3" (UID: "b736e009-d002-4177-8f79-8f50cd5325f3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:32:35.385978 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:35.385935    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/826b92f5-161c-4f6a-9055-d0c838c9ce09-kube-api-access-dq79r" pod "826b92f5-161c-4f6a-9055-d0c838c9ce09" (UID: "826b92f5-161c-4f6a-9055-d0c838c9ce09"). InnerVolumeSpecName "kube-api-access-dq79r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:32:35.386262 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:35.386058    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b736e009-d002-4177-8f79-8f50cd5325f3-istio-envoy" pod "b736e009-d002-4177-8f79-8f50cd5325f3" (UID: "b736e009-d002-4177-8f79-8f50cd5325f3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:32:35.386262 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:35.386075    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2c5f43a-3a0c-444b-afc1-e2bf090dc05c-istio-token" pod "d2c5f43a-3a0c-444b-afc1-e2bf090dc05c" (UID: "d2c5f43a-3a0c-444b-afc1-e2bf090dc05c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:32:35.386362 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:35.386327    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3b2832d-2aad-406f-86b5-de405575f9b2-istio-envoy" pod "d3b2832d-2aad-406f-86b5-de405575f9b2" (UID: "d3b2832d-2aad-406f-86b5-de405575f9b2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:32:35.387122 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:35.387094    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/826b92f5-161c-4f6a-9055-d0c838c9ce09-istio-envoy" pod "826b92f5-161c-4f6a-9055-d0c838c9ce09" (UID: "826b92f5-161c-4f6a-9055-d0c838c9ce09"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:32:35.481613 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:35.481579    4559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b736e009-d002-4177-8f79-8f50cd5325f3-istio-envoy\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:32:35.481613 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:35.481607    4559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/826b92f5-161c-4f6a-9055-d0c838c9ce09-istio-podinfo\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:32:35.481613 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:35.481619    4559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2c5f43a-3a0c-444b-afc1-e2bf090dc05c-istio-podinfo\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:32:35.481854 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:35.481628    4559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b736e009-d002-4177-8f79-8f50cd5325f3-credential-socket\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:32:35.481854 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:35.481662    4559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2c5f43a-3a0c-444b-afc1-e2bf090dc05c-istio-envoy\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:32:35.481854 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:35.481676    4559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2c5f43a-3a0c-444b-afc1-e2bf090dc05c-workload-socket\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:32:35.481854 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:35.481685    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dq79r\" (UniqueName: \"kubernetes.io/projected/826b92f5-161c-4f6a-9055-d0c838c9ce09-kube-api-access-dq79r\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:32:35.481854 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:35.481693    4559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3b2832d-2aad-406f-86b5-de405575f9b2-workload-socket\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:32:35.481854 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:35.481703    4559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/826b92f5-161c-4f6a-9055-d0c838c9ce09-istio-data\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:32:35.481854 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:35.481713    4559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b736e009-d002-4177-8f79-8f50cd5325f3-istio-token\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:32:35.481854 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:35.481721    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-znb9l\" (UniqueName: \"kubernetes.io/projected/d2c5f43a-3a0c-444b-afc1-e2bf090dc05c-kube-api-access-znb9l\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:32:35.481854 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:35.481729    4559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/826b92f5-161c-4f6a-9055-d0c838c9ce09-istio-envoy\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:32:35.481854 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:35.481741    4559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3b2832d-2aad-406f-86b5-de405575f9b2-istio-token\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:32:35.481854 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:35.481751    4559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b736e009-d002-4177-8f79-8f50cd5325f3-istio-podinfo\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:32:35.481854 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:35.481758    4559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3b2832d-2aad-406f-86b5-de405575f9b2-credential-socket\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:32:35.481854 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:35.481767    4559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3b2832d-2aad-406f-86b5-de405575f9b2-istio-podinfo\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:32:35.481854 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:35.481775    4559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2c5f43a-3a0c-444b-afc1-e2bf090dc05c-istio-token\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:32:35.481854 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:35.481784    4559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3b2832d-2aad-406f-86b5-de405575f9b2-workload-certs\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:32:35.481854 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:35.481792    4559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b736e009-d002-4177-8f79-8f50cd5325f3-workload-certs\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:32:35.481854 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:35.481799    4559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3b2832d-2aad-406f-86b5-de405575f9b2-istio-envoy\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:32:35.804888 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:35.804851    4559 generic.go:356] "Generic (PLEG): container finished" podID="d2c5f43a-3a0c-444b-afc1-e2bf090dc05c" containerID="96ed21c4883f0d9a981d09f6cefccbd271ccb36cf3d4a1146353b01d1680067a" exitCode=0
Nov 02 13:32:35.805072 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:35.804899    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-xj4xk"
Nov 02 13:32:35.805072 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:35.804935    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-xj4xk" event={"ID":"d2c5f43a-3a0c-444b-afc1-e2bf090dc05c","Type":"ContainerDied","Data":"96ed21c4883f0d9a981d09f6cefccbd271ccb36cf3d4a1146353b01d1680067a"}
Nov 02 13:32:35.805072 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:35.804996    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-xj4xk" event={"ID":"d2c5f43a-3a0c-444b-afc1-e2bf090dc05c","Type":"ContainerDied","Data":"534dda2fb3c1a0513a827ead60da78c811c71e9ebb60f1ede628359096d82bb2"}
Nov 02 13:32:35.805072 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:35.805018    4559 scope.go:122] "RemoveContainer" containerID="96ed21c4883f0d9a981d09f6cefccbd271ccb36cf3d4a1146353b01d1680067a"
Nov 02 13:32:35.806421 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:35.806399    4559 generic.go:356] "Generic (PLEG): container finished" podID="d3b2832d-2aad-406f-86b5-de405575f9b2" containerID="e11beb411d5d8cb9ad1ee54ee6367d6245a564a1de071e3b6a623192e83fa165" exitCode=0
Nov 02 13:32:35.806564 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:35.806455    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-k5zn8"
Nov 02 13:32:35.806564 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:35.806497    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-k5zn8" event={"ID":"d3b2832d-2aad-406f-86b5-de405575f9b2","Type":"ContainerDied","Data":"e11beb411d5d8cb9ad1ee54ee6367d6245a564a1de071e3b6a623192e83fa165"}
Nov 02 13:32:35.806564 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:35.806541    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-k5zn8" event={"ID":"d3b2832d-2aad-406f-86b5-de405575f9b2","Type":"ContainerDied","Data":"0ac7d05e8eec691f2f96cd98958f8d22940289c439831f0aa5eed905fd61c8e7"}
Nov 02 13:32:35.808015 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:35.807991    4559 generic.go:356] "Generic (PLEG): container finished" podID="826b92f5-161c-4f6a-9055-d0c838c9ce09" containerID="bd06e4fbfb52ae78f4d1d88b88740481e7fb5e29e35825a9512b1bec633649b4" exitCode=0
Nov 02 13:32:35.808148 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:35.808027    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ckslt" event={"ID":"826b92f5-161c-4f6a-9055-d0c838c9ce09","Type":"ContainerDied","Data":"bd06e4fbfb52ae78f4d1d88b88740481e7fb5e29e35825a9512b1bec633649b4"}
Nov 02 13:32:35.808148 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:35.808043    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ckslt"
Nov 02 13:32:35.808148 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:35.808076    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ckslt" event={"ID":"826b92f5-161c-4f6a-9055-d0c838c9ce09","Type":"ContainerDied","Data":"63d729291d2408508b974a1dfb81e28eb6361ef4cf776ba39a410e68f9d30fd3"}
Nov 02 13:32:35.809426 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:35.809406    4559 generic.go:356] "Generic (PLEG): container finished" podID="b736e009-d002-4177-8f79-8f50cd5325f3" containerID="cd4b240992256d01908e7156df44d1661f5a7926377344fdd68067abb3c3f87e" exitCode=0
Nov 02 13:32:35.809541 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:35.809444    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7kvjr"
Nov 02 13:32:35.809541 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:35.809463    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7kvjr" event={"ID":"b736e009-d002-4177-8f79-8f50cd5325f3","Type":"ContainerDied","Data":"cd4b240992256d01908e7156df44d1661f5a7926377344fdd68067abb3c3f87e"}
Nov 02 13:32:35.809541 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:35.809502    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7kvjr" event={"ID":"b736e009-d002-4177-8f79-8f50cd5325f3","Type":"ContainerDied","Data":"220f5d89aed037e4349a2e1d847cd44191f43aa7e990587b689babf07244c8fa"}
Nov 02 13:32:35.821569 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:35.821546    4559 scope.go:122] "RemoveContainer" containerID="96ed21c4883f0d9a981d09f6cefccbd271ccb36cf3d4a1146353b01d1680067a"
Nov 02 13:32:35.824673 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:32:35.821967    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96ed21c4883f0d9a981d09f6cefccbd271ccb36cf3d4a1146353b01d1680067a\": container with ID starting with 96ed21c4883f0d9a981d09f6cefccbd271ccb36cf3d4a1146353b01d1680067a not found: ID does not exist" containerID="96ed21c4883f0d9a981d09f6cefccbd271ccb36cf3d4a1146353b01d1680067a"
Nov 02 13:32:35.824673 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:35.822015    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96ed21c4883f0d9a981d09f6cefccbd271ccb36cf3d4a1146353b01d1680067a"} err="failed to get container status \"96ed21c4883f0d9a981d09f6cefccbd271ccb36cf3d4a1146353b01d1680067a\": rpc error: code = NotFound desc = could not find container \"96ed21c4883f0d9a981d09f6cefccbd271ccb36cf3d4a1146353b01d1680067a\": container with ID starting with 96ed21c4883f0d9a981d09f6cefccbd271ccb36cf3d4a1146353b01d1680067a not found: ID does not exist"
Nov 02 13:32:35.824673 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:35.822039    4559 scope.go:122] "RemoveContainer" containerID="e11beb411d5d8cb9ad1ee54ee6367d6245a564a1de071e3b6a623192e83fa165"
Nov 02 13:32:35.836848 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:35.836825    4559 scope.go:122] "RemoveContainer" containerID="e11beb411d5d8cb9ad1ee54ee6367d6245a564a1de071e3b6a623192e83fa165"
Nov 02 13:32:35.837106 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:32:35.837084    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e11beb411d5d8cb9ad1ee54ee6367d6245a564a1de071e3b6a623192e83fa165\": container with ID starting with e11beb411d5d8cb9ad1ee54ee6367d6245a564a1de071e3b6a623192e83fa165 not found: ID does not exist" containerID="e11beb411d5d8cb9ad1ee54ee6367d6245a564a1de071e3b6a623192e83fa165"
Nov 02 13:32:35.837178 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:35.837119    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e11beb411d5d8cb9ad1ee54ee6367d6245a564a1de071e3b6a623192e83fa165"} err="failed to get container status \"e11beb411d5d8cb9ad1ee54ee6367d6245a564a1de071e3b6a623192e83fa165\": rpc error: code = NotFound desc = could not find container \"e11beb411d5d8cb9ad1ee54ee6367d6245a564a1de071e3b6a623192e83fa165\": container with ID starting with e11beb411d5d8cb9ad1ee54ee6367d6245a564a1de071e3b6a623192e83fa165 not found: ID does not exist"
Nov 02 13:32:35.837178 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:35.837142    4559 scope.go:122] "RemoveContainer" containerID="bd06e4fbfb52ae78f4d1d88b88740481e7fb5e29e35825a9512b1bec633649b4"
Nov 02 13:32:35.846319 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:35.846295    4559 scope.go:122] "RemoveContainer" containerID="bd06e4fbfb52ae78f4d1d88b88740481e7fb5e29e35825a9512b1bec633649b4"
Nov 02 13:32:35.846679 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:32:35.846629    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd06e4fbfb52ae78f4d1d88b88740481e7fb5e29e35825a9512b1bec633649b4\": container with ID starting with bd06e4fbfb52ae78f4d1d88b88740481e7fb5e29e35825a9512b1bec633649b4 not found: ID does not exist" containerID="bd06e4fbfb52ae78f4d1d88b88740481e7fb5e29e35825a9512b1bec633649b4"
Nov 02 13:32:35.846761 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:35.846688    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd06e4fbfb52ae78f4d1d88b88740481e7fb5e29e35825a9512b1bec633649b4"} err="failed to get container status \"bd06e4fbfb52ae78f4d1d88b88740481e7fb5e29e35825a9512b1bec633649b4\": rpc error: code = NotFound desc = could not find container \"bd06e4fbfb52ae78f4d1d88b88740481e7fb5e29e35825a9512b1bec633649b4\": container with ID starting with bd06e4fbfb52ae78f4d1d88b88740481e7fb5e29e35825a9512b1bec633649b4 not found: ID does not exist"
Nov 02 13:32:35.846761 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:35.846705    4559 scope.go:122] "RemoveContainer" containerID="cd4b240992256d01908e7156df44d1661f5a7926377344fdd68067abb3c3f87e"
Nov 02 13:32:35.857274 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:35.857258    4559 scope.go:122] "RemoveContainer" containerID="cd4b240992256d01908e7156df44d1661f5a7926377344fdd68067abb3c3f87e"
Nov 02 13:32:35.857553 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:32:35.857526    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd4b240992256d01908e7156df44d1661f5a7926377344fdd68067abb3c3f87e\": container with ID starting with cd4b240992256d01908e7156df44d1661f5a7926377344fdd68067abb3c3f87e not found: ID does not exist" containerID="cd4b240992256d01908e7156df44d1661f5a7926377344fdd68067abb3c3f87e"
Nov 02 13:32:35.857614 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:35.857562    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd4b240992256d01908e7156df44d1661f5a7926377344fdd68067abb3c3f87e"} err="failed to get container status \"cd4b240992256d01908e7156df44d1661f5a7926377344fdd68067abb3c3f87e\": rpc error: code = NotFound desc = could not find container \"cd4b240992256d01908e7156df44d1661f5a7926377344fdd68067abb3c3f87e\": container with ID starting with cd4b240992256d01908e7156df44d1661f5a7926377344fdd68067abb3c3f87e not found: ID does not exist"
Nov 02 13:32:35.878261 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:35.878233    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-xj4xk"]
Nov 02 13:32:35.896027 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:35.895999    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-xj4xk"]
Nov 02 13:32:35.950319 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:35.950292    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ckslt"]
Nov 02 13:32:35.954395 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:35.954374    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ckslt"]
Nov 02 13:32:35.991988 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:35.991966    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7kvjr"]
Nov 02 13:32:35.995500 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:35.995477    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7kvjr"]
Nov 02 13:32:36.033897 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:36.033873    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-k5zn8"]
Nov 02 13:32:36.047168 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:36.047147    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-k5zn8"]
Nov 02 13:32:37.893382 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:37.893344    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="826b92f5-161c-4f6a-9055-d0c838c9ce09" path="/var/lib/kubelet/pods/826b92f5-161c-4f6a-9055-d0c838c9ce09/volumes"
Nov 02 13:32:37.894046 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:37.894024    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b736e009-d002-4177-8f79-8f50cd5325f3" path="/var/lib/kubelet/pods/b736e009-d002-4177-8f79-8f50cd5325f3/volumes"
Nov 02 13:32:37.894511 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:37.894489    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2c5f43a-3a0c-444b-afc1-e2bf090dc05c" path="/var/lib/kubelet/pods/d2c5f43a-3a0c-444b-afc1-e2bf090dc05c/volumes"
Nov 02 13:32:37.895286 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:37.895265    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3b2832d-2aad-406f-86b5-de405575f9b2" path="/var/lib/kubelet/pods/d3b2832d-2aad-406f-86b5-de405575f9b2/volumes"
Nov 02 13:32:44.180820 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:44.180778    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-740f09-776cf8d88b-gtp87"]
Nov 02 13:32:44.181376 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:44.181001    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-740f09-776cf8d88b-gtp87" podUID="c482bbc5-fff9-42e7-a147-f9664750fc8a" containerName="maas-api" containerID="cri-o://6a4bb787b18fac6145882b8b6aad289a5dd481a8a1429ca200260f503be59a45" gracePeriod=30
Nov 02 13:32:44.457813 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:44.457792    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-740f09-776cf8d88b-gtp87"
Nov 02 13:32:44.582387 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:44.582348    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c482bbc5-fff9-42e7-a147-f9664750fc8a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c482bbc5-fff9-42e7-a147-f9664750fc8a-maas-api-tls\") pod \"c482bbc5-fff9-42e7-a147-f9664750fc8a\" (UID: \"c482bbc5-fff9-42e7-a147-f9664750fc8a\") "
Nov 02 13:32:44.582539 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:44.582478    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c482bbc5-fff9-42e7-a147-f9664750fc8a-kube-api-access-n5k8z\" (UniqueName: \"kubernetes.io/projected/c482bbc5-fff9-42e7-a147-f9664750fc8a-kube-api-access-n5k8z\") pod \"c482bbc5-fff9-42e7-a147-f9664750fc8a\" (UID: \"c482bbc5-fff9-42e7-a147-f9664750fc8a\") "
Nov 02 13:32:44.582539 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:44.582509    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c482bbc5-fff9-42e7-a147-f9664750fc8a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c482bbc5-fff9-42e7-a147-f9664750fc8a-metrics-certs\") pod \"c482bbc5-fff9-42e7-a147-f9664750fc8a\" (UID: \"c482bbc5-fff9-42e7-a147-f9664750fc8a\") "
Nov 02 13:32:44.582655 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:44.582579    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c482bbc5-fff9-42e7-a147-f9664750fc8a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c482bbc5-fff9-42e7-a147-f9664750fc8a-openshift-service-ca\") pod \"c482bbc5-fff9-42e7-a147-f9664750fc8a\" (UID: \"c482bbc5-fff9-42e7-a147-f9664750fc8a\") "
Nov 02 13:32:44.583020 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:44.582991    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c482bbc5-fff9-42e7-a147-f9664750fc8a-openshift-service-ca" pod "c482bbc5-fff9-42e7-a147-f9664750fc8a" (UID: "c482bbc5-fff9-42e7-a147-f9664750fc8a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:32:44.584487 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:44.584460    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c482bbc5-fff9-42e7-a147-f9664750fc8a-maas-api-tls" pod "c482bbc5-fff9-42e7-a147-f9664750fc8a" (UID: "c482bbc5-fff9-42e7-a147-f9664750fc8a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:32:44.584689 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:44.584664    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c482bbc5-fff9-42e7-a147-f9664750fc8a-kube-api-access-n5k8z" pod "c482bbc5-fff9-42e7-a147-f9664750fc8a" (UID: "c482bbc5-fff9-42e7-a147-f9664750fc8a"). InnerVolumeSpecName "kube-api-access-n5k8z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:32:44.584775 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:44.584709    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c482bbc5-fff9-42e7-a147-f9664750fc8a-metrics-certs" pod "c482bbc5-fff9-42e7-a147-f9664750fc8a" (UID: "c482bbc5-fff9-42e7-a147-f9664750fc8a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:32:44.684017 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:44.683986    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c482bbc5-fff9-42e7-a147-f9664750fc8a-maas-api-tls\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:32:44.684017 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:44.684016    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n5k8z\" (UniqueName: \"kubernetes.io/projected/c482bbc5-fff9-42e7-a147-f9664750fc8a-kube-api-access-n5k8z\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:32:44.684017 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:44.684025    4559 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c482bbc5-fff9-42e7-a147-f9664750fc8a-metrics-certs\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:32:44.684254 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:44.684035    4559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c482bbc5-fff9-42e7-a147-f9664750fc8a-openshift-service-ca\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:32:44.871023 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:44.870948    4559 generic.go:356] "Generic (PLEG): container finished" podID="c482bbc5-fff9-42e7-a147-f9664750fc8a" containerID="6a4bb787b18fac6145882b8b6aad289a5dd481a8a1429ca200260f503be59a45" exitCode=0
Nov 02 13:32:44.871023 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:44.870989    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-740f09-776cf8d88b-gtp87" event={"ID":"c482bbc5-fff9-42e7-a147-f9664750fc8a","Type":"ContainerDied","Data":"6a4bb787b18fac6145882b8b6aad289a5dd481a8a1429ca200260f503be59a45"}
Nov 02 13:32:44.871023 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:44.870998    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-740f09-776cf8d88b-gtp87"
Nov 02 13:32:44.871292 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:44.871028    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-740f09-776cf8d88b-gtp87" event={"ID":"c482bbc5-fff9-42e7-a147-f9664750fc8a","Type":"ContainerDied","Data":"b13839c1504f582db3c517a85f90c1b8be688d24d794dc7a15f263881084ae40"}
Nov 02 13:32:44.871292 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:44.871050    4559 scope.go:122] "RemoveContainer" containerID="6a4bb787b18fac6145882b8b6aad289a5dd481a8a1429ca200260f503be59a45"
Nov 02 13:32:44.883539 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:44.883513    4559 scope.go:122] "RemoveContainer" containerID="6a4bb787b18fac6145882b8b6aad289a5dd481a8a1429ca200260f503be59a45"
Nov 02 13:32:44.883873 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:32:44.883846    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a4bb787b18fac6145882b8b6aad289a5dd481a8a1429ca200260f503be59a45\": container with ID starting with 6a4bb787b18fac6145882b8b6aad289a5dd481a8a1429ca200260f503be59a45 not found: ID does not exist" containerID="6a4bb787b18fac6145882b8b6aad289a5dd481a8a1429ca200260f503be59a45"
Nov 02 13:32:44.883964 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:44.883877    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a4bb787b18fac6145882b8b6aad289a5dd481a8a1429ca200260f503be59a45"} err="failed to get container status \"6a4bb787b18fac6145882b8b6aad289a5dd481a8a1429ca200260f503be59a45\": rpc error: code = NotFound desc = could not find container \"6a4bb787b18fac6145882b8b6aad289a5dd481a8a1429ca200260f503be59a45\": container with ID starting with 6a4bb787b18fac6145882b8b6aad289a5dd481a8a1429ca200260f503be59a45 not found: ID does not exist"
Nov 02 13:32:44.924456 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:44.924430    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-740f09-776cf8d88b-gtp87"]
Nov 02 13:32:44.937108 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:44.937084    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-740f09-776cf8d88b-gtp87"]
Nov 02 13:32:45.087664 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:32:45.087613    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqcmj" containerName="extractor"
Nov 02 13:32:45.891651 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:45.891617    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c482bbc5-fff9-42e7-a147-f9664750fc8a" path="/var/lib/kubelet/pods/c482bbc5-fff9-42e7-a147-f9664750fc8a/volumes"
Nov 02 13:32:48.463143 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:48.463110    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w0-8032f3/sub-test-model-8032f3-kserve-76b9878597-5hq8k"]
Nov 02 13:32:48.463687 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:48.463667    4559 cpu_manager.go:414] "containerMap: removing container" podUID="4aa05b9d-f18c-4c47-82e9-bfe95d1afe48" containerName="cleanup"
Nov 02 13:32:48.463687 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:48.463682    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4aa05b9d-f18c-4c47-82e9-bfe95d1afe48" containerName="cleanup"
Nov 02 13:32:48.463822 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:48.463694    4559 cpu_manager.go:414] "containerMap: removing container" podUID="826b92f5-161c-4f6a-9055-d0c838c9ce09" containerName="istio-proxy"
Nov 02 13:32:48.463822 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:48.463702    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="826b92f5-161c-4f6a-9055-d0c838c9ce09" containerName="istio-proxy"
Nov 02 13:32:48.463822 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:48.463716    4559 cpu_manager.go:414] "containerMap: removing container" podUID="10370825-cfb5-4493-83bd-a883abf2d789" containerName="discovery"
Nov 02 13:32:48.463822 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:48.463721    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10370825-cfb5-4493-83bd-a883abf2d789" containerName="discovery"
Nov 02 13:32:48.463822 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:48.463728    4559 cpu_manager.go:414] "containerMap: removing container" podUID="eede656a-d78e-4cf2-b98d-14d172fd36c6" containerName="cleanup"
Nov 02 13:32:48.463822 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:48.463733    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eede656a-d78e-4cf2-b98d-14d172fd36c6" containerName="cleanup"
Nov 02 13:32:48.463822 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:48.463740    4559 cpu_manager.go:414] "containerMap: removing container" podUID="46747055-1220-4a7e-8177-915d463a9f10" containerName="cleanup"
Nov 02 13:32:48.463822 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:48.463745    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46747055-1220-4a7e-8177-915d463a9f10" containerName="cleanup"
Nov 02 13:32:48.463822 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:48.463750    4559 cpu_manager.go:414] "containerMap: removing container" podUID="4aa05b9d-f18c-4c47-82e9-bfe95d1afe48" containerName="cleanup"
Nov 02 13:32:48.463822 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:48.463755    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4aa05b9d-f18c-4c47-82e9-bfe95d1afe48" containerName="cleanup"
Nov 02 13:32:48.463822 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:48.463761    4559 cpu_manager.go:414] "containerMap: removing container" podUID="d2c5f43a-3a0c-444b-afc1-e2bf090dc05c" containerName="istio-proxy"
Nov 02 13:32:48.463822 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:48.463765    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2c5f43a-3a0c-444b-afc1-e2bf090dc05c" containerName="istio-proxy"
Nov 02 13:32:48.463822 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:48.463781    4559 cpu_manager.go:414] "containerMap: removing container" podUID="b736e009-d002-4177-8f79-8f50cd5325f3" containerName="istio-proxy"
Nov 02 13:32:48.463822 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:48.463786    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b736e009-d002-4177-8f79-8f50cd5325f3" containerName="istio-proxy"
Nov 02 13:32:48.463822 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:48.463796    4559 cpu_manager.go:414] "containerMap: removing container" podUID="4aa05b9d-f18c-4c47-82e9-bfe95d1afe48" containerName="cleanup"
Nov 02 13:32:48.463822 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:48.463802    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4aa05b9d-f18c-4c47-82e9-bfe95d1afe48" containerName="cleanup"
Nov 02 13:32:48.463822 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:48.463808    4559 cpu_manager.go:414] "containerMap: removing container" podUID="4293ef7c-ce26-475f-98e1-ebf007c35f00" containerName="istio-proxy"
Nov 02 13:32:48.463822 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:48.463813    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4293ef7c-ce26-475f-98e1-ebf007c35f00" containerName="istio-proxy"
Nov 02 13:32:48.463822 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:48.463821    4559 cpu_manager.go:414] "containerMap: removing container" podUID="226d3b62-6d8e-4f46-bf0e-870d7891314f" containerName="payload-processing"
Nov 02 13:32:48.463822 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:48.463825    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="226d3b62-6d8e-4f46-bf0e-870d7891314f" containerName="payload-processing"
Nov 02 13:32:48.463822 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:48.463832    4559 cpu_manager.go:414] "containerMap: removing container" podUID="ee4f0fd1-9b89-4c29-ba5f-3cf47a18d2be" containerName="istio-proxy"
Nov 02 13:32:48.463822 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:48.463837    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee4f0fd1-9b89-4c29-ba5f-3cf47a18d2be" containerName="istio-proxy"
Nov 02 13:32:48.465046 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:48.463846    4559 cpu_manager.go:414] "containerMap: removing container" podUID="82e5d483-e95b-43e6-9135-af960823427a" containerName="main"
Nov 02 13:32:48.465046 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:48.463854    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82e5d483-e95b-43e6-9135-af960823427a" containerName="main"
Nov 02 13:32:48.465046 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:48.463867    4559 cpu_manager.go:414] "containerMap: removing container" podUID="1972d657-0f0b-40b8-85ba-ac651e09d31f" containerName="payload-pre-processing"
Nov 02 13:32:48.465046 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:48.463872    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1972d657-0f0b-40b8-85ba-ac651e09d31f" containerName="payload-pre-processing"
Nov 02 13:32:48.465046 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:48.463878    4559 cpu_manager.go:414] "containerMap: removing container" podUID="459db7e8-1294-4758-93a4-1fc9b9320595" containerName="payload-pre-processing"
Nov 02 13:32:48.465046 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:48.463883    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="459db7e8-1294-4758-93a4-1fc9b9320595" containerName="payload-pre-processing"
Nov 02 13:32:48.465046 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:48.463892    4559 cpu_manager.go:414] "containerMap: removing container" podUID="c68bfb23-6abb-4a96-979b-6b00ea4d5f7e" containerName="maas-api"
Nov 02 13:32:48.465046 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:48.463896    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c68bfb23-6abb-4a96-979b-6b00ea4d5f7e" containerName="maas-api"
Nov 02 13:32:48.465046 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:48.463902    4559 cpu_manager.go:414] "containerMap: removing container" podUID="d3b2832d-2aad-406f-86b5-de405575f9b2" containerName="istio-proxy"
Nov 02 13:32:48.465046 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:48.463907    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3b2832d-2aad-406f-86b5-de405575f9b2" containerName="istio-proxy"
Nov 02 13:32:48.465046 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:48.463915    4559 cpu_manager.go:414] "containerMap: removing container" podUID="eede656a-d78e-4cf2-b98d-14d172fd36c6" containerName="cleanup"
Nov 02 13:32:48.465046 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:48.463920    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eede656a-d78e-4cf2-b98d-14d172fd36c6" containerName="cleanup"
Nov 02 13:32:48.465046 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:48.463934    4559 cpu_manager.go:414] "containerMap: removing container" podUID="d7505d42-8673-4ed0-8a6e-b6923d8ca25e" containerName="cleanup"
Nov 02 13:32:48.465046 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:48.463938    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7505d42-8673-4ed0-8a6e-b6923d8ca25e" containerName="cleanup"
Nov 02 13:32:48.465046 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:48.463943    4559 cpu_manager.go:414] "containerMap: removing container" podUID="46747055-1220-4a7e-8177-915d463a9f10" containerName="cleanup"
Nov 02 13:32:48.465046 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:48.463948    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46747055-1220-4a7e-8177-915d463a9f10" containerName="cleanup"
Nov 02 13:32:48.465046 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:48.463953    4559 cpu_manager.go:414] "containerMap: removing container" podUID="d7505d42-8673-4ed0-8a6e-b6923d8ca25e" containerName="cleanup"
Nov 02 13:32:48.465046 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:48.463957    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7505d42-8673-4ed0-8a6e-b6923d8ca25e" containerName="cleanup"
Nov 02 13:32:48.465046 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:48.463964    4559 cpu_manager.go:414] "containerMap: removing container" podUID="c482bbc5-fff9-42e7-a147-f9664750fc8a" containerName="maas-api"
Nov 02 13:32:48.465046 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:48.463968    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c482bbc5-fff9-42e7-a147-f9664750fc8a" containerName="maas-api"
Nov 02 13:32:48.465046 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:48.463975    4559 cpu_manager.go:414] "containerMap: removing container" podUID="d7505d42-8673-4ed0-8a6e-b6923d8ca25e" containerName="cleanup"
Nov 02 13:32:48.465046 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:48.463979    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7505d42-8673-4ed0-8a6e-b6923d8ca25e" containerName="cleanup"
Nov 02 13:32:48.465046 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:48.464047    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="c482bbc5-fff9-42e7-a147-f9664750fc8a" containerName="maas-api"
Nov 02 13:32:48.465046 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:48.464058    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7505d42-8673-4ed0-8a6e-b6923d8ca25e" containerName="cleanup"
Nov 02 13:32:48.465046 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:48.464063    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="4293ef7c-ce26-475f-98e1-ebf007c35f00" containerName="istio-proxy"
Nov 02 13:32:48.465046 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:48.464068    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="46747055-1220-4a7e-8177-915d463a9f10" containerName="cleanup"
Nov 02 13:32:48.465046 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:48.464075    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="4aa05b9d-f18c-4c47-82e9-bfe95d1afe48" containerName="cleanup"
Nov 02 13:32:48.465046 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:48.464084    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="826b92f5-161c-4f6a-9055-d0c838c9ce09" containerName="istio-proxy"
Nov 02 13:32:48.465046 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:48.464093    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="10370825-cfb5-4493-83bd-a883abf2d789" containerName="discovery"
Nov 02 13:32:48.465046 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:48.464098    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="eede656a-d78e-4cf2-b98d-14d172fd36c6" containerName="cleanup"
Nov 02 13:32:48.465046 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:48.464102    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="46747055-1220-4a7e-8177-915d463a9f10" containerName="cleanup"
Nov 02 13:32:48.465046 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:48.464106    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="4aa05b9d-f18c-4c47-82e9-bfe95d1afe48" containerName="cleanup"
Nov 02 13:32:48.465046 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:48.464110    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2c5f43a-3a0c-444b-afc1-e2bf090dc05c" containerName="istio-proxy"
Nov 02 13:32:48.465046 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:48.464115    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="b736e009-d002-4177-8f79-8f50cd5325f3" containerName="istio-proxy"
Nov 02 13:32:48.465046 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:48.464121    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="4aa05b9d-f18c-4c47-82e9-bfe95d1afe48" containerName="cleanup"
Nov 02 13:32:48.465046 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:48.464127    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="4293ef7c-ce26-475f-98e1-ebf007c35f00" containerName="istio-proxy"
Nov 02 13:32:48.465046 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:48.464132    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="226d3b62-6d8e-4f46-bf0e-870d7891314f" containerName="payload-processing"
Nov 02 13:32:48.465046 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:48.464137    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee4f0fd1-9b89-4c29-ba5f-3cf47a18d2be" containerName="istio-proxy"
Nov 02 13:32:48.465046 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:48.464142    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="82e5d483-e95b-43e6-9135-af960823427a" containerName="main"
Nov 02 13:32:48.465046 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:48.464147    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="1972d657-0f0b-40b8-85ba-ac651e09d31f" containerName="payload-pre-processing"
Nov 02 13:32:48.465046 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:48.464153    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="459db7e8-1294-4758-93a4-1fc9b9320595" containerName="payload-pre-processing"
Nov 02 13:32:48.465046 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:48.464162    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="c68bfb23-6abb-4a96-979b-6b00ea4d5f7e" containerName="maas-api"
Nov 02 13:32:48.465046 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:48.464168    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3b2832d-2aad-406f-86b5-de405575f9b2" containerName="istio-proxy"
Nov 02 13:32:48.465046 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:48.464173    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="eede656a-d78e-4cf2-b98d-14d172fd36c6" containerName="cleanup"
Nov 02 13:32:48.465046 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:48.464178    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="eede656a-d78e-4cf2-b98d-14d172fd36c6" containerName="cleanup"
Nov 02 13:32:48.465046 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:48.464184    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7505d42-8673-4ed0-8a6e-b6923d8ca25e" containerName="cleanup"
Nov 02 13:32:48.465046 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:48.464188    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7505d42-8673-4ed0-8a6e-b6923d8ca25e" containerName="cleanup"
Nov 02 13:32:48.469116 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:48.469097    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-8032f3/sub-test-model-8032f3-kserve-76b9878597-5hq8k"
Nov 02 13:32:48.498481 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:48.498451    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-8032f3\"/\"openshift-service-ca.crt\""
Nov 02 13:32:48.498481 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:48.498468    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-8032f3\"/\"kube-root-ca.crt\""
Nov 02 13:32:48.498724 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:48.498707    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-8032f3\"/\"default-dockercfg-28fjx\""
Nov 02 13:32:48.498795 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:48.498760    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-8032f3\"/\"sub-test-model-8032f3-kserve-self-signed-certs\""
Nov 02 13:32:48.501168 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:48.501142    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-8032f3/sub-test-model-8032f3-kserve-76b9878597-5hq8k"]
Nov 02 13:32:48.529577 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:48.529547    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f549d5fa-af79-4498-8eab-ee04abaa7ab4-tls-certs\") pod \"sub-test-model-8032f3-kserve-76b9878597-5hq8k\" (UID: \"f549d5fa-af79-4498-8eab-ee04abaa7ab4\") " pod="ai-tenant-e2e-shared-a-w0-8032f3/sub-test-model-8032f3-kserve-76b9878597-5hq8k"
Nov 02 13:32:48.529695 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:48.529605    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4f99\" (UniqueName: \"kubernetes.io/projected/f549d5fa-af79-4498-8eab-ee04abaa7ab4-kube-api-access-j4f99\") pod \"sub-test-model-8032f3-kserve-76b9878597-5hq8k\" (UID: \"f549d5fa-af79-4498-8eab-ee04abaa7ab4\") " pod="ai-tenant-e2e-shared-a-w0-8032f3/sub-test-model-8032f3-kserve-76b9878597-5hq8k"
Nov 02 13:32:48.529777 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:48.529755    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f549d5fa-af79-4498-8eab-ee04abaa7ab4-dshm\") pod \"sub-test-model-8032f3-kserve-76b9878597-5hq8k\" (UID: \"f549d5fa-af79-4498-8eab-ee04abaa7ab4\") " pod="ai-tenant-e2e-shared-a-w0-8032f3/sub-test-model-8032f3-kserve-76b9878597-5hq8k"
Nov 02 13:32:48.529828 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:48.529809    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f549d5fa-af79-4498-8eab-ee04abaa7ab4-model-cache\") pod \"sub-test-model-8032f3-kserve-76b9878597-5hq8k\" (UID: \"f549d5fa-af79-4498-8eab-ee04abaa7ab4\") " pod="ai-tenant-e2e-shared-a-w0-8032f3/sub-test-model-8032f3-kserve-76b9878597-5hq8k"
Nov 02 13:32:48.529867 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:48.529853    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f549d5fa-af79-4498-8eab-ee04abaa7ab4-home\") pod \"sub-test-model-8032f3-kserve-76b9878597-5hq8k\" (UID: \"f549d5fa-af79-4498-8eab-ee04abaa7ab4\") " pod="ai-tenant-e2e-shared-a-w0-8032f3/sub-test-model-8032f3-kserve-76b9878597-5hq8k"
Nov 02 13:32:48.529939 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:48.529925    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f549d5fa-af79-4498-8eab-ee04abaa7ab4-tmp-dir\") pod \"sub-test-model-8032f3-kserve-76b9878597-5hq8k\" (UID: \"f549d5fa-af79-4498-8eab-ee04abaa7ab4\") " pod="ai-tenant-e2e-shared-a-w0-8032f3/sub-test-model-8032f3-kserve-76b9878597-5hq8k"
Nov 02 13:32:48.631298 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:48.631276    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f549d5fa-af79-4498-8eab-ee04abaa7ab4-home\") pod \"sub-test-model-8032f3-kserve-76b9878597-5hq8k\" (UID: \"f549d5fa-af79-4498-8eab-ee04abaa7ab4\") " pod="ai-tenant-e2e-shared-a-w0-8032f3/sub-test-model-8032f3-kserve-76b9878597-5hq8k"
Nov 02 13:32:48.631412 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:48.631318    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f549d5fa-af79-4498-8eab-ee04abaa7ab4-tmp-dir\") pod \"sub-test-model-8032f3-kserve-76b9878597-5hq8k\" (UID: \"f549d5fa-af79-4498-8eab-ee04abaa7ab4\") " pod="ai-tenant-e2e-shared-a-w0-8032f3/sub-test-model-8032f3-kserve-76b9878597-5hq8k"
Nov 02 13:32:48.631471 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:48.631418    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f549d5fa-af79-4498-8eab-ee04abaa7ab4-tls-certs\") pod \"sub-test-model-8032f3-kserve-76b9878597-5hq8k\" (UID: \"f549d5fa-af79-4498-8eab-ee04abaa7ab4\") " pod="ai-tenant-e2e-shared-a-w0-8032f3/sub-test-model-8032f3-kserve-76b9878597-5hq8k"
Nov 02 13:32:48.631471 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:48.631453    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4f99\" (UniqueName: \"kubernetes.io/projected/f549d5fa-af79-4498-8eab-ee04abaa7ab4-kube-api-access-j4f99\") pod \"sub-test-model-8032f3-kserve-76b9878597-5hq8k\" (UID: \"f549d5fa-af79-4498-8eab-ee04abaa7ab4\") " pod="ai-tenant-e2e-shared-a-w0-8032f3/sub-test-model-8032f3-kserve-76b9878597-5hq8k"
Nov 02 13:32:48.631668 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:48.631604    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f549d5fa-af79-4498-8eab-ee04abaa7ab4-dshm\") pod \"sub-test-model-8032f3-kserve-76b9878597-5hq8k\" (UID: \"f549d5fa-af79-4498-8eab-ee04abaa7ab4\") " pod="ai-tenant-e2e-shared-a-w0-8032f3/sub-test-model-8032f3-kserve-76b9878597-5hq8k"
Nov 02 13:32:48.631755 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:48.631671    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f549d5fa-af79-4498-8eab-ee04abaa7ab4-model-cache\") pod \"sub-test-model-8032f3-kserve-76b9878597-5hq8k\" (UID: \"f549d5fa-af79-4498-8eab-ee04abaa7ab4\") " pod="ai-tenant-e2e-shared-a-w0-8032f3/sub-test-model-8032f3-kserve-76b9878597-5hq8k"
Nov 02 13:32:48.631815 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:48.631769    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f549d5fa-af79-4498-8eab-ee04abaa7ab4-home\") pod \"sub-test-model-8032f3-kserve-76b9878597-5hq8k\" (UID: \"f549d5fa-af79-4498-8eab-ee04abaa7ab4\") " pod="ai-tenant-e2e-shared-a-w0-8032f3/sub-test-model-8032f3-kserve-76b9878597-5hq8k"
Nov 02 13:32:48.631877 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:48.631830    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f549d5fa-af79-4498-8eab-ee04abaa7ab4-tmp-dir\") pod \"sub-test-model-8032f3-kserve-76b9878597-5hq8k\" (UID: \"f549d5fa-af79-4498-8eab-ee04abaa7ab4\") " pod="ai-tenant-e2e-shared-a-w0-8032f3/sub-test-model-8032f3-kserve-76b9878597-5hq8k"
Nov 02 13:32:48.631992 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:48.631973    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f549d5fa-af79-4498-8eab-ee04abaa7ab4-model-cache\") pod \"sub-test-model-8032f3-kserve-76b9878597-5hq8k\" (UID: \"f549d5fa-af79-4498-8eab-ee04abaa7ab4\") " pod="ai-tenant-e2e-shared-a-w0-8032f3/sub-test-model-8032f3-kserve-76b9878597-5hq8k"
Nov 02 13:32:48.633817 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:48.633787    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f549d5fa-af79-4498-8eab-ee04abaa7ab4-dshm\") pod \"sub-test-model-8032f3-kserve-76b9878597-5hq8k\" (UID: \"f549d5fa-af79-4498-8eab-ee04abaa7ab4\") " pod="ai-tenant-e2e-shared-a-w0-8032f3/sub-test-model-8032f3-kserve-76b9878597-5hq8k"
Nov 02 13:32:48.634056 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:48.634038    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f549d5fa-af79-4498-8eab-ee04abaa7ab4-tls-certs\") pod \"sub-test-model-8032f3-kserve-76b9878597-5hq8k\" (UID: \"f549d5fa-af79-4498-8eab-ee04abaa7ab4\") " pod="ai-tenant-e2e-shared-a-w0-8032f3/sub-test-model-8032f3-kserve-76b9878597-5hq8k"
Nov 02 13:32:48.654801 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:48.654778    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4f99\" (UniqueName: \"kubernetes.io/projected/f549d5fa-af79-4498-8eab-ee04abaa7ab4-kube-api-access-j4f99\") pod \"sub-test-model-8032f3-kserve-76b9878597-5hq8k\" (UID: \"f549d5fa-af79-4498-8eab-ee04abaa7ab4\") " pod="ai-tenant-e2e-shared-a-w0-8032f3/sub-test-model-8032f3-kserve-76b9878597-5hq8k"
Nov 02 13:32:48.782220 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:48.782136    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-8032f3/sub-test-model-8032f3-kserve-76b9878597-5hq8k"
Nov 02 13:32:49.083062 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:49.083033    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-8032f3/sub-test-model-8032f3-kserve-76b9878597-5hq8k"]
Nov 02 13:32:49.083189 ip-10-0-128-132 kubenswrapper[4559]: W1102 13:32:49.083038    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf549d5fa_af79_4498_8eab_ee04abaa7ab4.slice/crio-a442ebc664a4ea8074e4eced230b6aed1d7cd34f852e4286020f4f2e0ea94470 WatchSource:0}: Error finding container a442ebc664a4ea8074e4eced230b6aed1d7cd34f852e4286020f4f2e0ea94470: Status 404 returned error can't find the container with id a442ebc664a4ea8074e4eced230b6aed1d7cd34f852e4286020f4f2e0ea94470
Nov 02 13:32:49.906319 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:49.906279    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-8032f3/sub-test-model-8032f3-kserve-76b9878597-5hq8k" event={"ID":"f549d5fa-af79-4498-8eab-ee04abaa7ab4","Type":"ContainerStarted","Data":"cf8e5088471bd6bc9aa36bc68156de20e4288c0445971ab5a5e9e073175af59c"}
Nov 02 13:32:49.906319 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:49.906326    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-8032f3/sub-test-model-8032f3-kserve-76b9878597-5hq8k" event={"ID":"f549d5fa-af79-4498-8eab-ee04abaa7ab4","Type":"ContainerStarted","Data":"a442ebc664a4ea8074e4eced230b6aed1d7cd34f852e4286020f4f2e0ea94470"}
Nov 02 13:32:49.950301 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:49.950248    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w0-8032f3/sub-test-model-8032f3-kserve-76b9878597-5hq8k" podStartSLOduration=1.950234371 podStartE2EDuration="1.950234371s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:32:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:32:49.948956608 +0000 UTC m=+3966.204355703" watchObservedRunningTime="2026-11-02 13:32:49.950234371 +0000 UTC m=+3966.205633465"
Nov 02 13:32:50.782722 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:50.782689    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w0-8032f3/sub-test-model-8032f3-kserve-76b9878597-5hq8k"
Nov 02 13:32:53.991754 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:53.991718    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-740f09-openshift-default-786896ffcc-lmqlz"]
Nov 02 13:32:53.992246 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:53.991980    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-740f09-openshift-default-786896ffcc-lmqlz" podUID="8f2bb155-8114-4c86-a74d-479522616e10" containerName="istio-proxy" containerID="cri-o://168d71daca827a8ae4b13fa24b65d231dfd8557291998d1e831016a70371da98" gracePeriod=30
Nov 02 13:32:58.328274 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:58.328241    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"]
Nov 02 13:32:58.483624 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:58.483578    4559 patch_prober.go:29] interesting pod/e2e-extmod-a-740f09-openshift-default-786896ffcc-lmqlz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 13:32:58.483976 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:58.483928    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-740f09-openshift-default-786896ffcc-lmqlz" podUID="8f2bb155-8114-4c86-a74d-479522616e10" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 13:32:58.782494 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:58.782462    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w0-8032f3/sub-test-model-8032f3-kserve-76b9878597-5hq8k"
Nov 02 13:32:58.796407 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:58.796378    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w0-8032f3/sub-test-model-8032f3-kserve-76b9878597-5hq8k"
Nov 02 13:32:58.993722 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:58.993687    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w0-8032f3/sub-test-model-8032f3-kserve-76b9878597-5hq8k"
Nov 02 13:32:59.528662 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:59.527805    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-740f09-openshift-default-786896ffcc-lmqlz"
Nov 02 13:32:59.707669 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:59.698011    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f2bb155-8114-4c86-a74d-479522616e10-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f2bb155-8114-4c86-a74d-479522616e10-istio-data\") pod \"8f2bb155-8114-4c86-a74d-479522616e10\" (UID: \"8f2bb155-8114-4c86-a74d-479522616e10\") "
Nov 02 13:32:59.707669 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:59.698097    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f2bb155-8114-4c86-a74d-479522616e10-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f2bb155-8114-4c86-a74d-479522616e10-istio-envoy\") pod \"8f2bb155-8114-4c86-a74d-479522616e10\" (UID: \"8f2bb155-8114-4c86-a74d-479522616e10\") "
Nov 02 13:32:59.707669 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:59.698161    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f2bb155-8114-4c86-a74d-479522616e10-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f2bb155-8114-4c86-a74d-479522616e10-workload-socket\") pod \"8f2bb155-8114-4c86-a74d-479522616e10\" (UID: \"8f2bb155-8114-4c86-a74d-479522616e10\") "
Nov 02 13:32:59.707669 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:59.698256    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f2bb155-8114-4c86-a74d-479522616e10-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f2bb155-8114-4c86-a74d-479522616e10-istiod-ca-cert\") pod \"8f2bb155-8114-4c86-a74d-479522616e10\" (UID: \"8f2bb155-8114-4c86-a74d-479522616e10\") "
Nov 02 13:32:59.707669 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:59.698321    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f2bb155-8114-4c86-a74d-479522616e10-istio-token\" (UniqueName: \"kubernetes.io/projected/8f2bb155-8114-4c86-a74d-479522616e10-istio-token\") pod \"8f2bb155-8114-4c86-a74d-479522616e10\" (UID: \"8f2bb155-8114-4c86-a74d-479522616e10\") "
Nov 02 13:32:59.707669 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:59.698349    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8f2bb155-8114-4c86-a74d-479522616e10-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f2bb155-8114-4c86-a74d-479522616e10-istio-podinfo\") pod \"8f2bb155-8114-4c86-a74d-479522616e10\" (UID: \"8f2bb155-8114-4c86-a74d-479522616e10\") "
Nov 02 13:32:59.707669 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:59.698393    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f2bb155-8114-4c86-a74d-479522616e10-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f2bb155-8114-4c86-a74d-479522616e10-credential-socket\") pod \"8f2bb155-8114-4c86-a74d-479522616e10\" (UID: \"8f2bb155-8114-4c86-a74d-479522616e10\") "
Nov 02 13:32:59.707669 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:59.698427    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f2bb155-8114-4c86-a74d-479522616e10-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f2bb155-8114-4c86-a74d-479522616e10-workload-certs\") pod \"8f2bb155-8114-4c86-a74d-479522616e10\" (UID: \"8f2bb155-8114-4c86-a74d-479522616e10\") "
Nov 02 13:32:59.707669 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:59.698495    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f2bb155-8114-4c86-a74d-479522616e10-kube-api-access-q99sk\" (UniqueName: \"kubernetes.io/projected/8f2bb155-8114-4c86-a74d-479522616e10-kube-api-access-q99sk\") pod \"8f2bb155-8114-4c86-a74d-479522616e10\" (UID: \"8f2bb155-8114-4c86-a74d-479522616e10\") "
Nov 02 13:32:59.733888 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:59.733623    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f2bb155-8114-4c86-a74d-479522616e10-istio-data" pod "8f2bb155-8114-4c86-a74d-479522616e10" (UID: "8f2bb155-8114-4c86-a74d-479522616e10"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:32:59.734071 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:59.734030    4559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f2bb155-8114-4c86-a74d-479522616e10-istio-data\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:32:59.735136 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:59.735105    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f2bb155-8114-4c86-a74d-479522616e10-credential-socket" pod "8f2bb155-8114-4c86-a74d-479522616e10" (UID: "8f2bb155-8114-4c86-a74d-479522616e10"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:32:59.735313 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:59.735282    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f2bb155-8114-4c86-a74d-479522616e10-workload-socket" pod "8f2bb155-8114-4c86-a74d-479522616e10" (UID: "8f2bb155-8114-4c86-a74d-479522616e10"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:32:59.735952 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:59.735930    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f2bb155-8114-4c86-a74d-479522616e10-workload-certs" pod "8f2bb155-8114-4c86-a74d-479522616e10" (UID: "8f2bb155-8114-4c86-a74d-479522616e10"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:32:59.736101 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:59.736046    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f2bb155-8114-4c86-a74d-479522616e10-istiod-ca-cert" pod "8f2bb155-8114-4c86-a74d-479522616e10" (UID: "8f2bb155-8114-4c86-a74d-479522616e10"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:32:59.739507 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:59.739466    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f2bb155-8114-4c86-a74d-479522616e10-istio-token" pod "8f2bb155-8114-4c86-a74d-479522616e10" (UID: "8f2bb155-8114-4c86-a74d-479522616e10"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:32:59.740887 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:59.740854    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8f2bb155-8114-4c86-a74d-479522616e10-istio-podinfo" pod "8f2bb155-8114-4c86-a74d-479522616e10" (UID: "8f2bb155-8114-4c86-a74d-479522616e10"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 13:32:59.741431 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:59.741400    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f2bb155-8114-4c86-a74d-479522616e10-kube-api-access-q99sk" pod "8f2bb155-8114-4c86-a74d-479522616e10" (UID: "8f2bb155-8114-4c86-a74d-479522616e10"). InnerVolumeSpecName "kube-api-access-q99sk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:32:59.742082 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:59.742050    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f2bb155-8114-4c86-a74d-479522616e10-istio-envoy" pod "8f2bb155-8114-4c86-a74d-479522616e10" (UID: "8f2bb155-8114-4c86-a74d-479522616e10"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:32:59.835394 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:59.835368    4559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f2bb155-8114-4c86-a74d-479522616e10-istio-envoy\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:32:59.835394 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:59.835392    4559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f2bb155-8114-4c86-a74d-479522616e10-workload-socket\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:32:59.835592 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:59.835401    4559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f2bb155-8114-4c86-a74d-479522616e10-istiod-ca-cert\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:32:59.835592 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:59.835410    4559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f2bb155-8114-4c86-a74d-479522616e10-istio-token\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:32:59.835592 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:59.835418    4559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f2bb155-8114-4c86-a74d-479522616e10-istio-podinfo\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:32:59.835592 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:59.835426    4559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f2bb155-8114-4c86-a74d-479522616e10-credential-socket\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:32:59.835592 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:59.835434    4559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f2bb155-8114-4c86-a74d-479522616e10-workload-certs\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:32:59.835592 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:59.835448    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q99sk\" (UniqueName: \"kubernetes.io/projected/8f2bb155-8114-4c86-a74d-479522616e10-kube-api-access-q99sk\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:32:59.979321 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:59.979245    4559 generic.go:356] "Generic (PLEG): container finished" podID="8f2bb155-8114-4c86-a74d-479522616e10" containerID="168d71daca827a8ae4b13fa24b65d231dfd8557291998d1e831016a70371da98" exitCode=0
Nov 02 13:32:59.979478 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:59.979315    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-740f09-openshift-default-786896ffcc-lmqlz"
Nov 02 13:32:59.979478 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:59.979324    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-740f09-openshift-default-786896ffcc-lmqlz" event={"ID":"8f2bb155-8114-4c86-a74d-479522616e10","Type":"ContainerDied","Data":"168d71daca827a8ae4b13fa24b65d231dfd8557291998d1e831016a70371da98"}
Nov 02 13:32:59.979478 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:59.979398    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-740f09-openshift-default-786896ffcc-lmqlz" event={"ID":"8f2bb155-8114-4c86-a74d-479522616e10","Type":"ContainerDied","Data":"176b8adc4a7afe836783302c0db1184b60fdf7c5cab538e063e1080c0b6f3b08"}
Nov 02 13:32:59.979478 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:59.979420    4559 scope.go:122] "RemoveContainer" containerID="168d71daca827a8ae4b13fa24b65d231dfd8557291998d1e831016a70371da98"
Nov 02 13:32:59.989524 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:59.989494    4559 scope.go:122] "RemoveContainer" containerID="815680021cae23ccb7232840479478c9c9425105ef0b77eee0ee0e05076164ef"
Nov 02 13:32:59.999264 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:59.999247    4559 scope.go:122] "RemoveContainer" containerID="168d71daca827a8ae4b13fa24b65d231dfd8557291998d1e831016a70371da98"
Nov 02 13:32:59.999523 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:32:59.999497    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"168d71daca827a8ae4b13fa24b65d231dfd8557291998d1e831016a70371da98\": container with ID starting with 168d71daca827a8ae4b13fa24b65d231dfd8557291998d1e831016a70371da98 not found: ID does not exist" containerID="168d71daca827a8ae4b13fa24b65d231dfd8557291998d1e831016a70371da98"
Nov 02 13:32:59.999584 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:59.999524    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"168d71daca827a8ae4b13fa24b65d231dfd8557291998d1e831016a70371da98"} err="failed to get container status \"168d71daca827a8ae4b13fa24b65d231dfd8557291998d1e831016a70371da98\": rpc error: code = NotFound desc = could not find container \"168d71daca827a8ae4b13fa24b65d231dfd8557291998d1e831016a70371da98\": container with ID starting with 168d71daca827a8ae4b13fa24b65d231dfd8557291998d1e831016a70371da98 not found: ID does not exist"
Nov 02 13:32:59.999584 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:59.999541    4559 scope.go:122] "RemoveContainer" containerID="815680021cae23ccb7232840479478c9c9425105ef0b77eee0ee0e05076164ef"
Nov 02 13:32:59.999815 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:32:59.999796    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"815680021cae23ccb7232840479478c9c9425105ef0b77eee0ee0e05076164ef\": container with ID starting with 815680021cae23ccb7232840479478c9c9425105ef0b77eee0ee0e05076164ef not found: ID does not exist" containerID="815680021cae23ccb7232840479478c9c9425105ef0b77eee0ee0e05076164ef"
Nov 02 13:32:59.999878 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:32:59.999824    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"815680021cae23ccb7232840479478c9c9425105ef0b77eee0ee0e05076164ef"} err="failed to get container status \"815680021cae23ccb7232840479478c9c9425105ef0b77eee0ee0e05076164ef\": rpc error: code = NotFound desc = could not find container \"815680021cae23ccb7232840479478c9c9425105ef0b77eee0ee0e05076164ef\": container with ID starting with 815680021cae23ccb7232840479478c9c9425105ef0b77eee0ee0e05076164ef not found: ID does not exist"
Nov 02 13:33:00.039015 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:00.038993    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-740f09-openshift-default-786896ffcc-lmqlz"]
Nov 02 13:33:00.057865 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:00.057841    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-740f09-openshift-default-786896ffcc-lmqlz"]
Nov 02 13:33:01.892817 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:01.892781    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f2bb155-8114-4c86-a74d-479522616e10" path="/var/lib/kubelet/pods/8f2bb155-8114-4c86-a74d-479522616e10/volumes"
Nov 02 13:33:12.990561 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:12.990530    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-90cda9-5f86b75b95-bnzqk"]
Nov 02 13:33:12.991014 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:12.990773    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-90cda9-5f86b75b95-bnzqk" podUID="8acbad7b-5320-4a6c-80b1-b1766c55f340" containerName="maas-api" containerID="cri-o://9a34e36329226612c7729d6dbca102816c2e41091846d9b166e3718d76347c0a" gracePeriod=30
Nov 02 13:33:13.256490 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:13.256407    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-90cda9-5f86b75b95-bnzqk"
Nov 02 13:33:13.373968 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:13.373934    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8acbad7b-5320-4a6c-80b1-b1766c55f340-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8acbad7b-5320-4a6c-80b1-b1766c55f340-metrics-certs\") pod \"8acbad7b-5320-4a6c-80b1-b1766c55f340\" (UID: \"8acbad7b-5320-4a6c-80b1-b1766c55f340\") "
Nov 02 13:33:13.373968 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:13.373972    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8acbad7b-5320-4a6c-80b1-b1766c55f340-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8acbad7b-5320-4a6c-80b1-b1766c55f340-maas-api-tls\") pod \"8acbad7b-5320-4a6c-80b1-b1766c55f340\" (UID: \"8acbad7b-5320-4a6c-80b1-b1766c55f340\") "
Nov 02 13:33:13.374176 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:13.374023    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8acbad7b-5320-4a6c-80b1-b1766c55f340-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8acbad7b-5320-4a6c-80b1-b1766c55f340-openshift-service-ca\") pod \"8acbad7b-5320-4a6c-80b1-b1766c55f340\" (UID: \"8acbad7b-5320-4a6c-80b1-b1766c55f340\") "
Nov 02 13:33:13.374176 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:13.374068    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8acbad7b-5320-4a6c-80b1-b1766c55f340-kube-api-access-pv49h\" (UniqueName: \"kubernetes.io/projected/8acbad7b-5320-4a6c-80b1-b1766c55f340-kube-api-access-pv49h\") pod \"8acbad7b-5320-4a6c-80b1-b1766c55f340\" (UID: \"8acbad7b-5320-4a6c-80b1-b1766c55f340\") "
Nov 02 13:33:13.374434 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:13.374408    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8acbad7b-5320-4a6c-80b1-b1766c55f340-openshift-service-ca" pod "8acbad7b-5320-4a6c-80b1-b1766c55f340" (UID: "8acbad7b-5320-4a6c-80b1-b1766c55f340"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:33:13.376227 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:13.376203    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8acbad7b-5320-4a6c-80b1-b1766c55f340-metrics-certs" pod "8acbad7b-5320-4a6c-80b1-b1766c55f340" (UID: "8acbad7b-5320-4a6c-80b1-b1766c55f340"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:33:13.376317 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:13.376226    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8acbad7b-5320-4a6c-80b1-b1766c55f340-maas-api-tls" pod "8acbad7b-5320-4a6c-80b1-b1766c55f340" (UID: "8acbad7b-5320-4a6c-80b1-b1766c55f340"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:33:13.376359 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:13.376318    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8acbad7b-5320-4a6c-80b1-b1766c55f340-kube-api-access-pv49h" pod "8acbad7b-5320-4a6c-80b1-b1766c55f340" (UID: "8acbad7b-5320-4a6c-80b1-b1766c55f340"). InnerVolumeSpecName "kube-api-access-pv49h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:33:13.475284 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:13.475245    4559 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8acbad7b-5320-4a6c-80b1-b1766c55f340-metrics-certs\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:33:13.475284 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:13.475278    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8acbad7b-5320-4a6c-80b1-b1766c55f340-maas-api-tls\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:33:13.475284 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:13.475287    4559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8acbad7b-5320-4a6c-80b1-b1766c55f340-openshift-service-ca\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:33:13.475518 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:13.475296    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pv49h\" (UniqueName: \"kubernetes.io/projected/8acbad7b-5320-4a6c-80b1-b1766c55f340-kube-api-access-pv49h\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:33:14.060368 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:14.060336    4559 generic.go:356] "Generic (PLEG): container finished" podID="8acbad7b-5320-4a6c-80b1-b1766c55f340" containerID="9a34e36329226612c7729d6dbca102816c2e41091846d9b166e3718d76347c0a" exitCode=0
Nov 02 13:33:14.060855 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:14.060377    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-90cda9-5f86b75b95-bnzqk"
Nov 02 13:33:14.060855 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:14.060429    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-90cda9-5f86b75b95-bnzqk" event={"ID":"8acbad7b-5320-4a6c-80b1-b1766c55f340","Type":"ContainerDied","Data":"9a34e36329226612c7729d6dbca102816c2e41091846d9b166e3718d76347c0a"}
Nov 02 13:33:14.060855 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:14.060485    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-90cda9-5f86b75b95-bnzqk" event={"ID":"8acbad7b-5320-4a6c-80b1-b1766c55f340","Type":"ContainerDied","Data":"39e48fd80e03cafa3375ed50f62f3f99c003a9867230455adcdf352407ab0594"}
Nov 02 13:33:14.060855 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:14.060513    4559 scope.go:122] "RemoveContainer" containerID="9a34e36329226612c7729d6dbca102816c2e41091846d9b166e3718d76347c0a"
Nov 02 13:33:14.070655 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:14.070614    4559 scope.go:122] "RemoveContainer" containerID="9a34e36329226612c7729d6dbca102816c2e41091846d9b166e3718d76347c0a"
Nov 02 13:33:14.070964 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:33:14.070943    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a34e36329226612c7729d6dbca102816c2e41091846d9b166e3718d76347c0a\": container with ID starting with 9a34e36329226612c7729d6dbca102816c2e41091846d9b166e3718d76347c0a not found: ID does not exist" containerID="9a34e36329226612c7729d6dbca102816c2e41091846d9b166e3718d76347c0a"
Nov 02 13:33:14.071039 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:14.070975    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a34e36329226612c7729d6dbca102816c2e41091846d9b166e3718d76347c0a"} err="failed to get container status \"9a34e36329226612c7729d6dbca102816c2e41091846d9b166e3718d76347c0a\": rpc error: code = NotFound desc = could not find container \"9a34e36329226612c7729d6dbca102816c2e41091846d9b166e3718d76347c0a\": container with ID starting with 9a34e36329226612c7729d6dbca102816c2e41091846d9b166e3718d76347c0a not found: ID does not exist"
Nov 02 13:33:14.113370 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:14.113345    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-90cda9-5f86b75b95-bnzqk"]
Nov 02 13:33:14.126044 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:14.126022    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-90cda9-5f86b75b95-bnzqk"]
Nov 02 13:33:15.090758 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:33:15.090702    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqcmj" containerName="extractor"
Nov 02 13:33:15.828846 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:15.828813    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w0-43be82/sub-test-model-43be82-kserve-5dc888468f-9lf8t"]
Nov 02 13:33:15.829355 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:15.829338    4559 cpu_manager.go:414] "containerMap: removing container" podUID="8f2bb155-8114-4c86-a74d-479522616e10" containerName="istio-proxy"
Nov 02 13:33:15.829355 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:15.829354    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f2bb155-8114-4c86-a74d-479522616e10" containerName="istio-proxy"
Nov 02 13:33:15.829516 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:15.829365    4559 cpu_manager.go:414] "containerMap: removing container" podUID="eede656a-d78e-4cf2-b98d-14d172fd36c6" containerName="cleanup"
Nov 02 13:33:15.829516 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:15.829370    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eede656a-d78e-4cf2-b98d-14d172fd36c6" containerName="cleanup"
Nov 02 13:33:15.829516 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:15.829387    4559 cpu_manager.go:414] "containerMap: removing container" podUID="4293ef7c-ce26-475f-98e1-ebf007c35f00" containerName="istio-proxy"
Nov 02 13:33:15.829516 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:15.829391    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4293ef7c-ce26-475f-98e1-ebf007c35f00" containerName="istio-proxy"
Nov 02 13:33:15.829516 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:15.829416    4559 cpu_manager.go:414] "containerMap: removing container" podUID="46747055-1220-4a7e-8177-915d463a9f10" containerName="cleanup"
Nov 02 13:33:15.829516 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:15.829422    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46747055-1220-4a7e-8177-915d463a9f10" containerName="cleanup"
Nov 02 13:33:15.829516 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:15.829434    4559 cpu_manager.go:414] "containerMap: removing container" podUID="8acbad7b-5320-4a6c-80b1-b1766c55f340" containerName="maas-api"
Nov 02 13:33:15.829516 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:15.829440    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8acbad7b-5320-4a6c-80b1-b1766c55f340" containerName="maas-api"
Nov 02 13:33:15.829516 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:15.829513    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f2bb155-8114-4c86-a74d-479522616e10" containerName="istio-proxy"
Nov 02 13:33:15.829516 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:15.829521    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="46747055-1220-4a7e-8177-915d463a9f10" containerName="cleanup"
Nov 02 13:33:15.830128 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:15.829533    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f2bb155-8114-4c86-a74d-479522616e10" containerName="istio-proxy"
Nov 02 13:33:15.830128 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:15.829539    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="8acbad7b-5320-4a6c-80b1-b1766c55f340" containerName="maas-api"
Nov 02 13:33:15.834438 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:15.834413    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-43be82/sub-test-model-43be82-kserve-5dc888468f-9lf8t"
Nov 02 13:33:15.854467 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:15.854419    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-43be82/sub-test-model-43be82-kserve-5dc888468f-9lf8t"]
Nov 02 13:33:15.864209 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:15.864185    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-43be82\"/\"sub-test-model-43be82-kserve-self-signed-certs\""
Nov 02 13:33:15.864345 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:15.864263    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-43be82\"/\"default-dockercfg-w5zw7\""
Nov 02 13:33:15.864801 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:15.864783    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-43be82\"/\"openshift-service-ca.crt\""
Nov 02 13:33:15.864919 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:15.864804    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-43be82\"/\"kube-root-ca.crt\""
Nov 02 13:33:15.895044 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:15.895015    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8acbad7b-5320-4a6c-80b1-b1766c55f340" path="/var/lib/kubelet/pods/8acbad7b-5320-4a6c-80b1-b1766c55f340/volumes"
Nov 02 13:33:15.899557 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:15.899532    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9322192d-bc03-41d8-a556-7057c92fa5ba-home\") pod \"sub-test-model-43be82-kserve-5dc888468f-9lf8t\" (UID: \"9322192d-bc03-41d8-a556-7057c92fa5ba\") " pod="ai-tenant-e2e-shared-b-w0-43be82/sub-test-model-43be82-kserve-5dc888468f-9lf8t"
Nov 02 13:33:15.899672 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:15.899583    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9322192d-bc03-41d8-a556-7057c92fa5ba-tls-certs\") pod \"sub-test-model-43be82-kserve-5dc888468f-9lf8t\" (UID: \"9322192d-bc03-41d8-a556-7057c92fa5ba\") " pod="ai-tenant-e2e-shared-b-w0-43be82/sub-test-model-43be82-kserve-5dc888468f-9lf8t"
Nov 02 13:33:15.899719 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:15.899705    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vq6xm\" (UniqueName: \"kubernetes.io/projected/9322192d-bc03-41d8-a556-7057c92fa5ba-kube-api-access-vq6xm\") pod \"sub-test-model-43be82-kserve-5dc888468f-9lf8t\" (UID: \"9322192d-bc03-41d8-a556-7057c92fa5ba\") " pod="ai-tenant-e2e-shared-b-w0-43be82/sub-test-model-43be82-kserve-5dc888468f-9lf8t"
Nov 02 13:33:15.899754 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:15.899740    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9322192d-bc03-41d8-a556-7057c92fa5ba-tmp-dir\") pod \"sub-test-model-43be82-kserve-5dc888468f-9lf8t\" (UID: \"9322192d-bc03-41d8-a556-7057c92fa5ba\") " pod="ai-tenant-e2e-shared-b-w0-43be82/sub-test-model-43be82-kserve-5dc888468f-9lf8t"
Nov 02 13:33:15.899792 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:15.899784    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9322192d-bc03-41d8-a556-7057c92fa5ba-dshm\") pod \"sub-test-model-43be82-kserve-5dc888468f-9lf8t\" (UID: \"9322192d-bc03-41d8-a556-7057c92fa5ba\") " pod="ai-tenant-e2e-shared-b-w0-43be82/sub-test-model-43be82-kserve-5dc888468f-9lf8t"
Nov 02 13:33:15.899886 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:15.899873    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9322192d-bc03-41d8-a556-7057c92fa5ba-model-cache\") pod \"sub-test-model-43be82-kserve-5dc888468f-9lf8t\" (UID: \"9322192d-bc03-41d8-a556-7057c92fa5ba\") " pod="ai-tenant-e2e-shared-b-w0-43be82/sub-test-model-43be82-kserve-5dc888468f-9lf8t"
Nov 02 13:33:16.000885 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:16.000855    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9322192d-bc03-41d8-a556-7057c92fa5ba-model-cache\") pod \"sub-test-model-43be82-kserve-5dc888468f-9lf8t\" (UID: \"9322192d-bc03-41d8-a556-7057c92fa5ba\") " pod="ai-tenant-e2e-shared-b-w0-43be82/sub-test-model-43be82-kserve-5dc888468f-9lf8t"
Nov 02 13:33:16.001033 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:16.000907    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9322192d-bc03-41d8-a556-7057c92fa5ba-home\") pod \"sub-test-model-43be82-kserve-5dc888468f-9lf8t\" (UID: \"9322192d-bc03-41d8-a556-7057c92fa5ba\") " pod="ai-tenant-e2e-shared-b-w0-43be82/sub-test-model-43be82-kserve-5dc888468f-9lf8t"
Nov 02 13:33:16.001033 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:16.000952    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9322192d-bc03-41d8-a556-7057c92fa5ba-tls-certs\") pod \"sub-test-model-43be82-kserve-5dc888468f-9lf8t\" (UID: \"9322192d-bc03-41d8-a556-7057c92fa5ba\") " pod="ai-tenant-e2e-shared-b-w0-43be82/sub-test-model-43be82-kserve-5dc888468f-9lf8t"
Nov 02 13:33:16.001033 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:16.000998    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vq6xm\" (UniqueName: \"kubernetes.io/projected/9322192d-bc03-41d8-a556-7057c92fa5ba-kube-api-access-vq6xm\") pod \"sub-test-model-43be82-kserve-5dc888468f-9lf8t\" (UID: \"9322192d-bc03-41d8-a556-7057c92fa5ba\") " pod="ai-tenant-e2e-shared-b-w0-43be82/sub-test-model-43be82-kserve-5dc888468f-9lf8t"
Nov 02 13:33:16.001033 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:16.001030    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9322192d-bc03-41d8-a556-7057c92fa5ba-tmp-dir\") pod \"sub-test-model-43be82-kserve-5dc888468f-9lf8t\" (UID: \"9322192d-bc03-41d8-a556-7057c92fa5ba\") " pod="ai-tenant-e2e-shared-b-w0-43be82/sub-test-model-43be82-kserve-5dc888468f-9lf8t"
Nov 02 13:33:16.001269 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:16.001108    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9322192d-bc03-41d8-a556-7057c92fa5ba-dshm\") pod \"sub-test-model-43be82-kserve-5dc888468f-9lf8t\" (UID: \"9322192d-bc03-41d8-a556-7057c92fa5ba\") " pod="ai-tenant-e2e-shared-b-w0-43be82/sub-test-model-43be82-kserve-5dc888468f-9lf8t"
Nov 02 13:33:16.001352 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:16.001318    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9322192d-bc03-41d8-a556-7057c92fa5ba-model-cache\") pod \"sub-test-model-43be82-kserve-5dc888468f-9lf8t\" (UID: \"9322192d-bc03-41d8-a556-7057c92fa5ba\") " pod="ai-tenant-e2e-shared-b-w0-43be82/sub-test-model-43be82-kserve-5dc888468f-9lf8t"
Nov 02 13:33:16.001421 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:16.001362    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9322192d-bc03-41d8-a556-7057c92fa5ba-home\") pod \"sub-test-model-43be82-kserve-5dc888468f-9lf8t\" (UID: \"9322192d-bc03-41d8-a556-7057c92fa5ba\") " pod="ai-tenant-e2e-shared-b-w0-43be82/sub-test-model-43be82-kserve-5dc888468f-9lf8t"
Nov 02 13:33:16.001691 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:16.001625    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9322192d-bc03-41d8-a556-7057c92fa5ba-tmp-dir\") pod \"sub-test-model-43be82-kserve-5dc888468f-9lf8t\" (UID: \"9322192d-bc03-41d8-a556-7057c92fa5ba\") " pod="ai-tenant-e2e-shared-b-w0-43be82/sub-test-model-43be82-kserve-5dc888468f-9lf8t"
Nov 02 13:33:16.003331 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:16.003307    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9322192d-bc03-41d8-a556-7057c92fa5ba-dshm\") pod \"sub-test-model-43be82-kserve-5dc888468f-9lf8t\" (UID: \"9322192d-bc03-41d8-a556-7057c92fa5ba\") " pod="ai-tenant-e2e-shared-b-w0-43be82/sub-test-model-43be82-kserve-5dc888468f-9lf8t"
Nov 02 13:33:16.003497 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:16.003476    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9322192d-bc03-41d8-a556-7057c92fa5ba-tls-certs\") pod \"sub-test-model-43be82-kserve-5dc888468f-9lf8t\" (UID: \"9322192d-bc03-41d8-a556-7057c92fa5ba\") " pod="ai-tenant-e2e-shared-b-w0-43be82/sub-test-model-43be82-kserve-5dc888468f-9lf8t"
Nov 02 13:33:16.024748 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:16.024721    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vq6xm\" (UniqueName: \"kubernetes.io/projected/9322192d-bc03-41d8-a556-7057c92fa5ba-kube-api-access-vq6xm\") pod \"sub-test-model-43be82-kserve-5dc888468f-9lf8t\" (UID: \"9322192d-bc03-41d8-a556-7057c92fa5ba\") " pod="ai-tenant-e2e-shared-b-w0-43be82/sub-test-model-43be82-kserve-5dc888468f-9lf8t"
Nov 02 13:33:16.150614 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:16.150535    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-43be82/sub-test-model-43be82-kserve-5dc888468f-9lf8t"
Nov 02 13:33:16.454127 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:16.454098    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-43be82/sub-test-model-43be82-kserve-5dc888468f-9lf8t"]
Nov 02 13:33:16.454844 ip-10-0-128-132 kubenswrapper[4559]: W1102 13:33:16.454804    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9322192d_bc03_41d8_a556_7057c92fa5ba.slice/crio-de5d1675a909ee0a8419122a83a72404b77e80c3a54a8a0b625105075c9da1ed WatchSource:0}: Error finding container de5d1675a909ee0a8419122a83a72404b77e80c3a54a8a0b625105075c9da1ed: Status 404 returned error can't find the container with id de5d1675a909ee0a8419122a83a72404b77e80c3a54a8a0b625105075c9da1ed
Nov 02 13:33:17.095781 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:17.095745    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-43be82/sub-test-model-43be82-kserve-5dc888468f-9lf8t" event={"ID":"9322192d-bc03-41d8-a556-7057c92fa5ba","Type":"ContainerStarted","Data":"e80a23b58b99a6d967a0196ec5badc56b48991f18f40c8f9ac0bcb058932d597"}
Nov 02 13:33:17.095944 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:17.095787    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-43be82/sub-test-model-43be82-kserve-5dc888468f-9lf8t" event={"ID":"9322192d-bc03-41d8-a556-7057c92fa5ba","Type":"ContainerStarted","Data":"de5d1675a909ee0a8419122a83a72404b77e80c3a54a8a0b625105075c9da1ed"}
Nov 02 13:33:17.151312 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:17.151281    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w0-43be82/sub-test-model-43be82-kserve-5dc888468f-9lf8t"
Nov 02 13:33:17.207793 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:17.207744    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w0-43be82/sub-test-model-43be82-kserve-5dc888468f-9lf8t" podStartSLOduration=2.207729863 podStartE2EDuration="2.207729863s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:33:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:33:17.206954617 +0000 UTC m=+3993.462353716" watchObservedRunningTime="2026-11-02 13:33:17.207729863 +0000 UTC m=+3993.463128958"
Nov 02 13:33:17.809189 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:17.809148    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-90cda9-openshift-default-57f8b6f957-b5jzx"]
Nov 02 13:33:17.809939 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:17.809397    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-90cda9-openshift-default-57f8b6f957-b5jzx" podUID="bb3c761d-8641-496c-9d3a-531e8117e7dd" containerName="istio-proxy" containerID="cri-o://e59de548036ec32daed50a5119084a5473e6be9275af17787db33e25b4b53425" gracePeriod=30
Nov 02 13:33:20.571027 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:20.570992    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-25f7a6-595fd6dd77-72r7x"]
Nov 02 13:33:20.571471 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:20.571189    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-25f7a6-595fd6dd77-72r7x" podUID="e97a314b-6891-4a14-9f4f-c7131a59e231" containerName="maas-api" containerID="cri-o://d20c487cb4f07ea33ee2373c66b22491e0e2a889fc56abf45850d777988d848f" gracePeriod=30
Nov 02 13:33:20.820255 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:20.820230    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-25f7a6-595fd6dd77-72r7x"
Nov 02 13:33:20.956646 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:20.956614    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e97a314b-6891-4a14-9f4f-c7131a59e231-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e97a314b-6891-4a14-9f4f-c7131a59e231-openshift-service-ca\") pod \"e97a314b-6891-4a14-9f4f-c7131a59e231\" (UID: \"e97a314b-6891-4a14-9f4f-c7131a59e231\") "
Nov 02 13:33:20.956856 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:20.956760    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e97a314b-6891-4a14-9f4f-c7131a59e231-kube-api-access-dk7p5\" (UniqueName: \"kubernetes.io/projected/e97a314b-6891-4a14-9f4f-c7131a59e231-kube-api-access-dk7p5\") pod \"e97a314b-6891-4a14-9f4f-c7131a59e231\" (UID: \"e97a314b-6891-4a14-9f4f-c7131a59e231\") "
Nov 02 13:33:20.956856 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:20.956829    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e97a314b-6891-4a14-9f4f-c7131a59e231-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e97a314b-6891-4a14-9f4f-c7131a59e231-metrics-certs\") pod \"e97a314b-6891-4a14-9f4f-c7131a59e231\" (UID: \"e97a314b-6891-4a14-9f4f-c7131a59e231\") "
Nov 02 13:33:20.956856 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:20.956849    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e97a314b-6891-4a14-9f4f-c7131a59e231-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e97a314b-6891-4a14-9f4f-c7131a59e231-maas-api-tls\") pod \"e97a314b-6891-4a14-9f4f-c7131a59e231\" (UID: \"e97a314b-6891-4a14-9f4f-c7131a59e231\") "
Nov 02 13:33:20.957057 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:20.956991    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e97a314b-6891-4a14-9f4f-c7131a59e231-openshift-service-ca" pod "e97a314b-6891-4a14-9f4f-c7131a59e231" (UID: "e97a314b-6891-4a14-9f4f-c7131a59e231"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:33:20.957237 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:20.957210    4559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e97a314b-6891-4a14-9f4f-c7131a59e231-openshift-service-ca\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:33:20.958979 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:20.958955    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e97a314b-6891-4a14-9f4f-c7131a59e231-maas-api-tls" pod "e97a314b-6891-4a14-9f4f-c7131a59e231" (UID: "e97a314b-6891-4a14-9f4f-c7131a59e231"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:33:20.959341 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:20.959318    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e97a314b-6891-4a14-9f4f-c7131a59e231-metrics-certs" pod "e97a314b-6891-4a14-9f4f-c7131a59e231" (UID: "e97a314b-6891-4a14-9f4f-c7131a59e231"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:33:20.959425 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:20.959354    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e97a314b-6891-4a14-9f4f-c7131a59e231-kube-api-access-dk7p5" pod "e97a314b-6891-4a14-9f4f-c7131a59e231" (UID: "e97a314b-6891-4a14-9f4f-c7131a59e231"). InnerVolumeSpecName "kube-api-access-dk7p5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:33:20.977436 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:20.977411    4559 patch_prober.go:29] interesting pod/e2e-extmod-b-90cda9-openshift-default-57f8b6f957-b5jzx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 13:33:20.977524 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:20.977471    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-90cda9-openshift-default-57f8b6f957-b5jzx" podUID="bb3c761d-8641-496c-9d3a-531e8117e7dd" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 13:33:21.059141 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:21.059099    4559 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e97a314b-6891-4a14-9f4f-c7131a59e231-metrics-certs\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:33:21.059141 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:21.059139    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e97a314b-6891-4a14-9f4f-c7131a59e231-maas-api-tls\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:33:21.059402 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:21.059155    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dk7p5\" (UniqueName: \"kubernetes.io/projected/e97a314b-6891-4a14-9f4f-c7131a59e231-kube-api-access-dk7p5\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:33:21.118223 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:21.118192    4559 generic.go:356] "Generic (PLEG): container finished" podID="e97a314b-6891-4a14-9f4f-c7131a59e231" containerID="d20c487cb4f07ea33ee2373c66b22491e0e2a889fc56abf45850d777988d848f" exitCode=0
Nov 02 13:33:21.118425 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:21.118240    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-25f7a6-595fd6dd77-72r7x"
Nov 02 13:33:21.118425 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:21.118276    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-25f7a6-595fd6dd77-72r7x" event={"ID":"e97a314b-6891-4a14-9f4f-c7131a59e231","Type":"ContainerDied","Data":"d20c487cb4f07ea33ee2373c66b22491e0e2a889fc56abf45850d777988d848f"}
Nov 02 13:33:21.118425 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:21.118341    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-25f7a6-595fd6dd77-72r7x" event={"ID":"e97a314b-6891-4a14-9f4f-c7131a59e231","Type":"ContainerDied","Data":"d4cf84fb4166b0af9c3dc52ab9e69fbe3de435e97b9a51c26b9934f85506671a"}
Nov 02 13:33:21.118425 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:21.118369    4559 scope.go:122] "RemoveContainer" containerID="d20c487cb4f07ea33ee2373c66b22491e0e2a889fc56abf45850d777988d848f"
Nov 02 13:33:21.128886 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:21.128868    4559 scope.go:122] "RemoveContainer" containerID="d20c487cb4f07ea33ee2373c66b22491e0e2a889fc56abf45850d777988d848f"
Nov 02 13:33:21.129174 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:33:21.129148    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d20c487cb4f07ea33ee2373c66b22491e0e2a889fc56abf45850d777988d848f\": container with ID starting with d20c487cb4f07ea33ee2373c66b22491e0e2a889fc56abf45850d777988d848f not found: ID does not exist" containerID="d20c487cb4f07ea33ee2373c66b22491e0e2a889fc56abf45850d777988d848f"
Nov 02 13:33:21.129221 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:21.129183    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d20c487cb4f07ea33ee2373c66b22491e0e2a889fc56abf45850d777988d848f"} err="failed to get container status \"d20c487cb4f07ea33ee2373c66b22491e0e2a889fc56abf45850d777988d848f\": rpc error: code = NotFound desc = could not find container \"d20c487cb4f07ea33ee2373c66b22491e0e2a889fc56abf45850d777988d848f\": container with ID starting with d20c487cb4f07ea33ee2373c66b22491e0e2a889fc56abf45850d777988d848f not found: ID does not exist"
Nov 02 13:33:21.170970 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:21.170944    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-25f7a6-595fd6dd77-72r7x"]
Nov 02 13:33:21.176649 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:21.176617    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-25f7a6-595fd6dd77-72r7x"]
Nov 02 13:33:21.893090 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:21.893056    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e97a314b-6891-4a14-9f4f-c7131a59e231" path="/var/lib/kubelet/pods/e97a314b-6891-4a14-9f4f-c7131a59e231/volumes"
Nov 02 13:33:23.076380 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:23.076358    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-90cda9-openshift-default-57f8b6f957-b5jzx"
Nov 02 13:33:23.132721 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:23.132624    4559 generic.go:356] "Generic (PLEG): container finished" podID="bb3c761d-8641-496c-9d3a-531e8117e7dd" containerID="e59de548036ec32daed50a5119084a5473e6be9275af17787db33e25b4b53425" exitCode=0
Nov 02 13:33:23.132721 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:23.132695    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-90cda9-openshift-default-57f8b6f957-b5jzx"
Nov 02 13:33:23.132937 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:23.132712    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-90cda9-openshift-default-57f8b6f957-b5jzx" event={"ID":"bb3c761d-8641-496c-9d3a-531e8117e7dd","Type":"ContainerDied","Data":"e59de548036ec32daed50a5119084a5473e6be9275af17787db33e25b4b53425"}
Nov 02 13:33:23.132937 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:23.132776    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-90cda9-openshift-default-57f8b6f957-b5jzx" event={"ID":"bb3c761d-8641-496c-9d3a-531e8117e7dd","Type":"ContainerDied","Data":"0c83a6053ae23e43970b631436b163c787562ceb20c18eb8edd097fa1c11a4a3"}
Nov 02 13:33:23.132937 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:23.132811    4559 scope.go:122] "RemoveContainer" containerID="e59de548036ec32daed50a5119084a5473e6be9275af17787db33e25b4b53425"
Nov 02 13:33:23.143265 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:23.143237    4559 scope.go:122] "RemoveContainer" containerID="d421f86a3e5b5b24f5e2908085ec94c480c14b86b84e4c22495f03c365aa50fb"
Nov 02 13:33:23.155922 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:23.155901    4559 scope.go:122] "RemoveContainer" containerID="e59de548036ec32daed50a5119084a5473e6be9275af17787db33e25b4b53425"
Nov 02 13:33:23.156226 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:33:23.156207    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e59de548036ec32daed50a5119084a5473e6be9275af17787db33e25b4b53425\": container with ID starting with e59de548036ec32daed50a5119084a5473e6be9275af17787db33e25b4b53425 not found: ID does not exist" containerID="e59de548036ec32daed50a5119084a5473e6be9275af17787db33e25b4b53425"
Nov 02 13:33:23.156297 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:23.156232    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e59de548036ec32daed50a5119084a5473e6be9275af17787db33e25b4b53425"} err="failed to get container status \"e59de548036ec32daed50a5119084a5473e6be9275af17787db33e25b4b53425\": rpc error: code = NotFound desc = could not find container \"e59de548036ec32daed50a5119084a5473e6be9275af17787db33e25b4b53425\": container with ID starting with e59de548036ec32daed50a5119084a5473e6be9275af17787db33e25b4b53425 not found: ID does not exist"
Nov 02 13:33:23.156297 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:23.156248    4559 scope.go:122] "RemoveContainer" containerID="d421f86a3e5b5b24f5e2908085ec94c480c14b86b84e4c22495f03c365aa50fb"
Nov 02 13:33:23.156514 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:33:23.156484    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d421f86a3e5b5b24f5e2908085ec94c480c14b86b84e4c22495f03c365aa50fb\": container with ID starting with d421f86a3e5b5b24f5e2908085ec94c480c14b86b84e4c22495f03c365aa50fb not found: ID does not exist" containerID="d421f86a3e5b5b24f5e2908085ec94c480c14b86b84e4c22495f03c365aa50fb"
Nov 02 13:33:23.156614 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:23.156520    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d421f86a3e5b5b24f5e2908085ec94c480c14b86b84e4c22495f03c365aa50fb"} err="failed to get container status \"d421f86a3e5b5b24f5e2908085ec94c480c14b86b84e4c22495f03c365aa50fb\": rpc error: code = NotFound desc = could not find container \"d421f86a3e5b5b24f5e2908085ec94c480c14b86b84e4c22495f03c365aa50fb\": container with ID starting with d421f86a3e5b5b24f5e2908085ec94c480c14b86b84e4c22495f03c365aa50fb not found: ID does not exist"
Nov 02 13:33:23.179999 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:23.179976    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bb3c761d-8641-496c-9d3a-531e8117e7dd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb3c761d-8641-496c-9d3a-531e8117e7dd-istio-podinfo\") pod \"bb3c761d-8641-496c-9d3a-531e8117e7dd\" (UID: \"bb3c761d-8641-496c-9d3a-531e8117e7dd\") "
Nov 02 13:33:23.180124 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:23.180014    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb3c761d-8641-496c-9d3a-531e8117e7dd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb3c761d-8641-496c-9d3a-531e8117e7dd-workload-certs\") pod \"bb3c761d-8641-496c-9d3a-531e8117e7dd\" (UID: \"bb3c761d-8641-496c-9d3a-531e8117e7dd\") "
Nov 02 13:33:23.180124 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:23.180082    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb3c761d-8641-496c-9d3a-531e8117e7dd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb3c761d-8641-496c-9d3a-531e8117e7dd-workload-socket\") pod \"bb3c761d-8641-496c-9d3a-531e8117e7dd\" (UID: \"bb3c761d-8641-496c-9d3a-531e8117e7dd\") "
Nov 02 13:33:23.180124 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:23.180108    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb3c761d-8641-496c-9d3a-531e8117e7dd-istio-token\" (UniqueName: \"kubernetes.io/projected/bb3c761d-8641-496c-9d3a-531e8117e7dd-istio-token\") pod \"bb3c761d-8641-496c-9d3a-531e8117e7dd\" (UID: \"bb3c761d-8641-496c-9d3a-531e8117e7dd\") "
Nov 02 13:33:23.180299 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:23.180128    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb3c761d-8641-496c-9d3a-531e8117e7dd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb3c761d-8641-496c-9d3a-531e8117e7dd-istiod-ca-cert\") pod \"bb3c761d-8641-496c-9d3a-531e8117e7dd\" (UID: \"bb3c761d-8641-496c-9d3a-531e8117e7dd\") "
Nov 02 13:33:23.180299 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:23.180172    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb3c761d-8641-496c-9d3a-531e8117e7dd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb3c761d-8641-496c-9d3a-531e8117e7dd-istio-envoy\") pod \"bb3c761d-8641-496c-9d3a-531e8117e7dd\" (UID: \"bb3c761d-8641-496c-9d3a-531e8117e7dd\") "
Nov 02 13:33:23.180299 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:23.180210    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb3c761d-8641-496c-9d3a-531e8117e7dd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb3c761d-8641-496c-9d3a-531e8117e7dd-credential-socket\") pod \"bb3c761d-8641-496c-9d3a-531e8117e7dd\" (UID: \"bb3c761d-8641-496c-9d3a-531e8117e7dd\") "
Nov 02 13:33:23.180299 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:23.180237    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb3c761d-8641-496c-9d3a-531e8117e7dd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb3c761d-8641-496c-9d3a-531e8117e7dd-istio-data\") pod \"bb3c761d-8641-496c-9d3a-531e8117e7dd\" (UID: \"bb3c761d-8641-496c-9d3a-531e8117e7dd\") "
Nov 02 13:33:23.180531 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:23.180357    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb3c761d-8641-496c-9d3a-531e8117e7dd-workload-socket" pod "bb3c761d-8641-496c-9d3a-531e8117e7dd" (UID: "bb3c761d-8641-496c-9d3a-531e8117e7dd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:33:23.180531 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:23.180378    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb3c761d-8641-496c-9d3a-531e8117e7dd-workload-certs" pod "bb3c761d-8641-496c-9d3a-531e8117e7dd" (UID: "bb3c761d-8641-496c-9d3a-531e8117e7dd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:33:23.180531 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:23.180395    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb3c761d-8641-496c-9d3a-531e8117e7dd-kube-api-access-qdwxx\" (UniqueName: \"kubernetes.io/projected/bb3c761d-8641-496c-9d3a-531e8117e7dd-kube-api-access-qdwxx\") pod \"bb3c761d-8641-496c-9d3a-531e8117e7dd\" (UID: \"bb3c761d-8641-496c-9d3a-531e8117e7dd\") "
Nov 02 13:33:23.180723 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:23.180565    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb3c761d-8641-496c-9d3a-531e8117e7dd-istiod-ca-cert" pod "bb3c761d-8641-496c-9d3a-531e8117e7dd" (UID: "bb3c761d-8641-496c-9d3a-531e8117e7dd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:33:23.180723 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:23.180612    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb3c761d-8641-496c-9d3a-531e8117e7dd-credential-socket" pod "bb3c761d-8641-496c-9d3a-531e8117e7dd" (UID: "bb3c761d-8641-496c-9d3a-531e8117e7dd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:33:23.180723 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:23.180631    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb3c761d-8641-496c-9d3a-531e8117e7dd-istio-data" pod "bb3c761d-8641-496c-9d3a-531e8117e7dd" (UID: "bb3c761d-8641-496c-9d3a-531e8117e7dd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:33:23.181035 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:23.181012    4559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb3c761d-8641-496c-9d3a-531e8117e7dd-workload-certs\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:33:23.181108 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:23.181041    4559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb3c761d-8641-496c-9d3a-531e8117e7dd-workload-socket\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:33:23.181108 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:23.181056    4559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb3c761d-8641-496c-9d3a-531e8117e7dd-istiod-ca-cert\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:33:23.181108 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:23.181071    4559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb3c761d-8641-496c-9d3a-531e8117e7dd-credential-socket\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:33:23.181108 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:23.181085    4559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb3c761d-8641-496c-9d3a-531e8117e7dd-istio-data\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:33:23.182395 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:23.182371    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bb3c761d-8641-496c-9d3a-531e8117e7dd-istio-podinfo" pod "bb3c761d-8641-496c-9d3a-531e8117e7dd" (UID: "bb3c761d-8641-496c-9d3a-531e8117e7dd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 13:33:23.182602 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:23.182584    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb3c761d-8641-496c-9d3a-531e8117e7dd-istio-envoy" pod "bb3c761d-8641-496c-9d3a-531e8117e7dd" (UID: "bb3c761d-8641-496c-9d3a-531e8117e7dd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:33:23.182752 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:23.182732    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb3c761d-8641-496c-9d3a-531e8117e7dd-kube-api-access-qdwxx" pod "bb3c761d-8641-496c-9d3a-531e8117e7dd" (UID: "bb3c761d-8641-496c-9d3a-531e8117e7dd"). InnerVolumeSpecName "kube-api-access-qdwxx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:33:23.182817 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:23.182772    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb3c761d-8641-496c-9d3a-531e8117e7dd-istio-token" pod "bb3c761d-8641-496c-9d3a-531e8117e7dd" (UID: "bb3c761d-8641-496c-9d3a-531e8117e7dd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:33:23.282273 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:23.282245    4559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb3c761d-8641-496c-9d3a-531e8117e7dd-istio-envoy\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:33:23.282424 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:23.282277    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qdwxx\" (UniqueName: \"kubernetes.io/projected/bb3c761d-8641-496c-9d3a-531e8117e7dd-kube-api-access-qdwxx\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:33:23.282424 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:23.282291    4559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb3c761d-8641-496c-9d3a-531e8117e7dd-istio-podinfo\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:33:23.282424 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:23.282302    4559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb3c761d-8641-496c-9d3a-531e8117e7dd-istio-token\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:33:23.482389 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:23.482357    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-90cda9-openshift-default-57f8b6f957-b5jzx"]
Nov 02 13:33:23.494850 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:23.494825    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-90cda9-openshift-default-57f8b6f957-b5jzx"]
Nov 02 13:33:23.892134 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:23.892061    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb3c761d-8641-496c-9d3a-531e8117e7dd" path="/var/lib/kubelet/pods/bb3c761d-8641-496c-9d3a-531e8117e7dd/volumes"
Nov 02 13:33:24.620705 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:24.620674    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a6f2c0-d46bdcf55-7nw8f"]
Nov 02 13:33:25.741706 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:25.741668    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"]
Nov 02 13:33:26.150776 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:26.150696    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w0-43be82/sub-test-model-43be82-kserve-5dc888468f-9lf8t"
Nov 02 13:33:26.164552 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:26.164519    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w0-43be82/sub-test-model-43be82-kserve-5dc888468f-9lf8t"
Nov 02 13:33:27.162441 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:27.162405    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w0-43be82/sub-test-model-43be82-kserve-5dc888468f-9lf8t"
Nov 02 13:33:29.533246 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:29.533212    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-smxns"]
Nov 02 13:33:29.533716 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:29.533433    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-smxns" podUID="d4eca390-56ae-40c6-a669-de202c2483de" containerName="istio-proxy" containerID="cri-o://d8d47f9a5cc855553a22756baceb40d8528085583a6cd8e324c40ece47aec249" gracePeriod=30
Nov 02 13:33:29.621389 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:29.621347    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a6f2c0-d46bdcf55-7nw8f" podUID="da10c6a3-b295-4997-84fc-ffe8ebb90061" containerName="payload-pre-processing" containerID="cri-o://cc72ae59767debf7cdb18e6bb565deeba6674c7c578e00332ae45d2325d9480c" gracePeriod=25
Nov 02 13:33:31.607232 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:31.607192    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a6f2c0-d46bdcf55-7nw8f" podUID="da10c6a3-b295-4997-84fc-ffe8ebb90061" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 13:33:34.801148 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:34.801087    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-smxns"
Nov 02 13:33:34.822758 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:34.822730    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4eca390-56ae-40c6-a669-de202c2483de-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4eca390-56ae-40c6-a669-de202c2483de-workload-certs\") pod \"d4eca390-56ae-40c6-a669-de202c2483de\" (UID: \"d4eca390-56ae-40c6-a669-de202c2483de\") "
Nov 02 13:33:34.822910 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:34.822764    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4eca390-56ae-40c6-a669-de202c2483de-istio-token\" (UniqueName: \"kubernetes.io/projected/d4eca390-56ae-40c6-a669-de202c2483de-istio-token\") pod \"d4eca390-56ae-40c6-a669-de202c2483de\" (UID: \"d4eca390-56ae-40c6-a669-de202c2483de\") "
Nov 02 13:33:34.822910 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:34.822806    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d4eca390-56ae-40c6-a669-de202c2483de-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4eca390-56ae-40c6-a669-de202c2483de-istio-podinfo\") pod \"d4eca390-56ae-40c6-a669-de202c2483de\" (UID: \"d4eca390-56ae-40c6-a669-de202c2483de\") "
Nov 02 13:33:34.822910 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:34.822875    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4eca390-56ae-40c6-a669-de202c2483de-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4eca390-56ae-40c6-a669-de202c2483de-istiod-ca-cert\") pod \"d4eca390-56ae-40c6-a669-de202c2483de\" (UID: \"d4eca390-56ae-40c6-a669-de202c2483de\") "
Nov 02 13:33:34.823099 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:34.823003    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4eca390-56ae-40c6-a669-de202c2483de-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4eca390-56ae-40c6-a669-de202c2483de-istio-data\") pod \"d4eca390-56ae-40c6-a669-de202c2483de\" (UID: \"d4eca390-56ae-40c6-a669-de202c2483de\") "
Nov 02 13:33:34.823099 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:34.823030    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4eca390-56ae-40c6-a669-de202c2483de-workload-certs" pod "d4eca390-56ae-40c6-a669-de202c2483de" (UID: "d4eca390-56ae-40c6-a669-de202c2483de"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:33:34.823099 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:34.823062    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4eca390-56ae-40c6-a669-de202c2483de-kube-api-access-s6q86\" (UniqueName: \"kubernetes.io/projected/d4eca390-56ae-40c6-a669-de202c2483de-kube-api-access-s6q86\") pod \"d4eca390-56ae-40c6-a669-de202c2483de\" (UID: \"d4eca390-56ae-40c6-a669-de202c2483de\") "
Nov 02 13:33:34.823274 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:34.823109    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4eca390-56ae-40c6-a669-de202c2483de-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4eca390-56ae-40c6-a669-de202c2483de-istio-envoy\") pod \"d4eca390-56ae-40c6-a669-de202c2483de\" (UID: \"d4eca390-56ae-40c6-a669-de202c2483de\") "
Nov 02 13:33:34.823274 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:34.823222    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4eca390-56ae-40c6-a669-de202c2483de-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4eca390-56ae-40c6-a669-de202c2483de-credential-socket\") pod \"d4eca390-56ae-40c6-a669-de202c2483de\" (UID: \"d4eca390-56ae-40c6-a669-de202c2483de\") "
Nov 02 13:33:34.823274 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:34.823227    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4eca390-56ae-40c6-a669-de202c2483de-istio-data" pod "d4eca390-56ae-40c6-a669-de202c2483de" (UID: "d4eca390-56ae-40c6-a669-de202c2483de"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:33:34.823274 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:34.823252    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4eca390-56ae-40c6-a669-de202c2483de-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4eca390-56ae-40c6-a669-de202c2483de-workload-socket\") pod \"d4eca390-56ae-40c6-a669-de202c2483de\" (UID: \"d4eca390-56ae-40c6-a669-de202c2483de\") "
Nov 02 13:33:34.823274 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:34.823254    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4eca390-56ae-40c6-a669-de202c2483de-istiod-ca-cert" pod "d4eca390-56ae-40c6-a669-de202c2483de" (UID: "d4eca390-56ae-40c6-a669-de202c2483de"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:33:34.823556 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:34.823472    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4eca390-56ae-40c6-a669-de202c2483de-credential-socket" pod "d4eca390-56ae-40c6-a669-de202c2483de" (UID: "d4eca390-56ae-40c6-a669-de202c2483de"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:33:34.823623 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:34.823599    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4eca390-56ae-40c6-a669-de202c2483de-workload-socket" pod "d4eca390-56ae-40c6-a669-de202c2483de" (UID: "d4eca390-56ae-40c6-a669-de202c2483de"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:33:34.823825 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:34.823806    4559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4eca390-56ae-40c6-a669-de202c2483de-workload-certs\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:33:34.823934 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:34.823829    4559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4eca390-56ae-40c6-a669-de202c2483de-istiod-ca-cert\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:33:34.823934 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:34.823842    4559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4eca390-56ae-40c6-a669-de202c2483de-istio-data\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:33:34.823934 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:34.823854    4559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4eca390-56ae-40c6-a669-de202c2483de-credential-socket\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:33:34.823934 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:34.823867    4559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4eca390-56ae-40c6-a669-de202c2483de-workload-socket\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:33:34.825361 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:34.825335    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4eca390-56ae-40c6-a669-de202c2483de-istio-envoy" pod "d4eca390-56ae-40c6-a669-de202c2483de" (UID: "d4eca390-56ae-40c6-a669-de202c2483de"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:33:34.825449 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:34.825400    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d4eca390-56ae-40c6-a669-de202c2483de-istio-podinfo" pod "d4eca390-56ae-40c6-a669-de202c2483de" (UID: "d4eca390-56ae-40c6-a669-de202c2483de"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 13:33:34.825572 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:34.825547    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4eca390-56ae-40c6-a669-de202c2483de-kube-api-access-s6q86" pod "d4eca390-56ae-40c6-a669-de202c2483de" (UID: "d4eca390-56ae-40c6-a669-de202c2483de"). InnerVolumeSpecName "kube-api-access-s6q86". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:33:34.825633 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:34.825584    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4eca390-56ae-40c6-a669-de202c2483de-istio-token" pod "d4eca390-56ae-40c6-a669-de202c2483de" (UID: "d4eca390-56ae-40c6-a669-de202c2483de"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:33:34.925375 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:34.925233    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s6q86\" (UniqueName: \"kubernetes.io/projected/d4eca390-56ae-40c6-a669-de202c2483de-kube-api-access-s6q86\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:33:34.925375 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:34.925266    4559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4eca390-56ae-40c6-a669-de202c2483de-istio-envoy\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:33:34.925375 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:34.925281    4559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4eca390-56ae-40c6-a669-de202c2483de-istio-token\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:33:34.925375 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:34.925294    4559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4eca390-56ae-40c6-a669-de202c2483de-istio-podinfo\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:33:35.208944 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:35.208913    4559 generic.go:356] "Generic (PLEG): container finished" podID="d4eca390-56ae-40c6-a669-de202c2483de" containerID="d8d47f9a5cc855553a22756baceb40d8528085583a6cd8e324c40ece47aec249" exitCode=0
Nov 02 13:33:35.209139 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:35.208960    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-smxns"
Nov 02 13:33:35.209139 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:35.208997    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-smxns" event={"ID":"d4eca390-56ae-40c6-a669-de202c2483de","Type":"ContainerDied","Data":"d8d47f9a5cc855553a22756baceb40d8528085583a6cd8e324c40ece47aec249"}
Nov 02 13:33:35.209139 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:35.209056    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-smxns" event={"ID":"d4eca390-56ae-40c6-a669-de202c2483de","Type":"ContainerDied","Data":"35f7eee78639776f3e926101807cd6ae214f5bc67eb0a649e436ee0e5996547d"}
Nov 02 13:33:35.209139 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:35.209084    4559 scope.go:122] "RemoveContainer" containerID="d8d47f9a5cc855553a22756baceb40d8528085583a6cd8e324c40ece47aec249"
Nov 02 13:33:35.219488 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:35.219470    4559 scope.go:122] "RemoveContainer" containerID="d8d47f9a5cc855553a22756baceb40d8528085583a6cd8e324c40ece47aec249"
Nov 02 13:33:35.219716 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:33:35.219697    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8d47f9a5cc855553a22756baceb40d8528085583a6cd8e324c40ece47aec249\": container with ID starting with d8d47f9a5cc855553a22756baceb40d8528085583a6cd8e324c40ece47aec249 not found: ID does not exist" containerID="d8d47f9a5cc855553a22756baceb40d8528085583a6cd8e324c40ece47aec249"
Nov 02 13:33:35.219792 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:35.219724    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8d47f9a5cc855553a22756baceb40d8528085583a6cd8e324c40ece47aec249"} err="failed to get container status \"d8d47f9a5cc855553a22756baceb40d8528085583a6cd8e324c40ece47aec249\": rpc error: code = NotFound desc = could not find container \"d8d47f9a5cc855553a22756baceb40d8528085583a6cd8e324c40ece47aec249\": container with ID starting with d8d47f9a5cc855553a22756baceb40d8528085583a6cd8e324c40ece47aec249 not found: ID does not exist"
Nov 02 13:33:35.262546 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:35.262512    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-smxns"]
Nov 02 13:33:35.269537 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:35.269514    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-smxns"]
Nov 02 13:33:35.892087 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:35.892055    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4eca390-56ae-40c6-a669-de202c2483de" path="/var/lib/kubelet/pods/d4eca390-56ae-40c6-a669-de202c2483de/volumes"
Nov 02 13:33:41.606892 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:41.606854    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a6f2c0-d46bdcf55-7nw8f" podUID="da10c6a3-b295-4997-84fc-ffe8ebb90061" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 13:33:45.086667 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:33:45.086610    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqcmj" containerName="extractor"
Nov 02 13:33:49.878313 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:49.878254    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a6f2c0-d46bdcf55-7nw8f"
Nov 02 13:33:49.983811 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:49.983770    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da10c6a3-b295-4997-84fc-ffe8ebb90061-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da10c6a3-b295-4997-84fc-ffe8ebb90061-plugins-config-volume\") pod \"da10c6a3-b295-4997-84fc-ffe8ebb90061\" (UID: \"da10c6a3-b295-4997-84fc-ffe8ebb90061\") "
Nov 02 13:33:49.983990 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:49.983907    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da10c6a3-b295-4997-84fc-ffe8ebb90061-kube-api-access-xbsx9\" (UniqueName: \"kubernetes.io/projected/da10c6a3-b295-4997-84fc-ffe8ebb90061-kube-api-access-xbsx9\") pod \"da10c6a3-b295-4997-84fc-ffe8ebb90061\" (UID: \"da10c6a3-b295-4997-84fc-ffe8ebb90061\") "
Nov 02 13:33:49.984194 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:49.984167    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da10c6a3-b295-4997-84fc-ffe8ebb90061-plugins-config-volume" pod "da10c6a3-b295-4997-84fc-ffe8ebb90061" (UID: "da10c6a3-b295-4997-84fc-ffe8ebb90061"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:33:49.984289 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:49.984270    4559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da10c6a3-b295-4997-84fc-ffe8ebb90061-plugins-config-volume\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:33:49.985956 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:49.985935    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da10c6a3-b295-4997-84fc-ffe8ebb90061-kube-api-access-xbsx9" pod "da10c6a3-b295-4997-84fc-ffe8ebb90061" (UID: "da10c6a3-b295-4997-84fc-ffe8ebb90061"). InnerVolumeSpecName "kube-api-access-xbsx9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:33:50.085179 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:50.085144    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xbsx9\" (UniqueName: \"kubernetes.io/projected/da10c6a3-b295-4997-84fc-ffe8ebb90061-kube-api-access-xbsx9\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:33:50.291411 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:50.291376    4559 generic.go:356] "Generic (PLEG): container finished" podID="da10c6a3-b295-4997-84fc-ffe8ebb90061" containerID="cc72ae59767debf7cdb18e6bb565deeba6674c7c578e00332ae45d2325d9480c" exitCode=0
Nov 02 13:33:50.291608 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:50.291418    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a6f2c0-d46bdcf55-7nw8f"
Nov 02 13:33:50.291608 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:50.291462    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a6f2c0-d46bdcf55-7nw8f" event={"ID":"da10c6a3-b295-4997-84fc-ffe8ebb90061","Type":"ContainerDied","Data":"cc72ae59767debf7cdb18e6bb565deeba6674c7c578e00332ae45d2325d9480c"}
Nov 02 13:33:50.291608 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:50.291515    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a6f2c0-d46bdcf55-7nw8f" event={"ID":"da10c6a3-b295-4997-84fc-ffe8ebb90061","Type":"ContainerDied","Data":"91d1e848e333f8376301d59e7a8df48327583ab99364547a3da981760d95c957"}
Nov 02 13:33:50.291608 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:50.291537    4559 scope.go:122] "RemoveContainer" containerID="cc72ae59767debf7cdb18e6bb565deeba6674c7c578e00332ae45d2325d9480c"
Nov 02 13:33:50.301549 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:50.301533    4559 scope.go:122] "RemoveContainer" containerID="cc72ae59767debf7cdb18e6bb565deeba6674c7c578e00332ae45d2325d9480c"
Nov 02 13:33:50.301840 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:33:50.301813    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc72ae59767debf7cdb18e6bb565deeba6674c7c578e00332ae45d2325d9480c\": container with ID starting with cc72ae59767debf7cdb18e6bb565deeba6674c7c578e00332ae45d2325d9480c not found: ID does not exist" containerID="cc72ae59767debf7cdb18e6bb565deeba6674c7c578e00332ae45d2325d9480c"
Nov 02 13:33:50.301921 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:50.301841    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc72ae59767debf7cdb18e6bb565deeba6674c7c578e00332ae45d2325d9480c"} err="failed to get container status \"cc72ae59767debf7cdb18e6bb565deeba6674c7c578e00332ae45d2325d9480c\": rpc error: code = NotFound desc = could not find container \"cc72ae59767debf7cdb18e6bb565deeba6674c7c578e00332ae45d2325d9480c\": container with ID starting with cc72ae59767debf7cdb18e6bb565deeba6674c7c578e00332ae45d2325d9480c not found: ID does not exist"
Nov 02 13:33:50.345986 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:50.345962    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a6f2c0-d46bdcf55-7nw8f"]
Nov 02 13:33:50.350160 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:50.350143    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a6f2c0-d46bdcf55-7nw8f"]
Nov 02 13:33:51.892849 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:51.892818    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da10c6a3-b295-4997-84fc-ffe8ebb90061" path="/var/lib/kubelet/pods/da10c6a3-b295-4997-84fc-ffe8ebb90061/volumes"
Nov 02 13:33:56.240332 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:56.240295    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-1aa18c-openshift-default-7d7f44f669-xz5m4"]
Nov 02 13:33:56.240785 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:56.240555    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-1aa18c-openshift-default-7d7f44f669-xz5m4" podUID="8debb7bd-d249-47d9-8776-f2be8f0d4f27" containerName="istio-proxy" containerID="cri-o://7e6c60c6cf892fd38b50a857246c2b4ee3fd1cf261261951a149d41e12dfb557" gracePeriod=30
Nov 02 13:33:58.111406 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:58.111365    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-ed56b9-7bc9f55c5-qjvr7"]
Nov 02 13:33:58.180134 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:58.180099    4559 patch_prober.go:29] interesting pod/e2e-shared-b-w1-1aa18c-openshift-default-7d7f44f669-xz5m4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 13:33:58.180307 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:58.180162    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w1-1aa18c-openshift-default-7d7f44f669-xz5m4" podUID="8debb7bd-d249-47d9-8776-f2be8f0d4f27" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 13:33:59.841106 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:33:59.841066    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"]
Nov 02 13:34:01.360156 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:01.360125    4559 generic.go:356] "Generic (PLEG): container finished" podID="8debb7bd-d249-47d9-8776-f2be8f0d4f27" containerID="7e6c60c6cf892fd38b50a857246c2b4ee3fd1cf261261951a149d41e12dfb557" exitCode=0
Nov 02 13:34:01.360468 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:01.360192    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-1aa18c-openshift-default-7d7f44f669-xz5m4" event={"ID":"8debb7bd-d249-47d9-8776-f2be8f0d4f27","Type":"ContainerDied","Data":"7e6c60c6cf892fd38b50a857246c2b4ee3fd1cf261261951a149d41e12dfb557"}
Nov 02 13:34:01.360468 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:01.360264    4559 scope.go:122] "RemoveContainer" containerID="66a2714cf5cd54c62bed7a6eb34297b3139d6f16a0ad9d88a2404ca6ef03071e"
Nov 02 13:34:01.503479 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:01.503426    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-1aa18c-openshift-default-7d7f44f669-xz5m4"
Nov 02 13:34:01.607586 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:01.607558    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8debb7bd-d249-47d9-8776-f2be8f0d4f27-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8debb7bd-d249-47d9-8776-f2be8f0d4f27-istio-data\") pod \"8debb7bd-d249-47d9-8776-f2be8f0d4f27\" (UID: \"8debb7bd-d249-47d9-8776-f2be8f0d4f27\") "
Nov 02 13:34:01.607805 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:01.607614    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8debb7bd-d249-47d9-8776-f2be8f0d4f27-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8debb7bd-d249-47d9-8776-f2be8f0d4f27-istio-podinfo\") pod \"8debb7bd-d249-47d9-8776-f2be8f0d4f27\" (UID: \"8debb7bd-d249-47d9-8776-f2be8f0d4f27\") "
Nov 02 13:34:01.607805 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:01.607663    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8debb7bd-d249-47d9-8776-f2be8f0d4f27-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8debb7bd-d249-47d9-8776-f2be8f0d4f27-workload-certs\") pod \"8debb7bd-d249-47d9-8776-f2be8f0d4f27\" (UID: \"8debb7bd-d249-47d9-8776-f2be8f0d4f27\") "
Nov 02 13:34:01.607805 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:01.607691    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8debb7bd-d249-47d9-8776-f2be8f0d4f27-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8debb7bd-d249-47d9-8776-f2be8f0d4f27-istiod-ca-cert\") pod \"8debb7bd-d249-47d9-8776-f2be8f0d4f27\" (UID: \"8debb7bd-d249-47d9-8776-f2be8f0d4f27\") "
Nov 02 13:34:01.607805 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:01.607761    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8debb7bd-d249-47d9-8776-f2be8f0d4f27-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8debb7bd-d249-47d9-8776-f2be8f0d4f27-workload-socket\") pod \"8debb7bd-d249-47d9-8776-f2be8f0d4f27\" (UID: \"8debb7bd-d249-47d9-8776-f2be8f0d4f27\") "
Nov 02 13:34:01.607805 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:01.607789    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8debb7bd-d249-47d9-8776-f2be8f0d4f27-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8debb7bd-d249-47d9-8776-f2be8f0d4f27-istio-envoy\") pod \"8debb7bd-d249-47d9-8776-f2be8f0d4f27\" (UID: \"8debb7bd-d249-47d9-8776-f2be8f0d4f27\") "
Nov 02 13:34:01.608103 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:01.607826    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8debb7bd-d249-47d9-8776-f2be8f0d4f27-kube-api-access-lp29f\" (UniqueName: \"kubernetes.io/projected/8debb7bd-d249-47d9-8776-f2be8f0d4f27-kube-api-access-lp29f\") pod \"8debb7bd-d249-47d9-8776-f2be8f0d4f27\" (UID: \"8debb7bd-d249-47d9-8776-f2be8f0d4f27\") "
Nov 02 13:34:01.608103 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:01.607831    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8debb7bd-d249-47d9-8776-f2be8f0d4f27-istio-data" pod "8debb7bd-d249-47d9-8776-f2be8f0d4f27" (UID: "8debb7bd-d249-47d9-8776-f2be8f0d4f27"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:34:01.608103 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:01.607934    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8debb7bd-d249-47d9-8776-f2be8f0d4f27-workload-certs" pod "8debb7bd-d249-47d9-8776-f2be8f0d4f27" (UID: "8debb7bd-d249-47d9-8776-f2be8f0d4f27"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:34:01.608103 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:01.607940    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8debb7bd-d249-47d9-8776-f2be8f0d4f27-istio-token\" (UniqueName: \"kubernetes.io/projected/8debb7bd-d249-47d9-8776-f2be8f0d4f27-istio-token\") pod \"8debb7bd-d249-47d9-8776-f2be8f0d4f27\" (UID: \"8debb7bd-d249-47d9-8776-f2be8f0d4f27\") "
Nov 02 13:34:01.608103 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:01.607999    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8debb7bd-d249-47d9-8776-f2be8f0d4f27-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8debb7bd-d249-47d9-8776-f2be8f0d4f27-credential-socket\") pod \"8debb7bd-d249-47d9-8776-f2be8f0d4f27\" (UID: \"8debb7bd-d249-47d9-8776-f2be8f0d4f27\") "
Nov 02 13:34:01.608365 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:01.608127    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8debb7bd-d249-47d9-8776-f2be8f0d4f27-workload-socket" pod "8debb7bd-d249-47d9-8776-f2be8f0d4f27" (UID: "8debb7bd-d249-47d9-8776-f2be8f0d4f27"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:34:01.608365 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:01.608254    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8debb7bd-d249-47d9-8776-f2be8f0d4f27-credential-socket" pod "8debb7bd-d249-47d9-8776-f2be8f0d4f27" (UID: "8debb7bd-d249-47d9-8776-f2be8f0d4f27"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:34:01.608365 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:01.608278    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8debb7bd-d249-47d9-8776-f2be8f0d4f27-istiod-ca-cert" pod "8debb7bd-d249-47d9-8776-f2be8f0d4f27" (UID: "8debb7bd-d249-47d9-8776-f2be8f0d4f27"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:34:01.608531 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:01.608495    4559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8debb7bd-d249-47d9-8776-f2be8f0d4f27-credential-socket\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:34:01.608531 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:01.608526    4559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8debb7bd-d249-47d9-8776-f2be8f0d4f27-istio-data\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:34:01.608686 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:01.608542    4559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8debb7bd-d249-47d9-8776-f2be8f0d4f27-workload-certs\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:34:01.608686 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:01.608554    4559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8debb7bd-d249-47d9-8776-f2be8f0d4f27-istiod-ca-cert\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:34:01.608686 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:01.608566    4559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8debb7bd-d249-47d9-8776-f2be8f0d4f27-workload-socket\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:34:01.609998 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:01.609972    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8debb7bd-d249-47d9-8776-f2be8f0d4f27-istio-podinfo" pod "8debb7bd-d249-47d9-8776-f2be8f0d4f27" (UID: "8debb7bd-d249-47d9-8776-f2be8f0d4f27"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 13:34:01.609998 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:01.609991    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8debb7bd-d249-47d9-8776-f2be8f0d4f27-istio-envoy" pod "8debb7bd-d249-47d9-8776-f2be8f0d4f27" (UID: "8debb7bd-d249-47d9-8776-f2be8f0d4f27"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:34:01.610147 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:01.610012    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8debb7bd-d249-47d9-8776-f2be8f0d4f27-kube-api-access-lp29f" pod "8debb7bd-d249-47d9-8776-f2be8f0d4f27" (UID: "8debb7bd-d249-47d9-8776-f2be8f0d4f27"). InnerVolumeSpecName "kube-api-access-lp29f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:34:01.610201 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:01.610185    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8debb7bd-d249-47d9-8776-f2be8f0d4f27-istio-token" pod "8debb7bd-d249-47d9-8776-f2be8f0d4f27" (UID: "8debb7bd-d249-47d9-8776-f2be8f0d4f27"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:34:01.695335 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:01.695309    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-61e341/distinct-2-w1-61e341-kserve-954cf956f-n7jmg"]
Nov 02 13:34:01.700451 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:01.700428    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-61e341/distinct-w1-61e341-kserve-787cd566f4-cjj8s"]
Nov 02 13:34:01.710219 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:01.710193    4559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8debb7bd-d249-47d9-8776-f2be8f0d4f27-istio-token\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:34:01.710329 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:01.710222    4559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8debb7bd-d249-47d9-8776-f2be8f0d4f27-istio-podinfo\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:34:01.710329 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:01.710236    4559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8debb7bd-d249-47d9-8776-f2be8f0d4f27-istio-envoy\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:34:01.710329 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:01.710250    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lp29f\" (UniqueName: \"kubernetes.io/projected/8debb7bd-d249-47d9-8776-f2be8f0d4f27-kube-api-access-lp29f\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:34:01.710471 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:01.710386    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-61e341/premium-w1-61e341-kserve-746477b99-645g6"]
Nov 02 13:34:02.366685 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:02.366633    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-1aa18c-openshift-default-7d7f44f669-xz5m4"
Nov 02 13:34:02.366685 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:02.366668    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-1aa18c-openshift-default-7d7f44f669-xz5m4" event={"ID":"8debb7bd-d249-47d9-8776-f2be8f0d4f27","Type":"ContainerDied","Data":"ba55c7015f5e24ddbd03269a9ed034111edb4193bb60d2051785143a169e76e5"}
Nov 02 13:34:02.367216 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:02.366724    4559 scope.go:122] "RemoveContainer" containerID="7e6c60c6cf892fd38b50a857246c2b4ee3fd1cf261261951a149d41e12dfb557"
Nov 02 13:34:02.433425 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:02.433397    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-1aa18c-openshift-default-7d7f44f669-xz5m4"]
Nov 02 13:34:02.446451 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:02.446426    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-1aa18c-openshift-default-7d7f44f669-xz5m4"]
Nov 02 13:34:03.112098 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:03.112041    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-ed56b9-7bc9f55c5-qjvr7" podUID="7c89b44f-fa2c-4379-a933-9a8e7d010bfd" containerName="payload-processing" containerID="cri-o://839ab0ddb5712c3a205d8812562e16fb9303e8177aa61abb94a511641adb89e5" gracePeriod=25
Nov 02 13:34:03.892760 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:03.892721    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8debb7bd-d249-47d9-8776-f2be8f0d4f27" path="/var/lib/kubelet/pods/8debb7bd-d249-47d9-8776-f2be8f0d4f27/volumes"
Nov 02 13:34:03.966745 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:03.966711    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ed56b9-5f957cb677-8j2rp"]
Nov 02 13:34:03.966981 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:03.966955    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ed56b9-5f957cb677-8j2rp" podUID="95acd739-ef82-4350-a68c-7a40ae0acbe5" containerName="maas-api" containerID="cri-o://b2367a193246f6c5ac4b1037fe0eea981729682519ea7bc275635d2729097323" gracePeriod=30
Nov 02 13:34:04.224937 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:04.224907    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ed56b9-5f957cb677-8j2rp"
Nov 02 13:34:04.235031 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:04.235006    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/95acd739-ef82-4350-a68c-7a40ae0acbe5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/95acd739-ef82-4350-a68c-7a40ae0acbe5-metrics-certs\") pod \"95acd739-ef82-4350-a68c-7a40ae0acbe5\" (UID: \"95acd739-ef82-4350-a68c-7a40ae0acbe5\") "
Nov 02 13:34:04.235168 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:04.235038    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/95acd739-ef82-4350-a68c-7a40ae0acbe5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/95acd739-ef82-4350-a68c-7a40ae0acbe5-maas-api-tls\") pod \"95acd739-ef82-4350-a68c-7a40ae0acbe5\" (UID: \"95acd739-ef82-4350-a68c-7a40ae0acbe5\") "
Nov 02 13:34:04.235168 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:04.235089    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95acd739-ef82-4350-a68c-7a40ae0acbe5-kube-api-access-wjqgm\" (UniqueName: \"kubernetes.io/projected/95acd739-ef82-4350-a68c-7a40ae0acbe5-kube-api-access-wjqgm\") pod \"95acd739-ef82-4350-a68c-7a40ae0acbe5\" (UID: \"95acd739-ef82-4350-a68c-7a40ae0acbe5\") "
Nov 02 13:34:04.235168 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:04.235134    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95acd739-ef82-4350-a68c-7a40ae0acbe5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/95acd739-ef82-4350-a68c-7a40ae0acbe5-openshift-service-ca\") pod \"95acd739-ef82-4350-a68c-7a40ae0acbe5\" (UID: \"95acd739-ef82-4350-a68c-7a40ae0acbe5\") "
Nov 02 13:34:04.235560 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:04.235534    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95acd739-ef82-4350-a68c-7a40ae0acbe5-openshift-service-ca" pod "95acd739-ef82-4350-a68c-7a40ae0acbe5" (UID: "95acd739-ef82-4350-a68c-7a40ae0acbe5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:34:04.237806 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:04.237751    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95acd739-ef82-4350-a68c-7a40ae0acbe5-metrics-certs" pod "95acd739-ef82-4350-a68c-7a40ae0acbe5" (UID: "95acd739-ef82-4350-a68c-7a40ae0acbe5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:34:04.237933 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:04.237856    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95acd739-ef82-4350-a68c-7a40ae0acbe5-maas-api-tls" pod "95acd739-ef82-4350-a68c-7a40ae0acbe5" (UID: "95acd739-ef82-4350-a68c-7a40ae0acbe5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:34:04.238015 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:04.237950    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95acd739-ef82-4350-a68c-7a40ae0acbe5-kube-api-access-wjqgm" pod "95acd739-ef82-4350-a68c-7a40ae0acbe5" (UID: "95acd739-ef82-4350-a68c-7a40ae0acbe5"). InnerVolumeSpecName "kube-api-access-wjqgm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:34:04.336261 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:04.336214    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wjqgm\" (UniqueName: \"kubernetes.io/projected/95acd739-ef82-4350-a68c-7a40ae0acbe5-kube-api-access-wjqgm\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:34:04.336261 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:04.336258    4559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/95acd739-ef82-4350-a68c-7a40ae0acbe5-openshift-service-ca\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:34:04.336261 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:04.336271    4559 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/95acd739-ef82-4350-a68c-7a40ae0acbe5-metrics-certs\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:34:04.336497 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:04.336280    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/95acd739-ef82-4350-a68c-7a40ae0acbe5-maas-api-tls\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:34:04.379886 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:04.379857    4559 generic.go:356] "Generic (PLEG): container finished" podID="95acd739-ef82-4350-a68c-7a40ae0acbe5" containerID="b2367a193246f6c5ac4b1037fe0eea981729682519ea7bc275635d2729097323" exitCode=0
Nov 02 13:34:04.380039 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:04.379900    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ed56b9-5f957cb677-8j2rp"
Nov 02 13:34:04.380039 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:04.379927    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ed56b9-5f957cb677-8j2rp" event={"ID":"95acd739-ef82-4350-a68c-7a40ae0acbe5","Type":"ContainerDied","Data":"b2367a193246f6c5ac4b1037fe0eea981729682519ea7bc275635d2729097323"}
Nov 02 13:34:04.380039 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:04.379982    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ed56b9-5f957cb677-8j2rp" event={"ID":"95acd739-ef82-4350-a68c-7a40ae0acbe5","Type":"ContainerDied","Data":"1491d35cf638ec42190d9ccffff7f789fe938263d6862596adf1fbf5f9e52ac3"}
Nov 02 13:34:04.380039 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:04.380014    4559 scope.go:122] "RemoveContainer" containerID="b2367a193246f6c5ac4b1037fe0eea981729682519ea7bc275635d2729097323"
Nov 02 13:34:04.390592 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:04.390574    4559 scope.go:122] "RemoveContainer" containerID="b2367a193246f6c5ac4b1037fe0eea981729682519ea7bc275635d2729097323"
Nov 02 13:34:04.390873 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:34:04.390852    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2367a193246f6c5ac4b1037fe0eea981729682519ea7bc275635d2729097323\": container with ID starting with b2367a193246f6c5ac4b1037fe0eea981729682519ea7bc275635d2729097323 not found: ID does not exist" containerID="b2367a193246f6c5ac4b1037fe0eea981729682519ea7bc275635d2729097323"
Nov 02 13:34:04.390993 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:04.390881    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2367a193246f6c5ac4b1037fe0eea981729682519ea7bc275635d2729097323"} err="failed to get container status \"b2367a193246f6c5ac4b1037fe0eea981729682519ea7bc275635d2729097323\": rpc error: code = NotFound desc = could not find container \"b2367a193246f6c5ac4b1037fe0eea981729682519ea7bc275635d2729097323\": container with ID starting with b2367a193246f6c5ac4b1037fe0eea981729682519ea7bc275635d2729097323 not found: ID does not exist"
Nov 02 13:34:04.433529 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:04.433498    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ed56b9-5f957cb677-8j2rp"]
Nov 02 13:34:04.437507 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:04.437477    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ed56b9-5f957cb677-8j2rp"]
Nov 02 13:34:04.739818 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:04.739777    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-ed56b9-7bc9f55c5-qjvr7" podUID="7c89b44f-fa2c-4379-a933-9a8e7d010bfd" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 13:34:05.892876 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:05.892844    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95acd739-ef82-4350-a68c-7a40ae0acbe5" path="/var/lib/kubelet/pods/95acd739-ef82-4350-a68c-7a40ae0acbe5/volumes"
Nov 02 13:34:07.872536 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:07.872488    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-ed56b9-openshift-default-789859d697-5f5lv"]
Nov 02 13:34:07.872968 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:07.872729    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-ed56b9-openshift-default-789859d697-5f5lv" podUID="abd3e462-db3a-4c9a-9fbc-1b5cb3550ef9" containerName="istio-proxy" containerID="cri-o://fc7c8f5f7a49d000b0f3bfcd56620a16dac5d7e80d0f3a450144f5c9a4196cdc" gracePeriod=30
Nov 02 13:34:13.122597 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:13.122573    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-ed56b9-openshift-default-789859d697-5f5lv"
Nov 02 13:34:13.231419 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:13.231384    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abd3e462-db3a-4c9a-9fbc-1b5cb3550ef9-istio-token\" (UniqueName: \"kubernetes.io/projected/abd3e462-db3a-4c9a-9fbc-1b5cb3550ef9-istio-token\") pod \"abd3e462-db3a-4c9a-9fbc-1b5cb3550ef9\" (UID: \"abd3e462-db3a-4c9a-9fbc-1b5cb3550ef9\") "
Nov 02 13:34:13.231624 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:13.231434    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/abd3e462-db3a-4c9a-9fbc-1b5cb3550ef9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abd3e462-db3a-4c9a-9fbc-1b5cb3550ef9-istiod-ca-cert\") pod \"abd3e462-db3a-4c9a-9fbc-1b5cb3550ef9\" (UID: \"abd3e462-db3a-4c9a-9fbc-1b5cb3550ef9\") "
Nov 02 13:34:13.231624 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:13.231519    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abd3e462-db3a-4c9a-9fbc-1b5cb3550ef9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abd3e462-db3a-4c9a-9fbc-1b5cb3550ef9-istio-envoy\") pod \"abd3e462-db3a-4c9a-9fbc-1b5cb3550ef9\" (UID: \"abd3e462-db3a-4c9a-9fbc-1b5cb3550ef9\") "
Nov 02 13:34:13.231624 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:13.231566    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abd3e462-db3a-4c9a-9fbc-1b5cb3550ef9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abd3e462-db3a-4c9a-9fbc-1b5cb3550ef9-workload-certs\") pod \"abd3e462-db3a-4c9a-9fbc-1b5cb3550ef9\" (UID: \"abd3e462-db3a-4c9a-9fbc-1b5cb3550ef9\") "
Nov 02 13:34:13.231624 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:13.231602    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abd3e462-db3a-4c9a-9fbc-1b5cb3550ef9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abd3e462-db3a-4c9a-9fbc-1b5cb3550ef9-credential-socket\") pod \"abd3e462-db3a-4c9a-9fbc-1b5cb3550ef9\" (UID: \"abd3e462-db3a-4c9a-9fbc-1b5cb3550ef9\") "
Nov 02 13:34:13.231904 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:13.231631    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abd3e462-db3a-4c9a-9fbc-1b5cb3550ef9-kube-api-access-9hzsq\" (UniqueName: \"kubernetes.io/projected/abd3e462-db3a-4c9a-9fbc-1b5cb3550ef9-kube-api-access-9hzsq\") pod \"abd3e462-db3a-4c9a-9fbc-1b5cb3550ef9\" (UID: \"abd3e462-db3a-4c9a-9fbc-1b5cb3550ef9\") "
Nov 02 13:34:13.231904 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:13.231703    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abd3e462-db3a-4c9a-9fbc-1b5cb3550ef9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abd3e462-db3a-4c9a-9fbc-1b5cb3550ef9-istio-data\") pod \"abd3e462-db3a-4c9a-9fbc-1b5cb3550ef9\" (UID: \"abd3e462-db3a-4c9a-9fbc-1b5cb3550ef9\") "
Nov 02 13:34:13.231904 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:13.231732    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/abd3e462-db3a-4c9a-9fbc-1b5cb3550ef9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abd3e462-db3a-4c9a-9fbc-1b5cb3550ef9-istio-podinfo\") pod \"abd3e462-db3a-4c9a-9fbc-1b5cb3550ef9\" (UID: \"abd3e462-db3a-4c9a-9fbc-1b5cb3550ef9\") "
Nov 02 13:34:13.231904 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:13.231784    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abd3e462-db3a-4c9a-9fbc-1b5cb3550ef9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abd3e462-db3a-4c9a-9fbc-1b5cb3550ef9-workload-socket\") pod \"abd3e462-db3a-4c9a-9fbc-1b5cb3550ef9\" (UID: \"abd3e462-db3a-4c9a-9fbc-1b5cb3550ef9\") "
Nov 02 13:34:13.232115 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:13.231900    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abd3e462-db3a-4c9a-9fbc-1b5cb3550ef9-credential-socket" pod "abd3e462-db3a-4c9a-9fbc-1b5cb3550ef9" (UID: "abd3e462-db3a-4c9a-9fbc-1b5cb3550ef9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:34:13.232115 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:13.231913    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abd3e462-db3a-4c9a-9fbc-1b5cb3550ef9-workload-certs" pod "abd3e462-db3a-4c9a-9fbc-1b5cb3550ef9" (UID: "abd3e462-db3a-4c9a-9fbc-1b5cb3550ef9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:34:13.232115 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:13.231929    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abd3e462-db3a-4c9a-9fbc-1b5cb3550ef9-istiod-ca-cert" pod "abd3e462-db3a-4c9a-9fbc-1b5cb3550ef9" (UID: "abd3e462-db3a-4c9a-9fbc-1b5cb3550ef9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:34:13.232115 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:13.232036    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abd3e462-db3a-4c9a-9fbc-1b5cb3550ef9-istio-data" pod "abd3e462-db3a-4c9a-9fbc-1b5cb3550ef9" (UID: "abd3e462-db3a-4c9a-9fbc-1b5cb3550ef9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:34:13.232331 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:13.232181    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abd3e462-db3a-4c9a-9fbc-1b5cb3550ef9-workload-socket" pod "abd3e462-db3a-4c9a-9fbc-1b5cb3550ef9" (UID: "abd3e462-db3a-4c9a-9fbc-1b5cb3550ef9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:34:13.232331 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:13.232247    4559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abd3e462-db3a-4c9a-9fbc-1b5cb3550ef9-istiod-ca-cert\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:34:13.232331 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:13.232265    4559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abd3e462-db3a-4c9a-9fbc-1b5cb3550ef9-workload-certs\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:34:13.232331 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:13.232289    4559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abd3e462-db3a-4c9a-9fbc-1b5cb3550ef9-credential-socket\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:34:13.232331 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:13.232303    4559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abd3e462-db3a-4c9a-9fbc-1b5cb3550ef9-istio-data\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:34:13.233901 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:13.233876    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/abd3e462-db3a-4c9a-9fbc-1b5cb3550ef9-istio-podinfo" pod "abd3e462-db3a-4c9a-9fbc-1b5cb3550ef9" (UID: "abd3e462-db3a-4c9a-9fbc-1b5cb3550ef9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 13:34:13.234045 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:13.233901    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abd3e462-db3a-4c9a-9fbc-1b5cb3550ef9-kube-api-access-9hzsq" pod "abd3e462-db3a-4c9a-9fbc-1b5cb3550ef9" (UID: "abd3e462-db3a-4c9a-9fbc-1b5cb3550ef9"). InnerVolumeSpecName "kube-api-access-9hzsq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:34:13.234045 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:13.233984    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abd3e462-db3a-4c9a-9fbc-1b5cb3550ef9-istio-token" pod "abd3e462-db3a-4c9a-9fbc-1b5cb3550ef9" (UID: "abd3e462-db3a-4c9a-9fbc-1b5cb3550ef9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:34:13.234243 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:13.234218    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abd3e462-db3a-4c9a-9fbc-1b5cb3550ef9-istio-envoy" pod "abd3e462-db3a-4c9a-9fbc-1b5cb3550ef9" (UID: "abd3e462-db3a-4c9a-9fbc-1b5cb3550ef9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:34:13.333919 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:13.333882    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9hzsq\" (UniqueName: \"kubernetes.io/projected/abd3e462-db3a-4c9a-9fbc-1b5cb3550ef9-kube-api-access-9hzsq\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:34:13.333919 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:13.333919    4559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abd3e462-db3a-4c9a-9fbc-1b5cb3550ef9-istio-podinfo\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:34:13.334207 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:13.333934    4559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abd3e462-db3a-4c9a-9fbc-1b5cb3550ef9-workload-socket\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:34:13.334207 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:13.333948    4559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abd3e462-db3a-4c9a-9fbc-1b5cb3550ef9-istio-token\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:34:13.334207 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:13.333961    4559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abd3e462-db3a-4c9a-9fbc-1b5cb3550ef9-istio-envoy\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:34:13.428132 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:13.428098    4559 generic.go:356] "Generic (PLEG): container finished" podID="abd3e462-db3a-4c9a-9fbc-1b5cb3550ef9" containerID="fc7c8f5f7a49d000b0f3bfcd56620a16dac5d7e80d0f3a450144f5c9a4196cdc" exitCode=0
Nov 02 13:34:13.428329 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:13.428152    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-ed56b9-openshift-default-789859d697-5f5lv"
Nov 02 13:34:13.428329 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:13.428184    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-ed56b9-openshift-default-789859d697-5f5lv" event={"ID":"abd3e462-db3a-4c9a-9fbc-1b5cb3550ef9","Type":"ContainerDied","Data":"fc7c8f5f7a49d000b0f3bfcd56620a16dac5d7e80d0f3a450144f5c9a4196cdc"}
Nov 02 13:34:13.428329 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:13.428274    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-ed56b9-openshift-default-789859d697-5f5lv" event={"ID":"abd3e462-db3a-4c9a-9fbc-1b5cb3550ef9","Type":"ContainerDied","Data":"34dc62a4a4e26e0cbfe078f3f914f46263cd7d1c5a7ae7a9d0ff30779c47666d"}
Nov 02 13:34:13.428329 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:13.428303    4559 scope.go:122] "RemoveContainer" containerID="fc7c8f5f7a49d000b0f3bfcd56620a16dac5d7e80d0f3a450144f5c9a4196cdc"
Nov 02 13:34:13.467769 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:13.467710    4559 scope.go:122] "RemoveContainer" containerID="087d74e1c8b7ed7ed4a40876cfb5f43cc55df856350c1acc8542fd8efadee767"
Nov 02 13:34:13.477407 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:13.477384    4559 scope.go:122] "RemoveContainer" containerID="fc7c8f5f7a49d000b0f3bfcd56620a16dac5d7e80d0f3a450144f5c9a4196cdc"
Nov 02 13:34:13.477776 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:34:13.477752    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc7c8f5f7a49d000b0f3bfcd56620a16dac5d7e80d0f3a450144f5c9a4196cdc\": container with ID starting with fc7c8f5f7a49d000b0f3bfcd56620a16dac5d7e80d0f3a450144f5c9a4196cdc not found: ID does not exist" containerID="fc7c8f5f7a49d000b0f3bfcd56620a16dac5d7e80d0f3a450144f5c9a4196cdc"
Nov 02 13:34:13.477884 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:13.477793    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc7c8f5f7a49d000b0f3bfcd56620a16dac5d7e80d0f3a450144f5c9a4196cdc"} err="failed to get container status \"fc7c8f5f7a49d000b0f3bfcd56620a16dac5d7e80d0f3a450144f5c9a4196cdc\": rpc error: code = NotFound desc = could not find container \"fc7c8f5f7a49d000b0f3bfcd56620a16dac5d7e80d0f3a450144f5c9a4196cdc\": container with ID starting with fc7c8f5f7a49d000b0f3bfcd56620a16dac5d7e80d0f3a450144f5c9a4196cdc not found: ID does not exist"
Nov 02 13:34:13.477884 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:13.477817    4559 scope.go:122] "RemoveContainer" containerID="087d74e1c8b7ed7ed4a40876cfb5f43cc55df856350c1acc8542fd8efadee767"
Nov 02 13:34:13.478113 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:34:13.478094    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"087d74e1c8b7ed7ed4a40876cfb5f43cc55df856350c1acc8542fd8efadee767\": container with ID starting with 087d74e1c8b7ed7ed4a40876cfb5f43cc55df856350c1acc8542fd8efadee767 not found: ID does not exist" containerID="087d74e1c8b7ed7ed4a40876cfb5f43cc55df856350c1acc8542fd8efadee767"
Nov 02 13:34:13.478168 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:13.478124    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"087d74e1c8b7ed7ed4a40876cfb5f43cc55df856350c1acc8542fd8efadee767"} err="failed to get container status \"087d74e1c8b7ed7ed4a40876cfb5f43cc55df856350c1acc8542fd8efadee767\": rpc error: code = NotFound desc = could not find container \"087d74e1c8b7ed7ed4a40876cfb5f43cc55df856350c1acc8542fd8efadee767\": container with ID starting with 087d74e1c8b7ed7ed4a40876cfb5f43cc55df856350c1acc8542fd8efadee767 not found: ID does not exist"
Nov 02 13:34:13.483135 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:13.483075    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-ed56b9-openshift-default-789859d697-5f5lv"]
Nov 02 13:34:13.487871 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:13.487834    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-ed56b9-openshift-default-789859d697-5f5lv"]
Nov 02 13:34:13.891961 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:13.891882    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="abd3e462-db3a-4c9a-9fbc-1b5cb3550ef9" path="/var/lib/kubelet/pods/abd3e462-db3a-4c9a-9fbc-1b5cb3550ef9/volumes"
Nov 02 13:34:14.740609 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:14.740566    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-ed56b9-7bc9f55c5-qjvr7" podUID="7c89b44f-fa2c-4379-a933-9a8e7d010bfd" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 13:34:15.088723 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:34:15.088626    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqcmj" containerName="extractor"
Nov 02 13:34:16.325062 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:16.325027    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"]
Nov 02 13:34:16.719807 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:16.719737    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-61e341/distinct-2-w1-61e341-kserve-954cf956f-n7jmg" podUID="c5e1a5f8-97b4-469d-be9a-eae7ca995283" containerName="main" containerID="cri-o://b415dc77973de5552aa58690105309e69a20126b11eb57daa9718d17c7d22f87" gracePeriod=45
Nov 02 13:34:16.723392 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:16.723355    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-61e341/distinct-w1-61e341-kserve-787cd566f4-cjj8s" podUID="6886ce28-5800-402d-b893-f92109e289a9" containerName="main" containerID="cri-o://bf5d6ed86bcb2065ce47c40320410f0302766b65c3e6f92ee835402075446aa1" gracePeriod=45
Nov 02 13:34:16.733083 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:16.733052    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-61e341/premium-w1-61e341-kserve-746477b99-645g6" podUID="02ff21d3-5a37-493a-ab95-8e4784c29123" containerName="main" containerID="cri-o://39de71ad3ca784367dec8aca9a9aa93ad9d1d9d85bc63dbe5da57936c20ba0af" gracePeriod=45
Nov 02 13:34:17.017873 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:17.017836    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-61e341/distinct-w1-61e341-kserve-787cd566f4-cjj8s"
Nov 02 13:34:17.050291 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:17.050254    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-61e341/distinct-2-w1-61e341-kserve-954cf956f-n7jmg"
Nov 02 13:34:17.052794 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:17.052770    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-61e341/premium-w1-61e341-kserve-746477b99-645g6"
Nov 02 13:34:17.073271 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:17.073225    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6886ce28-5800-402d-b893-f92109e289a9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6886ce28-5800-402d-b893-f92109e289a9-dshm\") pod \"6886ce28-5800-402d-b893-f92109e289a9\" (UID: \"6886ce28-5800-402d-b893-f92109e289a9\") "
Nov 02 13:34:17.073511 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:17.073279    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6886ce28-5800-402d-b893-f92109e289a9-kube-api-access-gj5mj\" (UniqueName: \"kubernetes.io/projected/6886ce28-5800-402d-b893-f92109e289a9-kube-api-access-gj5mj\") pod \"6886ce28-5800-402d-b893-f92109e289a9\" (UID: \"6886ce28-5800-402d-b893-f92109e289a9\") "
Nov 02 13:34:17.073511 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:17.073339    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6886ce28-5800-402d-b893-f92109e289a9-home\" (UniqueName: \"kubernetes.io/empty-dir/6886ce28-5800-402d-b893-f92109e289a9-home\") pod \"6886ce28-5800-402d-b893-f92109e289a9\" (UID: \"6886ce28-5800-402d-b893-f92109e289a9\") "
Nov 02 13:34:17.073511 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:17.073397    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6886ce28-5800-402d-b893-f92109e289a9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6886ce28-5800-402d-b893-f92109e289a9-model-cache\") pod \"6886ce28-5800-402d-b893-f92109e289a9\" (UID: \"6886ce28-5800-402d-b893-f92109e289a9\") "
Nov 02 13:34:17.073511 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:17.073443    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6886ce28-5800-402d-b893-f92109e289a9-tls-certs\" (UniqueName: \"kubernetes.io/secret/6886ce28-5800-402d-b893-f92109e289a9-tls-certs\") pod \"6886ce28-5800-402d-b893-f92109e289a9\" (UID: \"6886ce28-5800-402d-b893-f92109e289a9\") "
Nov 02 13:34:17.073794 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:17.073548    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6886ce28-5800-402d-b893-f92109e289a9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6886ce28-5800-402d-b893-f92109e289a9-tmp-dir\") pod \"6886ce28-5800-402d-b893-f92109e289a9\" (UID: \"6886ce28-5800-402d-b893-f92109e289a9\") "
Nov 02 13:34:17.074072 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:17.074048    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6886ce28-5800-402d-b893-f92109e289a9-model-cache" pod "6886ce28-5800-402d-b893-f92109e289a9" (UID: "6886ce28-5800-402d-b893-f92109e289a9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:34:17.074402 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:17.074378    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6886ce28-5800-402d-b893-f92109e289a9-home" pod "6886ce28-5800-402d-b893-f92109e289a9" (UID: "6886ce28-5800-402d-b893-f92109e289a9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:34:17.074596 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:17.074561    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6886ce28-5800-402d-b893-f92109e289a9-tmp-dir" pod "6886ce28-5800-402d-b893-f92109e289a9" (UID: "6886ce28-5800-402d-b893-f92109e289a9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:34:17.075227 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:17.074802    4559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6886ce28-5800-402d-b893-f92109e289a9-home\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:34:17.075227 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:17.074829    4559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6886ce28-5800-402d-b893-f92109e289a9-model-cache\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:34:17.075227 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:17.074845    4559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6886ce28-5800-402d-b893-f92109e289a9-tmp-dir\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:34:17.079780 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:17.078075    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6886ce28-5800-402d-b893-f92109e289a9-tls-certs" pod "6886ce28-5800-402d-b893-f92109e289a9" (UID: "6886ce28-5800-402d-b893-f92109e289a9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:34:17.079780 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:17.079058    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6886ce28-5800-402d-b893-f92109e289a9-dshm" pod "6886ce28-5800-402d-b893-f92109e289a9" (UID: "6886ce28-5800-402d-b893-f92109e289a9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:34:17.079997 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:17.079973    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6886ce28-5800-402d-b893-f92109e289a9-kube-api-access-gj5mj" pod "6886ce28-5800-402d-b893-f92109e289a9" (UID: "6886ce28-5800-402d-b893-f92109e289a9"). InnerVolumeSpecName "kube-api-access-gj5mj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:34:17.176103 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:17.176064    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5e1a5f8-97b4-469d-be9a-eae7ca995283-kube-api-access-h7x74\" (UniqueName: \"kubernetes.io/projected/c5e1a5f8-97b4-469d-be9a-eae7ca995283-kube-api-access-h7x74\") pod \"c5e1a5f8-97b4-469d-be9a-eae7ca995283\" (UID: \"c5e1a5f8-97b4-469d-be9a-eae7ca995283\") "
Nov 02 13:34:17.176383 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:17.176152    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/02ff21d3-5a37-493a-ab95-8e4784c29123-tls-certs\" (UniqueName: \"kubernetes.io/secret/02ff21d3-5a37-493a-ab95-8e4784c29123-tls-certs\") pod \"02ff21d3-5a37-493a-ab95-8e4784c29123\" (UID: \"02ff21d3-5a37-493a-ab95-8e4784c29123\") "
Nov 02 13:34:17.176383 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:17.176216    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5e1a5f8-97b4-469d-be9a-eae7ca995283-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c5e1a5f8-97b4-469d-be9a-eae7ca995283-dshm\") pod \"c5e1a5f8-97b4-469d-be9a-eae7ca995283\" (UID: \"c5e1a5f8-97b4-469d-be9a-eae7ca995283\") "
Nov 02 13:34:17.176383 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:17.176271    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02ff21d3-5a37-493a-ab95-8e4784c29123-kube-api-access-bn9pn\" (UniqueName: \"kubernetes.io/projected/02ff21d3-5a37-493a-ab95-8e4784c29123-kube-api-access-bn9pn\") pod \"02ff21d3-5a37-493a-ab95-8e4784c29123\" (UID: \"02ff21d3-5a37-493a-ab95-8e4784c29123\") "
Nov 02 13:34:17.176383 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:17.176303    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5e1a5f8-97b4-469d-be9a-eae7ca995283-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c5e1a5f8-97b4-469d-be9a-eae7ca995283-tmp-dir\") pod \"c5e1a5f8-97b4-469d-be9a-eae7ca995283\" (UID: \"c5e1a5f8-97b4-469d-be9a-eae7ca995283\") "
Nov 02 13:34:17.176383 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:17.176367    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02ff21d3-5a37-493a-ab95-8e4784c29123-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/02ff21d3-5a37-493a-ab95-8e4784c29123-model-cache\") pod \"02ff21d3-5a37-493a-ab95-8e4784c29123\" (UID: \"02ff21d3-5a37-493a-ab95-8e4784c29123\") "
Nov 02 13:34:17.176756 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:17.176407    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02ff21d3-5a37-493a-ab95-8e4784c29123-dshm\" (UniqueName: \"kubernetes.io/empty-dir/02ff21d3-5a37-493a-ab95-8e4784c29123-dshm\") pod \"02ff21d3-5a37-493a-ab95-8e4784c29123\" (UID: \"02ff21d3-5a37-493a-ab95-8e4784c29123\") "
Nov 02 13:34:17.176756 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:17.176487    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02ff21d3-5a37-493a-ab95-8e4784c29123-home\" (UniqueName: \"kubernetes.io/empty-dir/02ff21d3-5a37-493a-ab95-8e4784c29123-home\") pod \"02ff21d3-5a37-493a-ab95-8e4784c29123\" (UID: \"02ff21d3-5a37-493a-ab95-8e4784c29123\") "
Nov 02 13:34:17.176756 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:17.176519    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5e1a5f8-97b4-469d-be9a-eae7ca995283-home\" (UniqueName: \"kubernetes.io/empty-dir/c5e1a5f8-97b4-469d-be9a-eae7ca995283-home\") pod \"c5e1a5f8-97b4-469d-be9a-eae7ca995283\" (UID: \"c5e1a5f8-97b4-469d-be9a-eae7ca995283\") "
Nov 02 13:34:17.176756 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:17.176567    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02ff21d3-5a37-493a-ab95-8e4784c29123-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02ff21d3-5a37-493a-ab95-8e4784c29123-tmp-dir\") pod \"02ff21d3-5a37-493a-ab95-8e4784c29123\" (UID: \"02ff21d3-5a37-493a-ab95-8e4784c29123\") "
Nov 02 13:34:17.176756 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:17.176596    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c5e1a5f8-97b4-469d-be9a-eae7ca995283-tls-certs\" (UniqueName: \"kubernetes.io/secret/c5e1a5f8-97b4-469d-be9a-eae7ca995283-tls-certs\") pod \"c5e1a5f8-97b4-469d-be9a-eae7ca995283\" (UID: \"c5e1a5f8-97b4-469d-be9a-eae7ca995283\") "
Nov 02 13:34:17.176756 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:17.176685    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5e1a5f8-97b4-469d-be9a-eae7ca995283-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c5e1a5f8-97b4-469d-be9a-eae7ca995283-model-cache\") pod \"c5e1a5f8-97b4-469d-be9a-eae7ca995283\" (UID: \"c5e1a5f8-97b4-469d-be9a-eae7ca995283\") "
Nov 02 13:34:17.177104 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:17.176787    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02ff21d3-5a37-493a-ab95-8e4784c29123-model-cache" pod "02ff21d3-5a37-493a-ab95-8e4784c29123" (UID: "02ff21d3-5a37-493a-ab95-8e4784c29123"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:34:17.177104 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:17.176985    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5e1a5f8-97b4-469d-be9a-eae7ca995283-tmp-dir" pod "c5e1a5f8-97b4-469d-be9a-eae7ca995283" (UID: "c5e1a5f8-97b4-469d-be9a-eae7ca995283"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:34:17.177220 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:17.177116    4559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6886ce28-5800-402d-b893-f92109e289a9-tls-certs\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:34:17.177220 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:17.177135    4559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c5e1a5f8-97b4-469d-be9a-eae7ca995283-tmp-dir\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:34:17.177220 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:17.177148    4559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/02ff21d3-5a37-493a-ab95-8e4784c29123-model-cache\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:34:17.177220 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:17.177161    4559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6886ce28-5800-402d-b893-f92109e289a9-dshm\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:34:17.177220 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:17.177175    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gj5mj\" (UniqueName: \"kubernetes.io/projected/6886ce28-5800-402d-b893-f92109e289a9-kube-api-access-gj5mj\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:34:17.177493 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:17.177372    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02ff21d3-5a37-493a-ab95-8e4784c29123-tmp-dir" pod "02ff21d3-5a37-493a-ab95-8e4784c29123" (UID: "02ff21d3-5a37-493a-ab95-8e4784c29123"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:34:17.177557 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:17.177542    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02ff21d3-5a37-493a-ab95-8e4784c29123-home" pod "02ff21d3-5a37-493a-ab95-8e4784c29123" (UID: "02ff21d3-5a37-493a-ab95-8e4784c29123"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:34:17.177862 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:17.177739    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5e1a5f8-97b4-469d-be9a-eae7ca995283-home" pod "c5e1a5f8-97b4-469d-be9a-eae7ca995283" (UID: "c5e1a5f8-97b4-469d-be9a-eae7ca995283"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:34:17.178071 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:17.178035    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5e1a5f8-97b4-469d-be9a-eae7ca995283-model-cache" pod "c5e1a5f8-97b4-469d-be9a-eae7ca995283" (UID: "c5e1a5f8-97b4-469d-be9a-eae7ca995283"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:34:17.180892 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:17.180864    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5e1a5f8-97b4-469d-be9a-eae7ca995283-tls-certs" pod "c5e1a5f8-97b4-469d-be9a-eae7ca995283" (UID: "c5e1a5f8-97b4-469d-be9a-eae7ca995283"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:34:17.181119 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:17.181007    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5e1a5f8-97b4-469d-be9a-eae7ca995283-dshm" pod "c5e1a5f8-97b4-469d-be9a-eae7ca995283" (UID: "c5e1a5f8-97b4-469d-be9a-eae7ca995283"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:34:17.181119 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:17.181020    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02ff21d3-5a37-493a-ab95-8e4784c29123-tls-certs" pod "02ff21d3-5a37-493a-ab95-8e4784c29123" (UID: "02ff21d3-5a37-493a-ab95-8e4784c29123"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:34:17.181119 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:17.181094    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5e1a5f8-97b4-469d-be9a-eae7ca995283-kube-api-access-h7x74" pod "c5e1a5f8-97b4-469d-be9a-eae7ca995283" (UID: "c5e1a5f8-97b4-469d-be9a-eae7ca995283"). InnerVolumeSpecName "kube-api-access-h7x74". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:34:17.182514 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:17.182486    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02ff21d3-5a37-493a-ab95-8e4784c29123-kube-api-access-bn9pn" pod "02ff21d3-5a37-493a-ab95-8e4784c29123" (UID: "02ff21d3-5a37-493a-ab95-8e4784c29123"). InnerVolumeSpecName "kube-api-access-bn9pn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:34:17.182660 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:17.182531    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02ff21d3-5a37-493a-ab95-8e4784c29123-dshm" pod "02ff21d3-5a37-493a-ab95-8e4784c29123" (UID: "02ff21d3-5a37-493a-ab95-8e4784c29123"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:34:17.278434 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:17.278343    4559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/02ff21d3-5a37-493a-ab95-8e4784c29123-home\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:34:17.278434 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:17.278377    4559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c5e1a5f8-97b4-469d-be9a-eae7ca995283-home\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:34:17.278434 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:17.278387    4559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02ff21d3-5a37-493a-ab95-8e4784c29123-tmp-dir\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:34:17.278434 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:17.278397    4559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c5e1a5f8-97b4-469d-be9a-eae7ca995283-tls-certs\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:34:17.278434 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:17.278406    4559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c5e1a5f8-97b4-469d-be9a-eae7ca995283-model-cache\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:34:17.278434 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:17.278416    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h7x74\" (UniqueName: \"kubernetes.io/projected/c5e1a5f8-97b4-469d-be9a-eae7ca995283-kube-api-access-h7x74\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:34:17.278434 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:17.278425    4559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02ff21d3-5a37-493a-ab95-8e4784c29123-tls-certs\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:34:17.278434 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:17.278433    4559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c5e1a5f8-97b4-469d-be9a-eae7ca995283-dshm\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:34:17.278434 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:17.278441    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bn9pn\" (UniqueName: \"kubernetes.io/projected/02ff21d3-5a37-493a-ab95-8e4784c29123-kube-api-access-bn9pn\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:34:17.278868 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:17.278449    4559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/02ff21d3-5a37-493a-ab95-8e4784c29123-dshm\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:34:17.452485 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:17.452457    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-61e341/distinct-2-w1-61e341-kserve-954cf956f-n7jmg"
Nov 02 13:34:17.452485 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:17.452482    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-61e341/distinct-2-w1-61e341-kserve-954cf956f-n7jmg" event={"ID":"c5e1a5f8-97b4-469d-be9a-eae7ca995283","Type":"ContainerDied","Data":"b415dc77973de5552aa58690105309e69a20126b11eb57daa9718d17c7d22f87"}
Nov 02 13:34:17.453062 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:17.452524    4559 scope.go:122] "RemoveContainer" containerID="b415dc77973de5552aa58690105309e69a20126b11eb57daa9718d17c7d22f87"
Nov 02 13:34:17.453062 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:17.452455    4559 generic.go:356] "Generic (PLEG): container finished" podID="c5e1a5f8-97b4-469d-be9a-eae7ca995283" containerID="b415dc77973de5552aa58690105309e69a20126b11eb57daa9718d17c7d22f87" exitCode=0
Nov 02 13:34:17.453062 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:17.452672    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-61e341/distinct-2-w1-61e341-kserve-954cf956f-n7jmg" event={"ID":"c5e1a5f8-97b4-469d-be9a-eae7ca995283","Type":"ContainerDied","Data":"bd4811df291f82e384b5fe6bf9faeb0703c2ea90657c5060bd957eb955a7f6a4"}
Nov 02 13:34:17.454095 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:17.454064    4559 generic.go:356] "Generic (PLEG): container finished" podID="02ff21d3-5a37-493a-ab95-8e4784c29123" containerID="39de71ad3ca784367dec8aca9a9aa93ad9d1d9d85bc63dbe5da57936c20ba0af" exitCode=0
Nov 02 13:34:17.454185 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:17.454094    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-61e341/premium-w1-61e341-kserve-746477b99-645g6" event={"ID":"02ff21d3-5a37-493a-ab95-8e4784c29123","Type":"ContainerDied","Data":"39de71ad3ca784367dec8aca9a9aa93ad9d1d9d85bc63dbe5da57936c20ba0af"}
Nov 02 13:34:17.454185 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:17.454112    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-61e341/premium-w1-61e341-kserve-746477b99-645g6"
Nov 02 13:34:17.454185 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:17.454130    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-61e341/premium-w1-61e341-kserve-746477b99-645g6" event={"ID":"02ff21d3-5a37-493a-ab95-8e4784c29123","Type":"ContainerDied","Data":"cb2d84e264030d9f5304e00fe71820d17cafbedf5c006ae4feb22391625a5f59"}
Nov 02 13:34:17.455670 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:17.455609    4559 generic.go:356] "Generic (PLEG): container finished" podID="6886ce28-5800-402d-b893-f92109e289a9" containerID="bf5d6ed86bcb2065ce47c40320410f0302766b65c3e6f92ee835402075446aa1" exitCode=0
Nov 02 13:34:17.455796 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:17.455678    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-61e341/distinct-w1-61e341-kserve-787cd566f4-cjj8s" event={"ID":"6886ce28-5800-402d-b893-f92109e289a9","Type":"ContainerDied","Data":"bf5d6ed86bcb2065ce47c40320410f0302766b65c3e6f92ee835402075446aa1"}
Nov 02 13:34:17.455796 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:17.455689    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-61e341/distinct-w1-61e341-kserve-787cd566f4-cjj8s"
Nov 02 13:34:17.455796 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:17.455705    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-61e341/distinct-w1-61e341-kserve-787cd566f4-cjj8s" event={"ID":"6886ce28-5800-402d-b893-f92109e289a9","Type":"ContainerDied","Data":"bae6e6622784432b8d0a25e4d00141e64ea18e6063492e5c0f4f7e3216467763"}
Nov 02 13:34:17.465015 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:17.464998    4559 scope.go:122] "RemoveContainer" containerID="b415dc77973de5552aa58690105309e69a20126b11eb57daa9718d17c7d22f87"
Nov 02 13:34:17.465330 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:34:17.465304    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b415dc77973de5552aa58690105309e69a20126b11eb57daa9718d17c7d22f87\": container with ID starting with b415dc77973de5552aa58690105309e69a20126b11eb57daa9718d17c7d22f87 not found: ID does not exist" containerID="b415dc77973de5552aa58690105309e69a20126b11eb57daa9718d17c7d22f87"
Nov 02 13:34:17.465398 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:17.465344    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b415dc77973de5552aa58690105309e69a20126b11eb57daa9718d17c7d22f87"} err="failed to get container status \"b415dc77973de5552aa58690105309e69a20126b11eb57daa9718d17c7d22f87\": rpc error: code = NotFound desc = could not find container \"b415dc77973de5552aa58690105309e69a20126b11eb57daa9718d17c7d22f87\": container with ID starting with b415dc77973de5552aa58690105309e69a20126b11eb57daa9718d17c7d22f87 not found: ID does not exist"
Nov 02 13:34:17.465398 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:17.465370    4559 scope.go:122] "RemoveContainer" containerID="39de71ad3ca784367dec8aca9a9aa93ad9d1d9d85bc63dbe5da57936c20ba0af"
Nov 02 13:34:17.476880 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:17.476858    4559 scope.go:122] "RemoveContainer" containerID="39de71ad3ca784367dec8aca9a9aa93ad9d1d9d85bc63dbe5da57936c20ba0af"
Nov 02 13:34:17.477207 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:34:17.477186    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39de71ad3ca784367dec8aca9a9aa93ad9d1d9d85bc63dbe5da57936c20ba0af\": container with ID starting with 39de71ad3ca784367dec8aca9a9aa93ad9d1d9d85bc63dbe5da57936c20ba0af not found: ID does not exist" containerID="39de71ad3ca784367dec8aca9a9aa93ad9d1d9d85bc63dbe5da57936c20ba0af"
Nov 02 13:34:17.477275 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:17.477216    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39de71ad3ca784367dec8aca9a9aa93ad9d1d9d85bc63dbe5da57936c20ba0af"} err="failed to get container status \"39de71ad3ca784367dec8aca9a9aa93ad9d1d9d85bc63dbe5da57936c20ba0af\": rpc error: code = NotFound desc = could not find container \"39de71ad3ca784367dec8aca9a9aa93ad9d1d9d85bc63dbe5da57936c20ba0af\": container with ID starting with 39de71ad3ca784367dec8aca9a9aa93ad9d1d9d85bc63dbe5da57936c20ba0af not found: ID does not exist"
Nov 02 13:34:17.477275 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:17.477233    4559 scope.go:122] "RemoveContainer" containerID="bf5d6ed86bcb2065ce47c40320410f0302766b65c3e6f92ee835402075446aa1"
Nov 02 13:34:17.486161 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:17.486140    4559 scope.go:122] "RemoveContainer" containerID="bf5d6ed86bcb2065ce47c40320410f0302766b65c3e6f92ee835402075446aa1"
Nov 02 13:34:17.486518 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:34:17.486486    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf5d6ed86bcb2065ce47c40320410f0302766b65c3e6f92ee835402075446aa1\": container with ID starting with bf5d6ed86bcb2065ce47c40320410f0302766b65c3e6f92ee835402075446aa1 not found: ID does not exist" containerID="bf5d6ed86bcb2065ce47c40320410f0302766b65c3e6f92ee835402075446aa1"
Nov 02 13:34:17.486688 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:17.486527    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf5d6ed86bcb2065ce47c40320410f0302766b65c3e6f92ee835402075446aa1"} err="failed to get container status \"bf5d6ed86bcb2065ce47c40320410f0302766b65c3e6f92ee835402075446aa1\": rpc error: code = NotFound desc = could not find container \"bf5d6ed86bcb2065ce47c40320410f0302766b65c3e6f92ee835402075446aa1\": container with ID starting with bf5d6ed86bcb2065ce47c40320410f0302766b65c3e6f92ee835402075446aa1 not found: ID does not exist"
Nov 02 13:34:17.510516 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:17.510485    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-61e341/distinct-2-w1-61e341-kserve-954cf956f-n7jmg"]
Nov 02 13:34:17.523108 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:17.523072    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-61e341/distinct-2-w1-61e341-kserve-954cf956f-n7jmg"]
Nov 02 13:34:17.551662 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:17.551566    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-61e341/distinct-w1-61e341-kserve-787cd566f4-cjj8s"]
Nov 02 13:34:17.555999 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:17.555968    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-61e341/distinct-w1-61e341-kserve-787cd566f4-cjj8s"]
Nov 02 13:34:17.596788 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:17.596742    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-61e341/premium-w1-61e341-kserve-746477b99-645g6"]
Nov 02 13:34:17.599732 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:17.599705    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-61e341/premium-w1-61e341-kserve-746477b99-645g6"]
Nov 02 13:34:17.892882 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:17.892805    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02ff21d3-5a37-493a-ab95-8e4784c29123" path="/var/lib/kubelet/pods/02ff21d3-5a37-493a-ab95-8e4784c29123/volumes"
Nov 02 13:34:17.893227 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:17.893213    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6886ce28-5800-402d-b893-f92109e289a9" path="/var/lib/kubelet/pods/6886ce28-5800-402d-b893-f92109e289a9/volumes"
Nov 02 13:34:17.893584 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:17.893571    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5e1a5f8-97b4-469d-be9a-eae7ca995283" path="/var/lib/kubelet/pods/c5e1a5f8-97b4-469d-be9a-eae7ca995283/volumes"
Nov 02 13:34:23.364508 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:23.364484    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-ed56b9-7bc9f55c5-qjvr7"
Nov 02 13:34:23.442519 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:23.442493    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c89b44f-fa2c-4379-a933-9a8e7d010bfd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c89b44f-fa2c-4379-a933-9a8e7d010bfd-plugins-config-volume\") pod \"7c89b44f-fa2c-4379-a933-9a8e7d010bfd\" (UID: \"7c89b44f-fa2c-4379-a933-9a8e7d010bfd\") "
Nov 02 13:34:23.442701 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:23.442610    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c89b44f-fa2c-4379-a933-9a8e7d010bfd-kube-api-access-t4m6q\" (UniqueName: \"kubernetes.io/projected/7c89b44f-fa2c-4379-a933-9a8e7d010bfd-kube-api-access-t4m6q\") pod \"7c89b44f-fa2c-4379-a933-9a8e7d010bfd\" (UID: \"7c89b44f-fa2c-4379-a933-9a8e7d010bfd\") "
Nov 02 13:34:23.442945 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:23.442921    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c89b44f-fa2c-4379-a933-9a8e7d010bfd-plugins-config-volume" pod "7c89b44f-fa2c-4379-a933-9a8e7d010bfd" (UID: "7c89b44f-fa2c-4379-a933-9a8e7d010bfd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:34:23.444598 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:23.444580    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c89b44f-fa2c-4379-a933-9a8e7d010bfd-kube-api-access-t4m6q" pod "7c89b44f-fa2c-4379-a933-9a8e7d010bfd" (UID: "7c89b44f-fa2c-4379-a933-9a8e7d010bfd"). InnerVolumeSpecName "kube-api-access-t4m6q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:34:23.490433 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:23.490405    4559 generic.go:356] "Generic (PLEG): container finished" podID="7c89b44f-fa2c-4379-a933-9a8e7d010bfd" containerID="839ab0ddb5712c3a205d8812562e16fb9303e8177aa61abb94a511641adb89e5" exitCode=0
Nov 02 13:34:23.490578 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:23.490448    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-ed56b9-7bc9f55c5-qjvr7"
Nov 02 13:34:23.490578 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:23.490466    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-ed56b9-7bc9f55c5-qjvr7" event={"ID":"7c89b44f-fa2c-4379-a933-9a8e7d010bfd","Type":"ContainerDied","Data":"839ab0ddb5712c3a205d8812562e16fb9303e8177aa61abb94a511641adb89e5"}
Nov 02 13:34:23.490578 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:23.490520    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-ed56b9-7bc9f55c5-qjvr7" event={"ID":"7c89b44f-fa2c-4379-a933-9a8e7d010bfd","Type":"ContainerDied","Data":"ca81d9dd4f68fc85f585cf9c1cd2a9d5174fdb36a592fa196b08919fe3216f33"}
Nov 02 13:34:23.490578 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:23.490551    4559 scope.go:122] "RemoveContainer" containerID="839ab0ddb5712c3a205d8812562e16fb9303e8177aa61abb94a511641adb89e5"
Nov 02 13:34:23.500604 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:23.500586    4559 scope.go:122] "RemoveContainer" containerID="839ab0ddb5712c3a205d8812562e16fb9303e8177aa61abb94a511641adb89e5"
Nov 02 13:34:23.500878 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:34:23.500861    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"839ab0ddb5712c3a205d8812562e16fb9303e8177aa61abb94a511641adb89e5\": container with ID starting with 839ab0ddb5712c3a205d8812562e16fb9303e8177aa61abb94a511641adb89e5 not found: ID does not exist" containerID="839ab0ddb5712c3a205d8812562e16fb9303e8177aa61abb94a511641adb89e5"
Nov 02 13:34:23.500952 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:23.500885    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"839ab0ddb5712c3a205d8812562e16fb9303e8177aa61abb94a511641adb89e5"} err="failed to get container status \"839ab0ddb5712c3a205d8812562e16fb9303e8177aa61abb94a511641adb89e5\": rpc error: code = NotFound desc = could not find container \"839ab0ddb5712c3a205d8812562e16fb9303e8177aa61abb94a511641adb89e5\": container with ID starting with 839ab0ddb5712c3a205d8812562e16fb9303e8177aa61abb94a511641adb89e5 not found: ID does not exist"
Nov 02 13:34:23.543174 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:23.543153    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-ed56b9-7bc9f55c5-qjvr7"]
Nov 02 13:34:23.543445 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:23.543429    4559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c89b44f-fa2c-4379-a933-9a8e7d010bfd-plugins-config-volume\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:34:23.543496 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:23.543448    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t4m6q\" (UniqueName: \"kubernetes.io/projected/7c89b44f-fa2c-4379-a933-9a8e7d010bfd-kube-api-access-t4m6q\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:34:23.547378 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:23.547360    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-ed56b9-7bc9f55c5-qjvr7"]
Nov 02 13:34:23.891766 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:23.891685    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c89b44f-fa2c-4379-a933-9a8e7d010bfd" path="/var/lib/kubelet/pods/7c89b44f-fa2c-4379-a933-9a8e7d010bfd/volumes"
Nov 02 13:34:25.456277 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:25.456249    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-1e14d4-7f57bfcbd8-9l28c"]
Nov 02 13:34:25.456776 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:25.456502    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-1e14d4-7f57bfcbd8-9l28c" podUID="6ec4ed15-a5c5-461f-b839-ef61b09f6309" containerName="maas-api" containerID="cri-o://348a76bf1dba8e6bd0880ef63793384582f20e48c82aa7156f577cacbecd85ec" gracePeriod=30
Nov 02 13:34:25.718348 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:25.718300    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-1e14d4-7f57bfcbd8-9l28c"
Nov 02 13:34:25.865813 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:25.865770    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ec4ed15-a5c5-461f-b839-ef61b09f6309-kube-api-access-n22f4\" (UniqueName: \"kubernetes.io/projected/6ec4ed15-a5c5-461f-b839-ef61b09f6309-kube-api-access-n22f4\") pod \"6ec4ed15-a5c5-461f-b839-ef61b09f6309\" (UID: \"6ec4ed15-a5c5-461f-b839-ef61b09f6309\") "
Nov 02 13:34:25.866020 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:25.865898    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6ec4ed15-a5c5-461f-b839-ef61b09f6309-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ec4ed15-a5c5-461f-b839-ef61b09f6309-maas-api-tls\") pod \"6ec4ed15-a5c5-461f-b839-ef61b09f6309\" (UID: \"6ec4ed15-a5c5-461f-b839-ef61b09f6309\") "
Nov 02 13:34:25.866020 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:25.865942    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ec4ed15-a5c5-461f-b839-ef61b09f6309-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ec4ed15-a5c5-461f-b839-ef61b09f6309-openshift-service-ca\") pod \"6ec4ed15-a5c5-461f-b839-ef61b09f6309\" (UID: \"6ec4ed15-a5c5-461f-b839-ef61b09f6309\") "
Nov 02 13:34:25.866020 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:25.865965    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6ec4ed15-a5c5-461f-b839-ef61b09f6309-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ec4ed15-a5c5-461f-b839-ef61b09f6309-metrics-certs\") pod \"6ec4ed15-a5c5-461f-b839-ef61b09f6309\" (UID: \"6ec4ed15-a5c5-461f-b839-ef61b09f6309\") "
Nov 02 13:34:25.866295 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:25.866273    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ec4ed15-a5c5-461f-b839-ef61b09f6309-openshift-service-ca" pod "6ec4ed15-a5c5-461f-b839-ef61b09f6309" (UID: "6ec4ed15-a5c5-461f-b839-ef61b09f6309"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:34:25.866475 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:25.866450    4559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ec4ed15-a5c5-461f-b839-ef61b09f6309-openshift-service-ca\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:34:25.868032 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:25.868002    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ec4ed15-a5c5-461f-b839-ef61b09f6309-maas-api-tls" pod "6ec4ed15-a5c5-461f-b839-ef61b09f6309" (UID: "6ec4ed15-a5c5-461f-b839-ef61b09f6309"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:34:25.868032 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:25.868018    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ec4ed15-a5c5-461f-b839-ef61b09f6309-metrics-certs" pod "6ec4ed15-a5c5-461f-b839-ef61b09f6309" (UID: "6ec4ed15-a5c5-461f-b839-ef61b09f6309"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:34:25.868184 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:25.868076    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ec4ed15-a5c5-461f-b839-ef61b09f6309-kube-api-access-n22f4" pod "6ec4ed15-a5c5-461f-b839-ef61b09f6309" (UID: "6ec4ed15-a5c5-461f-b839-ef61b09f6309"). InnerVolumeSpecName "kube-api-access-n22f4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:34:25.968196 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:25.968157    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n22f4\" (UniqueName: \"kubernetes.io/projected/6ec4ed15-a5c5-461f-b839-ef61b09f6309-kube-api-access-n22f4\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:34:25.968196 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:25.968191    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ec4ed15-a5c5-461f-b839-ef61b09f6309-maas-api-tls\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:34:25.968385 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:25.968206    4559 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ec4ed15-a5c5-461f-b839-ef61b09f6309-metrics-certs\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:34:26.517284 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:26.517250    4559 generic.go:356] "Generic (PLEG): container finished" podID="6ec4ed15-a5c5-461f-b839-ef61b09f6309" containerID="348a76bf1dba8e6bd0880ef63793384582f20e48c82aa7156f577cacbecd85ec" exitCode=0
Nov 02 13:34:26.517735 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:26.517319    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-1e14d4-7f57bfcbd8-9l28c"
Nov 02 13:34:26.517735 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:26.517334    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-1e14d4-7f57bfcbd8-9l28c" event={"ID":"6ec4ed15-a5c5-461f-b839-ef61b09f6309","Type":"ContainerDied","Data":"348a76bf1dba8e6bd0880ef63793384582f20e48c82aa7156f577cacbecd85ec"}
Nov 02 13:34:26.517735 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:26.517411    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-1e14d4-7f57bfcbd8-9l28c" event={"ID":"6ec4ed15-a5c5-461f-b839-ef61b09f6309","Type":"ContainerDied","Data":"7707c01c642dc767ebec3de6d0e8c90f2d0b2bd6aea25b9d270ab6a4e561055b"}
Nov 02 13:34:26.517735 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:26.517433    4559 scope.go:122] "RemoveContainer" containerID="348a76bf1dba8e6bd0880ef63793384582f20e48c82aa7156f577cacbecd85ec"
Nov 02 13:34:26.526667 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:26.526633    4559 scope.go:122] "RemoveContainer" containerID="348a76bf1dba8e6bd0880ef63793384582f20e48c82aa7156f577cacbecd85ec"
Nov 02 13:34:26.527020 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:34:26.526991    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"348a76bf1dba8e6bd0880ef63793384582f20e48c82aa7156f577cacbecd85ec\": container with ID starting with 348a76bf1dba8e6bd0880ef63793384582f20e48c82aa7156f577cacbecd85ec not found: ID does not exist" containerID="348a76bf1dba8e6bd0880ef63793384582f20e48c82aa7156f577cacbecd85ec"
Nov 02 13:34:26.527106 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:26.527047    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"348a76bf1dba8e6bd0880ef63793384582f20e48c82aa7156f577cacbecd85ec"} err="failed to get container status \"348a76bf1dba8e6bd0880ef63793384582f20e48c82aa7156f577cacbecd85ec\": rpc error: code = NotFound desc = could not find container \"348a76bf1dba8e6bd0880ef63793384582f20e48c82aa7156f577cacbecd85ec\": container with ID starting with 348a76bf1dba8e6bd0880ef63793384582f20e48c82aa7156f577cacbecd85ec not found: ID does not exist"
Nov 02 13:34:26.570511 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:26.570490    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-1e14d4-7f57bfcbd8-9l28c"]
Nov 02 13:34:26.574616 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:26.574596    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-1e14d4-7f57bfcbd8-9l28c"]
Nov 02 13:34:27.892593 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:27.892559    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ec4ed15-a5c5-461f-b839-ef61b09f6309" path="/var/lib/kubelet/pods/6ec4ed15-a5c5-461f-b839-ef61b09f6309/volumes"
Nov 02 13:34:28.900608 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:28.900572    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-cmmcs"]
Nov 02 13:34:28.901026 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:28.900967    4559 cpu_manager.go:414] "containerMap: removing container" podUID="7c89b44f-fa2c-4379-a933-9a8e7d010bfd" containerName="payload-processing"
Nov 02 13:34:28.901026 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:28.900978    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c89b44f-fa2c-4379-a933-9a8e7d010bfd" containerName="payload-processing"
Nov 02 13:34:28.901026 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:28.900987    4559 cpu_manager.go:414] "containerMap: removing container" podUID="6886ce28-5800-402d-b893-f92109e289a9" containerName="main"
Nov 02 13:34:28.901026 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:28.900992    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6886ce28-5800-402d-b893-f92109e289a9" containerName="main"
Nov 02 13:34:28.901026 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:28.901004    4559 cpu_manager.go:414] "containerMap: removing container" podUID="d4eca390-56ae-40c6-a669-de202c2483de" containerName="istio-proxy"
Nov 02 13:34:28.901026 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:28.901009    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4eca390-56ae-40c6-a669-de202c2483de" containerName="istio-proxy"
Nov 02 13:34:28.901026 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:28.901017    4559 cpu_manager.go:414] "containerMap: removing container" podUID="02ff21d3-5a37-493a-ab95-8e4784c29123" containerName="main"
Nov 02 13:34:28.901026 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:28.901022    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02ff21d3-5a37-493a-ab95-8e4784c29123" containerName="main"
Nov 02 13:34:28.901026 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:28.901029    4559 cpu_manager.go:414] "containerMap: removing container" podUID="bb3c761d-8641-496c-9d3a-531e8117e7dd" containerName="istio-proxy"
Nov 02 13:34:28.901330 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:28.901034    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb3c761d-8641-496c-9d3a-531e8117e7dd" containerName="istio-proxy"
Nov 02 13:34:28.901330 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:28.901040    4559 cpu_manager.go:414] "containerMap: removing container" podUID="6ec4ed15-a5c5-461f-b839-ef61b09f6309" containerName="maas-api"
Nov 02 13:34:28.901330 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:28.901045    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ec4ed15-a5c5-461f-b839-ef61b09f6309" containerName="maas-api"
Nov 02 13:34:28.901330 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:28.901052    4559 cpu_manager.go:414] "containerMap: removing container" podUID="abd3e462-db3a-4c9a-9fbc-1b5cb3550ef9" containerName="istio-proxy"
Nov 02 13:34:28.901330 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:28.901058    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abd3e462-db3a-4c9a-9fbc-1b5cb3550ef9" containerName="istio-proxy"
Nov 02 13:34:28.901330 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:28.901068    4559 cpu_manager.go:414] "containerMap: removing container" podUID="8f2bb155-8114-4c86-a74d-479522616e10" containerName="istio-proxy"
Nov 02 13:34:28.901330 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:28.901073    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f2bb155-8114-4c86-a74d-479522616e10" containerName="istio-proxy"
Nov 02 13:34:28.901330 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:28.901083    4559 cpu_manager.go:414] "containerMap: removing container" podUID="e97a314b-6891-4a14-9f4f-c7131a59e231" containerName="maas-api"
Nov 02 13:34:28.901330 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:28.901088    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e97a314b-6891-4a14-9f4f-c7131a59e231" containerName="maas-api"
Nov 02 13:34:28.901330 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:28.901096    4559 cpu_manager.go:414] "containerMap: removing container" podUID="95acd739-ef82-4350-a68c-7a40ae0acbe5" containerName="maas-api"
Nov 02 13:34:28.901330 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:28.901101    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95acd739-ef82-4350-a68c-7a40ae0acbe5" containerName="maas-api"
Nov 02 13:34:28.901330 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:28.901107    4559 cpu_manager.go:414] "containerMap: removing container" podUID="bb3c761d-8641-496c-9d3a-531e8117e7dd" containerName="istio-proxy"
Nov 02 13:34:28.901330 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:28.901112    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb3c761d-8641-496c-9d3a-531e8117e7dd" containerName="istio-proxy"
Nov 02 13:34:28.901330 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:28.901119    4559 cpu_manager.go:414] "containerMap: removing container" podUID="da10c6a3-b295-4997-84fc-ffe8ebb90061" containerName="payload-pre-processing"
Nov 02 13:34:28.901330 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:28.901125    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da10c6a3-b295-4997-84fc-ffe8ebb90061" containerName="payload-pre-processing"
Nov 02 13:34:28.901330 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:28.901136    4559 cpu_manager.go:414] "containerMap: removing container" podUID="8debb7bd-d249-47d9-8776-f2be8f0d4f27" containerName="istio-proxy"
Nov 02 13:34:28.901330 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:28.901140    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8debb7bd-d249-47d9-8776-f2be8f0d4f27" containerName="istio-proxy"
Nov 02 13:34:28.901330 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:28.901146    4559 cpu_manager.go:414] "containerMap: removing container" podUID="abd3e462-db3a-4c9a-9fbc-1b5cb3550ef9" containerName="istio-proxy"
Nov 02 13:34:28.901330 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:28.901151    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abd3e462-db3a-4c9a-9fbc-1b5cb3550ef9" containerName="istio-proxy"
Nov 02 13:34:28.901330 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:28.901163    4559 cpu_manager.go:414] "containerMap: removing container" podUID="c5e1a5f8-97b4-469d-be9a-eae7ca995283" containerName="main"
Nov 02 13:34:28.901330 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:28.901167    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5e1a5f8-97b4-469d-be9a-eae7ca995283" containerName="main"
Nov 02 13:34:28.901330 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:28.901175    4559 cpu_manager.go:414] "containerMap: removing container" podUID="8debb7bd-d249-47d9-8776-f2be8f0d4f27" containerName="istio-proxy"
Nov 02 13:34:28.901330 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:28.901179    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8debb7bd-d249-47d9-8776-f2be8f0d4f27" containerName="istio-proxy"
Nov 02 13:34:28.901330 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:28.901231    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="8debb7bd-d249-47d9-8776-f2be8f0d4f27" containerName="istio-proxy"
Nov 02 13:34:28.901330 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:28.901238    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="abd3e462-db3a-4c9a-9fbc-1b5cb3550ef9" containerName="istio-proxy"
Nov 02 13:34:28.901330 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:28.901246    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5e1a5f8-97b4-469d-be9a-eae7ca995283" containerName="main"
Nov 02 13:34:28.901330 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:28.901253    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c89b44f-fa2c-4379-a933-9a8e7d010bfd" containerName="payload-processing"
Nov 02 13:34:28.901330 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:28.901258    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="6886ce28-5800-402d-b893-f92109e289a9" containerName="main"
Nov 02 13:34:28.901330 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:28.901264    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4eca390-56ae-40c6-a669-de202c2483de" containerName="istio-proxy"
Nov 02 13:34:28.901330 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:28.901270    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="02ff21d3-5a37-493a-ab95-8e4784c29123" containerName="main"
Nov 02 13:34:28.901330 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:28.901277    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb3c761d-8641-496c-9d3a-531e8117e7dd" containerName="istio-proxy"
Nov 02 13:34:28.901330 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:28.901282    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ec4ed15-a5c5-461f-b839-ef61b09f6309" containerName="maas-api"
Nov 02 13:34:28.901330 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:28.901288    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="abd3e462-db3a-4c9a-9fbc-1b5cb3550ef9" containerName="istio-proxy"
Nov 02 13:34:28.901330 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:28.901294    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="e97a314b-6891-4a14-9f4f-c7131a59e231" containerName="maas-api"
Nov 02 13:34:28.901330 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:28.901299    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="95acd739-ef82-4350-a68c-7a40ae0acbe5" containerName="maas-api"
Nov 02 13:34:28.901330 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:28.901304    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb3c761d-8641-496c-9d3a-531e8117e7dd" containerName="istio-proxy"
Nov 02 13:34:28.901330 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:28.901309    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="da10c6a3-b295-4997-84fc-ffe8ebb90061" containerName="payload-pre-processing"
Nov 02 13:34:28.906157 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:28.906140    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-cmmcs"
Nov 02 13:34:28.921893 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:28.921872    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-61e341-openshift-default-dockercfg-jcvsw\""
Nov 02 13:34:28.937341 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:28.937317    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-cmmcs"]
Nov 02 13:34:28.996089 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:28.996064    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06d7fd9a-2b1f-4970-b9c2-fc1e4bb58652-credential-socket\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-cmmcs\" (UID: \"06d7fd9a-2b1f-4970-b9c2-fc1e4bb58652\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-cmmcs"
Nov 02 13:34:28.996201 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:28.996094    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwrsx\" (UniqueName: \"kubernetes.io/projected/06d7fd9a-2b1f-4970-b9c2-fc1e4bb58652-kube-api-access-mwrsx\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-cmmcs\" (UID: \"06d7fd9a-2b1f-4970-b9c2-fc1e4bb58652\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-cmmcs"
Nov 02 13:34:28.996201 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:28.996125    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06d7fd9a-2b1f-4970-b9c2-fc1e4bb58652-istio-data\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-cmmcs\" (UID: \"06d7fd9a-2b1f-4970-b9c2-fc1e4bb58652\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-cmmcs"
Nov 02 13:34:28.996284 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:28.996262    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06d7fd9a-2b1f-4970-b9c2-fc1e4bb58652-istio-envoy\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-cmmcs\" (UID: \"06d7fd9a-2b1f-4970-b9c2-fc1e4bb58652\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-cmmcs"
Nov 02 13:34:28.996327 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:28.996299    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06d7fd9a-2b1f-4970-b9c2-fc1e4bb58652-workload-certs\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-cmmcs\" (UID: \"06d7fd9a-2b1f-4970-b9c2-fc1e4bb58652\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-cmmcs"
Nov 02 13:34:28.996391 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:28.996374    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06d7fd9a-2b1f-4970-b9c2-fc1e4bb58652-istio-podinfo\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-cmmcs\" (UID: \"06d7fd9a-2b1f-4970-b9c2-fc1e4bb58652\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-cmmcs"
Nov 02 13:34:28.996428 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:28.996405    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06d7fd9a-2b1f-4970-b9c2-fc1e4bb58652-workload-socket\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-cmmcs\" (UID: \"06d7fd9a-2b1f-4970-b9c2-fc1e4bb58652\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-cmmcs"
Nov 02 13:34:28.996428 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:28.996424    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06d7fd9a-2b1f-4970-b9c2-fc1e4bb58652-istiod-ca-cert\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-cmmcs\" (UID: \"06d7fd9a-2b1f-4970-b9c2-fc1e4bb58652\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-cmmcs"
Nov 02 13:34:28.996501 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:28.996454    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06d7fd9a-2b1f-4970-b9c2-fc1e4bb58652-istio-token\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-cmmcs\" (UID: \"06d7fd9a-2b1f-4970-b9c2-fc1e4bb58652\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-cmmcs"
Nov 02 13:34:29.097423 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:29.097390    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06d7fd9a-2b1f-4970-b9c2-fc1e4bb58652-istio-data\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-cmmcs\" (UID: \"06d7fd9a-2b1f-4970-b9c2-fc1e4bb58652\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-cmmcs"
Nov 02 13:34:29.097579 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:29.097467    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06d7fd9a-2b1f-4970-b9c2-fc1e4bb58652-istio-envoy\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-cmmcs\" (UID: \"06d7fd9a-2b1f-4970-b9c2-fc1e4bb58652\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-cmmcs"
Nov 02 13:34:29.097579 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:29.097492    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06d7fd9a-2b1f-4970-b9c2-fc1e4bb58652-workload-certs\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-cmmcs\" (UID: \"06d7fd9a-2b1f-4970-b9c2-fc1e4bb58652\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-cmmcs"
Nov 02 13:34:29.097579 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:29.097525    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06d7fd9a-2b1f-4970-b9c2-fc1e4bb58652-istio-podinfo\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-cmmcs\" (UID: \"06d7fd9a-2b1f-4970-b9c2-fc1e4bb58652\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-cmmcs"
Nov 02 13:34:29.097579 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:29.097555    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06d7fd9a-2b1f-4970-b9c2-fc1e4bb58652-workload-socket\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-cmmcs\" (UID: \"06d7fd9a-2b1f-4970-b9c2-fc1e4bb58652\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-cmmcs"
Nov 02 13:34:29.097825 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:29.097689    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06d7fd9a-2b1f-4970-b9c2-fc1e4bb58652-istiod-ca-cert\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-cmmcs\" (UID: \"06d7fd9a-2b1f-4970-b9c2-fc1e4bb58652\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-cmmcs"
Nov 02 13:34:29.097825 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:29.097757    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06d7fd9a-2b1f-4970-b9c2-fc1e4bb58652-istio-token\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-cmmcs\" (UID: \"06d7fd9a-2b1f-4970-b9c2-fc1e4bb58652\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-cmmcs"
Nov 02 13:34:29.097825 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:29.097808    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06d7fd9a-2b1f-4970-b9c2-fc1e4bb58652-credential-socket\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-cmmcs\" (UID: \"06d7fd9a-2b1f-4970-b9c2-fc1e4bb58652\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-cmmcs"
Nov 02 13:34:29.097986 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:29.097835    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06d7fd9a-2b1f-4970-b9c2-fc1e4bb58652-istio-data\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-cmmcs\" (UID: \"06d7fd9a-2b1f-4970-b9c2-fc1e4bb58652\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-cmmcs"
Nov 02 13:34:29.097986 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:29.097837    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwrsx\" (UniqueName: \"kubernetes.io/projected/06d7fd9a-2b1f-4970-b9c2-fc1e4bb58652-kube-api-access-mwrsx\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-cmmcs\" (UID: \"06d7fd9a-2b1f-4970-b9c2-fc1e4bb58652\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-cmmcs"
Nov 02 13:34:29.097986 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:29.097876    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06d7fd9a-2b1f-4970-b9c2-fc1e4bb58652-workload-certs\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-cmmcs\" (UID: \"06d7fd9a-2b1f-4970-b9c2-fc1e4bb58652\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-cmmcs"
Nov 02 13:34:29.097986 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:29.097937    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06d7fd9a-2b1f-4970-b9c2-fc1e4bb58652-workload-socket\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-cmmcs\" (UID: \"06d7fd9a-2b1f-4970-b9c2-fc1e4bb58652\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-cmmcs"
Nov 02 13:34:29.098229 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:29.098206    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06d7fd9a-2b1f-4970-b9c2-fc1e4bb58652-credential-socket\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-cmmcs\" (UID: \"06d7fd9a-2b1f-4970-b9c2-fc1e4bb58652\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-cmmcs"
Nov 02 13:34:29.098407 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:29.098390    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06d7fd9a-2b1f-4970-b9c2-fc1e4bb58652-istiod-ca-cert\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-cmmcs\" (UID: \"06d7fd9a-2b1f-4970-b9c2-fc1e4bb58652\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-cmmcs"
Nov 02 13:34:29.100490 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:29.100461    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06d7fd9a-2b1f-4970-b9c2-fc1e4bb58652-istio-envoy\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-cmmcs\" (UID: \"06d7fd9a-2b1f-4970-b9c2-fc1e4bb58652\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-cmmcs"
Nov 02 13:34:29.100622 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:29.100603    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06d7fd9a-2b1f-4970-b9c2-fc1e4bb58652-istio-podinfo\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-cmmcs\" (UID: \"06d7fd9a-2b1f-4970-b9c2-fc1e4bb58652\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-cmmcs"
Nov 02 13:34:29.126249 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:29.126222    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06d7fd9a-2b1f-4970-b9c2-fc1e4bb58652-istio-token\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-cmmcs\" (UID: \"06d7fd9a-2b1f-4970-b9c2-fc1e4bb58652\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-cmmcs"
Nov 02 13:34:29.126374 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:29.126340    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwrsx\" (UniqueName: \"kubernetes.io/projected/06d7fd9a-2b1f-4970-b9c2-fc1e4bb58652-kube-api-access-mwrsx\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-cmmcs\" (UID: \"06d7fd9a-2b1f-4970-b9c2-fc1e4bb58652\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-cmmcs"
Nov 02 13:34:29.219231 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:29.219207    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-cmmcs"
Nov 02 13:34:29.539278 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:29.539249    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-cmmcs"]
Nov 02 13:34:29.539748 ip-10-0-128-132 kubenswrapper[4559]: W1102 13:34:29.539727    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06d7fd9a_2b1f_4970_b9c2_fc1e4bb58652.slice/crio-4346019ad5c2c66dd15727d1706ac3272c742e6a14bbf335baf57cac7ed269d5 WatchSource:0}: Error finding container 4346019ad5c2c66dd15727d1706ac3272c742e6a14bbf335baf57cac7ed269d5: Status 404 returned error can't find the container with id 4346019ad5c2c66dd15727d1706ac3272c742e6a14bbf335baf57cac7ed269d5
Nov 02 13:34:29.541904 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:29.541870    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 13:34:29.541990 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:29.541935    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 13:34:29.541990 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:29.541967    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 13:34:30.551025 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:30.550993    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-cmmcs" event={"ID":"06d7fd9a-2b1f-4970-b9c2-fc1e4bb58652","Type":"ContainerStarted","Data":"1b14be44ba212aa83409be14519113a51a08a1c45ad3dbad8265aa8e92e470f6"}
Nov 02 13:34:30.551424 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:30.551067    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-cmmcs" event={"ID":"06d7fd9a-2b1f-4970-b9c2-fc1e4bb58652","Type":"ContainerStarted","Data":"4346019ad5c2c66dd15727d1706ac3272c742e6a14bbf335baf57cac7ed269d5"}
Nov 02 13:34:30.625225 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:30.625168    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-cmmcs" podStartSLOduration=2.625152357 podStartE2EDuration="2.625152357s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:34:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:34:30.611007639 +0000 UTC m=+4066.866406733" watchObservedRunningTime="2026-11-02 13:34:30.625152357 +0000 UTC m=+4066.880551530"
Nov 02 13:34:31.219880 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:31.219847    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-cmmcs"
Nov 02 13:34:31.221174 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:31.221149    4559 patch_prober.go:29] interesting pod/e2e-worker-w1-61e341-openshift-default-59d97476dc-cmmcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.3:15021/healthz/ready\": dial tcp 10.133.1.3:15021: connect: connection refused" start-of-body=
Nov 02 13:34:31.221293 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:31.221211    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-cmmcs" podUID="06d7fd9a-2b1f-4970-b9c2-fc1e4bb58652" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.3:15021/healthz/ready\": dial tcp 10.133.1.3:15021: connect: connection refused"
Nov 02 13:34:32.219716 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:32.219679    4559 patch_prober.go:29] interesting pod/e2e-worker-w1-61e341-openshift-default-59d97476dc-cmmcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.3:15021/healthz/ready\": dial tcp 10.133.1.3:15021: connect: connection refused" start-of-body=
Nov 02 13:34:32.220147 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:32.219739    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-cmmcs" podUID="06d7fd9a-2b1f-4970-b9c2-fc1e4bb58652" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.3:15021/healthz/ready\": dial tcp 10.133.1.3:15021: connect: connection refused"
Nov 02 13:34:33.219759 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:33.219718    4559 patch_prober.go:29] interesting pod/e2e-worker-w1-61e341-openshift-default-59d97476dc-cmmcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.3:15021/healthz/ready\": dial tcp 10.133.1.3:15021: connect: connection refused" start-of-body=
Nov 02 13:34:33.220166 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:33.219792    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-cmmcs" podUID="06d7fd9a-2b1f-4970-b9c2-fc1e4bb58652" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.3:15021/healthz/ready\": dial tcp 10.133.1.3:15021: connect: connection refused"
Nov 02 13:34:34.220235 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:34.220199    4559 patch_prober.go:29] interesting pod/e2e-worker-w1-61e341-openshift-default-59d97476dc-cmmcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.3:15021/healthz/ready\": dial tcp 10.133.1.3:15021: connect: connection refused" start-of-body=
Nov 02 13:34:34.220659 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:34.220268    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-cmmcs" podUID="06d7fd9a-2b1f-4970-b9c2-fc1e4bb58652" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.3:15021/healthz/ready\": dial tcp 10.133.1.3:15021: connect: connection refused"
Nov 02 13:34:35.219731 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:35.219699    4559 patch_prober.go:29] interesting pod/e2e-worker-w1-61e341-openshift-default-59d97476dc-cmmcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.3:15021/healthz/ready\": dial tcp 10.133.1.3:15021: connect: connection refused" start-of-body=
Nov 02 13:34:35.219915 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:35.219749    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-cmmcs" podUID="06d7fd9a-2b1f-4970-b9c2-fc1e4bb58652" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.3:15021/healthz/ready\": dial tcp 10.133.1.3:15021: connect: connection refused"
Nov 02 13:34:36.220206 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:36.220170    4559 patch_prober.go:29] interesting pod/e2e-worker-w1-61e341-openshift-default-59d97476dc-cmmcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.3:15021/healthz/ready\": dial tcp 10.133.1.3:15021: connect: connection refused" start-of-body=
Nov 02 13:34:36.220597 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:36.220218    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-cmmcs" podUID="06d7fd9a-2b1f-4970-b9c2-fc1e4bb58652" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.3:15021/healthz/ready\": dial tcp 10.133.1.3:15021: connect: connection refused"
Nov 02 13:34:37.237893 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:37.237862    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-cmmcs"
Nov 02 13:34:37.238411 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:37.238227    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-cmmcs"
Nov 02 13:34:37.239098 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:37.239081    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-cmmcs"
Nov 02 13:34:41.109203 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:41.109167    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"]
Nov 02 13:34:43.347130 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:43.347094    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-cmmcs"]
Nov 02 13:34:43.347925 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:43.347885    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-cmmcs" podUID="06d7fd9a-2b1f-4970-b9c2-fc1e4bb58652" containerName="istio-proxy" containerID="cri-o://1b14be44ba212aa83409be14519113a51a08a1c45ad3dbad8265aa8e92e470f6" gracePeriod=30
Nov 02 13:34:43.420759 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:43.420725    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-95ee26-5f96f56f79-2zqdd"]
Nov 02 13:34:45.085075 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:34:45.085031    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqcmj" containerName="extractor"
Nov 02 13:34:48.422003 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:48.421961    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w2-95ee26-5f96f56f79-2zqdd" podUID="57c75efe-f588-45c2-b5ea-2f948d0e6e13" containerName="payload-processing" containerID="cri-o://c6ee188467a1f51ac30d421c5c1528e70aa6d6757253b757faed5e2768b2e0b6" gracePeriod=25
Nov 02 13:34:48.458617 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:48.458585    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-95ee26-5f96f56f79-2zqdd" podUID="57c75efe-f588-45c2-b5ea-2f948d0e6e13" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 13:34:48.599021 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:48.598991    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-cmmcs"
Nov 02 13:34:48.643794 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:48.643740    4559 generic.go:356] "Generic (PLEG): container finished" podID="06d7fd9a-2b1f-4970-b9c2-fc1e4bb58652" containerID="1b14be44ba212aa83409be14519113a51a08a1c45ad3dbad8265aa8e92e470f6" exitCode=0
Nov 02 13:34:48.643794 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:48.643787    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-cmmcs"
Nov 02 13:34:48.643956 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:48.643804    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-cmmcs" event={"ID":"06d7fd9a-2b1f-4970-b9c2-fc1e4bb58652","Type":"ContainerDied","Data":"1b14be44ba212aa83409be14519113a51a08a1c45ad3dbad8265aa8e92e470f6"}
Nov 02 13:34:48.643956 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:48.643844    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-cmmcs" event={"ID":"06d7fd9a-2b1f-4970-b9c2-fc1e4bb58652","Type":"ContainerDied","Data":"4346019ad5c2c66dd15727d1706ac3272c742e6a14bbf335baf57cac7ed269d5"}
Nov 02 13:34:48.643956 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:48.643866    4559 scope.go:122] "RemoveContainer" containerID="1b14be44ba212aa83409be14519113a51a08a1c45ad3dbad8265aa8e92e470f6"
Nov 02 13:34:48.652324 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:48.652308    4559 scope.go:122] "RemoveContainer" containerID="1b14be44ba212aa83409be14519113a51a08a1c45ad3dbad8265aa8e92e470f6"
Nov 02 13:34:48.652587 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:34:48.652569    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b14be44ba212aa83409be14519113a51a08a1c45ad3dbad8265aa8e92e470f6\": container with ID starting with 1b14be44ba212aa83409be14519113a51a08a1c45ad3dbad8265aa8e92e470f6 not found: ID does not exist" containerID="1b14be44ba212aa83409be14519113a51a08a1c45ad3dbad8265aa8e92e470f6"
Nov 02 13:34:48.652664 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:48.652595    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b14be44ba212aa83409be14519113a51a08a1c45ad3dbad8265aa8e92e470f6"} err="failed to get container status \"1b14be44ba212aa83409be14519113a51a08a1c45ad3dbad8265aa8e92e470f6\": rpc error: code = NotFound desc = could not find container \"1b14be44ba212aa83409be14519113a51a08a1c45ad3dbad8265aa8e92e470f6\": container with ID starting with 1b14be44ba212aa83409be14519113a51a08a1c45ad3dbad8265aa8e92e470f6 not found: ID does not exist"
Nov 02 13:34:48.688650 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:48.688622    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06d7fd9a-2b1f-4970-b9c2-fc1e4bb58652-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06d7fd9a-2b1f-4970-b9c2-fc1e4bb58652-credential-socket\") pod \"06d7fd9a-2b1f-4970-b9c2-fc1e4bb58652\" (UID: \"06d7fd9a-2b1f-4970-b9c2-fc1e4bb58652\") "
Nov 02 13:34:48.688765 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:48.688693    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06d7fd9a-2b1f-4970-b9c2-fc1e4bb58652-istio-token\" (UniqueName: \"kubernetes.io/projected/06d7fd9a-2b1f-4970-b9c2-fc1e4bb58652-istio-token\") pod \"06d7fd9a-2b1f-4970-b9c2-fc1e4bb58652\" (UID: \"06d7fd9a-2b1f-4970-b9c2-fc1e4bb58652\") "
Nov 02 13:34:48.688765 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:48.688719    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06d7fd9a-2b1f-4970-b9c2-fc1e4bb58652-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06d7fd9a-2b1f-4970-b9c2-fc1e4bb58652-workload-certs\") pod \"06d7fd9a-2b1f-4970-b9c2-fc1e4bb58652\" (UID: \"06d7fd9a-2b1f-4970-b9c2-fc1e4bb58652\") "
Nov 02 13:34:48.688765 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:48.688739    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06d7fd9a-2b1f-4970-b9c2-fc1e4bb58652-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06d7fd9a-2b1f-4970-b9c2-fc1e4bb58652-istio-data\") pod \"06d7fd9a-2b1f-4970-b9c2-fc1e4bb58652\" (UID: \"06d7fd9a-2b1f-4970-b9c2-fc1e4bb58652\") "
Nov 02 13:34:48.688940 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:48.688796    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06d7fd9a-2b1f-4970-b9c2-fc1e4bb58652-kube-api-access-mwrsx\" (UniqueName: \"kubernetes.io/projected/06d7fd9a-2b1f-4970-b9c2-fc1e4bb58652-kube-api-access-mwrsx\") pod \"06d7fd9a-2b1f-4970-b9c2-fc1e4bb58652\" (UID: \"06d7fd9a-2b1f-4970-b9c2-fc1e4bb58652\") "
Nov 02 13:34:48.688940 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:48.688835    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06d7fd9a-2b1f-4970-b9c2-fc1e4bb58652-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06d7fd9a-2b1f-4970-b9c2-fc1e4bb58652-workload-socket\") pod \"06d7fd9a-2b1f-4970-b9c2-fc1e4bb58652\" (UID: \"06d7fd9a-2b1f-4970-b9c2-fc1e4bb58652\") "
Nov 02 13:34:48.688940 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:48.688872    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/06d7fd9a-2b1f-4970-b9c2-fc1e4bb58652-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06d7fd9a-2b1f-4970-b9c2-fc1e4bb58652-istio-podinfo\") pod \"06d7fd9a-2b1f-4970-b9c2-fc1e4bb58652\" (UID: \"06d7fd9a-2b1f-4970-b9c2-fc1e4bb58652\") "
Nov 02 13:34:48.688940 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:48.688903    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06d7fd9a-2b1f-4970-b9c2-fc1e4bb58652-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06d7fd9a-2b1f-4970-b9c2-fc1e4bb58652-istiod-ca-cert\") pod \"06d7fd9a-2b1f-4970-b9c2-fc1e4bb58652\" (UID: \"06d7fd9a-2b1f-4970-b9c2-fc1e4bb58652\") "
Nov 02 13:34:48.689160 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:48.688935    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06d7fd9a-2b1f-4970-b9c2-fc1e4bb58652-credential-socket" pod "06d7fd9a-2b1f-4970-b9c2-fc1e4bb58652" (UID: "06d7fd9a-2b1f-4970-b9c2-fc1e4bb58652"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:34:48.689160 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:48.688979    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06d7fd9a-2b1f-4970-b9c2-fc1e4bb58652-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06d7fd9a-2b1f-4970-b9c2-fc1e4bb58652-istio-envoy\") pod \"06d7fd9a-2b1f-4970-b9c2-fc1e4bb58652\" (UID: \"06d7fd9a-2b1f-4970-b9c2-fc1e4bb58652\") "
Nov 02 13:34:48.689160 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:48.689068    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06d7fd9a-2b1f-4970-b9c2-fc1e4bb58652-workload-certs" pod "06d7fd9a-2b1f-4970-b9c2-fc1e4bb58652" (UID: "06d7fd9a-2b1f-4970-b9c2-fc1e4bb58652"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:34:48.689160 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:48.689078    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06d7fd9a-2b1f-4970-b9c2-fc1e4bb58652-istio-data" pod "06d7fd9a-2b1f-4970-b9c2-fc1e4bb58652" (UID: "06d7fd9a-2b1f-4970-b9c2-fc1e4bb58652"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:34:48.689395 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:48.689188    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06d7fd9a-2b1f-4970-b9c2-fc1e4bb58652-workload-socket" pod "06d7fd9a-2b1f-4970-b9c2-fc1e4bb58652" (UID: "06d7fd9a-2b1f-4970-b9c2-fc1e4bb58652"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:34:48.689474 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:48.689443    4559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06d7fd9a-2b1f-4970-b9c2-fc1e4bb58652-credential-socket\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:34:48.689474 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:48.689465    4559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06d7fd9a-2b1f-4970-b9c2-fc1e4bb58652-workload-certs\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:34:48.689598 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:48.689479    4559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06d7fd9a-2b1f-4970-b9c2-fc1e4bb58652-istio-data\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:34:48.689598 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:48.689478    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06d7fd9a-2b1f-4970-b9c2-fc1e4bb58652-istiod-ca-cert" pod "06d7fd9a-2b1f-4970-b9c2-fc1e4bb58652" (UID: "06d7fd9a-2b1f-4970-b9c2-fc1e4bb58652"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:34:48.689598 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:48.689493    4559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06d7fd9a-2b1f-4970-b9c2-fc1e4bb58652-workload-socket\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:34:48.691300 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:48.691265    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06d7fd9a-2b1f-4970-b9c2-fc1e4bb58652-kube-api-access-mwrsx" pod "06d7fd9a-2b1f-4970-b9c2-fc1e4bb58652" (UID: "06d7fd9a-2b1f-4970-b9c2-fc1e4bb58652"). InnerVolumeSpecName "kube-api-access-mwrsx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:34:48.691413 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:48.691398    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/06d7fd9a-2b1f-4970-b9c2-fc1e4bb58652-istio-podinfo" pod "06d7fd9a-2b1f-4970-b9c2-fc1e4bb58652" (UID: "06d7fd9a-2b1f-4970-b9c2-fc1e4bb58652"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 13:34:48.691778 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:48.691755    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06d7fd9a-2b1f-4970-b9c2-fc1e4bb58652-istio-envoy" pod "06d7fd9a-2b1f-4970-b9c2-fc1e4bb58652" (UID: "06d7fd9a-2b1f-4970-b9c2-fc1e4bb58652"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:34:48.691858 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:48.691779    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06d7fd9a-2b1f-4970-b9c2-fc1e4bb58652-istio-token" pod "06d7fd9a-2b1f-4970-b9c2-fc1e4bb58652" (UID: "06d7fd9a-2b1f-4970-b9c2-fc1e4bb58652"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:34:48.790808 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:48.790630    4559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06d7fd9a-2b1f-4970-b9c2-fc1e4bb58652-istio-token\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:34:48.790930 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:48.790815    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mwrsx\" (UniqueName: \"kubernetes.io/projected/06d7fd9a-2b1f-4970-b9c2-fc1e4bb58652-kube-api-access-mwrsx\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:34:48.790930 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:48.790827    4559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06d7fd9a-2b1f-4970-b9c2-fc1e4bb58652-istio-podinfo\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:34:48.790930 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:48.790835    4559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06d7fd9a-2b1f-4970-b9c2-fc1e4bb58652-istiod-ca-cert\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:34:48.790930 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:48.790844    4559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06d7fd9a-2b1f-4970-b9c2-fc1e4bb58652-istio-envoy\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:34:48.991412 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:48.991383    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-cmmcs"]
Nov 02 13:34:48.996893 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:48.996868    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-cmmcs"]
Nov 02 13:34:49.892009 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:49.891975    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06d7fd9a-2b1f-4970-b9c2-fc1e4bb58652" path="/var/lib/kubelet/pods/06d7fd9a-2b1f-4970-b9c2-fc1e4bb58652/volumes"
Nov 02 13:34:57.971801 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:57.971721    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-6992-a1cce2"]
Nov 02 13:34:57.972310 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:57.972288    4559 cpu_manager.go:414] "containerMap: removing container" podUID="06d7fd9a-2b1f-4970-b9c2-fc1e4bb58652" containerName="istio-proxy"
Nov 02 13:34:57.972401 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:57.972312    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06d7fd9a-2b1f-4970-b9c2-fc1e4bb58652" containerName="istio-proxy"
Nov 02 13:34:57.972462 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:57.972444    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="8debb7bd-d249-47d9-8776-f2be8f0d4f27" containerName="istio-proxy"
Nov 02 13:34:57.972525 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:57.972467    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="06d7fd9a-2b1f-4970-b9c2-fc1e4bb58652" containerName="istio-proxy"
Nov 02 13:34:57.975543 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:57.975524    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6992-a1cce2"
Nov 02 13:34:57.992614 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:57.992457    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-hfcns\""
Nov 02 13:34:58.005781 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:58.005759    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6992-a1cce2"]
Nov 02 13:34:58.081955 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:58.081926    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcxtn\" (UniqueName: \"kubernetes.io/projected/7d0dc3a4-9ced-49cb-86bc-9538932f1682-kube-api-access-kcxtn\") pod \"test-curl-6992-a1cce2\" (UID: \"7d0dc3a4-9ced-49cb-86bc-9538932f1682\") " pod="opendatahub/test-curl-6992-a1cce2"
Nov 02 13:34:58.182693 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:58.182661    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcxtn\" (UniqueName: \"kubernetes.io/projected/7d0dc3a4-9ced-49cb-86bc-9538932f1682-kube-api-access-kcxtn\") pod \"test-curl-6992-a1cce2\" (UID: \"7d0dc3a4-9ced-49cb-86bc-9538932f1682\") " pod="opendatahub/test-curl-6992-a1cce2"
Nov 02 13:34:58.222653 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:58.222573    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcxtn\" (UniqueName: \"kubernetes.io/projected/7d0dc3a4-9ced-49cb-86bc-9538932f1682-kube-api-access-kcxtn\") pod \"test-curl-6992-a1cce2\" (UID: \"7d0dc3a4-9ced-49cb-86bc-9538932f1682\") " pod="opendatahub/test-curl-6992-a1cce2"
Nov 02 13:34:58.287324 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:58.287300    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6992-a1cce2"
Nov 02 13:34:58.457880 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:58.457839    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-95ee26-5f96f56f79-2zqdd" podUID="57c75efe-f588-45c2-b5ea-2f948d0e6e13" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 13:34:58.585088 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:58.585059    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6992-a1cce2"]
Nov 02 13:34:58.586063 ip-10-0-128-132 kubenswrapper[4559]: W1102 13:34:58.586024    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7d0dc3a4_9ced_49cb_86bc_9538932f1682.slice/crio-dc814f7e36477cc30b475df39d5c1ac06478839d6d2c1556ff844cae0427d36a WatchSource:0}: Error finding container dc814f7e36477cc30b475df39d5c1ac06478839d6d2c1556ff844cae0427d36a: Status 404 returned error can't find the container with id dc814f7e36477cc30b475df39d5c1ac06478839d6d2c1556ff844cae0427d36a
Nov 02 13:34:58.711859 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:58.711828    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6992-a1cce2" event={"ID":"7d0dc3a4-9ced-49cb-86bc-9538932f1682","Type":"ContainerStarted","Data":"06516eff570f17825d2ab8309c6ad7d1630ff8ef3a10f9c969ce75140246ff89"}
Nov 02 13:34:58.712008 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:58.711870    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6992-a1cce2" event={"ID":"7d0dc3a4-9ced-49cb-86bc-9538932f1682","Type":"ContainerStarted","Data":"dc814f7e36477cc30b475df39d5c1ac06478839d6d2c1556ff844cae0427d36a"}
Nov 02 13:34:58.754880 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:58.754831    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-6992-a1cce2" podStartSLOduration=1.754813098 podStartE2EDuration="1.754813098s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:34:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:34:58.753022285 +0000 UTC m=+4095.008421380" watchObservedRunningTime="2026-11-02 13:34:58.754813098 +0000 UTC m=+4095.010212196"
Nov 02 13:34:59.634430 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:59.634396    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-6992-a1cce2"]
Nov 02 13:34:59.636798 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:59.636773    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-6992-a1cce2"]
Nov 02 13:34:59.787276 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:59.787247    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-6992-fefa6a"]
Nov 02 13:34:59.787698 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:59.787684    4559 cpu_manager.go:414] "containerMap: removing container" podUID="7d0dc3a4-9ced-49cb-86bc-9538932f1682" containerName="test-curl-6992-a1cce2"
Nov 02 13:34:59.787755 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:59.787699    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d0dc3a4-9ced-49cb-86bc-9538932f1682" containerName="test-curl-6992-a1cce2"
Nov 02 13:34:59.787793 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:59.787775    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d0dc3a4-9ced-49cb-86bc-9538932f1682" containerName="test-curl-6992-a1cce2"
Nov 02 13:34:59.791006 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:59.790990    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6992-fefa6a"
Nov 02 13:34:59.807219 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:59.807192    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6992-fefa6a"]
Nov 02 13:34:59.897837 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:59.897811    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6b4m\" (UniqueName: \"kubernetes.io/projected/738b99e2-1698-4101-a6a9-2c2778bdfe2b-kube-api-access-w6b4m\") pod \"test-curl-6992-fefa6a\" (UID: \"738b99e2-1698-4101-a6a9-2c2778bdfe2b\") " pod="opendatahub/test-curl-6992-fefa6a"
Nov 02 13:34:59.998593 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:34:59.998562    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6b4m\" (UniqueName: \"kubernetes.io/projected/738b99e2-1698-4101-a6a9-2c2778bdfe2b-kube-api-access-w6b4m\") pod \"test-curl-6992-fefa6a\" (UID: \"738b99e2-1698-4101-a6a9-2c2778bdfe2b\") " pod="opendatahub/test-curl-6992-fefa6a"
Nov 02 13:35:00.022447 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:35:00.022410    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6b4m\" (UniqueName: \"kubernetes.io/projected/738b99e2-1698-4101-a6a9-2c2778bdfe2b-kube-api-access-w6b4m\") pod \"test-curl-6992-fefa6a\" (UID: \"738b99e2-1698-4101-a6a9-2c2778bdfe2b\") " pod="opendatahub/test-curl-6992-fefa6a"
Nov 02 13:35:00.102580 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:35:00.102549    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6992-fefa6a"
Nov 02 13:35:00.397735 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:35:00.397707    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6992-fefa6a"]
Nov 02 13:35:00.398392 ip-10-0-128-132 kubenswrapper[4559]: W1102 13:35:00.398367    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod738b99e2_1698_4101_a6a9_2c2778bdfe2b.slice/crio-7f5960f94b398b6777b391027cf8bd802f2c508ebcc94bc3894de7218a6c1ee4 WatchSource:0}: Error finding container 7f5960f94b398b6777b391027cf8bd802f2c508ebcc94bc3894de7218a6c1ee4: Status 404 returned error can't find the container with id 7f5960f94b398b6777b391027cf8bd802f2c508ebcc94bc3894de7218a6c1ee4
Nov 02 13:35:00.723545 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:35:00.723506    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6992-fefa6a" event={"ID":"738b99e2-1698-4101-a6a9-2c2778bdfe2b","Type":"ContainerStarted","Data":"b9065e1540aa8f02ef28ccde4f2c51a6f1f0b3fdfbda830de14e9449a9e4f410"}
Nov 02 13:35:00.723972 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:35:00.723551    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6992-fefa6a" event={"ID":"738b99e2-1698-4101-a6a9-2c2778bdfe2b","Type":"ContainerStarted","Data":"7f5960f94b398b6777b391027cf8bd802f2c508ebcc94bc3894de7218a6c1ee4"}
Nov 02 13:35:00.723972 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:35:00.723538    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-6992-a1cce2" podUID="7d0dc3a4-9ced-49cb-86bc-9538932f1682" containerName="test-curl-6992-a1cce2" containerID="cri-o://06516eff570f17825d2ab8309c6ad7d1630ff8ef3a10f9c969ce75140246ff89" gracePeriod=2
Nov 02 13:35:00.764952 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:35:00.764900    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-6992-fefa6a" podStartSLOduration=1.764883535 podStartE2EDuration="1.764883535s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:34:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:35:00.762599121 +0000 UTC m=+4097.017998221" watchObservedRunningTime="2026-11-02 13:35:00.764883535 +0000 UTC m=+4097.020282628"
Nov 02 13:35:01.599742 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:35:01.599709    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"]
Nov 02 13:35:01.611975 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:35:01.611948    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-6992-fefa6a"]
Nov 02 13:35:01.616720 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:35:01.616697    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-6992-fefa6a"]
Nov 02 13:35:02.733594 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:35:02.733559    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-6992-fefa6a" podUID="738b99e2-1698-4101-a6a9-2c2778bdfe2b" containerName="test-curl-6992-fefa6a" containerID="cri-o://b9065e1540aa8f02ef28ccde4f2c51a6f1f0b3fdfbda830de14e9449a9e4f410" gracePeriod=2
Nov 02 13:35:02.922987 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:35:02.922964    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6992-a1cce2"
Nov 02 13:35:03.026974 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:35:03.026916    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d0dc3a4-9ced-49cb-86bc-9538932f1682-kube-api-access-kcxtn\" (UniqueName: \"kubernetes.io/projected/7d0dc3a4-9ced-49cb-86bc-9538932f1682-kube-api-access-kcxtn\") pod \"7d0dc3a4-9ced-49cb-86bc-9538932f1682\" (UID: \"7d0dc3a4-9ced-49cb-86bc-9538932f1682\") "
Nov 02 13:35:03.028884 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:35:03.028857    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d0dc3a4-9ced-49cb-86bc-9538932f1682-kube-api-access-kcxtn" pod "7d0dc3a4-9ced-49cb-86bc-9538932f1682" (UID: "7d0dc3a4-9ced-49cb-86bc-9538932f1682"). InnerVolumeSpecName "kube-api-access-kcxtn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:35:03.128334 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:35:03.128299    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kcxtn\" (UniqueName: \"kubernetes.io/projected/7d0dc3a4-9ced-49cb-86bc-9538932f1682-kube-api-access-kcxtn\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:35:03.739530 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:35:03.739499    4559 generic.go:356] "Generic (PLEG): container finished" podID="7d0dc3a4-9ced-49cb-86bc-9538932f1682" containerID="06516eff570f17825d2ab8309c6ad7d1630ff8ef3a10f9c969ce75140246ff89" exitCode=137
Nov 02 13:35:03.740043 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:35:03.739554    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6992-a1cce2"
Nov 02 13:35:03.740043 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:35:03.739566    4559 scope.go:122] "RemoveContainer" containerID="06516eff570f17825d2ab8309c6ad7d1630ff8ef3a10f9c969ce75140246ff89"
Nov 02 13:35:03.749787 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:35:03.749767    4559 scope.go:122] "RemoveContainer" containerID="06516eff570f17825d2ab8309c6ad7d1630ff8ef3a10f9c969ce75140246ff89"
Nov 02 13:35:03.750039 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:35:03.750020    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06516eff570f17825d2ab8309c6ad7d1630ff8ef3a10f9c969ce75140246ff89\": container with ID starting with 06516eff570f17825d2ab8309c6ad7d1630ff8ef3a10f9c969ce75140246ff89 not found: ID does not exist" containerID="06516eff570f17825d2ab8309c6ad7d1630ff8ef3a10f9c969ce75140246ff89"
Nov 02 13:35:03.750098 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:35:03.750045    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06516eff570f17825d2ab8309c6ad7d1630ff8ef3a10f9c969ce75140246ff89"} err="failed to get container status \"06516eff570f17825d2ab8309c6ad7d1630ff8ef3a10f9c969ce75140246ff89\": rpc error: code = NotFound desc = could not find container \"06516eff570f17825d2ab8309c6ad7d1630ff8ef3a10f9c969ce75140246ff89\": container with ID starting with 06516eff570f17825d2ab8309c6ad7d1630ff8ef3a10f9c969ce75140246ff89 not found: ID does not exist"
Nov 02 13:35:03.763911 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:35:03.763878    4559 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6992-a1cce2\" is forbidden: User \"system:node:ip-10-0-128-132.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-132.ec2.internal' and this object" podUID="7d0dc3a4-9ced-49cb-86bc-9538932f1682" pod="opendatahub/test-curl-6992-a1cce2"
Nov 02 13:35:03.891571 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:35:03.891545    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d0dc3a4-9ced-49cb-86bc-9538932f1682" path="/var/lib/kubelet/pods/7d0dc3a4-9ced-49cb-86bc-9538932f1682/volumes"
Nov 02 13:35:03.902177 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:35:03.902152    4559 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6992-a1cce2\" is forbidden: User \"system:node:ip-10-0-128-132.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-132.ec2.internal' and this object" podUID="7d0dc3a4-9ced-49cb-86bc-9538932f1682" pod="opendatahub/test-curl-6992-a1cce2"
Nov 02 13:35:04.922978 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:35:04.922954    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6992-fefa6a"
Nov 02 13:35:05.050263 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:35:05.050200    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/738b99e2-1698-4101-a6a9-2c2778bdfe2b-kube-api-access-w6b4m\" (UniqueName: \"kubernetes.io/projected/738b99e2-1698-4101-a6a9-2c2778bdfe2b-kube-api-access-w6b4m\") pod \"738b99e2-1698-4101-a6a9-2c2778bdfe2b\" (UID: \"738b99e2-1698-4101-a6a9-2c2778bdfe2b\") "
Nov 02 13:35:05.052178 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:35:05.052155    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/738b99e2-1698-4101-a6a9-2c2778bdfe2b-kube-api-access-w6b4m" pod "738b99e2-1698-4101-a6a9-2c2778bdfe2b" (UID: "738b99e2-1698-4101-a6a9-2c2778bdfe2b"). InnerVolumeSpecName "kube-api-access-w6b4m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:35:05.151498 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:35:05.151472    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w6b4m\" (UniqueName: \"kubernetes.io/projected/738b99e2-1698-4101-a6a9-2c2778bdfe2b-kube-api-access-w6b4m\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:35:05.758345 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:35:05.758310    4559 generic.go:356] "Generic (PLEG): container finished" podID="738b99e2-1698-4101-a6a9-2c2778bdfe2b" containerID="b9065e1540aa8f02ef28ccde4f2c51a6f1f0b3fdfbda830de14e9449a9e4f410" exitCode=137
Nov 02 13:35:05.758566 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:35:05.758353    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6992-fefa6a"
Nov 02 13:35:05.758566 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:35:05.758412    4559 scope.go:122] "RemoveContainer" containerID="b9065e1540aa8f02ef28ccde4f2c51a6f1f0b3fdfbda830de14e9449a9e4f410"
Nov 02 13:35:05.767938 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:35:05.767917    4559 scope.go:122] "RemoveContainer" containerID="b9065e1540aa8f02ef28ccde4f2c51a6f1f0b3fdfbda830de14e9449a9e4f410"
Nov 02 13:35:05.768208 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:35:05.768184    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9065e1540aa8f02ef28ccde4f2c51a6f1f0b3fdfbda830de14e9449a9e4f410\": container with ID starting with b9065e1540aa8f02ef28ccde4f2c51a6f1f0b3fdfbda830de14e9449a9e4f410 not found: ID does not exist" containerID="b9065e1540aa8f02ef28ccde4f2c51a6f1f0b3fdfbda830de14e9449a9e4f410"
Nov 02 13:35:05.768285 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:35:05.768220    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9065e1540aa8f02ef28ccde4f2c51a6f1f0b3fdfbda830de14e9449a9e4f410"} err="failed to get container status \"b9065e1540aa8f02ef28ccde4f2c51a6f1f0b3fdfbda830de14e9449a9e4f410\": rpc error: code = NotFound desc = could not find container \"b9065e1540aa8f02ef28ccde4f2c51a6f1f0b3fdfbda830de14e9449a9e4f410\": container with ID starting with b9065e1540aa8f02ef28ccde4f2c51a6f1f0b3fdfbda830de14e9449a9e4f410 not found: ID does not exist"
Nov 02 13:35:05.780820 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:35:05.780793    4559 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6992-fefa6a\" is forbidden: User \"system:node:ip-10-0-128-132.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-132.ec2.internal' and this object" podUID="738b99e2-1698-4101-a6a9-2c2778bdfe2b" pod="opendatahub/test-curl-6992-fefa6a"
Nov 02 13:35:05.891930 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:35:05.891896    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="738b99e2-1698-4101-a6a9-2c2778bdfe2b" path="/var/lib/kubelet/pods/738b99e2-1698-4101-a6a9-2c2778bdfe2b/volumes"
Nov 02 13:35:06.924681 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:35:06.924620    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-95ee26-769ff594c9-wdwnp"]
Nov 02 13:35:06.925139 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:35:06.924814    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-95ee26-769ff594c9-wdwnp" podUID="90adff48-e3d2-445f-9c4c-4133e5820b26" containerName="maas-api" containerID="cri-o://fd10839dd8698d770627382c12113ca42d22a8196b2aa329f9a362fd8853d2b2" gracePeriod=30
Nov 02 13:35:07.161508 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:35:07.161481    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-95ee26-769ff594c9-wdwnp"
Nov 02 13:35:07.269047 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:35:07.268963    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90adff48-e3d2-445f-9c4c-4133e5820b26-kube-api-access-tw2fb\" (UniqueName: \"kubernetes.io/projected/90adff48-e3d2-445f-9c4c-4133e5820b26-kube-api-access-tw2fb\") pod \"90adff48-e3d2-445f-9c4c-4133e5820b26\" (UID: \"90adff48-e3d2-445f-9c4c-4133e5820b26\") "
Nov 02 13:35:07.269209 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:35:07.269068    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/90adff48-e3d2-445f-9c4c-4133e5820b26-metrics-certs\" (UniqueName: \"kubernetes.io/secret/90adff48-e3d2-445f-9c4c-4133e5820b26-metrics-certs\") pod \"90adff48-e3d2-445f-9c4c-4133e5820b26\" (UID: \"90adff48-e3d2-445f-9c4c-4133e5820b26\") "
Nov 02 13:35:07.269209 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:35:07.269147    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/90adff48-e3d2-445f-9c4c-4133e5820b26-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90adff48-e3d2-445f-9c4c-4133e5820b26-maas-api-tls\") pod \"90adff48-e3d2-445f-9c4c-4133e5820b26\" (UID: \"90adff48-e3d2-445f-9c4c-4133e5820b26\") "
Nov 02 13:35:07.269209 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:35:07.269193    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90adff48-e3d2-445f-9c4c-4133e5820b26-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/90adff48-e3d2-445f-9c4c-4133e5820b26-openshift-service-ca\") pod \"90adff48-e3d2-445f-9c4c-4133e5820b26\" (UID: \"90adff48-e3d2-445f-9c4c-4133e5820b26\") "
Nov 02 13:35:07.269658 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:35:07.269602    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90adff48-e3d2-445f-9c4c-4133e5820b26-openshift-service-ca" pod "90adff48-e3d2-445f-9c4c-4133e5820b26" (UID: "90adff48-e3d2-445f-9c4c-4133e5820b26"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:35:07.271156 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:35:07.271131    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90adff48-e3d2-445f-9c4c-4133e5820b26-maas-api-tls" pod "90adff48-e3d2-445f-9c4c-4133e5820b26" (UID: "90adff48-e3d2-445f-9c4c-4133e5820b26"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:35:07.271258 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:35:07.271171    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90adff48-e3d2-445f-9c4c-4133e5820b26-metrics-certs" pod "90adff48-e3d2-445f-9c4c-4133e5820b26" (UID: "90adff48-e3d2-445f-9c4c-4133e5820b26"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:35:07.271301 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:35:07.271267    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90adff48-e3d2-445f-9c4c-4133e5820b26-kube-api-access-tw2fb" pod "90adff48-e3d2-445f-9c4c-4133e5820b26" (UID: "90adff48-e3d2-445f-9c4c-4133e5820b26"). InnerVolumeSpecName "kube-api-access-tw2fb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:35:07.370188 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:35:07.370156    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90adff48-e3d2-445f-9c4c-4133e5820b26-maas-api-tls\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:35:07.370188 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:35:07.370184    4559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/90adff48-e3d2-445f-9c4c-4133e5820b26-openshift-service-ca\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:35:07.370188 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:35:07.370195    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tw2fb\" (UniqueName: \"kubernetes.io/projected/90adff48-e3d2-445f-9c4c-4133e5820b26-kube-api-access-tw2fb\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:35:07.370420 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:35:07.370204    4559 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90adff48-e3d2-445f-9c4c-4133e5820b26-metrics-certs\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:35:07.770276 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:35:07.770246    4559 generic.go:356] "Generic (PLEG): container finished" podID="90adff48-e3d2-445f-9c4c-4133e5820b26" containerID="fd10839dd8698d770627382c12113ca42d22a8196b2aa329f9a362fd8853d2b2" exitCode=0
Nov 02 13:35:07.770480 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:35:07.770292    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-95ee26-769ff594c9-wdwnp"
Nov 02 13:35:07.770480 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:35:07.770336    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-95ee26-769ff594c9-wdwnp" event={"ID":"90adff48-e3d2-445f-9c4c-4133e5820b26","Type":"ContainerDied","Data":"fd10839dd8698d770627382c12113ca42d22a8196b2aa329f9a362fd8853d2b2"}
Nov 02 13:35:07.770480 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:35:07.770391    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-95ee26-769ff594c9-wdwnp" event={"ID":"90adff48-e3d2-445f-9c4c-4133e5820b26","Type":"ContainerDied","Data":"602723e99fe84622a982c96bd2512e5ba8c094de95f92c5a085c8c9715ddd156"}
Nov 02 13:35:07.770480 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:35:07.770420    4559 scope.go:122] "RemoveContainer" containerID="fd10839dd8698d770627382c12113ca42d22a8196b2aa329f9a362fd8853d2b2"
Nov 02 13:35:07.780611 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:35:07.780596    4559 scope.go:122] "RemoveContainer" containerID="fd10839dd8698d770627382c12113ca42d22a8196b2aa329f9a362fd8853d2b2"
Nov 02 13:35:07.780880 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:35:07.780864    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd10839dd8698d770627382c12113ca42d22a8196b2aa329f9a362fd8853d2b2\": container with ID starting with fd10839dd8698d770627382c12113ca42d22a8196b2aa329f9a362fd8853d2b2 not found: ID does not exist" containerID="fd10839dd8698d770627382c12113ca42d22a8196b2aa329f9a362fd8853d2b2"
Nov 02 13:35:07.780921 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:35:07.780890    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd10839dd8698d770627382c12113ca42d22a8196b2aa329f9a362fd8853d2b2"} err="failed to get container status \"fd10839dd8698d770627382c12113ca42d22a8196b2aa329f9a362fd8853d2b2\": rpc error: code = NotFound desc = could not find container \"fd10839dd8698d770627382c12113ca42d22a8196b2aa329f9a362fd8853d2b2\": container with ID starting with fd10839dd8698d770627382c12113ca42d22a8196b2aa329f9a362fd8853d2b2 not found: ID does not exist"
Nov 02 13:35:07.824053 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:35:07.824023    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-95ee26-769ff594c9-wdwnp"]
Nov 02 13:35:07.836529 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:35:07.836509    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-95ee26-769ff594c9-wdwnp"]
Nov 02 13:35:07.890792 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:35:07.890766    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90adff48-e3d2-445f-9c4c-4133e5820b26" path="/var/lib/kubelet/pods/90adff48-e3d2-445f-9c4c-4133e5820b26/volumes"
Nov 02 13:35:08.457396 ip-10-0-128-132 kubenswrapper[4559]: W1102 13:35:08.457375    4559 logging.go:55] [core] [Channel #12219 SubChannel #12220]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.51:50052", ServerName: "10.133.0.51:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.51:50052: connect: connection refused"
Nov 02 13:35:08.684723 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:35:08.684627    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-95ee26-5f96f56f79-2zqdd"
Nov 02 13:35:08.777554 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:35:08.777524    4559 generic.go:356] "Generic (PLEG): container finished" podID="57c75efe-f588-45c2-b5ea-2f948d0e6e13" containerID="c6ee188467a1f51ac30d421c5c1528e70aa6d6757253b757faed5e2768b2e0b6" exitCode=0
Nov 02 13:35:08.777764 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:35:08.777574    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-95ee26-5f96f56f79-2zqdd"
Nov 02 13:35:08.777764 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:35:08.777576    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-95ee26-5f96f56f79-2zqdd" event={"ID":"57c75efe-f588-45c2-b5ea-2f948d0e6e13","Type":"ContainerDied","Data":"c6ee188467a1f51ac30d421c5c1528e70aa6d6757253b757faed5e2768b2e0b6"}
Nov 02 13:35:08.777764 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:35:08.777728    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-95ee26-5f96f56f79-2zqdd" event={"ID":"57c75efe-f588-45c2-b5ea-2f948d0e6e13","Type":"ContainerDied","Data":"87863d1d01353658282220caa9108ed5df45a0788a29303d91151e46adc13bc8"}
Nov 02 13:35:08.777764 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:35:08.777735    4559 scope.go:122] "RemoveContainer" containerID="c6ee188467a1f51ac30d421c5c1528e70aa6d6757253b757faed5e2768b2e0b6"
Nov 02 13:35:08.783304 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:35:08.783279    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57c75efe-f588-45c2-b5ea-2f948d0e6e13-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57c75efe-f588-45c2-b5ea-2f948d0e6e13-plugins-config-volume\") pod \"57c75efe-f588-45c2-b5ea-2f948d0e6e13\" (UID: \"57c75efe-f588-45c2-b5ea-2f948d0e6e13\") "
Nov 02 13:35:08.783440 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:35:08.783378    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57c75efe-f588-45c2-b5ea-2f948d0e6e13-kube-api-access-zlnpn\" (UniqueName: \"kubernetes.io/projected/57c75efe-f588-45c2-b5ea-2f948d0e6e13-kube-api-access-zlnpn\") pod \"57c75efe-f588-45c2-b5ea-2f948d0e6e13\" (UID: \"57c75efe-f588-45c2-b5ea-2f948d0e6e13\") "
Nov 02 13:35:08.783729 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:35:08.783696    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57c75efe-f588-45c2-b5ea-2f948d0e6e13-plugins-config-volume" pod "57c75efe-f588-45c2-b5ea-2f948d0e6e13" (UID: "57c75efe-f588-45c2-b5ea-2f948d0e6e13"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:35:08.785594 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:35:08.785568    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57c75efe-f588-45c2-b5ea-2f948d0e6e13-kube-api-access-zlnpn" pod "57c75efe-f588-45c2-b5ea-2f948d0e6e13" (UID: "57c75efe-f588-45c2-b5ea-2f948d0e6e13"). InnerVolumeSpecName "kube-api-access-zlnpn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:35:08.787322 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:35:08.787300    4559 scope.go:122] "RemoveContainer" containerID="c6ee188467a1f51ac30d421c5c1528e70aa6d6757253b757faed5e2768b2e0b6"
Nov 02 13:35:08.787625 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:35:08.787597    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6ee188467a1f51ac30d421c5c1528e70aa6d6757253b757faed5e2768b2e0b6\": container with ID starting with c6ee188467a1f51ac30d421c5c1528e70aa6d6757253b757faed5e2768b2e0b6 not found: ID does not exist" containerID="c6ee188467a1f51ac30d421c5c1528e70aa6d6757253b757faed5e2768b2e0b6"
Nov 02 13:35:08.787752 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:35:08.787633    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6ee188467a1f51ac30d421c5c1528e70aa6d6757253b757faed5e2768b2e0b6"} err="failed to get container status \"c6ee188467a1f51ac30d421c5c1528e70aa6d6757253b757faed5e2768b2e0b6\": rpc error: code = NotFound desc = could not find container \"c6ee188467a1f51ac30d421c5c1528e70aa6d6757253b757faed5e2768b2e0b6\": container with ID starting with c6ee188467a1f51ac30d421c5c1528e70aa6d6757253b757faed5e2768b2e0b6 not found: ID does not exist"
Nov 02 13:35:08.885351 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:35:08.885316    4559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57c75efe-f588-45c2-b5ea-2f948d0e6e13-plugins-config-volume\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:35:08.885351 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:35:08.885349    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zlnpn\" (UniqueName: \"kubernetes.io/projected/57c75efe-f588-45c2-b5ea-2f948d0e6e13-kube-api-access-zlnpn\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:35:09.127308 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:35:09.127274    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-95ee26-5f96f56f79-2zqdd"]
Nov 02 13:35:09.130950 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:35:09.130922    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-95ee26-5f96f56f79-2zqdd"]
Nov 02 13:35:09.457558 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:35:09.457511    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-95ee26-5f96f56f79-2zqdd" podUID="57c75efe-f588-45c2-b5ea-2f948d0e6e13" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.51:50052\" within 1s: context deadline exceeded"
Nov 02 13:35:09.893491 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:35:09.893414    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57c75efe-f588-45c2-b5ea-2f948d0e6e13" path="/var/lib/kubelet/pods/57c75efe-f588-45c2-b5ea-2f948d0e6e13/volumes"
Nov 02 13:35:10.728222 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:35:10.728192    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-5r6hx"]
Nov 02 13:35:10.728630 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:35:10.728612    4559 cpu_manager.go:414] "containerMap: removing container" podUID="90adff48-e3d2-445f-9c4c-4133e5820b26" containerName="maas-api"
Nov 02 13:35:10.728725 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:35:10.728635    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90adff48-e3d2-445f-9c4c-4133e5820b26" containerName="maas-api"
Nov 02 13:35:10.728725 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:35:10.728672    4559 cpu_manager.go:414] "containerMap: removing container" podUID="738b99e2-1698-4101-a6a9-2c2778bdfe2b" containerName="test-curl-6992-fefa6a"
Nov 02 13:35:10.728725 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:35:10.728681    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="738b99e2-1698-4101-a6a9-2c2778bdfe2b" containerName="test-curl-6992-fefa6a"
Nov 02 13:35:10.728849 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:35:10.728726    4559 cpu_manager.go:414] "containerMap: removing container" podUID="57c75efe-f588-45c2-b5ea-2f948d0e6e13" containerName="payload-processing"
Nov 02 13:35:10.728849 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:35:10.728734    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57c75efe-f588-45c2-b5ea-2f948d0e6e13" containerName="payload-processing"
Nov 02 13:35:10.728849 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:35:10.728834    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="738b99e2-1698-4101-a6a9-2c2778bdfe2b" containerName="test-curl-6992-fefa6a"
Nov 02 13:35:10.728994 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:35:10.728854    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="57c75efe-f588-45c2-b5ea-2f948d0e6e13" containerName="payload-processing"
Nov 02 13:35:10.728994 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:35:10.728868    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="90adff48-e3d2-445f-9c4c-4133e5820b26" containerName="maas-api"
Nov 02 13:35:10.733299 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:35:10.733281    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-5r6hx"
Nov 02 13:35:10.752187 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:35:10.752169    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-95ee26-openshift-default-dockercfg-xjh8d\""
Nov 02 13:35:10.767144 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:35:10.767120    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-5r6hx"]
Nov 02 13:35:10.904857 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:35:10.904821    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d649e5c1-cbc6-4cd7-ac2d-cb56e6fa38ca-istio-envoy\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-5r6hx\" (UID: \"d649e5c1-cbc6-4cd7-ac2d-cb56e6fa38ca\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-5r6hx"
Nov 02 13:35:10.904857 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:35:10.904862    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d649e5c1-cbc6-4cd7-ac2d-cb56e6fa38ca-credential-socket\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-5r6hx\" (UID: \"d649e5c1-cbc6-4cd7-ac2d-cb56e6fa38ca\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-5r6hx"
Nov 02 13:35:10.905052 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:35:10.904882    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zghjh\" (UniqueName: \"kubernetes.io/projected/d649e5c1-cbc6-4cd7-ac2d-cb56e6fa38ca-kube-api-access-zghjh\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-5r6hx\" (UID: \"d649e5c1-cbc6-4cd7-ac2d-cb56e6fa38ca\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-5r6hx"
Nov 02 13:35:10.905052 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:35:10.905026    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d649e5c1-cbc6-4cd7-ac2d-cb56e6fa38ca-istio-data\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-5r6hx\" (UID: \"d649e5c1-cbc6-4cd7-ac2d-cb56e6fa38ca\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-5r6hx"
Nov 02 13:35:10.905124 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:35:10.905057    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d649e5c1-cbc6-4cd7-ac2d-cb56e6fa38ca-workload-certs\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-5r6hx\" (UID: \"d649e5c1-cbc6-4cd7-ac2d-cb56e6fa38ca\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-5r6hx"
Nov 02 13:35:10.905124 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:35:10.905099    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d649e5c1-cbc6-4cd7-ac2d-cb56e6fa38ca-workload-socket\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-5r6hx\" (UID: \"d649e5c1-cbc6-4cd7-ac2d-cb56e6fa38ca\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-5r6hx"
Nov 02 13:35:10.905191 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:35:10.905147    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d649e5c1-cbc6-4cd7-ac2d-cb56e6fa38ca-istiod-ca-cert\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-5r6hx\" (UID: \"d649e5c1-cbc6-4cd7-ac2d-cb56e6fa38ca\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-5r6hx"
Nov 02 13:35:10.905191 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:35:10.905181    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d649e5c1-cbc6-4cd7-ac2d-cb56e6fa38ca-istio-podinfo\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-5r6hx\" (UID: \"d649e5c1-cbc6-4cd7-ac2d-cb56e6fa38ca\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-5r6hx"
Nov 02 13:35:10.905259 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:35:10.905234    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d649e5c1-cbc6-4cd7-ac2d-cb56e6fa38ca-istio-token\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-5r6hx\" (UID: \"d649e5c1-cbc6-4cd7-ac2d-cb56e6fa38ca\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-5r6hx"
Nov 02 13:35:11.006571 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:35:11.006510    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d649e5c1-cbc6-4cd7-ac2d-cb56e6fa38ca-istio-token\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-5r6hx\" (UID: \"d649e5c1-cbc6-4cd7-ac2d-cb56e6fa38ca\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-5r6hx"
Nov 02 13:35:11.006571 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:35:11.006554    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d649e5c1-cbc6-4cd7-ac2d-cb56e6fa38ca-istio-envoy\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-5r6hx\" (UID: \"d649e5c1-cbc6-4cd7-ac2d-cb56e6fa38ca\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-5r6hx"
Nov 02 13:35:11.006793 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:35:11.006572    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d649e5c1-cbc6-4cd7-ac2d-cb56e6fa38ca-credential-socket\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-5r6hx\" (UID: \"d649e5c1-cbc6-4cd7-ac2d-cb56e6fa38ca\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-5r6hx"
Nov 02 13:35:11.006793 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:35:11.006597    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zghjh\" (UniqueName: \"kubernetes.io/projected/d649e5c1-cbc6-4cd7-ac2d-cb56e6fa38ca-kube-api-access-zghjh\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-5r6hx\" (UID: \"d649e5c1-cbc6-4cd7-ac2d-cb56e6fa38ca\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-5r6hx"
Nov 02 13:35:11.006793 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:35:11.006625    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d649e5c1-cbc6-4cd7-ac2d-cb56e6fa38ca-istio-data\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-5r6hx\" (UID: \"d649e5c1-cbc6-4cd7-ac2d-cb56e6fa38ca\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-5r6hx"
Nov 02 13:35:11.006793 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:35:11.006685    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d649e5c1-cbc6-4cd7-ac2d-cb56e6fa38ca-workload-certs\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-5r6hx\" (UID: \"d649e5c1-cbc6-4cd7-ac2d-cb56e6fa38ca\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-5r6hx"
Nov 02 13:35:11.006793 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:35:11.006732    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d649e5c1-cbc6-4cd7-ac2d-cb56e6fa38ca-workload-socket\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-5r6hx\" (UID: \"d649e5c1-cbc6-4cd7-ac2d-cb56e6fa38ca\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-5r6hx"
Nov 02 13:35:11.006793 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:35:11.006765    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d649e5c1-cbc6-4cd7-ac2d-cb56e6fa38ca-istiod-ca-cert\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-5r6hx\" (UID: \"d649e5c1-cbc6-4cd7-ac2d-cb56e6fa38ca\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-5r6hx"
Nov 02 13:35:11.007132 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:35:11.006805    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d649e5c1-cbc6-4cd7-ac2d-cb56e6fa38ca-istio-podinfo\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-5r6hx\" (UID: \"d649e5c1-cbc6-4cd7-ac2d-cb56e6fa38ca\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-5r6hx"
Nov 02 13:35:11.007132 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:35:11.007106    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d649e5c1-cbc6-4cd7-ac2d-cb56e6fa38ca-credential-socket\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-5r6hx\" (UID: \"d649e5c1-cbc6-4cd7-ac2d-cb56e6fa38ca\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-5r6hx"
Nov 02 13:35:11.007234 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:35:11.007152    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d649e5c1-cbc6-4cd7-ac2d-cb56e6fa38ca-istio-data\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-5r6hx\" (UID: \"d649e5c1-cbc6-4cd7-ac2d-cb56e6fa38ca\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-5r6hx"
Nov 02 13:35:11.007234 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:35:11.007170    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d649e5c1-cbc6-4cd7-ac2d-cb56e6fa38ca-workload-certs\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-5r6hx\" (UID: \"d649e5c1-cbc6-4cd7-ac2d-cb56e6fa38ca\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-5r6hx"
Nov 02 13:35:11.007335 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:35:11.007320    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d649e5c1-cbc6-4cd7-ac2d-cb56e6fa38ca-workload-socket\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-5r6hx\" (UID: \"d649e5c1-cbc6-4cd7-ac2d-cb56e6fa38ca\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-5r6hx"
Nov 02 13:35:11.007697 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:35:11.007676    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d649e5c1-cbc6-4cd7-ac2d-cb56e6fa38ca-istiod-ca-cert\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-5r6hx\" (UID: \"d649e5c1-cbc6-4cd7-ac2d-cb56e6fa38ca\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-5r6hx"
Nov 02 13:35:11.008931 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:35:11.008912    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d649e5c1-cbc6-4cd7-ac2d-cb56e6fa38ca-istio-envoy\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-5r6hx\" (UID: \"d649e5c1-cbc6-4cd7-ac2d-cb56e6fa38ca\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-5r6hx"
Nov 02 13:35:11.009184 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:35:11.009166    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d649e5c1-cbc6-4cd7-ac2d-cb56e6fa38ca-istio-podinfo\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-5r6hx\" (UID: \"d649e5c1-cbc6-4cd7-ac2d-cb56e6fa38ca\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-5r6hx"
Nov 02 13:35:11.034036 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:35:11.034008    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d649e5c1-cbc6-4cd7-ac2d-cb56e6fa38ca-istio-token\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-5r6hx\" (UID: \"d649e5c1-cbc6-4cd7-ac2d-cb56e6fa38ca\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-5r6hx"
Nov 02 13:35:11.035208 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:35:11.035188    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zghjh\" (UniqueName: \"kubernetes.io/projected/d649e5c1-cbc6-4cd7-ac2d-cb56e6fa38ca-kube-api-access-zghjh\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-5r6hx\" (UID: \"d649e5c1-cbc6-4cd7-ac2d-cb56e6fa38ca\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-5r6hx"
Nov 02 13:35:11.046632 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:35:11.046611    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-5r6hx"
Nov 02 13:35:11.361031 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:35:11.360998    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-5r6hx"]
Nov 02 13:35:11.361540 ip-10-0-128-132 kubenswrapper[4559]: W1102 13:35:11.361516    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd649e5c1_cbc6_4cd7_ac2d_cb56e6fa38ca.slice/crio-333659169b88a8690bb8de20d9d9adc2dfbaa2f05059764d90eb943fe2c10434 WatchSource:0}: Error finding container 333659169b88a8690bb8de20d9d9adc2dfbaa2f05059764d90eb943fe2c10434: Status 404 returned error can't find the container with id 333659169b88a8690bb8de20d9d9adc2dfbaa2f05059764d90eb943fe2c10434
Nov 02 13:35:11.363794 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:35:11.363753    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 13:35:11.363898 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:35:11.363841    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 13:35:11.363898 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:35:11.363891    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 13:35:11.804674 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:35:11.803733    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-5r6hx" event={"ID":"d649e5c1-cbc6-4cd7-ac2d-cb56e6fa38ca","Type":"ContainerStarted","Data":"c50914fb740796bb9f5a918e39ffb5650fcf8d6654d4919ae8383018cac3d167"}
Nov 02 13:35:11.804674 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:35:11.803822    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-5r6hx" event={"ID":"d649e5c1-cbc6-4cd7-ac2d-cb56e6fa38ca","Type":"ContainerStarted","Data":"333659169b88a8690bb8de20d9d9adc2dfbaa2f05059764d90eb943fe2c10434"}
Nov 02 13:35:11.852741 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:35:11.852686    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-5r6hx" podStartSLOduration=1.852671424 podStartE2EDuration="1.852671424s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:35:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:35:11.850437837 +0000 UTC m=+4108.105836932" watchObservedRunningTime="2026-11-02 13:35:11.852671424 +0000 UTC m=+4108.108070519"
Nov 02 13:35:12.047274 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:35:12.047240    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-5r6hx"
Nov 02 13:35:12.048725 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:35:12.048704    4559 patch_prober.go:29] interesting pod/e2e-worker-w2-95ee26-openshift-default-66679d8f57-5r6hx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.6:15021/healthz/ready\": dial tcp 10.133.1.6:15021: connect: connection refused" start-of-body=
Nov 02 13:35:12.048798 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:35:12.048750    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-5r6hx" podUID="d649e5c1-cbc6-4cd7-ac2d-cb56e6fa38ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.6:15021/healthz/ready\": dial tcp 10.133.1.6:15021: connect: connection refused"
Nov 02 13:35:13.047961 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:35:13.047926    4559 patch_prober.go:29] interesting pod/e2e-worker-w2-95ee26-openshift-default-66679d8f57-5r6hx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.6:15021/healthz/ready\": dial tcp 10.133.1.6:15021: connect: connection refused" start-of-body=
Nov 02 13:35:13.048367 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:35:13.047991    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-5r6hx" podUID="d649e5c1-cbc6-4cd7-ac2d-cb56e6fa38ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.6:15021/healthz/ready\": dial tcp 10.133.1.6:15021: connect: connection refused"
Nov 02 13:35:14.047800 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:35:14.047766    4559 patch_prober.go:29] interesting pod/e2e-worker-w2-95ee26-openshift-default-66679d8f57-5r6hx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.6:15021/healthz/ready\": dial tcp 10.133.1.6:15021: connect: connection refused" start-of-body=
Nov 02 13:35:14.047990 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:35:14.047818    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-5r6hx" podUID="d649e5c1-cbc6-4cd7-ac2d-cb56e6fa38ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.6:15021/healthz/ready\": dial tcp 10.133.1.6:15021: connect: connection refused"
Nov 02 13:35:15.047956 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:35:15.047924    4559 patch_prober.go:29] interesting pod/e2e-worker-w2-95ee26-openshift-default-66679d8f57-5r6hx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.6:15021/healthz/ready\": dial tcp 10.133.1.6:15021: connect: connection refused" start-of-body=
Nov 02 13:35:15.048139 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:35:15.047977    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-5r6hx" podUID="d649e5c1-cbc6-4cd7-ac2d-cb56e6fa38ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.6:15021/healthz/ready\": dial tcp 10.133.1.6:15021: connect: connection refused"
Nov 02 13:35:15.114067 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:35:15.114033    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqcmj" containerName="extractor"
Nov 02 13:35:16.040021 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:35:16.039988    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"]
Nov 02 13:35:16.047679 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:35:16.047632    4559 patch_prober.go:29] interesting pod/e2e-worker-w2-95ee26-openshift-default-66679d8f57-5r6hx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.6:15021/healthz/ready\": dial tcp 10.133.1.6:15021: connect: connection refused" start-of-body=
Nov 02 13:35:16.047829 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:35:16.047736    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-5r6hx" podUID="d649e5c1-cbc6-4cd7-ac2d-cb56e6fa38ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.6:15021/healthz/ready\": dial tcp 10.133.1.6:15021: connect: connection refused"
Nov 02 13:35:16.146387 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:35:16.146354    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-5r6hx"]
Nov 02 13:35:16.146859 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:35:16.146598    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-5r6hx" podUID="d649e5c1-cbc6-4cd7-ac2d-cb56e6fa38ca" containerName="istio-proxy" containerID="cri-o://c50914fb740796bb9f5a918e39ffb5650fcf8d6654d4919ae8383018cac3d167" gracePeriod=30
Nov 02 13:35:23.407451 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:35:23.407420    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-5r6hx"
Nov 02 13:35:23.436074 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:35:23.436047    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d649e5c1-cbc6-4cd7-ac2d-cb56e6fa38ca-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d649e5c1-cbc6-4cd7-ac2d-cb56e6fa38ca-istio-podinfo\") pod \"d649e5c1-cbc6-4cd7-ac2d-cb56e6fa38ca\" (UID: \"d649e5c1-cbc6-4cd7-ac2d-cb56e6fa38ca\") "
Nov 02 13:35:23.436197 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:35:23.436093    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d649e5c1-cbc6-4cd7-ac2d-cb56e6fa38ca-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d649e5c1-cbc6-4cd7-ac2d-cb56e6fa38ca-istio-envoy\") pod \"d649e5c1-cbc6-4cd7-ac2d-cb56e6fa38ca\" (UID: \"d649e5c1-cbc6-4cd7-ac2d-cb56e6fa38ca\") "
Nov 02 13:35:23.436197 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:35:23.436126    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d649e5c1-cbc6-4cd7-ac2d-cb56e6fa38ca-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d649e5c1-cbc6-4cd7-ac2d-cb56e6fa38ca-credential-socket\") pod \"d649e5c1-cbc6-4cd7-ac2d-cb56e6fa38ca\" (UID: \"d649e5c1-cbc6-4cd7-ac2d-cb56e6fa38ca\") "
Nov 02 13:35:23.436197 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:35:23.436190    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d649e5c1-cbc6-4cd7-ac2d-cb56e6fa38ca-kube-api-access-zghjh\" (UniqueName: \"kubernetes.io/projected/d649e5c1-cbc6-4cd7-ac2d-cb56e6fa38ca-kube-api-access-zghjh\") pod \"d649e5c1-cbc6-4cd7-ac2d-cb56e6fa38ca\" (UID: \"d649e5c1-cbc6-4cd7-ac2d-cb56e6fa38ca\") "
Nov 02 13:35:23.436394 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:35:23.436219    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d649e5c1-cbc6-4cd7-ac2d-cb56e6fa38ca-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d649e5c1-cbc6-4cd7-ac2d-cb56e6fa38ca-workload-certs\") pod \"d649e5c1-cbc6-4cd7-ac2d-cb56e6fa38ca\" (UID: \"d649e5c1-cbc6-4cd7-ac2d-cb56e6fa38ca\") "
Nov 02 13:35:23.436394 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:35:23.436250    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d649e5c1-cbc6-4cd7-ac2d-cb56e6fa38ca-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d649e5c1-cbc6-4cd7-ac2d-cb56e6fa38ca-istio-data\") pod \"d649e5c1-cbc6-4cd7-ac2d-cb56e6fa38ca\" (UID: \"d649e5c1-cbc6-4cd7-ac2d-cb56e6fa38ca\") "
Nov 02 13:35:23.436394 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:35:23.436281    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d649e5c1-cbc6-4cd7-ac2d-cb56e6fa38ca-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d649e5c1-cbc6-4cd7-ac2d-cb56e6fa38ca-istiod-ca-cert\") pod \"d649e5c1-cbc6-4cd7-ac2d-cb56e6fa38ca\" (UID: \"d649e5c1-cbc6-4cd7-ac2d-cb56e6fa38ca\") "
Nov 02 13:35:23.436394 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:35:23.436358    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d649e5c1-cbc6-4cd7-ac2d-cb56e6fa38ca-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d649e5c1-cbc6-4cd7-ac2d-cb56e6fa38ca-workload-socket\") pod \"d649e5c1-cbc6-4cd7-ac2d-cb56e6fa38ca\" (UID: \"d649e5c1-cbc6-4cd7-ac2d-cb56e6fa38ca\") "
Nov 02 13:35:23.436394 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:35:23.436391    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d649e5c1-cbc6-4cd7-ac2d-cb56e6fa38ca-istio-token\" (UniqueName: \"kubernetes.io/projected/d649e5c1-cbc6-4cd7-ac2d-cb56e6fa38ca-istio-token\") pod \"d649e5c1-cbc6-4cd7-ac2d-cb56e6fa38ca\" (UID: \"d649e5c1-cbc6-4cd7-ac2d-cb56e6fa38ca\") "
Nov 02 13:35:23.436712 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:35:23.436476    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d649e5c1-cbc6-4cd7-ac2d-cb56e6fa38ca-credential-socket" pod "d649e5c1-cbc6-4cd7-ac2d-cb56e6fa38ca" (UID: "d649e5c1-cbc6-4cd7-ac2d-cb56e6fa38ca"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:35:23.436712 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:35:23.436500    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d649e5c1-cbc6-4cd7-ac2d-cb56e6fa38ca-workload-certs" pod "d649e5c1-cbc6-4cd7-ac2d-cb56e6fa38ca" (UID: "d649e5c1-cbc6-4cd7-ac2d-cb56e6fa38ca"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:35:23.436836 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:35:23.436741    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d649e5c1-cbc6-4cd7-ac2d-cb56e6fa38ca-istio-data" pod "d649e5c1-cbc6-4cd7-ac2d-cb56e6fa38ca" (UID: "d649e5c1-cbc6-4cd7-ac2d-cb56e6fa38ca"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:35:23.436836 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:35:23.436755    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d649e5c1-cbc6-4cd7-ac2d-cb56e6fa38ca-workload-socket" pod "d649e5c1-cbc6-4cd7-ac2d-cb56e6fa38ca" (UID: "d649e5c1-cbc6-4cd7-ac2d-cb56e6fa38ca"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:35:23.436955 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:35:23.436832    4559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d649e5c1-cbc6-4cd7-ac2d-cb56e6fa38ca-workload-certs\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:35:23.436955 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:35:23.436852    4559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d649e5c1-cbc6-4cd7-ac2d-cb56e6fa38ca-credential-socket\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:35:23.436955 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:35:23.436889    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d649e5c1-cbc6-4cd7-ac2d-cb56e6fa38ca-istiod-ca-cert" pod "d649e5c1-cbc6-4cd7-ac2d-cb56e6fa38ca" (UID: "d649e5c1-cbc6-4cd7-ac2d-cb56e6fa38ca"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:35:23.438258 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:35:23.438234    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d649e5c1-cbc6-4cd7-ac2d-cb56e6fa38ca-istio-envoy" pod "d649e5c1-cbc6-4cd7-ac2d-cb56e6fa38ca" (UID: "d649e5c1-cbc6-4cd7-ac2d-cb56e6fa38ca"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:35:23.438380 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:35:23.438281    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d649e5c1-cbc6-4cd7-ac2d-cb56e6fa38ca-istio-podinfo" pod "d649e5c1-cbc6-4cd7-ac2d-cb56e6fa38ca" (UID: "d649e5c1-cbc6-4cd7-ac2d-cb56e6fa38ca"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 13:35:23.438740 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:35:23.438719    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d649e5c1-cbc6-4cd7-ac2d-cb56e6fa38ca-kube-api-access-zghjh" pod "d649e5c1-cbc6-4cd7-ac2d-cb56e6fa38ca" (UID: "d649e5c1-cbc6-4cd7-ac2d-cb56e6fa38ca"). InnerVolumeSpecName "kube-api-access-zghjh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:35:23.438740 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:35:23.438729    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d649e5c1-cbc6-4cd7-ac2d-cb56e6fa38ca-istio-token" pod "d649e5c1-cbc6-4cd7-ac2d-cb56e6fa38ca" (UID: "d649e5c1-cbc6-4cd7-ac2d-cb56e6fa38ca"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:35:23.537786 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:35:23.537709    4559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d649e5c1-cbc6-4cd7-ac2d-cb56e6fa38ca-workload-socket\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:35:23.537786 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:35:23.537738    4559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d649e5c1-cbc6-4cd7-ac2d-cb56e6fa38ca-istio-token\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:35:23.537786 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:35:23.537751    4559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d649e5c1-cbc6-4cd7-ac2d-cb56e6fa38ca-istio-podinfo\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:35:23.537786 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:35:23.537762    4559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d649e5c1-cbc6-4cd7-ac2d-cb56e6fa38ca-istio-envoy\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:35:23.537786 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:35:23.537776    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zghjh\" (UniqueName: \"kubernetes.io/projected/d649e5c1-cbc6-4cd7-ac2d-cb56e6fa38ca-kube-api-access-zghjh\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:35:23.537786 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:35:23.537790    4559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d649e5c1-cbc6-4cd7-ac2d-cb56e6fa38ca-istio-data\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:35:23.538087 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:35:23.537802    4559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d649e5c1-cbc6-4cd7-ac2d-cb56e6fa38ca-istiod-ca-cert\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:35:23.879895 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:35:23.879816    4559 generic.go:356] "Generic (PLEG): container finished" podID="d649e5c1-cbc6-4cd7-ac2d-cb56e6fa38ca" containerID="c50914fb740796bb9f5a918e39ffb5650fcf8d6654d4919ae8383018cac3d167" exitCode=0
Nov 02 13:35:23.879895 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:35:23.879856    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-5r6hx" event={"ID":"d649e5c1-cbc6-4cd7-ac2d-cb56e6fa38ca","Type":"ContainerDied","Data":"c50914fb740796bb9f5a918e39ffb5650fcf8d6654d4919ae8383018cac3d167"}
Nov 02 13:35:23.879895 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:35:23.879861    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-5r6hx"
Nov 02 13:35:23.880158 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:35:23.879926    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-5r6hx" event={"ID":"d649e5c1-cbc6-4cd7-ac2d-cb56e6fa38ca","Type":"ContainerDied","Data":"333659169b88a8690bb8de20d9d9adc2dfbaa2f05059764d90eb943fe2c10434"}
Nov 02 13:35:23.880158 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:35:23.879950    4559 scope.go:122] "RemoveContainer" containerID="c50914fb740796bb9f5a918e39ffb5650fcf8d6654d4919ae8383018cac3d167"
Nov 02 13:35:23.890220 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:35:23.890201    4559 scope.go:122] "RemoveContainer" containerID="c50914fb740796bb9f5a918e39ffb5650fcf8d6654d4919ae8383018cac3d167"
Nov 02 13:35:23.890492 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:35:23.890471    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c50914fb740796bb9f5a918e39ffb5650fcf8d6654d4919ae8383018cac3d167\": container with ID starting with c50914fb740796bb9f5a918e39ffb5650fcf8d6654d4919ae8383018cac3d167 not found: ID does not exist" containerID="c50914fb740796bb9f5a918e39ffb5650fcf8d6654d4919ae8383018cac3d167"
Nov 02 13:35:23.890540 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:35:23.890500    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c50914fb740796bb9f5a918e39ffb5650fcf8d6654d4919ae8383018cac3d167"} err="failed to get container status \"c50914fb740796bb9f5a918e39ffb5650fcf8d6654d4919ae8383018cac3d167\": rpc error: code = NotFound desc = could not find container \"c50914fb740796bb9f5a918e39ffb5650fcf8d6654d4919ae8383018cac3d167\": container with ID starting with c50914fb740796bb9f5a918e39ffb5650fcf8d6654d4919ae8383018cac3d167 not found: ID does not exist"
Nov 02 13:35:23.937141 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:35:23.937118    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-5r6hx"]
Nov 02 13:35:23.949806 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:35:23.949784    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-5r6hx"]
Nov 02 13:35:25.892333 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:35:25.892299    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d649e5c1-cbc6-4cd7-ac2d-cb56e6fa38ca" path="/var/lib/kubelet/pods/d649e5c1-cbc6-4cd7-ac2d-cb56e6fa38ca/volumes"
Nov 02 13:35:32.700528 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:35:32.700492    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"]
Nov 02 13:35:43.736181 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:35:43.736146    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"]
Nov 02 13:35:44.822479 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:35:44.822448    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-8032f3/sub-test-model-8032f3-kserve-76b9878597-5hq8k"]
Nov 02 13:35:45.087003 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:35:45.086911    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqcmj" containerName="extractor"
Nov 02 13:35:52.923817 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:35:52.923774    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-43be82/sub-test-model-43be82-kserve-5dc888468f-9lf8t"]
Nov 02 13:35:55.725165 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:35:55.725131    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"]
Nov 02 13:35:59.841778 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:35:59.841720    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w0-8032f3/sub-test-model-8032f3-kserve-76b9878597-5hq8k" podUID="f549d5fa-af79-4498-8eab-ee04abaa7ab4" containerName="main" containerID="cri-o://cf8e5088471bd6bc9aa36bc68156de20e4288c0445971ab5a5e9e073175af59c" gracePeriod=45
Nov 02 13:35:59.971468 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:35:59.971426    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w0-8032f3/sub-test-model-8032f3-kserve-76b9878597-5hq8k" podUID="f549d5fa-af79-4498-8eab-ee04abaa7ab4" containerName="main" probeResult="failure" output="Get \"https://10.133.1.1:8000/ready\": dial tcp 10.133.1.1:8000: connect: connection refused"
Nov 02 13:36:00.053538 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:00.053504    4559 generic.go:356] "Generic (PLEG): container finished" podID="f549d5fa-af79-4498-8eab-ee04abaa7ab4" containerID="cf8e5088471bd6bc9aa36bc68156de20e4288c0445971ab5a5e9e073175af59c" exitCode=0
Nov 02 13:36:00.053723 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:00.053588    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-8032f3/sub-test-model-8032f3-kserve-76b9878597-5hq8k" event={"ID":"f549d5fa-af79-4498-8eab-ee04abaa7ab4","Type":"ContainerDied","Data":"cf8e5088471bd6bc9aa36bc68156de20e4288c0445971ab5a5e9e073175af59c"}
Nov 02 13:36:00.095583 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:00.095528    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-8032f3/sub-test-model-8032f3-kserve-76b9878597-5hq8k"
Nov 02 13:36:00.190829 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:00.190784    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f549d5fa-af79-4498-8eab-ee04abaa7ab4-kube-api-access-j4f99\" (UniqueName: \"kubernetes.io/projected/f549d5fa-af79-4498-8eab-ee04abaa7ab4-kube-api-access-j4f99\") pod \"f549d5fa-af79-4498-8eab-ee04abaa7ab4\" (UID: \"f549d5fa-af79-4498-8eab-ee04abaa7ab4\") "
Nov 02 13:36:00.190829 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:00.190835    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f549d5fa-af79-4498-8eab-ee04abaa7ab4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f549d5fa-af79-4498-8eab-ee04abaa7ab4-model-cache\") pod \"f549d5fa-af79-4498-8eab-ee04abaa7ab4\" (UID: \"f549d5fa-af79-4498-8eab-ee04abaa7ab4\") "
Nov 02 13:36:00.191098 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:00.190982    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f549d5fa-af79-4498-8eab-ee04abaa7ab4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f549d5fa-af79-4498-8eab-ee04abaa7ab4-dshm\") pod \"f549d5fa-af79-4498-8eab-ee04abaa7ab4\" (UID: \"f549d5fa-af79-4498-8eab-ee04abaa7ab4\") "
Nov 02 13:36:00.191098 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:00.191017    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f549d5fa-af79-4498-8eab-ee04abaa7ab4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f549d5fa-af79-4498-8eab-ee04abaa7ab4-tmp-dir\") pod \"f549d5fa-af79-4498-8eab-ee04abaa7ab4\" (UID: \"f549d5fa-af79-4498-8eab-ee04abaa7ab4\") "
Nov 02 13:36:00.191098 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:00.191089    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f549d5fa-af79-4498-8eab-ee04abaa7ab4-home\" (UniqueName: \"kubernetes.io/empty-dir/f549d5fa-af79-4498-8eab-ee04abaa7ab4-home\") pod \"f549d5fa-af79-4498-8eab-ee04abaa7ab4\" (UID: \"f549d5fa-af79-4498-8eab-ee04abaa7ab4\") "
Nov 02 13:36:00.191266 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:00.191096    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f549d5fa-af79-4498-8eab-ee04abaa7ab4-model-cache" pod "f549d5fa-af79-4498-8eab-ee04abaa7ab4" (UID: "f549d5fa-af79-4498-8eab-ee04abaa7ab4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:36:00.191266 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:00.191257    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f549d5fa-af79-4498-8eab-ee04abaa7ab4-tmp-dir" pod "f549d5fa-af79-4498-8eab-ee04abaa7ab4" (UID: "f549d5fa-af79-4498-8eab-ee04abaa7ab4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:36:00.191378 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:00.191359    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f549d5fa-af79-4498-8eab-ee04abaa7ab4-home" pod "f549d5fa-af79-4498-8eab-ee04abaa7ab4" (UID: "f549d5fa-af79-4498-8eab-ee04abaa7ab4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:36:00.191427 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:00.191373    4559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f549d5fa-af79-4498-8eab-ee04abaa7ab4-model-cache\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:36:00.191427 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:00.191394    4559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f549d5fa-af79-4498-8eab-ee04abaa7ab4-tmp-dir\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:36:00.192988 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:00.192964    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f549d5fa-af79-4498-8eab-ee04abaa7ab4-dshm" pod "f549d5fa-af79-4498-8eab-ee04abaa7ab4" (UID: "f549d5fa-af79-4498-8eab-ee04abaa7ab4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:36:00.193067 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:00.193023    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f549d5fa-af79-4498-8eab-ee04abaa7ab4-kube-api-access-j4f99" pod "f549d5fa-af79-4498-8eab-ee04abaa7ab4" (UID: "f549d5fa-af79-4498-8eab-ee04abaa7ab4"). InnerVolumeSpecName "kube-api-access-j4f99". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:36:00.292038 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:00.292003    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f549d5fa-af79-4498-8eab-ee04abaa7ab4-tls-certs\" (UniqueName: \"kubernetes.io/secret/f549d5fa-af79-4498-8eab-ee04abaa7ab4-tls-certs\") pod \"f549d5fa-af79-4498-8eab-ee04abaa7ab4\" (UID: \"f549d5fa-af79-4498-8eab-ee04abaa7ab4\") "
Nov 02 13:36:00.292254 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:00.292238    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4f99\" (UniqueName: \"kubernetes.io/projected/f549d5fa-af79-4498-8eab-ee04abaa7ab4-kube-api-access-j4f99\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:36:00.292300 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:00.292257    4559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f549d5fa-af79-4498-8eab-ee04abaa7ab4-dshm\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:36:00.292300 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:00.292267    4559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f549d5fa-af79-4498-8eab-ee04abaa7ab4-home\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:36:00.294026 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:00.294002    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f549d5fa-af79-4498-8eab-ee04abaa7ab4-tls-certs" pod "f549d5fa-af79-4498-8eab-ee04abaa7ab4" (UID: "f549d5fa-af79-4498-8eab-ee04abaa7ab4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:36:00.392830 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:00.392761    4559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f549d5fa-af79-4498-8eab-ee04abaa7ab4-tls-certs\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:36:01.059680 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:01.059630    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-8032f3/sub-test-model-8032f3-kserve-76b9878597-5hq8k"
Nov 02 13:36:01.059680 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:01.059665    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-8032f3/sub-test-model-8032f3-kserve-76b9878597-5hq8k" event={"ID":"f549d5fa-af79-4498-8eab-ee04abaa7ab4","Type":"ContainerDied","Data":"a442ebc664a4ea8074e4eced230b6aed1d7cd34f852e4286020f4f2e0ea94470"}
Nov 02 13:36:01.060324 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:01.059712    4559 scope.go:122] "RemoveContainer" containerID="cf8e5088471bd6bc9aa36bc68156de20e4288c0445971ab5a5e9e073175af59c"
Nov 02 13:36:01.114426 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:01.114396    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-8032f3/sub-test-model-8032f3-kserve-76b9878597-5hq8k"]
Nov 02 13:36:01.126707 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:01.126681    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w0-8032f3/sub-test-model-8032f3-kserve-76b9878597-5hq8k"]
Nov 02 13:36:01.892282 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:01.892249    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f549d5fa-af79-4498-8eab-ee04abaa7ab4" path="/var/lib/kubelet/pods/f549d5fa-af79-4498-8eab-ee04abaa7ab4/volumes"
Nov 02 13:36:03.868525 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:03.868490    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-14fcb6-7cd8b6b866-tdw6v"]
Nov 02 13:36:03.869002 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:03.868687    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-14fcb6-7cd8b6b866-tdw6v" podUID="b49dc55f-dea9-4a2d-be97-0bc8219e8fe1" containerName="maas-api" containerID="cri-o://ebaae3d5bfa136ba1ab72cdfe04326deebb3eb91aa991dab18cc972d97af0b6c" gracePeriod=30
Nov 02 13:36:04.076031 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:04.076001    4559 generic.go:356] "Generic (PLEG): container finished" podID="b49dc55f-dea9-4a2d-be97-0bc8219e8fe1" containerID="ebaae3d5bfa136ba1ab72cdfe04326deebb3eb91aa991dab18cc972d97af0b6c" exitCode=0
Nov 02 13:36:04.076215 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:04.076041    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-14fcb6-7cd8b6b866-tdw6v" event={"ID":"b49dc55f-dea9-4a2d-be97-0bc8219e8fe1","Type":"ContainerDied","Data":"ebaae3d5bfa136ba1ab72cdfe04326deebb3eb91aa991dab18cc972d97af0b6c"}
Nov 02 13:36:04.121252 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:04.121183    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-14fcb6-7cd8b6b866-tdw6v"
Nov 02 13:36:04.222784 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:04.222740    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b49dc55f-dea9-4a2d-be97-0bc8219e8fe1-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b49dc55f-dea9-4a2d-be97-0bc8219e8fe1-maas-api-tls\") pod \"b49dc55f-dea9-4a2d-be97-0bc8219e8fe1\" (UID: \"b49dc55f-dea9-4a2d-be97-0bc8219e8fe1\") "
Nov 02 13:36:04.222999 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:04.222789    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b49dc55f-dea9-4a2d-be97-0bc8219e8fe1-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b49dc55f-dea9-4a2d-be97-0bc8219e8fe1-openshift-service-ca\") pod \"b49dc55f-dea9-4a2d-be97-0bc8219e8fe1\" (UID: \"b49dc55f-dea9-4a2d-be97-0bc8219e8fe1\") "
Nov 02 13:36:04.222999 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:04.222876    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b49dc55f-dea9-4a2d-be97-0bc8219e8fe1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b49dc55f-dea9-4a2d-be97-0bc8219e8fe1-metrics-certs\") pod \"b49dc55f-dea9-4a2d-be97-0bc8219e8fe1\" (UID: \"b49dc55f-dea9-4a2d-be97-0bc8219e8fe1\") "
Nov 02 13:36:04.222999 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:04.222912    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b49dc55f-dea9-4a2d-be97-0bc8219e8fe1-kube-api-access-j5pfj\" (UniqueName: \"kubernetes.io/projected/b49dc55f-dea9-4a2d-be97-0bc8219e8fe1-kube-api-access-j5pfj\") pod \"b49dc55f-dea9-4a2d-be97-0bc8219e8fe1\" (UID: \"b49dc55f-dea9-4a2d-be97-0bc8219e8fe1\") "
Nov 02 13:36:04.223270 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:04.223237    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b49dc55f-dea9-4a2d-be97-0bc8219e8fe1-openshift-service-ca" pod "b49dc55f-dea9-4a2d-be97-0bc8219e8fe1" (UID: "b49dc55f-dea9-4a2d-be97-0bc8219e8fe1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:36:04.225045 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:04.225021    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b49dc55f-dea9-4a2d-be97-0bc8219e8fe1-maas-api-tls" pod "b49dc55f-dea9-4a2d-be97-0bc8219e8fe1" (UID: "b49dc55f-dea9-4a2d-be97-0bc8219e8fe1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:36:04.225146 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:04.225075    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b49dc55f-dea9-4a2d-be97-0bc8219e8fe1-metrics-certs" pod "b49dc55f-dea9-4a2d-be97-0bc8219e8fe1" (UID: "b49dc55f-dea9-4a2d-be97-0bc8219e8fe1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:36:04.225146 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:04.225100    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b49dc55f-dea9-4a2d-be97-0bc8219e8fe1-kube-api-access-j5pfj" pod "b49dc55f-dea9-4a2d-be97-0bc8219e8fe1" (UID: "b49dc55f-dea9-4a2d-be97-0bc8219e8fe1"). InnerVolumeSpecName "kube-api-access-j5pfj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:36:04.324075 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:04.324031    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b49dc55f-dea9-4a2d-be97-0bc8219e8fe1-maas-api-tls\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:36:04.324075 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:04.324069    4559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b49dc55f-dea9-4a2d-be97-0bc8219e8fe1-openshift-service-ca\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:36:04.324075 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:04.324079    4559 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b49dc55f-dea9-4a2d-be97-0bc8219e8fe1-metrics-certs\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:36:04.324075 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:04.324088    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j5pfj\" (UniqueName: \"kubernetes.io/projected/b49dc55f-dea9-4a2d-be97-0bc8219e8fe1-kube-api-access-j5pfj\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:36:05.081766 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:05.081730    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-14fcb6-7cd8b6b866-tdw6v" event={"ID":"b49dc55f-dea9-4a2d-be97-0bc8219e8fe1","Type":"ContainerDied","Data":"f3a37fdc25f755542f3959b01684dd875da9d652ac7169a06798dec546bf36ca"}
Nov 02 13:36:05.082220 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:05.081772    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-14fcb6-7cd8b6b866-tdw6v"
Nov 02 13:36:05.082220 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:05.081816    4559 scope.go:122] "RemoveContainer" containerID="ebaae3d5bfa136ba1ab72cdfe04326deebb3eb91aa991dab18cc972d97af0b6c"
Nov 02 13:36:05.135684 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:05.135633    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-14fcb6-7cd8b6b866-tdw6v"]
Nov 02 13:36:05.147681 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:05.147630    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-14fcb6-7cd8b6b866-tdw6v"]
Nov 02 13:36:05.891353 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:05.891318    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b49dc55f-dea9-4a2d-be97-0bc8219e8fe1" path="/var/lib/kubelet/pods/b49dc55f-dea9-4a2d-be97-0bc8219e8fe1/volumes"
Nov 02 13:36:07.842581 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:07.842547    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-14fcb6-openshift-default-568ddb87b9-tzf8f"]
Nov 02 13:36:07.843076 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:07.842787    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-14fcb6-openshift-default-568ddb87b9-tzf8f" podUID="0029ba38-71d3-4212-9b50-ded1a42abd1d" containerName="istio-proxy" containerID="cri-o://c8634cb2c9ca821baf4648fb98052f709ae9905476a7296de6da8b14e581e117" gracePeriod=30
Nov 02 13:36:07.947328 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:07.947285    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w0-43be82/sub-test-model-43be82-kserve-5dc888468f-9lf8t" podUID="9322192d-bc03-41d8-a556-7057c92fa5ba" containerName="main" containerID="cri-o://e80a23b58b99a6d967a0196ec5badc56b48991f18f40c8f9ac0bcb058932d597" gracePeriod=45
Nov 02 13:36:08.097736 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:08.097633    4559 generic.go:356] "Generic (PLEG): container finished" podID="9322192d-bc03-41d8-a556-7057c92fa5ba" containerID="e80a23b58b99a6d967a0196ec5badc56b48991f18f40c8f9ac0bcb058932d597" exitCode=0
Nov 02 13:36:08.097736 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:08.097707    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-43be82/sub-test-model-43be82-kserve-5dc888468f-9lf8t" event={"ID":"9322192d-bc03-41d8-a556-7057c92fa5ba","Type":"ContainerDied","Data":"e80a23b58b99a6d967a0196ec5badc56b48991f18f40c8f9ac0bcb058932d597"}
Nov 02 13:36:08.194301 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:08.194273    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-43be82/sub-test-model-43be82-kserve-5dc888468f-9lf8t"
Nov 02 13:36:08.261683 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:08.261632    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9322192d-bc03-41d8-a556-7057c92fa5ba-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9322192d-bc03-41d8-a556-7057c92fa5ba-dshm\") pod \"9322192d-bc03-41d8-a556-7057c92fa5ba\" (UID: \"9322192d-bc03-41d8-a556-7057c92fa5ba\") "
Nov 02 13:36:08.261823 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:08.261707    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9322192d-bc03-41d8-a556-7057c92fa5ba-tls-certs\" (UniqueName: \"kubernetes.io/secret/9322192d-bc03-41d8-a556-7057c92fa5ba-tls-certs\") pod \"9322192d-bc03-41d8-a556-7057c92fa5ba\" (UID: \"9322192d-bc03-41d8-a556-7057c92fa5ba\") "
Nov 02 13:36:08.261823 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:08.261795    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9322192d-bc03-41d8-a556-7057c92fa5ba-home\" (UniqueName: \"kubernetes.io/empty-dir/9322192d-bc03-41d8-a556-7057c92fa5ba-home\") pod \"9322192d-bc03-41d8-a556-7057c92fa5ba\" (UID: \"9322192d-bc03-41d8-a556-7057c92fa5ba\") "
Nov 02 13:36:08.261917 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:08.261834    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9322192d-bc03-41d8-a556-7057c92fa5ba-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9322192d-bc03-41d8-a556-7057c92fa5ba-model-cache\") pod \"9322192d-bc03-41d8-a556-7057c92fa5ba\" (UID: \"9322192d-bc03-41d8-a556-7057c92fa5ba\") "
Nov 02 13:36:08.261917 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:08.261896    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9322192d-bc03-41d8-a556-7057c92fa5ba-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9322192d-bc03-41d8-a556-7057c92fa5ba-tmp-dir\") pod \"9322192d-bc03-41d8-a556-7057c92fa5ba\" (UID: \"9322192d-bc03-41d8-a556-7057c92fa5ba\") "
Nov 02 13:36:08.262033 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:08.261937    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9322192d-bc03-41d8-a556-7057c92fa5ba-kube-api-access-vq6xm\" (UniqueName: \"kubernetes.io/projected/9322192d-bc03-41d8-a556-7057c92fa5ba-kube-api-access-vq6xm\") pod \"9322192d-bc03-41d8-a556-7057c92fa5ba\" (UID: \"9322192d-bc03-41d8-a556-7057c92fa5ba\") "
Nov 02 13:36:08.262125 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:08.262087    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9322192d-bc03-41d8-a556-7057c92fa5ba-home" pod "9322192d-bc03-41d8-a556-7057c92fa5ba" (UID: "9322192d-bc03-41d8-a556-7057c92fa5ba"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:36:08.262191 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:08.262169    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9322192d-bc03-41d8-a556-7057c92fa5ba-tmp-dir" pod "9322192d-bc03-41d8-a556-7057c92fa5ba" (UID: "9322192d-bc03-41d8-a556-7057c92fa5ba"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:36:08.262253 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:08.262183    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9322192d-bc03-41d8-a556-7057c92fa5ba-model-cache" pod "9322192d-bc03-41d8-a556-7057c92fa5ba" (UID: "9322192d-bc03-41d8-a556-7057c92fa5ba"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:36:08.262310 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:08.262284    4559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9322192d-bc03-41d8-a556-7057c92fa5ba-home\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:36:08.262310 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:08.262304    4559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9322192d-bc03-41d8-a556-7057c92fa5ba-model-cache\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:36:08.262429 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:08.262320    4559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9322192d-bc03-41d8-a556-7057c92fa5ba-tmp-dir\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:36:08.264024 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:08.264001    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9322192d-bc03-41d8-a556-7057c92fa5ba-dshm" pod "9322192d-bc03-41d8-a556-7057c92fa5ba" (UID: "9322192d-bc03-41d8-a556-7057c92fa5ba"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:36:08.264024 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:08.264005    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9322192d-bc03-41d8-a556-7057c92fa5ba-kube-api-access-vq6xm" pod "9322192d-bc03-41d8-a556-7057c92fa5ba" (UID: "9322192d-bc03-41d8-a556-7057c92fa5ba"). InnerVolumeSpecName "kube-api-access-vq6xm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:36:08.264266 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:08.264247    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9322192d-bc03-41d8-a556-7057c92fa5ba-tls-certs" pod "9322192d-bc03-41d8-a556-7057c92fa5ba" (UID: "9322192d-bc03-41d8-a556-7057c92fa5ba"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:36:08.362877 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:08.362798    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vq6xm\" (UniqueName: \"kubernetes.io/projected/9322192d-bc03-41d8-a556-7057c92fa5ba-kube-api-access-vq6xm\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:36:08.362877 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:08.362828    4559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9322192d-bc03-41d8-a556-7057c92fa5ba-dshm\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:36:08.362877 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:08.362838    4559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9322192d-bc03-41d8-a556-7057c92fa5ba-tls-certs\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:36:09.105041 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:09.105008    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-43be82/sub-test-model-43be82-kserve-5dc888468f-9lf8t" event={"ID":"9322192d-bc03-41d8-a556-7057c92fa5ba","Type":"ContainerDied","Data":"de5d1675a909ee0a8419122a83a72404b77e80c3a54a8a0b625105075c9da1ed"}
Nov 02 13:36:09.105478 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:09.105019    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-43be82/sub-test-model-43be82-kserve-5dc888468f-9lf8t"
Nov 02 13:36:09.105478 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:09.105054    4559 scope.go:122] "RemoveContainer" containerID="e80a23b58b99a6d967a0196ec5badc56b48991f18f40c8f9ac0bcb058932d597"
Nov 02 13:36:09.157258 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:09.157232    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-43be82/sub-test-model-43be82-kserve-5dc888468f-9lf8t"]
Nov 02 13:36:09.163314 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:09.163284    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w0-43be82/sub-test-model-43be82-kserve-5dc888468f-9lf8t" podUID="9322192d-bc03-41d8-a556-7057c92fa5ba" containerName="main" probeResult="failure" output="Get \"https://10.133.1.2:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:36:09.169916 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:09.169891    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w0-43be82/sub-test-model-43be82-kserve-5dc888468f-9lf8t"]
Nov 02 13:36:09.892234 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:09.892203    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9322192d-bc03-41d8-a556-7057c92fa5ba" path="/var/lib/kubelet/pods/9322192d-bc03-41d8-a556-7057c92fa5ba/volumes"
Nov 02 13:36:10.027391 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:10.027358    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"]
Nov 02 13:36:13.096224 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:13.096199    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-14fcb6-openshift-default-568ddb87b9-tzf8f"
Nov 02 13:36:13.126093 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:13.126034    4559 generic.go:356] "Generic (PLEG): container finished" podID="0029ba38-71d3-4212-9b50-ded1a42abd1d" containerID="c8634cb2c9ca821baf4648fb98052f709ae9905476a7296de6da8b14e581e117" exitCode=0
Nov 02 13:36:13.126211 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:13.126091    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-14fcb6-openshift-default-568ddb87b9-tzf8f"
Nov 02 13:36:13.126211 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:13.126116    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-14fcb6-openshift-default-568ddb87b9-tzf8f" event={"ID":"0029ba38-71d3-4212-9b50-ded1a42abd1d","Type":"ContainerDied","Data":"c8634cb2c9ca821baf4648fb98052f709ae9905476a7296de6da8b14e581e117"}
Nov 02 13:36:13.126211 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:13.126194    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-14fcb6-openshift-default-568ddb87b9-tzf8f" event={"ID":"0029ba38-71d3-4212-9b50-ded1a42abd1d","Type":"ContainerDied","Data":"8a261e86569eb681d412286740390f7f166b3b7619c421e3c6d0144931d7bdb9"}
Nov 02 13:36:13.126211 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:13.126206    4559 scope.go:122] "RemoveContainer" containerID="c8634cb2c9ca821baf4648fb98052f709ae9905476a7296de6da8b14e581e117"
Nov 02 13:36:13.135366 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:13.135344    4559 scope.go:122] "RemoveContainer" containerID="3de5c889a2ef40f69cbc4ea9aa9eda016b3925972ef4c9c11a190b59925b248e"
Nov 02 13:36:13.143536 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:13.143519    4559 scope.go:122] "RemoveContainer" containerID="c8634cb2c9ca821baf4648fb98052f709ae9905476a7296de6da8b14e581e117"
Nov 02 13:36:13.143898 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:36:13.143878    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8634cb2c9ca821baf4648fb98052f709ae9905476a7296de6da8b14e581e117\": container with ID starting with c8634cb2c9ca821baf4648fb98052f709ae9905476a7296de6da8b14e581e117 not found: ID does not exist" containerID="c8634cb2c9ca821baf4648fb98052f709ae9905476a7296de6da8b14e581e117"
Nov 02 13:36:13.143976 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:13.143907    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8634cb2c9ca821baf4648fb98052f709ae9905476a7296de6da8b14e581e117"} err="failed to get container status \"c8634cb2c9ca821baf4648fb98052f709ae9905476a7296de6da8b14e581e117\": rpc error: code = NotFound desc = could not find container \"c8634cb2c9ca821baf4648fb98052f709ae9905476a7296de6da8b14e581e117\": container with ID starting with c8634cb2c9ca821baf4648fb98052f709ae9905476a7296de6da8b14e581e117 not found: ID does not exist"
Nov 02 13:36:13.143976 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:13.143924    4559 scope.go:122] "RemoveContainer" containerID="3de5c889a2ef40f69cbc4ea9aa9eda016b3925972ef4c9c11a190b59925b248e"
Nov 02 13:36:13.144203 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:36:13.144182    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3de5c889a2ef40f69cbc4ea9aa9eda016b3925972ef4c9c11a190b59925b248e\": container with ID starting with 3de5c889a2ef40f69cbc4ea9aa9eda016b3925972ef4c9c11a190b59925b248e not found: ID does not exist" containerID="3de5c889a2ef40f69cbc4ea9aa9eda016b3925972ef4c9c11a190b59925b248e"
Nov 02 13:36:13.144264 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:13.144212    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3de5c889a2ef40f69cbc4ea9aa9eda016b3925972ef4c9c11a190b59925b248e"} err="failed to get container status \"3de5c889a2ef40f69cbc4ea9aa9eda016b3925972ef4c9c11a190b59925b248e\": rpc error: code = NotFound desc = could not find container \"3de5c889a2ef40f69cbc4ea9aa9eda016b3925972ef4c9c11a190b59925b248e\": container with ID starting with 3de5c889a2ef40f69cbc4ea9aa9eda016b3925972ef4c9c11a190b59925b248e not found: ID does not exist"
Nov 02 13:36:13.201064 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:13.201037    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0029ba38-71d3-4212-9b50-ded1a42abd1d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0029ba38-71d3-4212-9b50-ded1a42abd1d-istio-data\") pod \"0029ba38-71d3-4212-9b50-ded1a42abd1d\" (UID: \"0029ba38-71d3-4212-9b50-ded1a42abd1d\") "
Nov 02 13:36:13.201231 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:13.201093    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0029ba38-71d3-4212-9b50-ded1a42abd1d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0029ba38-71d3-4212-9b50-ded1a42abd1d-istio-podinfo\") pod \"0029ba38-71d3-4212-9b50-ded1a42abd1d\" (UID: \"0029ba38-71d3-4212-9b50-ded1a42abd1d\") "
Nov 02 13:36:13.201231 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:13.201124    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0029ba38-71d3-4212-9b50-ded1a42abd1d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0029ba38-71d3-4212-9b50-ded1a42abd1d-istiod-ca-cert\") pod \"0029ba38-71d3-4212-9b50-ded1a42abd1d\" (UID: \"0029ba38-71d3-4212-9b50-ded1a42abd1d\") "
Nov 02 13:36:13.201231 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:13.201173    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0029ba38-71d3-4212-9b50-ded1a42abd1d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0029ba38-71d3-4212-9b50-ded1a42abd1d-workload-socket\") pod \"0029ba38-71d3-4212-9b50-ded1a42abd1d\" (UID: \"0029ba38-71d3-4212-9b50-ded1a42abd1d\") "
Nov 02 13:36:13.201231 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:13.201206    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0029ba38-71d3-4212-9b50-ded1a42abd1d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0029ba38-71d3-4212-9b50-ded1a42abd1d-istio-envoy\") pod \"0029ba38-71d3-4212-9b50-ded1a42abd1d\" (UID: \"0029ba38-71d3-4212-9b50-ded1a42abd1d\") "
Nov 02 13:36:13.201231 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:13.201226    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0029ba38-71d3-4212-9b50-ded1a42abd1d-istio-token\" (UniqueName: \"kubernetes.io/projected/0029ba38-71d3-4212-9b50-ded1a42abd1d-istio-token\") pod \"0029ba38-71d3-4212-9b50-ded1a42abd1d\" (UID: \"0029ba38-71d3-4212-9b50-ded1a42abd1d\") "
Nov 02 13:36:13.201540 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:13.201257    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0029ba38-71d3-4212-9b50-ded1a42abd1d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0029ba38-71d3-4212-9b50-ded1a42abd1d-workload-certs\") pod \"0029ba38-71d3-4212-9b50-ded1a42abd1d\" (UID: \"0029ba38-71d3-4212-9b50-ded1a42abd1d\") "
Nov 02 13:36:13.201540 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:13.201277    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0029ba38-71d3-4212-9b50-ded1a42abd1d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0029ba38-71d3-4212-9b50-ded1a42abd1d-credential-socket\") pod \"0029ba38-71d3-4212-9b50-ded1a42abd1d\" (UID: \"0029ba38-71d3-4212-9b50-ded1a42abd1d\") "
Nov 02 13:36:13.201540 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:13.201297    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0029ba38-71d3-4212-9b50-ded1a42abd1d-kube-api-access-r6cm7\" (UniqueName: \"kubernetes.io/projected/0029ba38-71d3-4212-9b50-ded1a42abd1d-kube-api-access-r6cm7\") pod \"0029ba38-71d3-4212-9b50-ded1a42abd1d\" (UID: \"0029ba38-71d3-4212-9b50-ded1a42abd1d\") "
Nov 02 13:36:13.201540 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:13.201345    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0029ba38-71d3-4212-9b50-ded1a42abd1d-istio-data" pod "0029ba38-71d3-4212-9b50-ded1a42abd1d" (UID: "0029ba38-71d3-4212-9b50-ded1a42abd1d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:36:13.201540 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:13.201439    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0029ba38-71d3-4212-9b50-ded1a42abd1d-workload-socket" pod "0029ba38-71d3-4212-9b50-ded1a42abd1d" (UID: "0029ba38-71d3-4212-9b50-ded1a42abd1d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:36:13.201899 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:13.201620    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0029ba38-71d3-4212-9b50-ded1a42abd1d-istiod-ca-cert" pod "0029ba38-71d3-4212-9b50-ded1a42abd1d" (UID: "0029ba38-71d3-4212-9b50-ded1a42abd1d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:36:13.201899 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:13.201626    4559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0029ba38-71d3-4212-9b50-ded1a42abd1d-workload-socket\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:36:13.201899 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:13.201680    4559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0029ba38-71d3-4212-9b50-ded1a42abd1d-istio-data\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:36:13.201899 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:13.201611    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0029ba38-71d3-4212-9b50-ded1a42abd1d-workload-certs" pod "0029ba38-71d3-4212-9b50-ded1a42abd1d" (UID: "0029ba38-71d3-4212-9b50-ded1a42abd1d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:36:13.201899 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:13.201857    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0029ba38-71d3-4212-9b50-ded1a42abd1d-credential-socket" pod "0029ba38-71d3-4212-9b50-ded1a42abd1d" (UID: "0029ba38-71d3-4212-9b50-ded1a42abd1d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:36:13.203492 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:13.203463    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0029ba38-71d3-4212-9b50-ded1a42abd1d-istio-envoy" pod "0029ba38-71d3-4212-9b50-ded1a42abd1d" (UID: "0029ba38-71d3-4212-9b50-ded1a42abd1d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:36:13.203613 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:13.203554    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0029ba38-71d3-4212-9b50-ded1a42abd1d-istio-token" pod "0029ba38-71d3-4212-9b50-ded1a42abd1d" (UID: "0029ba38-71d3-4212-9b50-ded1a42abd1d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:36:13.203613 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:13.203562    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0029ba38-71d3-4212-9b50-ded1a42abd1d-istio-podinfo" pod "0029ba38-71d3-4212-9b50-ded1a42abd1d" (UID: "0029ba38-71d3-4212-9b50-ded1a42abd1d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 13:36:13.203613 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:13.203588    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0029ba38-71d3-4212-9b50-ded1a42abd1d-kube-api-access-r6cm7" pod "0029ba38-71d3-4212-9b50-ded1a42abd1d" (UID: "0029ba38-71d3-4212-9b50-ded1a42abd1d"). InnerVolumeSpecName "kube-api-access-r6cm7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:36:13.302703 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:13.302667    4559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0029ba38-71d3-4212-9b50-ded1a42abd1d-istio-podinfo\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:36:13.302703 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:13.302700    4559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0029ba38-71d3-4212-9b50-ded1a42abd1d-istiod-ca-cert\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:36:13.302703 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:13.302709    4559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0029ba38-71d3-4212-9b50-ded1a42abd1d-istio-envoy\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:36:13.302894 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:13.302717    4559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0029ba38-71d3-4212-9b50-ded1a42abd1d-istio-token\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:36:13.302894 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:13.302725    4559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0029ba38-71d3-4212-9b50-ded1a42abd1d-workload-certs\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:36:13.302894 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:13.302734    4559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0029ba38-71d3-4212-9b50-ded1a42abd1d-credential-socket\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:36:13.302894 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:13.302748    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r6cm7\" (UniqueName: \"kubernetes.io/projected/0029ba38-71d3-4212-9b50-ded1a42abd1d-kube-api-access-r6cm7\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:36:13.472778 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:13.472751    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-14fcb6-openshift-default-568ddb87b9-tzf8f"]
Nov 02 13:36:13.477169 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:13.477147    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-14fcb6-openshift-default-568ddb87b9-tzf8f"]
Nov 02 13:36:13.892559 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:13.892485    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0029ba38-71d3-4212-9b50-ded1a42abd1d" path="/var/lib/kubelet/pods/0029ba38-71d3-4212-9b50-ded1a42abd1d/volumes"
Nov 02 13:36:15.085208 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:36:15.085174    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqcmj" containerName="extractor"
Nov 02 13:36:24.612856 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:24.612818    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"]
Nov 02 13:36:36.206203 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:36.206165    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-8032f3-86595cb565-rrbnr"]
Nov 02 13:36:36.391129 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:36.391090    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-8032f3-74cb7cfb7d-56656"]
Nov 02 13:36:36.391331 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:36.391291    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-8032f3-74cb7cfb7d-56656" podUID="8b66c2e1-91f4-4204-be78-73c05fc42258" containerName="maas-api" containerID="cri-o://609e0c4adfd669ba77b10ea7556750ca4e78e3e8a8b59fbf40fd746e4c3489f8" gracePeriod=30
Nov 02 13:36:36.647976 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:36.647945    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-8032f3-74cb7cfb7d-56656"
Nov 02 13:36:36.723838 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:36.723808    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b66c2e1-91f4-4204-be78-73c05fc42258-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b66c2e1-91f4-4204-be78-73c05fc42258-openshift-service-ca\") pod \"8b66c2e1-91f4-4204-be78-73c05fc42258\" (UID: \"8b66c2e1-91f4-4204-be78-73c05fc42258\") "
Nov 02 13:36:36.724017 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:36.723857    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8b66c2e1-91f4-4204-be78-73c05fc42258-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b66c2e1-91f4-4204-be78-73c05fc42258-maas-api-tls\") pod \"8b66c2e1-91f4-4204-be78-73c05fc42258\" (UID: \"8b66c2e1-91f4-4204-be78-73c05fc42258\") "
Nov 02 13:36:36.724017 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:36.723924    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b66c2e1-91f4-4204-be78-73c05fc42258-kube-api-access-q65nn\" (UniqueName: \"kubernetes.io/projected/8b66c2e1-91f4-4204-be78-73c05fc42258-kube-api-access-q65nn\") pod \"8b66c2e1-91f4-4204-be78-73c05fc42258\" (UID: \"8b66c2e1-91f4-4204-be78-73c05fc42258\") "
Nov 02 13:36:36.724017 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:36.723990    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8b66c2e1-91f4-4204-be78-73c05fc42258-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b66c2e1-91f4-4204-be78-73c05fc42258-metrics-certs\") pod \"8b66c2e1-91f4-4204-be78-73c05fc42258\" (UID: \"8b66c2e1-91f4-4204-be78-73c05fc42258\") "
Nov 02 13:36:36.724211 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:36.724179    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b66c2e1-91f4-4204-be78-73c05fc42258-openshift-service-ca" pod "8b66c2e1-91f4-4204-be78-73c05fc42258" (UID: "8b66c2e1-91f4-4204-be78-73c05fc42258"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:36:36.724300 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:36.724282    4559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b66c2e1-91f4-4204-be78-73c05fc42258-openshift-service-ca\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:36:36.726189 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:36.726161    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b66c2e1-91f4-4204-be78-73c05fc42258-maas-api-tls" pod "8b66c2e1-91f4-4204-be78-73c05fc42258" (UID: "8b66c2e1-91f4-4204-be78-73c05fc42258"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:36:36.726495 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:36.726475    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b66c2e1-91f4-4204-be78-73c05fc42258-kube-api-access-q65nn" pod "8b66c2e1-91f4-4204-be78-73c05fc42258" (UID: "8b66c2e1-91f4-4204-be78-73c05fc42258"). InnerVolumeSpecName "kube-api-access-q65nn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:36:36.726558 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:36.726471    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b66c2e1-91f4-4204-be78-73c05fc42258-metrics-certs" pod "8b66c2e1-91f4-4204-be78-73c05fc42258" (UID: "8b66c2e1-91f4-4204-be78-73c05fc42258"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:36:36.825612 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:36.825522    4559 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b66c2e1-91f4-4204-be78-73c05fc42258-metrics-certs\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:36:36.825612 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:36.825554    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b66c2e1-91f4-4204-be78-73c05fc42258-maas-api-tls\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:36:36.825612 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:36.825570    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q65nn\" (UniqueName: \"kubernetes.io/projected/8b66c2e1-91f4-4204-be78-73c05fc42258-kube-api-access-q65nn\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:36:37.241942 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:37.241901    4559 generic.go:356] "Generic (PLEG): container finished" podID="8b66c2e1-91f4-4204-be78-73c05fc42258" containerID="609e0c4adfd669ba77b10ea7556750ca4e78e3e8a8b59fbf40fd746e4c3489f8" exitCode=0
Nov 02 13:36:37.242371 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:37.241956    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-8032f3-74cb7cfb7d-56656"
Nov 02 13:36:37.242371 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:37.241995    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-8032f3-74cb7cfb7d-56656" event={"ID":"8b66c2e1-91f4-4204-be78-73c05fc42258","Type":"ContainerDied","Data":"609e0c4adfd669ba77b10ea7556750ca4e78e3e8a8b59fbf40fd746e4c3489f8"}
Nov 02 13:36:37.242371 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:37.242048    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-8032f3-74cb7cfb7d-56656" event={"ID":"8b66c2e1-91f4-4204-be78-73c05fc42258","Type":"ContainerDied","Data":"84ddd7e55f6200462d42b8dfbc26e3ebd549695d5e1d301462b8008b7492a51d"}
Nov 02 13:36:37.242371 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:37.242072    4559 scope.go:122] "RemoveContainer" containerID="609e0c4adfd669ba77b10ea7556750ca4e78e3e8a8b59fbf40fd746e4c3489f8"
Nov 02 13:36:37.251836 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:37.251815    4559 scope.go:122] "RemoveContainer" containerID="609e0c4adfd669ba77b10ea7556750ca4e78e3e8a8b59fbf40fd746e4c3489f8"
Nov 02 13:36:37.252172 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:36:37.252128    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"609e0c4adfd669ba77b10ea7556750ca4e78e3e8a8b59fbf40fd746e4c3489f8\": container with ID starting with 609e0c4adfd669ba77b10ea7556750ca4e78e3e8a8b59fbf40fd746e4c3489f8 not found: ID does not exist" containerID="609e0c4adfd669ba77b10ea7556750ca4e78e3e8a8b59fbf40fd746e4c3489f8"
Nov 02 13:36:37.252172 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:37.252158    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"609e0c4adfd669ba77b10ea7556750ca4e78e3e8a8b59fbf40fd746e4c3489f8"} err="failed to get container status \"609e0c4adfd669ba77b10ea7556750ca4e78e3e8a8b59fbf40fd746e4c3489f8\": rpc error: code = NotFound desc = could not find container \"609e0c4adfd669ba77b10ea7556750ca4e78e3e8a8b59fbf40fd746e4c3489f8\": container with ID starting with 609e0c4adfd669ba77b10ea7556750ca4e78e3e8a8b59fbf40fd746e4c3489f8 not found: ID does not exist"
Nov 02 13:36:37.294890 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:37.294861    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-8032f3-74cb7cfb7d-56656"]
Nov 02 13:36:37.298797 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:37.298768    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-8032f3-74cb7cfb7d-56656"]
Nov 02 13:36:37.891017 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:37.890975    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b66c2e1-91f4-4204-be78-73c05fc42258" path="/var/lib/kubelet/pods/8b66c2e1-91f4-4204-be78-73c05fc42258/volumes"
Nov 02 13:36:39.816528 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:39.816469    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"]
Nov 02 13:36:41.206936 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:41.206863    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-8032f3-86595cb565-rrbnr" podUID="45b5f260-c257-485e-88bd-c87dbe6cb4d9" containerName="payload-processing" containerID="cri-o://f465c93f0524f69e3979c69ec7e5841f1b0d61460f3150d55188ebcb71a6f072" gracePeriod=25
Nov 02 13:36:44.314956 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:44.314920    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-8032f3-openshift-default-765d774fdc-mhf7j"]
Nov 02 13:36:44.315671 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:44.315147    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-8032f3-openshift-default-765d774fdc-mhf7j" podUID="5d73ae6a-aaa7-4946-baa5-03ab6d91d403" containerName="istio-proxy" containerID="cri-o://3ab56461a8d9cd117c6b68400d9e00469020ae5db9744c2fdd3177ce5c40d2d6" gracePeriod=30
Nov 02 13:36:45.085426 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:36:45.085382    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqcmj" containerName="extractor"
Nov 02 13:36:48.788632 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:48.788595    4559 patch_prober.go:29] interesting pod/e2e-shared-a-w0-8032f3-openshift-default-765d774fdc-mhf7j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 13:36:48.789049 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:48.788682    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w0-8032f3-openshift-default-765d774fdc-mhf7j" podUID="5d73ae6a-aaa7-4946-baa5-03ab6d91d403" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 13:36:49.479232 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:49.479187    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-8032f3-86595cb565-rrbnr" podUID="45b5f260-c257-485e-88bd-c87dbe6cb4d9" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 13:36:49.574883 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:49.574857    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-8032f3-openshift-default-765d774fdc-mhf7j"
Nov 02 13:36:49.746309 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:49.746281    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d73ae6a-aaa7-4946-baa5-03ab6d91d403-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d73ae6a-aaa7-4946-baa5-03ab6d91d403-istio-data\") pod \"5d73ae6a-aaa7-4946-baa5-03ab6d91d403\" (UID: \"5d73ae6a-aaa7-4946-baa5-03ab6d91d403\") "
Nov 02 13:36:49.746514 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:49.746321    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d73ae6a-aaa7-4946-baa5-03ab6d91d403-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d73ae6a-aaa7-4946-baa5-03ab6d91d403-credential-socket\") pod \"5d73ae6a-aaa7-4946-baa5-03ab6d91d403\" (UID: \"5d73ae6a-aaa7-4946-baa5-03ab6d91d403\") "
Nov 02 13:36:49.746514 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:49.746372    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d73ae6a-aaa7-4946-baa5-03ab6d91d403-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d73ae6a-aaa7-4946-baa5-03ab6d91d403-istio-envoy\") pod \"5d73ae6a-aaa7-4946-baa5-03ab6d91d403\" (UID: \"5d73ae6a-aaa7-4946-baa5-03ab6d91d403\") "
Nov 02 13:36:49.746514 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:49.746407    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d73ae6a-aaa7-4946-baa5-03ab6d91d403-kube-api-access-b5z47\" (UniqueName: \"kubernetes.io/projected/5d73ae6a-aaa7-4946-baa5-03ab6d91d403-kube-api-access-b5z47\") pod \"5d73ae6a-aaa7-4946-baa5-03ab6d91d403\" (UID: \"5d73ae6a-aaa7-4946-baa5-03ab6d91d403\") "
Nov 02 13:36:49.746514 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:49.746435    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d73ae6a-aaa7-4946-baa5-03ab6d91d403-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d73ae6a-aaa7-4946-baa5-03ab6d91d403-istiod-ca-cert\") pod \"5d73ae6a-aaa7-4946-baa5-03ab6d91d403\" (UID: \"5d73ae6a-aaa7-4946-baa5-03ab6d91d403\") "
Nov 02 13:36:49.746514 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:49.746481    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d73ae6a-aaa7-4946-baa5-03ab6d91d403-istio-token\" (UniqueName: \"kubernetes.io/projected/5d73ae6a-aaa7-4946-baa5-03ab6d91d403-istio-token\") pod \"5d73ae6a-aaa7-4946-baa5-03ab6d91d403\" (UID: \"5d73ae6a-aaa7-4946-baa5-03ab6d91d403\") "
Nov 02 13:36:49.746840 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:49.746538    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5d73ae6a-aaa7-4946-baa5-03ab6d91d403-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d73ae6a-aaa7-4946-baa5-03ab6d91d403-istio-podinfo\") pod \"5d73ae6a-aaa7-4946-baa5-03ab6d91d403\" (UID: \"5d73ae6a-aaa7-4946-baa5-03ab6d91d403\") "
Nov 02 13:36:49.746840 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:49.746567    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d73ae6a-aaa7-4946-baa5-03ab6d91d403-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d73ae6a-aaa7-4946-baa5-03ab6d91d403-workload-certs\") pod \"5d73ae6a-aaa7-4946-baa5-03ab6d91d403\" (UID: \"5d73ae6a-aaa7-4946-baa5-03ab6d91d403\") "
Nov 02 13:36:49.746840 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:49.746601    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d73ae6a-aaa7-4946-baa5-03ab6d91d403-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d73ae6a-aaa7-4946-baa5-03ab6d91d403-workload-socket\") pod \"5d73ae6a-aaa7-4946-baa5-03ab6d91d403\" (UID: \"5d73ae6a-aaa7-4946-baa5-03ab6d91d403\") "
Nov 02 13:36:49.746840 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:49.746601    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d73ae6a-aaa7-4946-baa5-03ab6d91d403-istio-data" pod "5d73ae6a-aaa7-4946-baa5-03ab6d91d403" (UID: "5d73ae6a-aaa7-4946-baa5-03ab6d91d403"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:36:49.746840 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:49.746697    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d73ae6a-aaa7-4946-baa5-03ab6d91d403-credential-socket" pod "5d73ae6a-aaa7-4946-baa5-03ab6d91d403" (UID: "5d73ae6a-aaa7-4946-baa5-03ab6d91d403"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:36:49.747084 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:49.746887    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d73ae6a-aaa7-4946-baa5-03ab6d91d403-workload-certs" pod "5d73ae6a-aaa7-4946-baa5-03ab6d91d403" (UID: "5d73ae6a-aaa7-4946-baa5-03ab6d91d403"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:36:49.747084 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:49.746913    4559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d73ae6a-aaa7-4946-baa5-03ab6d91d403-istio-data\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:36:49.747084 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:49.746932    4559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d73ae6a-aaa7-4946-baa5-03ab6d91d403-credential-socket\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:36:49.747084 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:49.747000    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d73ae6a-aaa7-4946-baa5-03ab6d91d403-istiod-ca-cert" pod "5d73ae6a-aaa7-4946-baa5-03ab6d91d403" (UID: "5d73ae6a-aaa7-4946-baa5-03ab6d91d403"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:36:49.747084 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:49.747013    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d73ae6a-aaa7-4946-baa5-03ab6d91d403-workload-socket" pod "5d73ae6a-aaa7-4946-baa5-03ab6d91d403" (UID: "5d73ae6a-aaa7-4946-baa5-03ab6d91d403"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:36:49.748746 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:49.748720    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d73ae6a-aaa7-4946-baa5-03ab6d91d403-kube-api-access-b5z47" pod "5d73ae6a-aaa7-4946-baa5-03ab6d91d403" (UID: "5d73ae6a-aaa7-4946-baa5-03ab6d91d403"). InnerVolumeSpecName "kube-api-access-b5z47". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:36:49.748913 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:49.748887    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5d73ae6a-aaa7-4946-baa5-03ab6d91d403-istio-podinfo" pod "5d73ae6a-aaa7-4946-baa5-03ab6d91d403" (UID: "5d73ae6a-aaa7-4946-baa5-03ab6d91d403"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 13:36:49.748993 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:49.748970    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d73ae6a-aaa7-4946-baa5-03ab6d91d403-istio-envoy" pod "5d73ae6a-aaa7-4946-baa5-03ab6d91d403" (UID: "5d73ae6a-aaa7-4946-baa5-03ab6d91d403"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:36:49.749117 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:49.749096    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d73ae6a-aaa7-4946-baa5-03ab6d91d403-istio-token" pod "5d73ae6a-aaa7-4946-baa5-03ab6d91d403" (UID: "5d73ae6a-aaa7-4946-baa5-03ab6d91d403"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:36:49.847670 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:49.847623    4559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d73ae6a-aaa7-4946-baa5-03ab6d91d403-workload-socket\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:36:49.847670 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:49.847667    4559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d73ae6a-aaa7-4946-baa5-03ab6d91d403-istio-envoy\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:36:49.847670 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:49.847678    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b5z47\" (UniqueName: \"kubernetes.io/projected/5d73ae6a-aaa7-4946-baa5-03ab6d91d403-kube-api-access-b5z47\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:36:49.848134 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:49.847688    4559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d73ae6a-aaa7-4946-baa5-03ab6d91d403-istiod-ca-cert\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:36:49.848134 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:49.847696    4559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d73ae6a-aaa7-4946-baa5-03ab6d91d403-istio-token\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:36:49.848134 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:49.847704    4559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d73ae6a-aaa7-4946-baa5-03ab6d91d403-istio-podinfo\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:36:49.848134 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:49.847712    4559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d73ae6a-aaa7-4946-baa5-03ab6d91d403-workload-certs\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:36:50.301085 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:50.301050    4559 generic.go:356] "Generic (PLEG): container finished" podID="5d73ae6a-aaa7-4946-baa5-03ab6d91d403" containerID="3ab56461a8d9cd117c6b68400d9e00469020ae5db9744c2fdd3177ce5c40d2d6" exitCode=0
Nov 02 13:36:50.301262 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:50.301107    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-8032f3-openshift-default-765d774fdc-mhf7j"
Nov 02 13:36:50.301262 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:50.301120    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-8032f3-openshift-default-765d774fdc-mhf7j" event={"ID":"5d73ae6a-aaa7-4946-baa5-03ab6d91d403","Type":"ContainerDied","Data":"3ab56461a8d9cd117c6b68400d9e00469020ae5db9744c2fdd3177ce5c40d2d6"}
Nov 02 13:36:50.301262 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:50.301172    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-8032f3-openshift-default-765d774fdc-mhf7j" event={"ID":"5d73ae6a-aaa7-4946-baa5-03ab6d91d403","Type":"ContainerDied","Data":"31c052c90dfeeb7b01d5bc09963bd57e26812fd159d35390bf98573ddd489d77"}
Nov 02 13:36:50.301262 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:50.301201    4559 scope.go:122] "RemoveContainer" containerID="3ab56461a8d9cd117c6b68400d9e00469020ae5db9744c2fdd3177ce5c40d2d6"
Nov 02 13:36:50.312633 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:50.312603    4559 scope.go:122] "RemoveContainer" containerID="6d770e61b37efd4981a0b48fd44b029c55d19eb0c7dd7b8fb5ce75a89a9b3034"
Nov 02 13:36:50.321263 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:50.321243    4559 scope.go:122] "RemoveContainer" containerID="3ab56461a8d9cd117c6b68400d9e00469020ae5db9744c2fdd3177ce5c40d2d6"
Nov 02 13:36:50.321527 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:36:50.321506    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ab56461a8d9cd117c6b68400d9e00469020ae5db9744c2fdd3177ce5c40d2d6\": container with ID starting with 3ab56461a8d9cd117c6b68400d9e00469020ae5db9744c2fdd3177ce5c40d2d6 not found: ID does not exist" containerID="3ab56461a8d9cd117c6b68400d9e00469020ae5db9744c2fdd3177ce5c40d2d6"
Nov 02 13:36:50.321577 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:50.321534    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ab56461a8d9cd117c6b68400d9e00469020ae5db9744c2fdd3177ce5c40d2d6"} err="failed to get container status \"3ab56461a8d9cd117c6b68400d9e00469020ae5db9744c2fdd3177ce5c40d2d6\": rpc error: code = NotFound desc = could not find container \"3ab56461a8d9cd117c6b68400d9e00469020ae5db9744c2fdd3177ce5c40d2d6\": container with ID starting with 3ab56461a8d9cd117c6b68400d9e00469020ae5db9744c2fdd3177ce5c40d2d6 not found: ID does not exist"
Nov 02 13:36:50.321577 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:50.321549    4559 scope.go:122] "RemoveContainer" containerID="6d770e61b37efd4981a0b48fd44b029c55d19eb0c7dd7b8fb5ce75a89a9b3034"
Nov 02 13:36:50.321810 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:36:50.321792    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d770e61b37efd4981a0b48fd44b029c55d19eb0c7dd7b8fb5ce75a89a9b3034\": container with ID starting with 6d770e61b37efd4981a0b48fd44b029c55d19eb0c7dd7b8fb5ce75a89a9b3034 not found: ID does not exist" containerID="6d770e61b37efd4981a0b48fd44b029c55d19eb0c7dd7b8fb5ce75a89a9b3034"
Nov 02 13:36:50.321870 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:50.321815    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d770e61b37efd4981a0b48fd44b029c55d19eb0c7dd7b8fb5ce75a89a9b3034"} err="failed to get container status \"6d770e61b37efd4981a0b48fd44b029c55d19eb0c7dd7b8fb5ce75a89a9b3034\": rpc error: code = NotFound desc = could not find container \"6d770e61b37efd4981a0b48fd44b029c55d19eb0c7dd7b8fb5ce75a89a9b3034\": container with ID starting with 6d770e61b37efd4981a0b48fd44b029c55d19eb0c7dd7b8fb5ce75a89a9b3034 not found: ID does not exist"
Nov 02 13:36:50.376952 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:50.376927    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-8032f3-openshift-default-765d774fdc-mhf7j"]
Nov 02 13:36:50.390916 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:50.390890    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-8032f3-openshift-default-765d774fdc-mhf7j"]
Nov 02 13:36:51.791976 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:51.791939    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-43be82-5d876bd44c-s6bm8"]
Nov 02 13:36:51.792511 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:51.792219    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-43be82-5d876bd44c-s6bm8" podUID="4030f96f-2e78-477c-8670-c47821b4a931" containerName="maas-api" containerID="cri-o://ed26216e8698be654b7de5665a74fd6c8e80238ef41c88133ad4e5eaba918ec3" gracePeriod=30
Nov 02 13:36:51.891242 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:51.891209    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d73ae6a-aaa7-4946-baa5-03ab6d91d403" path="/var/lib/kubelet/pods/5d73ae6a-aaa7-4946-baa5-03ab6d91d403/volumes"
Nov 02 13:36:52.040519 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:52.040494    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-43be82-5d876bd44c-s6bm8"
Nov 02 13:36:52.061268 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:52.061203    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4030f96f-2e78-477c-8670-c47821b4a931-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4030f96f-2e78-477c-8670-c47821b4a931-maas-api-tls\") pod \"4030f96f-2e78-477c-8670-c47821b4a931\" (UID: \"4030f96f-2e78-477c-8670-c47821b4a931\") "
Nov 02 13:36:52.061395 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:52.061274    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4030f96f-2e78-477c-8670-c47821b4a931-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4030f96f-2e78-477c-8670-c47821b4a931-metrics-certs\") pod \"4030f96f-2e78-477c-8670-c47821b4a931\" (UID: \"4030f96f-2e78-477c-8670-c47821b4a931\") "
Nov 02 13:36:52.061395 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:52.061346    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4030f96f-2e78-477c-8670-c47821b4a931-kube-api-access-g6h25\" (UniqueName: \"kubernetes.io/projected/4030f96f-2e78-477c-8670-c47821b4a931-kube-api-access-g6h25\") pod \"4030f96f-2e78-477c-8670-c47821b4a931\" (UID: \"4030f96f-2e78-477c-8670-c47821b4a931\") "
Nov 02 13:36:52.061395 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:52.061384    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4030f96f-2e78-477c-8670-c47821b4a931-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4030f96f-2e78-477c-8670-c47821b4a931-openshift-service-ca\") pod \"4030f96f-2e78-477c-8670-c47821b4a931\" (UID: \"4030f96f-2e78-477c-8670-c47821b4a931\") "
Nov 02 13:36:52.061788 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:52.061762    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4030f96f-2e78-477c-8670-c47821b4a931-openshift-service-ca" pod "4030f96f-2e78-477c-8670-c47821b4a931" (UID: "4030f96f-2e78-477c-8670-c47821b4a931"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:36:52.063349 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:52.063321    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4030f96f-2e78-477c-8670-c47821b4a931-maas-api-tls" pod "4030f96f-2e78-477c-8670-c47821b4a931" (UID: "4030f96f-2e78-477c-8670-c47821b4a931"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:36:52.063349 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:52.063340    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4030f96f-2e78-477c-8670-c47821b4a931-metrics-certs" pod "4030f96f-2e78-477c-8670-c47821b4a931" (UID: "4030f96f-2e78-477c-8670-c47821b4a931"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:36:52.063483 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:52.063410    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4030f96f-2e78-477c-8670-c47821b4a931-kube-api-access-g6h25" pod "4030f96f-2e78-477c-8670-c47821b4a931" (UID: "4030f96f-2e78-477c-8670-c47821b4a931"). InnerVolumeSpecName "kube-api-access-g6h25". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:36:52.162817 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:52.162778    4559 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4030f96f-2e78-477c-8670-c47821b4a931-metrics-certs\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:36:52.162817 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:52.162808    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g6h25\" (UniqueName: \"kubernetes.io/projected/4030f96f-2e78-477c-8670-c47821b4a931-kube-api-access-g6h25\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:36:52.162817 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:52.162820    4559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4030f96f-2e78-477c-8670-c47821b4a931-openshift-service-ca\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:36:52.163054 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:52.162828    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4030f96f-2e78-477c-8670-c47821b4a931-maas-api-tls\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:36:52.311627 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:52.311537    4559 generic.go:356] "Generic (PLEG): container finished" podID="4030f96f-2e78-477c-8670-c47821b4a931" containerID="ed26216e8698be654b7de5665a74fd6c8e80238ef41c88133ad4e5eaba918ec3" exitCode=0
Nov 02 13:36:52.311627 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:52.311588    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-43be82-5d876bd44c-s6bm8"
Nov 02 13:36:52.311887 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:52.311623    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-43be82-5d876bd44c-s6bm8" event={"ID":"4030f96f-2e78-477c-8670-c47821b4a931","Type":"ContainerDied","Data":"ed26216e8698be654b7de5665a74fd6c8e80238ef41c88133ad4e5eaba918ec3"}
Nov 02 13:36:52.311887 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:52.311697    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-43be82-5d876bd44c-s6bm8" event={"ID":"4030f96f-2e78-477c-8670-c47821b4a931","Type":"ContainerDied","Data":"caac0a093d16286229028ada29053bd4a72f3f7953e0d0aa679c724be423f3b2"}
Nov 02 13:36:52.311887 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:52.311722    4559 scope.go:122] "RemoveContainer" containerID="ed26216e8698be654b7de5665a74fd6c8e80238ef41c88133ad4e5eaba918ec3"
Nov 02 13:36:52.321429 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:52.321415    4559 scope.go:122] "RemoveContainer" containerID="ed26216e8698be654b7de5665a74fd6c8e80238ef41c88133ad4e5eaba918ec3"
Nov 02 13:36:52.321714 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:36:52.321696    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed26216e8698be654b7de5665a74fd6c8e80238ef41c88133ad4e5eaba918ec3\": container with ID starting with ed26216e8698be654b7de5665a74fd6c8e80238ef41c88133ad4e5eaba918ec3 not found: ID does not exist" containerID="ed26216e8698be654b7de5665a74fd6c8e80238ef41c88133ad4e5eaba918ec3"
Nov 02 13:36:52.321778 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:52.321719    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed26216e8698be654b7de5665a74fd6c8e80238ef41c88133ad4e5eaba918ec3"} err="failed to get container status \"ed26216e8698be654b7de5665a74fd6c8e80238ef41c88133ad4e5eaba918ec3\": rpc error: code = NotFound desc = could not find container \"ed26216e8698be654b7de5665a74fd6c8e80238ef41c88133ad4e5eaba918ec3\": container with ID starting with ed26216e8698be654b7de5665a74fd6c8e80238ef41c88133ad4e5eaba918ec3 not found: ID does not exist"
Nov 02 13:36:52.363398 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:52.363375    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-43be82-5d876bd44c-s6bm8"]
Nov 02 13:36:52.376240 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:52.376217    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-43be82-5d876bd44c-s6bm8"]
Nov 02 13:36:53.891432 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:53.891399    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4030f96f-2e78-477c-8670-c47821b4a931" path="/var/lib/kubelet/pods/4030f96f-2e78-477c-8670-c47821b4a931/volumes"
Nov 02 13:36:55.078998 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:55.078958    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"]
Nov 02 13:36:59.479086 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:36:59.479046    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-8032f3-86595cb565-rrbnr" podUID="45b5f260-c257-485e-88bd-c87dbe6cb4d9" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 13:37:01.352628 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:01.352598    4559 generic.go:356] "Generic (PLEG): container finished" podID="45b5f260-c257-485e-88bd-c87dbe6cb4d9" containerID="f465c93f0524f69e3979c69ec7e5841f1b0d61460f3150d55188ebcb71a6f072" exitCode=0
Nov 02 13:37:01.353017 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:01.352671    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-8032f3-86595cb565-rrbnr" event={"ID":"45b5f260-c257-485e-88bd-c87dbe6cb4d9","Type":"ContainerDied","Data":"f465c93f0524f69e3979c69ec7e5841f1b0d61460f3150d55188ebcb71a6f072"}
Nov 02 13:37:01.453741 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:01.453710    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-8032f3-86595cb565-rrbnr"
Nov 02 13:37:01.532410 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:01.532339    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45b5f260-c257-485e-88bd-c87dbe6cb4d9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45b5f260-c257-485e-88bd-c87dbe6cb4d9-plugins-config-volume\") pod \"45b5f260-c257-485e-88bd-c87dbe6cb4d9\" (UID: \"45b5f260-c257-485e-88bd-c87dbe6cb4d9\") "
Nov 02 13:37:01.532568 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:01.532437    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45b5f260-c257-485e-88bd-c87dbe6cb4d9-kube-api-access-x4cfq\" (UniqueName: \"kubernetes.io/projected/45b5f260-c257-485e-88bd-c87dbe6cb4d9-kube-api-access-x4cfq\") pod \"45b5f260-c257-485e-88bd-c87dbe6cb4d9\" (UID: \"45b5f260-c257-485e-88bd-c87dbe6cb4d9\") "
Nov 02 13:37:01.532757 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:01.532735    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45b5f260-c257-485e-88bd-c87dbe6cb4d9-plugins-config-volume" pod "45b5f260-c257-485e-88bd-c87dbe6cb4d9" (UID: "45b5f260-c257-485e-88bd-c87dbe6cb4d9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:37:01.534425 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:01.534403    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45b5f260-c257-485e-88bd-c87dbe6cb4d9-kube-api-access-x4cfq" pod "45b5f260-c257-485e-88bd-c87dbe6cb4d9" (UID: "45b5f260-c257-485e-88bd-c87dbe6cb4d9"). InnerVolumeSpecName "kube-api-access-x4cfq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:37:01.633945 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:01.633917    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x4cfq\" (UniqueName: \"kubernetes.io/projected/45b5f260-c257-485e-88bd-c87dbe6cb4d9-kube-api-access-x4cfq\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:37:01.633945 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:01.633946    4559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45b5f260-c257-485e-88bd-c87dbe6cb4d9-plugins-config-volume\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:37:02.358259 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:02.358225    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-8032f3-86595cb565-rrbnr" event={"ID":"45b5f260-c257-485e-88bd-c87dbe6cb4d9","Type":"ContainerDied","Data":"2f3b184e31e5b04d9c1289e8e4aedb29ac54b910fd0e58f1d4c2ff35c805f2ca"}
Nov 02 13:37:02.358259 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:02.358248    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-8032f3-86595cb565-rrbnr"
Nov 02 13:37:02.358757 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:02.358291    4559 scope.go:122] "RemoveContainer" containerID="f465c93f0524f69e3979c69ec7e5841f1b0d61460f3150d55188ebcb71a6f072"
Nov 02 13:37:02.413463 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:02.413434    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-8032f3-86595cb565-rrbnr"]
Nov 02 13:37:02.421448 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:02.421419    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-8032f3-86595cb565-rrbnr"]
Nov 02 13:37:03.890826 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:03.890792    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45b5f260-c257-485e-88bd-c87dbe6cb4d9" path="/var/lib/kubelet/pods/45b5f260-c257-485e-88bd-c87dbe6cb4d9/volumes"
Nov 02 13:37:07.629972 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:07.629938    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"]
Nov 02 13:37:12.463605 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:12.463561    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-5514e5-b884d6445-hpkbk"]
Nov 02 13:37:12.499784 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:12.499751    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-5514e5-cc9847d79-8jprz"]
Nov 02 13:37:15.085148 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:37:15.085108    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqcmj" containerName="extractor"
Nov 02 13:37:17.464495 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:17.464440    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-5514e5-b884d6445-hpkbk" podUID="323858c5-a582-47b3-b38f-af6a750580ca" containerName="payload-processing" containerID="cri-o://f8b322bb97c51daf956a8928056f0df68b3dc8d0d34e7174883bfb0fbac42762" gracePeriod=25
Nov 02 13:37:17.500071 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:17.500005    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-5514e5-cc9847d79-8jprz" podUID="7d5d728d-c371-4307-8c83-601a6e63d10c" containerName="payload-pre-processing" containerID="cri-o://9e1762642d89763777ed3a88948f10ddbb1af3415403bda5d91a996a88e58931" gracePeriod=25
Nov 02 13:37:19.573118 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:19.573077    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-5514e5-cc9847d79-8jprz" podUID="7d5d728d-c371-4307-8c83-601a6e63d10c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 13:37:19.573661 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:19.573122    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-5514e5-b884d6445-hpkbk" podUID="323858c5-a582-47b3-b38f-af6a750580ca" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 13:37:23.553561 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:23.553521    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-5514e5-75476c695b-8sl6l"]
Nov 02 13:37:23.554492 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:23.553847    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-5514e5-75476c695b-8sl6l" podUID="75a5e1c8-189e-4005-a8b7-48b007680900" containerName="maas-api" containerID="cri-o://2b8a153b09aec9dff5645bd1ab9491e455bf99a4ca0d8c0d25cad76eb46a7332" gracePeriod=30
Nov 02 13:37:23.803241 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:23.803216    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-5514e5-75476c695b-8sl6l"
Nov 02 13:37:23.926748 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:23.926724    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/75a5e1c8-189e-4005-a8b7-48b007680900-metrics-certs\" (UniqueName: \"kubernetes.io/secret/75a5e1c8-189e-4005-a8b7-48b007680900-metrics-certs\") pod \"75a5e1c8-189e-4005-a8b7-48b007680900\" (UID: \"75a5e1c8-189e-4005-a8b7-48b007680900\") "
Nov 02 13:37:23.926884 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:23.926808    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/75a5e1c8-189e-4005-a8b7-48b007680900-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/75a5e1c8-189e-4005-a8b7-48b007680900-maas-api-tls\") pod \"75a5e1c8-189e-4005-a8b7-48b007680900\" (UID: \"75a5e1c8-189e-4005-a8b7-48b007680900\") "
Nov 02 13:37:23.926884 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:23.926841    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75a5e1c8-189e-4005-a8b7-48b007680900-kube-api-access-xdfwj\" (UniqueName: \"kubernetes.io/projected/75a5e1c8-189e-4005-a8b7-48b007680900-kube-api-access-xdfwj\") pod \"75a5e1c8-189e-4005-a8b7-48b007680900\" (UID: \"75a5e1c8-189e-4005-a8b7-48b007680900\") "
Nov 02 13:37:23.926884 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:23.926877    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75a5e1c8-189e-4005-a8b7-48b007680900-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/75a5e1c8-189e-4005-a8b7-48b007680900-openshift-service-ca\") pod \"75a5e1c8-189e-4005-a8b7-48b007680900\" (UID: \"75a5e1c8-189e-4005-a8b7-48b007680900\") "
Nov 02 13:37:23.927295 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:23.927263    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75a5e1c8-189e-4005-a8b7-48b007680900-openshift-service-ca" pod "75a5e1c8-189e-4005-a8b7-48b007680900" (UID: "75a5e1c8-189e-4005-a8b7-48b007680900"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:37:23.928785 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:23.928766    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75a5e1c8-189e-4005-a8b7-48b007680900-metrics-certs" pod "75a5e1c8-189e-4005-a8b7-48b007680900" (UID: "75a5e1c8-189e-4005-a8b7-48b007680900"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:37:23.928887 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:23.928814    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75a5e1c8-189e-4005-a8b7-48b007680900-kube-api-access-xdfwj" pod "75a5e1c8-189e-4005-a8b7-48b007680900" (UID: "75a5e1c8-189e-4005-a8b7-48b007680900"). InnerVolumeSpecName "kube-api-access-xdfwj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:37:23.928887 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:23.928818    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75a5e1c8-189e-4005-a8b7-48b007680900-maas-api-tls" pod "75a5e1c8-189e-4005-a8b7-48b007680900" (UID: "75a5e1c8-189e-4005-a8b7-48b007680900"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:37:24.028317 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:24.028276    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/75a5e1c8-189e-4005-a8b7-48b007680900-maas-api-tls\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:37:24.028317 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:24.028321    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xdfwj\" (UniqueName: \"kubernetes.io/projected/75a5e1c8-189e-4005-a8b7-48b007680900-kube-api-access-xdfwj\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:37:24.028487 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:24.028336    4559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/75a5e1c8-189e-4005-a8b7-48b007680900-openshift-service-ca\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:37:24.028487 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:24.028351    4559 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75a5e1c8-189e-4005-a8b7-48b007680900-metrics-certs\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:37:24.195828 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:24.195762    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"]
Nov 02 13:37:24.455214 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:24.455133    4559 generic.go:356] "Generic (PLEG): container finished" podID="75a5e1c8-189e-4005-a8b7-48b007680900" containerID="2b8a153b09aec9dff5645bd1ab9491e455bf99a4ca0d8c0d25cad76eb46a7332" exitCode=0
Nov 02 13:37:24.455214 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:24.455178    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-5514e5-75476c695b-8sl6l"
Nov 02 13:37:24.455453 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:24.455220    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-5514e5-75476c695b-8sl6l" event={"ID":"75a5e1c8-189e-4005-a8b7-48b007680900","Type":"ContainerDied","Data":"2b8a153b09aec9dff5645bd1ab9491e455bf99a4ca0d8c0d25cad76eb46a7332"}
Nov 02 13:37:24.455453 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:24.455271    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-5514e5-75476c695b-8sl6l" event={"ID":"75a5e1c8-189e-4005-a8b7-48b007680900","Type":"ContainerDied","Data":"8a6c97821fa2b0ee40f427ea7d4e92022921003fc8fc379c8b3a6d98efae41ca"}
Nov 02 13:37:24.455453 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:24.455300    4559 scope.go:122] "RemoveContainer" containerID="2b8a153b09aec9dff5645bd1ab9491e455bf99a4ca0d8c0d25cad76eb46a7332"
Nov 02 13:37:24.465009 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:24.464988    4559 scope.go:122] "RemoveContainer" containerID="2b8a153b09aec9dff5645bd1ab9491e455bf99a4ca0d8c0d25cad76eb46a7332"
Nov 02 13:37:24.465244 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:37:24.465226    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b8a153b09aec9dff5645bd1ab9491e455bf99a4ca0d8c0d25cad76eb46a7332\": container with ID starting with 2b8a153b09aec9dff5645bd1ab9491e455bf99a4ca0d8c0d25cad76eb46a7332 not found: ID does not exist" containerID="2b8a153b09aec9dff5645bd1ab9491e455bf99a4ca0d8c0d25cad76eb46a7332"
Nov 02 13:37:24.465298 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:24.465251    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b8a153b09aec9dff5645bd1ab9491e455bf99a4ca0d8c0d25cad76eb46a7332"} err="failed to get container status \"2b8a153b09aec9dff5645bd1ab9491e455bf99a4ca0d8c0d25cad76eb46a7332\": rpc error: code = NotFound desc = could not find container \"2b8a153b09aec9dff5645bd1ab9491e455bf99a4ca0d8c0d25cad76eb46a7332\": container with ID starting with 2b8a153b09aec9dff5645bd1ab9491e455bf99a4ca0d8c0d25cad76eb46a7332 not found: ID does not exist"
Nov 02 13:37:24.507577 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:24.507552    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-5514e5-75476c695b-8sl6l"]
Nov 02 13:37:24.520251 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:24.520225    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-5514e5-75476c695b-8sl6l"]
Nov 02 13:37:25.890368 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:25.890333    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75a5e1c8-189e-4005-a8b7-48b007680900" path="/var/lib/kubelet/pods/75a5e1c8-189e-4005-a8b7-48b007680900/volumes"
Nov 02 13:37:29.572768 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:29.572721    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-5514e5-cc9847d79-8jprz" podUID="7d5d728d-c371-4307-8c83-601a6e63d10c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 13:37:29.573152 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:29.572721    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-5514e5-b884d6445-hpkbk" podUID="323858c5-a582-47b3-b38f-af6a750580ca" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 13:37:30.630809 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:30.630775    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"]
Nov 02 13:37:37.639741 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:37.639704    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-pbc9q"]
Nov 02 13:37:37.640147 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:37.640091    4559 cpu_manager.go:414] "containerMap: removing container" podUID="8b66c2e1-91f4-4204-be78-73c05fc42258" containerName="maas-api"
Nov 02 13:37:37.640147 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:37.640102    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b66c2e1-91f4-4204-be78-73c05fc42258" containerName="maas-api"
Nov 02 13:37:37.640147 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:37.640120    4559 cpu_manager.go:414] "containerMap: removing container" podUID="5d73ae6a-aaa7-4946-baa5-03ab6d91d403" containerName="istio-proxy"
Nov 02 13:37:37.640147 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:37.640126    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d73ae6a-aaa7-4946-baa5-03ab6d91d403" containerName="istio-proxy"
Nov 02 13:37:37.640147 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:37.640135    4559 cpu_manager.go:414] "containerMap: removing container" podUID="4030f96f-2e78-477c-8670-c47821b4a931" containerName="maas-api"
Nov 02 13:37:37.640147 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:37.640140    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4030f96f-2e78-477c-8670-c47821b4a931" containerName="maas-api"
Nov 02 13:37:37.640147 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:37.640150    4559 cpu_manager.go:414] "containerMap: removing container" podUID="75a5e1c8-189e-4005-a8b7-48b007680900" containerName="maas-api"
Nov 02 13:37:37.640416 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:37.640155    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75a5e1c8-189e-4005-a8b7-48b007680900" containerName="maas-api"
Nov 02 13:37:37.640416 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:37.640165    4559 cpu_manager.go:414] "containerMap: removing container" podUID="0029ba38-71d3-4212-9b50-ded1a42abd1d" containerName="istio-proxy"
Nov 02 13:37:37.640416 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:37.640169    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0029ba38-71d3-4212-9b50-ded1a42abd1d" containerName="istio-proxy"
Nov 02 13:37:37.640416 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:37.640191    4559 cpu_manager.go:414] "containerMap: removing container" podUID="45b5f260-c257-485e-88bd-c87dbe6cb4d9" containerName="payload-processing"
Nov 02 13:37:37.640416 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:37.640199    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45b5f260-c257-485e-88bd-c87dbe6cb4d9" containerName="payload-processing"
Nov 02 13:37:37.640416 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:37.640207    4559 cpu_manager.go:414] "containerMap: removing container" podUID="f549d5fa-af79-4498-8eab-ee04abaa7ab4" containerName="main"
Nov 02 13:37:37.640416 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:37.640212    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f549d5fa-af79-4498-8eab-ee04abaa7ab4" containerName="main"
Nov 02 13:37:37.640416 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:37.640218    4559 cpu_manager.go:414] "containerMap: removing container" podUID="b49dc55f-dea9-4a2d-be97-0bc8219e8fe1" containerName="maas-api"
Nov 02 13:37:37.640416 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:37.640223    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b49dc55f-dea9-4a2d-be97-0bc8219e8fe1" containerName="maas-api"
Nov 02 13:37:37.640416 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:37.640233    4559 cpu_manager.go:414] "containerMap: removing container" podUID="9322192d-bc03-41d8-a556-7057c92fa5ba" containerName="main"
Nov 02 13:37:37.640416 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:37.640237    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9322192d-bc03-41d8-a556-7057c92fa5ba" containerName="main"
Nov 02 13:37:37.640416 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:37.640243    4559 cpu_manager.go:414] "containerMap: removing container" podUID="d649e5c1-cbc6-4cd7-ac2d-cb56e6fa38ca" containerName="istio-proxy"
Nov 02 13:37:37.640416 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:37.640247    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d649e5c1-cbc6-4cd7-ac2d-cb56e6fa38ca" containerName="istio-proxy"
Nov 02 13:37:37.640416 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:37.640305    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="75a5e1c8-189e-4005-a8b7-48b007680900" containerName="maas-api"
Nov 02 13:37:37.640416 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:37.640314    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="0029ba38-71d3-4212-9b50-ded1a42abd1d" containerName="istio-proxy"
Nov 02 13:37:37.640416 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:37.640324    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d73ae6a-aaa7-4946-baa5-03ab6d91d403" containerName="istio-proxy"
Nov 02 13:37:37.640416 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:37.640331    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="45b5f260-c257-485e-88bd-c87dbe6cb4d9" containerName="payload-processing"
Nov 02 13:37:37.640416 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:37.640336    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="f549d5fa-af79-4498-8eab-ee04abaa7ab4" containerName="main"
Nov 02 13:37:37.640416 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:37.640342    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="b49dc55f-dea9-4a2d-be97-0bc8219e8fe1" containerName="maas-api"
Nov 02 13:37:37.640416 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:37.640348    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="0029ba38-71d3-4212-9b50-ded1a42abd1d" containerName="istio-proxy"
Nov 02 13:37:37.640416 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:37.640354    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="9322192d-bc03-41d8-a556-7057c92fa5ba" containerName="main"
Nov 02 13:37:37.640416 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:37.640361    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="d649e5c1-cbc6-4cd7-ac2d-cb56e6fa38ca" containerName="istio-proxy"
Nov 02 13:37:37.640416 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:37.640370    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b66c2e1-91f4-4204-be78-73c05fc42258" containerName="maas-api"
Nov 02 13:37:37.640416 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:37.640389    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d73ae6a-aaa7-4946-baa5-03ab6d91d403" containerName="istio-proxy"
Nov 02 13:37:37.640416 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:37.640394    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="4030f96f-2e78-477c-8670-c47821b4a931" containerName="maas-api"
Nov 02 13:37:37.643474 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:37.643453    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-pbc9q"
Nov 02 13:37:37.659216 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:37.659182    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Nov 02 13:37:37.672789 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:37.672738    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-5514e5-openshift-default-dockercfg-7v87s\""
Nov 02 13:37:37.678775 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:37.678735    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-pbc9q"]
Nov 02 13:37:37.735505 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:37.735469    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e35f7ad4-be0d-43c2-abc9-159989f041be-istio-podinfo\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-pbc9q\" (UID: \"e35f7ad4-be0d-43c2-abc9-159989f041be\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-pbc9q"
Nov 02 13:37:37.735656 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:37.735512    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e35f7ad4-be0d-43c2-abc9-159989f041be-credential-socket\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-pbc9q\" (UID: \"e35f7ad4-be0d-43c2-abc9-159989f041be\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-pbc9q"
Nov 02 13:37:37.735656 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:37.735608    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e35f7ad4-be0d-43c2-abc9-159989f041be-istio-token\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-pbc9q\" (UID: \"e35f7ad4-be0d-43c2-abc9-159989f041be\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-pbc9q"
Nov 02 13:37:37.735790 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:37.735692    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e35f7ad4-be0d-43c2-abc9-159989f041be-workload-certs\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-pbc9q\" (UID: \"e35f7ad4-be0d-43c2-abc9-159989f041be\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-pbc9q"
Nov 02 13:37:37.735790 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:37.735733    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e35f7ad4-be0d-43c2-abc9-159989f041be-istio-envoy\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-pbc9q\" (UID: \"e35f7ad4-be0d-43c2-abc9-159989f041be\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-pbc9q"
Nov 02 13:37:37.735903 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:37.735784    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e35f7ad4-be0d-43c2-abc9-159989f041be-istio-data\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-pbc9q\" (UID: \"e35f7ad4-be0d-43c2-abc9-159989f041be\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-pbc9q"
Nov 02 13:37:37.735903 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:37.735817    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hclcc\" (UniqueName: \"kubernetes.io/projected/e35f7ad4-be0d-43c2-abc9-159989f041be-kube-api-access-hclcc\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-pbc9q\" (UID: \"e35f7ad4-be0d-43c2-abc9-159989f041be\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-pbc9q"
Nov 02 13:37:37.735903 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:37.735846    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e35f7ad4-be0d-43c2-abc9-159989f041be-workload-socket\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-pbc9q\" (UID: \"e35f7ad4-be0d-43c2-abc9-159989f041be\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-pbc9q"
Nov 02 13:37:37.735903 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:37.735891    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e35f7ad4-be0d-43c2-abc9-159989f041be-istiod-ca-cert\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-pbc9q\" (UID: \"e35f7ad4-be0d-43c2-abc9-159989f041be\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-pbc9q"
Nov 02 13:37:37.748456 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:37.748435    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-5514e5-b884d6445-hpkbk"
Nov 02 13:37:37.773991 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:37.773975    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-5514e5-cc9847d79-8jprz"
Nov 02 13:37:37.836590 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:37.836559    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/323858c5-a582-47b3-b38f-af6a750580ca-kube-api-access-zlt5t\" (UniqueName: \"kubernetes.io/projected/323858c5-a582-47b3-b38f-af6a750580ca-kube-api-access-zlt5t\") pod \"323858c5-a582-47b3-b38f-af6a750580ca\" (UID: \"323858c5-a582-47b3-b38f-af6a750580ca\") "
Nov 02 13:37:37.836747 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:37.836624    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/323858c5-a582-47b3-b38f-af6a750580ca-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/323858c5-a582-47b3-b38f-af6a750580ca-plugins-config-volume\") pod \"323858c5-a582-47b3-b38f-af6a750580ca\" (UID: \"323858c5-a582-47b3-b38f-af6a750580ca\") "
Nov 02 13:37:37.836747 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:37.836719    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d5d728d-c371-4307-8c83-601a6e63d10c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d5d728d-c371-4307-8c83-601a6e63d10c-plugins-config-volume\") pod \"7d5d728d-c371-4307-8c83-601a6e63d10c\" (UID: \"7d5d728d-c371-4307-8c83-601a6e63d10c\") "
Nov 02 13:37:37.836890 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:37.836796    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d5d728d-c371-4307-8c83-601a6e63d10c-kube-api-access-ptfvf\" (UniqueName: \"kubernetes.io/projected/7d5d728d-c371-4307-8c83-601a6e63d10c-kube-api-access-ptfvf\") pod \"7d5d728d-c371-4307-8c83-601a6e63d10c\" (UID: \"7d5d728d-c371-4307-8c83-601a6e63d10c\") "
Nov 02 13:37:37.836960 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:37.836941    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e35f7ad4-be0d-43c2-abc9-159989f041be-istio-podinfo\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-pbc9q\" (UID: \"e35f7ad4-be0d-43c2-abc9-159989f041be\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-pbc9q"
Nov 02 13:37:37.837026 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:37.836980    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e35f7ad4-be0d-43c2-abc9-159989f041be-credential-socket\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-pbc9q\" (UID: \"e35f7ad4-be0d-43c2-abc9-159989f041be\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-pbc9q"
Nov 02 13:37:37.837026 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:37.837010    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e35f7ad4-be0d-43c2-abc9-159989f041be-istio-token\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-pbc9q\" (UID: \"e35f7ad4-be0d-43c2-abc9-159989f041be\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-pbc9q"
Nov 02 13:37:37.837145 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:37.837055    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e35f7ad4-be0d-43c2-abc9-159989f041be-workload-certs\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-pbc9q\" (UID: \"e35f7ad4-be0d-43c2-abc9-159989f041be\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-pbc9q"
Nov 02 13:37:37.837145 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:37.837098    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d5d728d-c371-4307-8c83-601a6e63d10c-plugins-config-volume" pod "7d5d728d-c371-4307-8c83-601a6e63d10c" (UID: "7d5d728d-c371-4307-8c83-601a6e63d10c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:37:37.837476 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:37.837446    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e35f7ad4-be0d-43c2-abc9-159989f041be-credential-socket\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-pbc9q\" (UID: \"e35f7ad4-be0d-43c2-abc9-159989f041be\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-pbc9q"
Nov 02 13:37:37.837476 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:37.837473    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e35f7ad4-be0d-43c2-abc9-159989f041be-istio-envoy\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-pbc9q\" (UID: \"e35f7ad4-be0d-43c2-abc9-159989f041be\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-pbc9q"
Nov 02 13:37:37.837689 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:37.837452    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e35f7ad4-be0d-43c2-abc9-159989f041be-workload-certs\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-pbc9q\" (UID: \"e35f7ad4-be0d-43c2-abc9-159989f041be\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-pbc9q"
Nov 02 13:37:37.837689 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:37.837541    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e35f7ad4-be0d-43c2-abc9-159989f041be-istio-data\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-pbc9q\" (UID: \"e35f7ad4-be0d-43c2-abc9-159989f041be\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-pbc9q"
Nov 02 13:37:37.837689 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:37.837574    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hclcc\" (UniqueName: \"kubernetes.io/projected/e35f7ad4-be0d-43c2-abc9-159989f041be-kube-api-access-hclcc\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-pbc9q\" (UID: \"e35f7ad4-be0d-43c2-abc9-159989f041be\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-pbc9q"
Nov 02 13:37:37.837689 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:37.837606    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e35f7ad4-be0d-43c2-abc9-159989f041be-workload-socket\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-pbc9q\" (UID: \"e35f7ad4-be0d-43c2-abc9-159989f041be\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-pbc9q"
Nov 02 13:37:37.837689 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:37.837673    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e35f7ad4-be0d-43c2-abc9-159989f041be-istiod-ca-cert\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-pbc9q\" (UID: \"e35f7ad4-be0d-43c2-abc9-159989f041be\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-pbc9q"
Nov 02 13:37:37.837969 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:37.837727    4559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d5d728d-c371-4307-8c83-601a6e63d10c-plugins-config-volume\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:37:37.838077 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:37.838051    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e35f7ad4-be0d-43c2-abc9-159989f041be-istio-data\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-pbc9q\" (UID: \"e35f7ad4-be0d-43c2-abc9-159989f041be\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-pbc9q"
Nov 02 13:37:37.838385 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:37.838331    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e35f7ad4-be0d-43c2-abc9-159989f041be-workload-socket\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-pbc9q\" (UID: \"e35f7ad4-be0d-43c2-abc9-159989f041be\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-pbc9q"
Nov 02 13:37:37.838478 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:37.838386    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e35f7ad4-be0d-43c2-abc9-159989f041be-istiod-ca-cert\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-pbc9q\" (UID: \"e35f7ad4-be0d-43c2-abc9-159989f041be\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-pbc9q"
Nov 02 13:37:37.838478 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:37.837718    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/323858c5-a582-47b3-b38f-af6a750580ca-plugins-config-volume" pod "323858c5-a582-47b3-b38f-af6a750580ca" (UID: "323858c5-a582-47b3-b38f-af6a750580ca"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:37:37.840838 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:37.840813    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e35f7ad4-be0d-43c2-abc9-159989f041be-istio-envoy\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-pbc9q\" (UID: \"e35f7ad4-be0d-43c2-abc9-159989f041be\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-pbc9q"
Nov 02 13:37:37.841072 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:37.841051    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/323858c5-a582-47b3-b38f-af6a750580ca-kube-api-access-zlt5t" pod "323858c5-a582-47b3-b38f-af6a750580ca" (UID: "323858c5-a582-47b3-b38f-af6a750580ca"). InnerVolumeSpecName "kube-api-access-zlt5t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:37:37.841139 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:37.841087    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e35f7ad4-be0d-43c2-abc9-159989f041be-istio-podinfo\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-pbc9q\" (UID: \"e35f7ad4-be0d-43c2-abc9-159989f041be\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-pbc9q"
Nov 02 13:37:37.841944 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:37.841921    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d5d728d-c371-4307-8c83-601a6e63d10c-kube-api-access-ptfvf" pod "7d5d728d-c371-4307-8c83-601a6e63d10c" (UID: "7d5d728d-c371-4307-8c83-601a6e63d10c"). InnerVolumeSpecName "kube-api-access-ptfvf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:37:37.860405 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:37.860377    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e35f7ad4-be0d-43c2-abc9-159989f041be-istio-token\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-pbc9q\" (UID: \"e35f7ad4-be0d-43c2-abc9-159989f041be\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-pbc9q"
Nov 02 13:37:37.861108 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:37.861091    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hclcc\" (UniqueName: \"kubernetes.io/projected/e35f7ad4-be0d-43c2-abc9-159989f041be-kube-api-access-hclcc\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-pbc9q\" (UID: \"e35f7ad4-be0d-43c2-abc9-159989f041be\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-pbc9q"
Nov 02 13:37:37.938996 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:37.938971    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zlt5t\" (UniqueName: \"kubernetes.io/projected/323858c5-a582-47b3-b38f-af6a750580ca-kube-api-access-zlt5t\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:37:37.938996 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:37.938994    4559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/323858c5-a582-47b3-b38f-af6a750580ca-plugins-config-volume\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:37:37.939131 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:37.939003    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ptfvf\" (UniqueName: \"kubernetes.io/projected/7d5d728d-c371-4307-8c83-601a6e63d10c-kube-api-access-ptfvf\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:37:37.958906 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:37.958884    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-pbc9q"
Nov 02 13:37:38.270585 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:38.270554    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-pbc9q"]
Nov 02 13:37:38.271445 ip-10-0-128-132 kubenswrapper[4559]: W1102 13:37:38.271408    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode35f7ad4_be0d_43c2_abc9_159989f041be.slice/crio-0c53d95c12eb6d79b48341889aa41657646ec085149c31ead426814b84a4b259 WatchSource:0}: Error finding container 0c53d95c12eb6d79b48341889aa41657646ec085149c31ead426814b84a4b259: Status 404 returned error can't find the container with id 0c53d95c12eb6d79b48341889aa41657646ec085149c31ead426814b84a4b259
Nov 02 13:37:38.273621 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:38.273586    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 13:37:38.273704 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:38.273683    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 13:37:38.273748 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:38.273717    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 13:37:38.516440 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:38.516408    4559 generic.go:356] "Generic (PLEG): container finished" podID="7d5d728d-c371-4307-8c83-601a6e63d10c" containerID="9e1762642d89763777ed3a88948f10ddbb1af3415403bda5d91a996a88e58931" exitCode=0
Nov 02 13:37:38.516617 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:38.516471    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-5514e5-cc9847d79-8jprz"
Nov 02 13:37:38.516617 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:38.516486    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-5514e5-cc9847d79-8jprz" event={"ID":"7d5d728d-c371-4307-8c83-601a6e63d10c","Type":"ContainerDied","Data":"9e1762642d89763777ed3a88948f10ddbb1af3415403bda5d91a996a88e58931"}
Nov 02 13:37:38.516617 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:38.516537    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-5514e5-cc9847d79-8jprz" event={"ID":"7d5d728d-c371-4307-8c83-601a6e63d10c","Type":"ContainerDied","Data":"5bb24b6c27243f6dd7513923f12c0a71c5aa6c3a8a57bff3e7175d0e10939e17"}
Nov 02 13:37:38.516617 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:38.516562    4559 scope.go:122] "RemoveContainer" containerID="9e1762642d89763777ed3a88948f10ddbb1af3415403bda5d91a996a88e58931"
Nov 02 13:37:38.518965 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:38.518239    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-pbc9q" event={"ID":"e35f7ad4-be0d-43c2-abc9-159989f041be","Type":"ContainerStarted","Data":"aa16490a6e6a76bc0a3ff085c141ddeca5409a4044a918218185bc973f8aaa1d"}
Nov 02 13:37:38.518965 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:38.518348    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-pbc9q" event={"ID":"e35f7ad4-be0d-43c2-abc9-159989f041be","Type":"ContainerStarted","Data":"0c53d95c12eb6d79b48341889aa41657646ec085149c31ead426814b84a4b259"}
Nov 02 13:37:38.520176 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:38.520150    4559 generic.go:356] "Generic (PLEG): container finished" podID="323858c5-a582-47b3-b38f-af6a750580ca" containerID="f8b322bb97c51daf956a8928056f0df68b3dc8d0d34e7174883bfb0fbac42762" exitCode=0
Nov 02 13:37:38.520298 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:38.520183    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-5514e5-b884d6445-hpkbk"
Nov 02 13:37:38.520298 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:38.520203    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-5514e5-b884d6445-hpkbk" event={"ID":"323858c5-a582-47b3-b38f-af6a750580ca","Type":"ContainerDied","Data":"f8b322bb97c51daf956a8928056f0df68b3dc8d0d34e7174883bfb0fbac42762"}
Nov 02 13:37:38.520298 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:38.520243    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-5514e5-b884d6445-hpkbk" event={"ID":"323858c5-a582-47b3-b38f-af6a750580ca","Type":"ContainerDied","Data":"810283fd84af18d5942bfda569ee4c04ac3353e8d65a76b2b7041d5d1125135a"}
Nov 02 13:37:38.528051 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:38.528021    4559 scope.go:122] "RemoveContainer" containerID="9e1762642d89763777ed3a88948f10ddbb1af3415403bda5d91a996a88e58931"
Nov 02 13:37:38.528414 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:37:38.528385    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e1762642d89763777ed3a88948f10ddbb1af3415403bda5d91a996a88e58931\": container with ID starting with 9e1762642d89763777ed3a88948f10ddbb1af3415403bda5d91a996a88e58931 not found: ID does not exist" containerID="9e1762642d89763777ed3a88948f10ddbb1af3415403bda5d91a996a88e58931"
Nov 02 13:37:38.528530 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:38.528422    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e1762642d89763777ed3a88948f10ddbb1af3415403bda5d91a996a88e58931"} err="failed to get container status \"9e1762642d89763777ed3a88948f10ddbb1af3415403bda5d91a996a88e58931\": rpc error: code = NotFound desc = could not find container \"9e1762642d89763777ed3a88948f10ddbb1af3415403bda5d91a996a88e58931\": container with ID starting with 9e1762642d89763777ed3a88948f10ddbb1af3415403bda5d91a996a88e58931 not found: ID does not exist"
Nov 02 13:37:38.528530 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:38.528446    4559 scope.go:122] "RemoveContainer" containerID="f8b322bb97c51daf956a8928056f0df68b3dc8d0d34e7174883bfb0fbac42762"
Nov 02 13:37:38.537461 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:38.537441    4559 scope.go:122] "RemoveContainer" containerID="f8b322bb97c51daf956a8928056f0df68b3dc8d0d34e7174883bfb0fbac42762"
Nov 02 13:37:38.537822 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:37:38.537796    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8b322bb97c51daf956a8928056f0df68b3dc8d0d34e7174883bfb0fbac42762\": container with ID starting with f8b322bb97c51daf956a8928056f0df68b3dc8d0d34e7174883bfb0fbac42762 not found: ID does not exist" containerID="f8b322bb97c51daf956a8928056f0df68b3dc8d0d34e7174883bfb0fbac42762"
Nov 02 13:37:38.537912 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:38.537833    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8b322bb97c51daf956a8928056f0df68b3dc8d0d34e7174883bfb0fbac42762"} err="failed to get container status \"f8b322bb97c51daf956a8928056f0df68b3dc8d0d34e7174883bfb0fbac42762\": rpc error: code = NotFound desc = could not find container \"f8b322bb97c51daf956a8928056f0df68b3dc8d0d34e7174883bfb0fbac42762\": container with ID starting with f8b322bb97c51daf956a8928056f0df68b3dc8d0d34e7174883bfb0fbac42762 not found: ID does not exist"
Nov 02 13:37:38.568950 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:38.568927    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-5514e5-b884d6445-hpkbk"]
Nov 02 13:37:38.572426 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:38.572408    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-5514e5-b884d6445-hpkbk"]
Nov 02 13:37:38.607342 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:38.607315    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-5514e5-cc9847d79-8jprz"]
Nov 02 13:37:38.611901 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:38.611870    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-5514e5-cc9847d79-8jprz"]
Nov 02 13:37:38.666080 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:38.666032    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-pbc9q" podStartSLOduration=1.666017799 podStartE2EDuration="1.666017799s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:37:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:37:38.665026969 +0000 UTC m=+4254.920426059" watchObservedRunningTime="2026-11-02 13:37:38.666017799 +0000 UTC m=+4254.921416894"
Nov 02 13:37:38.959179 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:38.959151    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-pbc9q"
Nov 02 13:37:39.911204 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:39.911170    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="323858c5-a582-47b3-b38f-af6a750580ca" path="/var/lib/kubelet/pods/323858c5-a582-47b3-b38f-af6a750580ca/volumes"
Nov 02 13:37:39.911592 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:39.911514    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d5d728d-c371-4307-8c83-601a6e63d10c" path="/var/lib/kubelet/pods/7d5d728d-c371-4307-8c83-601a6e63d10c/volumes"
Nov 02 13:37:39.960817 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:39.960791    4559 patch_prober.go:29] interesting pod/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-pbc9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.7:15021/healthz/ready\": dial tcp 10.133.1.7:15021: connect: connection refused" start-of-body=
Nov 02 13:37:39.960938 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:39.960835    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-pbc9q" podUID="e35f7ad4-be0d-43c2-abc9-159989f041be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.7:15021/healthz/ready\": dial tcp 10.133.1.7:15021: connect: connection refused"
Nov 02 13:37:40.959466 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:40.959429    4559 patch_prober.go:29] interesting pod/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-pbc9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.7:15021/healthz/ready\": dial tcp 10.133.1.7:15021: connect: connection refused" start-of-body=
Nov 02 13:37:40.959880 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:40.959479    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-pbc9q" podUID="e35f7ad4-be0d-43c2-abc9-159989f041be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.7:15021/healthz/ready\": dial tcp 10.133.1.7:15021: connect: connection refused"
Nov 02 13:37:41.960230 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:41.960192    4559 patch_prober.go:29] interesting pod/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-pbc9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.7:15021/healthz/ready\": dial tcp 10.133.1.7:15021: connect: connection refused" start-of-body=
Nov 02 13:37:41.960631 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:41.960246    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-pbc9q" podUID="e35f7ad4-be0d-43c2-abc9-159989f041be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.7:15021/healthz/ready\": dial tcp 10.133.1.7:15021: connect: connection refused"
Nov 02 13:37:42.959334 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:42.959301    4559 patch_prober.go:29] interesting pod/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-pbc9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.7:15021/healthz/ready\": dial tcp 10.133.1.7:15021: connect: connection refused" start-of-body=
Nov 02 13:37:42.959517 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:42.959356    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-pbc9q" podUID="e35f7ad4-be0d-43c2-abc9-159989f041be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.7:15021/healthz/ready\": dial tcp 10.133.1.7:15021: connect: connection refused"
Nov 02 13:37:43.960090 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:43.960056    4559 patch_prober.go:29] interesting pod/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-pbc9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.7:15021/healthz/ready\": dial tcp 10.133.1.7:15021: connect: connection refused" start-of-body=
Nov 02 13:37:43.960477 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:43.960107    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-pbc9q" podUID="e35f7ad4-be0d-43c2-abc9-159989f041be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.7:15021/healthz/ready\": dial tcp 10.133.1.7:15021: connect: connection refused"
Nov 02 13:37:44.963041 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:44.963011    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-pbc9q"
Nov 02 13:37:44.963533 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:44.963288    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-pbc9q"
Nov 02 13:37:44.964089 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:44.964071    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-pbc9q"
Nov 02 13:37:45.094128 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:37:45.094090    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqcmj" containerName="extractor"
Nov 02 13:37:46.124443 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:46.124407    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-pbc9q"]
Nov 02 13:37:46.562436 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:46.562398    4559 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-pbc9q" secret="" err="secret \"e2e-worker-w0-5514e5-openshift-default-dockercfg-7v87s\" not found"
Nov 02 13:37:47.249377 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:47.249344    4559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6768bb47c6-q87t7_a00817af-1ee2-4bad-b007-b058928a12b7/authorino/0.log"
Nov 02 13:37:47.566454 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:47.566411    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-pbc9q" podUID="e35f7ad4-be0d-43c2-abc9-159989f041be" containerName="istio-proxy" containerID="cri-o://aa16490a6e6a76bc0a3ff085c141ddeca5409a4044a918218185bc973f8aaa1d" gracePeriod=30
Nov 02 13:37:48.623340 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:48.623312    4559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-l9pbf_ae914cec-8158-4c77-9870-e0fa6573c7b6/limitador/0.log"
Nov 02 13:37:49.707443 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:49.707413    4559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-64577f84c6-5nmh8_4af87e02-6dc0-42ef-93df-26e174ec85df/manager/0.log"
Nov 02 13:37:52.808789 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:52.808763    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-pbc9q"
Nov 02 13:37:52.992896 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:52.992856    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e35f7ad4-be0d-43c2-abc9-159989f041be-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e35f7ad4-be0d-43c2-abc9-159989f041be-workload-socket\") pod \"e35f7ad4-be0d-43c2-abc9-159989f041be\" (UID: \"e35f7ad4-be0d-43c2-abc9-159989f041be\") "
Nov 02 13:37:52.993097 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:52.992917    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e35f7ad4-be0d-43c2-abc9-159989f041be-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e35f7ad4-be0d-43c2-abc9-159989f041be-istiod-ca-cert\") pod \"e35f7ad4-be0d-43c2-abc9-159989f041be\" (UID: \"e35f7ad4-be0d-43c2-abc9-159989f041be\") "
Nov 02 13:37:52.993097 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:52.992938    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e35f7ad4-be0d-43c2-abc9-159989f041be-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e35f7ad4-be0d-43c2-abc9-159989f041be-istio-data\") pod \"e35f7ad4-be0d-43c2-abc9-159989f041be\" (UID: \"e35f7ad4-be0d-43c2-abc9-159989f041be\") "
Nov 02 13:37:52.993097 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:52.992961    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e35f7ad4-be0d-43c2-abc9-159989f041be-istio-token\" (UniqueName: \"kubernetes.io/projected/e35f7ad4-be0d-43c2-abc9-159989f041be-istio-token\") pod \"e35f7ad4-be0d-43c2-abc9-159989f041be\" (UID: \"e35f7ad4-be0d-43c2-abc9-159989f041be\") "
Nov 02 13:37:52.993301 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:52.993122    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e35f7ad4-be0d-43c2-abc9-159989f041be-kube-api-access-hclcc\" (UniqueName: \"kubernetes.io/projected/e35f7ad4-be0d-43c2-abc9-159989f041be-kube-api-access-hclcc\") pod \"e35f7ad4-be0d-43c2-abc9-159989f041be\" (UID: \"e35f7ad4-be0d-43c2-abc9-159989f041be\") "
Nov 02 13:37:52.993301 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:52.993138    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e35f7ad4-be0d-43c2-abc9-159989f041be-workload-socket" pod "e35f7ad4-be0d-43c2-abc9-159989f041be" (UID: "e35f7ad4-be0d-43c2-abc9-159989f041be"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:37:52.993301 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:52.993159    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e35f7ad4-be0d-43c2-abc9-159989f041be-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e35f7ad4-be0d-43c2-abc9-159989f041be-istio-envoy\") pod \"e35f7ad4-be0d-43c2-abc9-159989f041be\" (UID: \"e35f7ad4-be0d-43c2-abc9-159989f041be\") "
Nov 02 13:37:52.993301 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:52.993207    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e35f7ad4-be0d-43c2-abc9-159989f041be-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e35f7ad4-be0d-43c2-abc9-159989f041be-credential-socket\") pod \"e35f7ad4-be0d-43c2-abc9-159989f041be\" (UID: \"e35f7ad4-be0d-43c2-abc9-159989f041be\") "
Nov 02 13:37:52.993301 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:52.993238    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e35f7ad4-be0d-43c2-abc9-159989f041be-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e35f7ad4-be0d-43c2-abc9-159989f041be-workload-certs\") pod \"e35f7ad4-be0d-43c2-abc9-159989f041be\" (UID: \"e35f7ad4-be0d-43c2-abc9-159989f041be\") "
Nov 02 13:37:52.993301 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:52.993268    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e35f7ad4-be0d-43c2-abc9-159989f041be-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e35f7ad4-be0d-43c2-abc9-159989f041be-istio-podinfo\") pod \"e35f7ad4-be0d-43c2-abc9-159989f041be\" (UID: \"e35f7ad4-be0d-43c2-abc9-159989f041be\") "
Nov 02 13:37:52.993301 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:52.993287    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e35f7ad4-be0d-43c2-abc9-159989f041be-istio-data" pod "e35f7ad4-be0d-43c2-abc9-159989f041be" (UID: "e35f7ad4-be0d-43c2-abc9-159989f041be"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:37:52.993696 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:52.993347    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e35f7ad4-be0d-43c2-abc9-159989f041be-istiod-ca-cert" pod "e35f7ad4-be0d-43c2-abc9-159989f041be" (UID: "e35f7ad4-be0d-43c2-abc9-159989f041be"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:37:52.993696 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:52.993508    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e35f7ad4-be0d-43c2-abc9-159989f041be-credential-socket" pod "e35f7ad4-be0d-43c2-abc9-159989f041be" (UID: "e35f7ad4-be0d-43c2-abc9-159989f041be"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:37:52.993696 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:52.993530    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e35f7ad4-be0d-43c2-abc9-159989f041be-workload-certs" pod "e35f7ad4-be0d-43c2-abc9-159989f041be" (UID: "e35f7ad4-be0d-43c2-abc9-159989f041be"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:37:52.993696 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:52.993627    4559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e35f7ad4-be0d-43c2-abc9-159989f041be-credential-socket\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:37:52.993696 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:52.993679    4559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e35f7ad4-be0d-43c2-abc9-159989f041be-workload-certs\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:37:52.993696 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:52.993695    4559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e35f7ad4-be0d-43c2-abc9-159989f041be-workload-socket\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:37:52.994048 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:52.993708    4559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e35f7ad4-be0d-43c2-abc9-159989f041be-istiod-ca-cert\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:37:52.994048 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:52.993721    4559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e35f7ad4-be0d-43c2-abc9-159989f041be-istio-data\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:37:52.995228 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:52.995200    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e35f7ad4-be0d-43c2-abc9-159989f041be-istio-token" pod "e35f7ad4-be0d-43c2-abc9-159989f041be" (UID: "e35f7ad4-be0d-43c2-abc9-159989f041be"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:37:52.995329 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:52.995258    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e35f7ad4-be0d-43c2-abc9-159989f041be-istio-envoy" pod "e35f7ad4-be0d-43c2-abc9-159989f041be" (UID: "e35f7ad4-be0d-43c2-abc9-159989f041be"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:37:52.995329 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:52.995287    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e35f7ad4-be0d-43c2-abc9-159989f041be-kube-api-access-hclcc" pod "e35f7ad4-be0d-43c2-abc9-159989f041be" (UID: "e35f7ad4-be0d-43c2-abc9-159989f041be"). InnerVolumeSpecName "kube-api-access-hclcc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:37:52.995421 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:52.995396    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e35f7ad4-be0d-43c2-abc9-159989f041be-istio-podinfo" pod "e35f7ad4-be0d-43c2-abc9-159989f041be" (UID: "e35f7ad4-be0d-43c2-abc9-159989f041be"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 13:37:53.095001 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:53.094968    4559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e35f7ad4-be0d-43c2-abc9-159989f041be-istio-token\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:37:53.095001 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:53.094995    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hclcc\" (UniqueName: \"kubernetes.io/projected/e35f7ad4-be0d-43c2-abc9-159989f041be-kube-api-access-hclcc\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:37:53.095001 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:53.095006    4559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e35f7ad4-be0d-43c2-abc9-159989f041be-istio-envoy\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:37:53.095224 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:53.095015    4559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e35f7ad4-be0d-43c2-abc9-159989f041be-istio-podinfo\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:37:53.591094 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:53.591057    4559 generic.go:356] "Generic (PLEG): container finished" podID="e35f7ad4-be0d-43c2-abc9-159989f041be" containerID="aa16490a6e6a76bc0a3ff085c141ddeca5409a4044a918218185bc973f8aaa1d" exitCode=0
Nov 02 13:37:53.591273 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:53.591105    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-pbc9q" event={"ID":"e35f7ad4-be0d-43c2-abc9-159989f041be","Type":"ContainerDied","Data":"aa16490a6e6a76bc0a3ff085c141ddeca5409a4044a918218185bc973f8aaa1d"}
Nov 02 13:37:53.591273 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:53.591107    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-pbc9q"
Nov 02 13:37:53.591273 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:53.591151    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-pbc9q" event={"ID":"e35f7ad4-be0d-43c2-abc9-159989f041be","Type":"ContainerDied","Data":"0c53d95c12eb6d79b48341889aa41657646ec085149c31ead426814b84a4b259"}
Nov 02 13:37:53.591273 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:53.591175    4559 scope.go:122] "RemoveContainer" containerID="aa16490a6e6a76bc0a3ff085c141ddeca5409a4044a918218185bc973f8aaa1d"
Nov 02 13:37:53.600542 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:53.600522    4559 scope.go:122] "RemoveContainer" containerID="aa16490a6e6a76bc0a3ff085c141ddeca5409a4044a918218185bc973f8aaa1d"
Nov 02 13:37:53.600837 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:37:53.600815    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa16490a6e6a76bc0a3ff085c141ddeca5409a4044a918218185bc973f8aaa1d\": container with ID starting with aa16490a6e6a76bc0a3ff085c141ddeca5409a4044a918218185bc973f8aaa1d not found: ID does not exist" containerID="aa16490a6e6a76bc0a3ff085c141ddeca5409a4044a918218185bc973f8aaa1d"
Nov 02 13:37:53.600917 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:53.600849    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa16490a6e6a76bc0a3ff085c141ddeca5409a4044a918218185bc973f8aaa1d"} err="failed to get container status \"aa16490a6e6a76bc0a3ff085c141ddeca5409a4044a918218185bc973f8aaa1d\": rpc error: code = NotFound desc = could not find container \"aa16490a6e6a76bc0a3ff085c141ddeca5409a4044a918218185bc973f8aaa1d\": container with ID starting with aa16490a6e6a76bc0a3ff085c141ddeca5409a4044a918218185bc973f8aaa1d not found: ID does not exist"
Nov 02 13:37:53.657360 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:53.657328    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-pbc9q"]
Nov 02 13:37:53.670015 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:53.669991    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-pbc9q"]
Nov 02 13:37:53.890648 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:53.890566    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e35f7ad4-be0d-43c2-abc9-159989f041be" path="/var/lib/kubelet/pods/e35f7ad4-be0d-43c2-abc9-159989f041be/volumes"
Nov 02 13:37:56.590308 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:56.590231    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-60a7443a-openshift-default-5f47b4f685-4flqv"]
Nov 02 13:37:56.590732 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:56.590701    4559 cpu_manager.go:414] "containerMap: removing container" podUID="5d73ae6a-aaa7-4946-baa5-03ab6d91d403" containerName="istio-proxy"
Nov 02 13:37:56.590732 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:56.590716    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d73ae6a-aaa7-4946-baa5-03ab6d91d403" containerName="istio-proxy"
Nov 02 13:37:56.590825 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:56.590735    4559 cpu_manager.go:414] "containerMap: removing container" podUID="323858c5-a582-47b3-b38f-af6a750580ca" containerName="payload-processing"
Nov 02 13:37:56.590825 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:56.590743    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="323858c5-a582-47b3-b38f-af6a750580ca" containerName="payload-processing"
Nov 02 13:37:56.590825 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:56.590758    4559 cpu_manager.go:414] "containerMap: removing container" podUID="e35f7ad4-be0d-43c2-abc9-159989f041be" containerName="istio-proxy"
Nov 02 13:37:56.590825 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:56.590765    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e35f7ad4-be0d-43c2-abc9-159989f041be" containerName="istio-proxy"
Nov 02 13:37:56.590825 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:56.590774    4559 cpu_manager.go:414] "containerMap: removing container" podUID="0029ba38-71d3-4212-9b50-ded1a42abd1d" containerName="istio-proxy"
Nov 02 13:37:56.590825 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:56.590782    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0029ba38-71d3-4212-9b50-ded1a42abd1d" containerName="istio-proxy"
Nov 02 13:37:56.590825 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:56.590808    4559 cpu_manager.go:414] "containerMap: removing container" podUID="7d5d728d-c371-4307-8c83-601a6e63d10c" containerName="payload-pre-processing"
Nov 02 13:37:56.590825 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:56.590815    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d5d728d-c371-4307-8c83-601a6e63d10c" containerName="payload-pre-processing"
Nov 02 13:37:56.591075 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:56.590912    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="323858c5-a582-47b3-b38f-af6a750580ca" containerName="payload-processing"
Nov 02 13:37:56.591075 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:56.590923    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="e35f7ad4-be0d-43c2-abc9-159989f041be" containerName="istio-proxy"
Nov 02 13:37:56.591075 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:56.590937    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d5d728d-c371-4307-8c83-601a6e63d10c" containerName="payload-pre-processing"
Nov 02 13:37:56.596059 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:56.596035    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-60a7443a-openshift-default-5f47b4f685-4flqv"
Nov 02 13:37:56.612301 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:56.612277    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Nov 02 13:37:56.612432 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:56.612419    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-60a7443a-openshift-default-dockercfg-b2hxt\""
Nov 02 13:37:56.625235 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:56.625210    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93596dfc-4540-4cd5-9c06-d8a790d8c58e-istio-envoy\") pod \"e2e-ait-60a7443a-openshift-default-5f47b4f685-4flqv\" (UID: \"93596dfc-4540-4cd5-9c06-d8a790d8c58e\") " pod="openshift-ingress/e2e-ait-60a7443a-openshift-default-5f47b4f685-4flqv"
Nov 02 13:37:56.625365 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:56.625254    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93596dfc-4540-4cd5-9c06-d8a790d8c58e-workload-certs\") pod \"e2e-ait-60a7443a-openshift-default-5f47b4f685-4flqv\" (UID: \"93596dfc-4540-4cd5-9c06-d8a790d8c58e\") " pod="openshift-ingress/e2e-ait-60a7443a-openshift-default-5f47b4f685-4flqv"
Nov 02 13:37:56.625365 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:56.625281    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93596dfc-4540-4cd5-9c06-d8a790d8c58e-istio-data\") pod \"e2e-ait-60a7443a-openshift-default-5f47b4f685-4flqv\" (UID: \"93596dfc-4540-4cd5-9c06-d8a790d8c58e\") " pod="openshift-ingress/e2e-ait-60a7443a-openshift-default-5f47b4f685-4flqv"
Nov 02 13:37:56.625365 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:56.625341    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84sfz\" (UniqueName: \"kubernetes.io/projected/93596dfc-4540-4cd5-9c06-d8a790d8c58e-kube-api-access-84sfz\") pod \"e2e-ait-60a7443a-openshift-default-5f47b4f685-4flqv\" (UID: \"93596dfc-4540-4cd5-9c06-d8a790d8c58e\") " pod="openshift-ingress/e2e-ait-60a7443a-openshift-default-5f47b4f685-4flqv"
Nov 02 13:37:56.625512 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:56.625403    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93596dfc-4540-4cd5-9c06-d8a790d8c58e-istio-podinfo\") pod \"e2e-ait-60a7443a-openshift-default-5f47b4f685-4flqv\" (UID: \"93596dfc-4540-4cd5-9c06-d8a790d8c58e\") " pod="openshift-ingress/e2e-ait-60a7443a-openshift-default-5f47b4f685-4flqv"
Nov 02 13:37:56.625512 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:56.625442    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93596dfc-4540-4cd5-9c06-d8a790d8c58e-credential-socket\") pod \"e2e-ait-60a7443a-openshift-default-5f47b4f685-4flqv\" (UID: \"93596dfc-4540-4cd5-9c06-d8a790d8c58e\") " pod="openshift-ingress/e2e-ait-60a7443a-openshift-default-5f47b4f685-4flqv"
Nov 02 13:37:56.625512 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:56.625497    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93596dfc-4540-4cd5-9c06-d8a790d8c58e-istio-token\") pod \"e2e-ait-60a7443a-openshift-default-5f47b4f685-4flqv\" (UID: \"93596dfc-4540-4cd5-9c06-d8a790d8c58e\") " pod="openshift-ingress/e2e-ait-60a7443a-openshift-default-5f47b4f685-4flqv"
Nov 02 13:37:56.625708 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:56.625531    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93596dfc-4540-4cd5-9c06-d8a790d8c58e-workload-socket\") pod \"e2e-ait-60a7443a-openshift-default-5f47b4f685-4flqv\" (UID: \"93596dfc-4540-4cd5-9c06-d8a790d8c58e\") " pod="openshift-ingress/e2e-ait-60a7443a-openshift-default-5f47b4f685-4flqv"
Nov 02 13:37:56.625708 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:56.625580    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93596dfc-4540-4cd5-9c06-d8a790d8c58e-istiod-ca-cert\") pod \"e2e-ait-60a7443a-openshift-default-5f47b4f685-4flqv\" (UID: \"93596dfc-4540-4cd5-9c06-d8a790d8c58e\") " pod="openshift-ingress/e2e-ait-60a7443a-openshift-default-5f47b4f685-4flqv"
Nov 02 13:37:56.626924 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:56.626903    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-60a7443a-openshift-default-5f47b4f685-4flqv"]
Nov 02 13:37:56.726377 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:56.726348    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93596dfc-4540-4cd5-9c06-d8a790d8c58e-istio-token\") pod \"e2e-ait-60a7443a-openshift-default-5f47b4f685-4flqv\" (UID: \"93596dfc-4540-4cd5-9c06-d8a790d8c58e\") " pod="openshift-ingress/e2e-ait-60a7443a-openshift-default-5f47b4f685-4flqv"
Nov 02 13:37:56.726513 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:56.726383    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93596dfc-4540-4cd5-9c06-d8a790d8c58e-workload-socket\") pod \"e2e-ait-60a7443a-openshift-default-5f47b4f685-4flqv\" (UID: \"93596dfc-4540-4cd5-9c06-d8a790d8c58e\") " pod="openshift-ingress/e2e-ait-60a7443a-openshift-default-5f47b4f685-4flqv"
Nov 02 13:37:56.726513 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:56.726408    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93596dfc-4540-4cd5-9c06-d8a790d8c58e-istiod-ca-cert\") pod \"e2e-ait-60a7443a-openshift-default-5f47b4f685-4flqv\" (UID: \"93596dfc-4540-4cd5-9c06-d8a790d8c58e\") " pod="openshift-ingress/e2e-ait-60a7443a-openshift-default-5f47b4f685-4flqv"
Nov 02 13:37:56.726513 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:56.726441    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93596dfc-4540-4cd5-9c06-d8a790d8c58e-istio-envoy\") pod \"e2e-ait-60a7443a-openshift-default-5f47b4f685-4flqv\" (UID: \"93596dfc-4540-4cd5-9c06-d8a790d8c58e\") " pod="openshift-ingress/e2e-ait-60a7443a-openshift-default-5f47b4f685-4flqv"
Nov 02 13:37:56.726513 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:56.726461    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93596dfc-4540-4cd5-9c06-d8a790d8c58e-workload-certs\") pod \"e2e-ait-60a7443a-openshift-default-5f47b4f685-4flqv\" (UID: \"93596dfc-4540-4cd5-9c06-d8a790d8c58e\") " pod="openshift-ingress/e2e-ait-60a7443a-openshift-default-5f47b4f685-4flqv"
Nov 02 13:37:56.726513 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:56.726475    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93596dfc-4540-4cd5-9c06-d8a790d8c58e-istio-data\") pod \"e2e-ait-60a7443a-openshift-default-5f47b4f685-4flqv\" (UID: \"93596dfc-4540-4cd5-9c06-d8a790d8c58e\") " pod="openshift-ingress/e2e-ait-60a7443a-openshift-default-5f47b4f685-4flqv"
Nov 02 13:37:56.726513 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:56.726491    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84sfz\" (UniqueName: \"kubernetes.io/projected/93596dfc-4540-4cd5-9c06-d8a790d8c58e-kube-api-access-84sfz\") pod \"e2e-ait-60a7443a-openshift-default-5f47b4f685-4flqv\" (UID: \"93596dfc-4540-4cd5-9c06-d8a790d8c58e\") " pod="openshift-ingress/e2e-ait-60a7443a-openshift-default-5f47b4f685-4flqv"
Nov 02 13:37:56.726878 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:56.726515    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93596dfc-4540-4cd5-9c06-d8a790d8c58e-istio-podinfo\") pod \"e2e-ait-60a7443a-openshift-default-5f47b4f685-4flqv\" (UID: \"93596dfc-4540-4cd5-9c06-d8a790d8c58e\") " pod="openshift-ingress/e2e-ait-60a7443a-openshift-default-5f47b4f685-4flqv"
Nov 02 13:37:56.726878 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:56.726545    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93596dfc-4540-4cd5-9c06-d8a790d8c58e-credential-socket\") pod \"e2e-ait-60a7443a-openshift-default-5f47b4f685-4flqv\" (UID: \"93596dfc-4540-4cd5-9c06-d8a790d8c58e\") " pod="openshift-ingress/e2e-ait-60a7443a-openshift-default-5f47b4f685-4flqv"
Nov 02 13:37:56.726878 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:56.726819    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93596dfc-4540-4cd5-9c06-d8a790d8c58e-workload-socket\") pod \"e2e-ait-60a7443a-openshift-default-5f47b4f685-4flqv\" (UID: \"93596dfc-4540-4cd5-9c06-d8a790d8c58e\") " pod="openshift-ingress/e2e-ait-60a7443a-openshift-default-5f47b4f685-4flqv"
Nov 02 13:37:56.727043 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:56.726971    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93596dfc-4540-4cd5-9c06-d8a790d8c58e-workload-certs\") pod \"e2e-ait-60a7443a-openshift-default-5f47b4f685-4flqv\" (UID: \"93596dfc-4540-4cd5-9c06-d8a790d8c58e\") " pod="openshift-ingress/e2e-ait-60a7443a-openshift-default-5f47b4f685-4flqv"
Nov 02 13:37:56.727043 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:56.727006    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93596dfc-4540-4cd5-9c06-d8a790d8c58e-credential-socket\") pod \"e2e-ait-60a7443a-openshift-default-5f47b4f685-4flqv\" (UID: \"93596dfc-4540-4cd5-9c06-d8a790d8c58e\") " pod="openshift-ingress/e2e-ait-60a7443a-openshift-default-5f47b4f685-4flqv"
Nov 02 13:37:56.727043 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:56.727031    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93596dfc-4540-4cd5-9c06-d8a790d8c58e-istio-data\") pod \"e2e-ait-60a7443a-openshift-default-5f47b4f685-4flqv\" (UID: \"93596dfc-4540-4cd5-9c06-d8a790d8c58e\") " pod="openshift-ingress/e2e-ait-60a7443a-openshift-default-5f47b4f685-4flqv"
Nov 02 13:37:56.727267 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:56.727246    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93596dfc-4540-4cd5-9c06-d8a790d8c58e-istiod-ca-cert\") pod \"e2e-ait-60a7443a-openshift-default-5f47b4f685-4flqv\" (UID: \"93596dfc-4540-4cd5-9c06-d8a790d8c58e\") " pod="openshift-ingress/e2e-ait-60a7443a-openshift-default-5f47b4f685-4flqv"
Nov 02 13:37:56.728711 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:56.728686    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93596dfc-4540-4cd5-9c06-d8a790d8c58e-istio-envoy\") pod \"e2e-ait-60a7443a-openshift-default-5f47b4f685-4flqv\" (UID: \"93596dfc-4540-4cd5-9c06-d8a790d8c58e\") " pod="openshift-ingress/e2e-ait-60a7443a-openshift-default-5f47b4f685-4flqv"
Nov 02 13:37:56.729198 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:56.729166    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93596dfc-4540-4cd5-9c06-d8a790d8c58e-istio-podinfo\") pod \"e2e-ait-60a7443a-openshift-default-5f47b4f685-4flqv\" (UID: \"93596dfc-4540-4cd5-9c06-d8a790d8c58e\") " pod="openshift-ingress/e2e-ait-60a7443a-openshift-default-5f47b4f685-4flqv"
Nov 02 13:37:56.750722 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:56.750694    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93596dfc-4540-4cd5-9c06-d8a790d8c58e-istio-token\") pod \"e2e-ait-60a7443a-openshift-default-5f47b4f685-4flqv\" (UID: \"93596dfc-4540-4cd5-9c06-d8a790d8c58e\") " pod="openshift-ingress/e2e-ait-60a7443a-openshift-default-5f47b4f685-4flqv"
Nov 02 13:37:56.751018 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:56.750998    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-84sfz\" (UniqueName: \"kubernetes.io/projected/93596dfc-4540-4cd5-9c06-d8a790d8c58e-kube-api-access-84sfz\") pod \"e2e-ait-60a7443a-openshift-default-5f47b4f685-4flqv\" (UID: \"93596dfc-4540-4cd5-9c06-d8a790d8c58e\") " pod="openshift-ingress/e2e-ait-60a7443a-openshift-default-5f47b4f685-4flqv"
Nov 02 13:37:56.912199 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:56.912134    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-60a7443a-openshift-default-5f47b4f685-4flqv"
Nov 02 13:37:57.222559 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:57.222537    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-60a7443a-openshift-default-5f47b4f685-4flqv"]
Nov 02 13:37:57.224503 ip-10-0-128-132 kubenswrapper[4559]: W1102 13:37:57.224477    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93596dfc_4540_4cd5_9c06_d8a790d8c58e.slice/crio-c682234c336d13fdac5bab2b535d33322fef8db3dff63257e4f235c3351cf6e5 WatchSource:0}: Error finding container c682234c336d13fdac5bab2b535d33322fef8db3dff63257e4f235c3351cf6e5: Status 404 returned error can't find the container with id c682234c336d13fdac5bab2b535d33322fef8db3dff63257e4f235c3351cf6e5
Nov 02 13:37:57.226503 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:57.226471    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 13:37:57.226578 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:57.226533    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 13:37:57.226578 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:57.226561    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 13:37:57.611500 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:57.611415    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-60a7443a-openshift-default-5f47b4f685-4flqv" event={"ID":"93596dfc-4540-4cd5-9c06-d8a790d8c58e","Type":"ContainerStarted","Data":"4f316cbdc38bc2039a6deeac2106aa7e1d6131a4fee60463c5027108f03ee23b"}
Nov 02 13:37:57.611500 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:57.611465    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-60a7443a-openshift-default-5f47b4f685-4flqv" event={"ID":"93596dfc-4540-4cd5-9c06-d8a790d8c58e","Type":"ContainerStarted","Data":"c682234c336d13fdac5bab2b535d33322fef8db3dff63257e4f235c3351cf6e5"}
Nov 02 13:37:57.657571 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:57.657523    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-60a7443a-openshift-default-5f47b4f685-4flqv" podStartSLOduration=1.65750705 podStartE2EDuration="1.65750705s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:37:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:37:57.656917825 +0000 UTC m=+4273.912316920" watchObservedRunningTime="2026-11-02 13:37:57.65750705 +0000 UTC m=+4273.912906146"
Nov 02 13:37:57.912460 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:57.912393    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-60a7443a-openshift-default-5f47b4f685-4flqv"
Nov 02 13:37:58.918110 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:58.918081    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-60a7443a-openshift-default-5f47b4f685-4flqv"
Nov 02 13:37:59.620698 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:59.620665    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-60a7443a-openshift-default-5f47b4f685-4flqv"
Nov 02 13:37:59.621800 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:37:59.621783    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-60a7443a-openshift-default-5f47b4f685-4flqv"
Nov 02 13:38:00.689164 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:00.689135    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-60a7443a-788d448c95-8vpr9"]
Nov 02 13:38:00.692706 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:00.692687    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-60a7443a-788d448c95-8vpr9"
Nov 02 13:38:00.708735 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:00.708712    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-60a7443a-serving-cert\""
Nov 02 13:38:00.721588 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:00.721563    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Nov 02 13:38:00.724053 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:00.724030    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 13:38:00.724312 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:00.724292    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 13:38:00.725999 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:00.725977    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-60a7443a-788d448c95-8vpr9"]
Nov 02 13:38:00.727370 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:00.727282    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-sr5s9\""
Nov 02 13:38:00.757575 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:00.757551    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lth67\" (UniqueName: \"kubernetes.io/projected/9059b137-efb5-4347-9201-653988b23682-kube-api-access-lth67\") pod \"maas-api-e2e-ait-60a7443a-788d448c95-8vpr9\" (UID: \"9059b137-efb5-4347-9201-653988b23682\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-60a7443a-788d448c95-8vpr9"
Nov 02 13:38:00.757724 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:00.757591    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9059b137-efb5-4347-9201-653988b23682-openshift-service-ca\") pod \"maas-api-e2e-ait-60a7443a-788d448c95-8vpr9\" (UID: \"9059b137-efb5-4347-9201-653988b23682\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-60a7443a-788d448c95-8vpr9"
Nov 02 13:38:00.757724 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:00.757625    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9059b137-efb5-4347-9201-653988b23682-metrics-certs\") pod \"maas-api-e2e-ait-60a7443a-788d448c95-8vpr9\" (UID: \"9059b137-efb5-4347-9201-653988b23682\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-60a7443a-788d448c95-8vpr9"
Nov 02 13:38:00.757816 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:00.757725    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9059b137-efb5-4347-9201-653988b23682-maas-api-tls\") pod \"maas-api-e2e-ait-60a7443a-788d448c95-8vpr9\" (UID: \"9059b137-efb5-4347-9201-653988b23682\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-60a7443a-788d448c95-8vpr9"
Nov 02 13:38:00.858170 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:00.858141    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lth67\" (UniqueName: \"kubernetes.io/projected/9059b137-efb5-4347-9201-653988b23682-kube-api-access-lth67\") pod \"maas-api-e2e-ait-60a7443a-788d448c95-8vpr9\" (UID: \"9059b137-efb5-4347-9201-653988b23682\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-60a7443a-788d448c95-8vpr9"
Nov 02 13:38:00.858298 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:00.858181    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9059b137-efb5-4347-9201-653988b23682-openshift-service-ca\") pod \"maas-api-e2e-ait-60a7443a-788d448c95-8vpr9\" (UID: \"9059b137-efb5-4347-9201-653988b23682\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-60a7443a-788d448c95-8vpr9"
Nov 02 13:38:00.858298 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:00.858202    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9059b137-efb5-4347-9201-653988b23682-metrics-certs\") pod \"maas-api-e2e-ait-60a7443a-788d448c95-8vpr9\" (UID: \"9059b137-efb5-4347-9201-653988b23682\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-60a7443a-788d448c95-8vpr9"
Nov 02 13:38:00.858439 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:00.858320    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9059b137-efb5-4347-9201-653988b23682-maas-api-tls\") pod \"maas-api-e2e-ait-60a7443a-788d448c95-8vpr9\" (UID: \"9059b137-efb5-4347-9201-653988b23682\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-60a7443a-788d448c95-8vpr9"
Nov 02 13:38:00.858900 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:00.858876    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9059b137-efb5-4347-9201-653988b23682-openshift-service-ca\") pod \"maas-api-e2e-ait-60a7443a-788d448c95-8vpr9\" (UID: \"9059b137-efb5-4347-9201-653988b23682\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-60a7443a-788d448c95-8vpr9"
Nov 02 13:38:00.860798 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:00.860766    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9059b137-efb5-4347-9201-653988b23682-metrics-certs\") pod \"maas-api-e2e-ait-60a7443a-788d448c95-8vpr9\" (UID: \"9059b137-efb5-4347-9201-653988b23682\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-60a7443a-788d448c95-8vpr9"
Nov 02 13:38:00.860890 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:00.860829    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9059b137-efb5-4347-9201-653988b23682-maas-api-tls\") pod \"maas-api-e2e-ait-60a7443a-788d448c95-8vpr9\" (UID: \"9059b137-efb5-4347-9201-653988b23682\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-60a7443a-788d448c95-8vpr9"
Nov 02 13:38:00.881887 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:00.881860    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lth67\" (UniqueName: \"kubernetes.io/projected/9059b137-efb5-4347-9201-653988b23682-kube-api-access-lth67\") pod \"maas-api-e2e-ait-60a7443a-788d448c95-8vpr9\" (UID: \"9059b137-efb5-4347-9201-653988b23682\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-60a7443a-788d448c95-8vpr9"
Nov 02 13:38:01.008472 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:01.008388    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-60a7443a-788d448c95-8vpr9"
Nov 02 13:38:01.183588 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:01.183552    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-60a7443a-5467958647-np56w"]
Nov 02 13:38:01.187990 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:01.187973    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-60a7443a-5467958647-np56w"
Nov 02 13:38:01.204470 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:01.204433    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-60a7443a\""
Nov 02 13:38:01.204835 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:01.204815    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-60a7443a-dockercfg-47s46\""
Nov 02 13:38:01.206413 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:01.206380    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-60a7443a-5467958647-np56w"]
Nov 02 13:38:01.262135 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:01.262049    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3014668-082f-41c1-abf3-bff163f64eba-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-60a7443a-5467958647-np56w\" (UID: \"e3014668-082f-41c1-abf3-bff163f64eba\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-60a7443a-5467958647-np56w"
Nov 02 13:38:01.262135 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:01.262099    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9mx6\" (UniqueName: \"kubernetes.io/projected/e3014668-082f-41c1-abf3-bff163f64eba-kube-api-access-b9mx6\") pod \"payload-pre-processing-e2e-ait-60a7443a-5467958647-np56w\" (UID: \"e3014668-082f-41c1-abf3-bff163f64eba\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-60a7443a-5467958647-np56w"
Nov 02 13:38:01.265887 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:01.265860    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-60a7443a-59485bdc55-8jjx5"]
Nov 02 13:38:01.269270 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:01.269252    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-60a7443a-59485bdc55-8jjx5"
Nov 02 13:38:01.290881 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:01.290857    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-60a7443a-59485bdc55-8jjx5"]
Nov 02 13:38:01.328353 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:01.328328    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-60a7443a-788d448c95-8vpr9"]
Nov 02 13:38:01.329402 ip-10-0-128-132 kubenswrapper[4559]: W1102 13:38:01.329376    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9059b137_efb5_4347_9201_653988b23682.slice/crio-0834466de2920445a9f9d185bf89690f4e4a72aae89b994d235fb6d742fc0277 WatchSource:0}: Error finding container 0834466de2920445a9f9d185bf89690f4e4a72aae89b994d235fb6d742fc0277: Status 404 returned error can't find the container with id 0834466de2920445a9f9d185bf89690f4e4a72aae89b994d235fb6d742fc0277
Nov 02 13:38:01.330846 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:01.330826    4559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 13:38:01.363106 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:01.363080    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db3bac31-5a93-4c18-b8d9-08f28fdd8688-plugins-config-volume\") pod \"payload-processing-e2e-ait-60a7443a-59485bdc55-8jjx5\" (UID: \"db3bac31-5a93-4c18-b8d9-08f28fdd8688\") " pod="openshift-ingress/payload-processing-e2e-ait-60a7443a-59485bdc55-8jjx5"
Nov 02 13:38:01.363227 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:01.363114    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k22xt\" (UniqueName: \"kubernetes.io/projected/db3bac31-5a93-4c18-b8d9-08f28fdd8688-kube-api-access-k22xt\") pod \"payload-processing-e2e-ait-60a7443a-59485bdc55-8jjx5\" (UID: \"db3bac31-5a93-4c18-b8d9-08f28fdd8688\") " pod="openshift-ingress/payload-processing-e2e-ait-60a7443a-59485bdc55-8jjx5"
Nov 02 13:38:01.363265 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:01.363226    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3014668-082f-41c1-abf3-bff163f64eba-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-60a7443a-5467958647-np56w\" (UID: \"e3014668-082f-41c1-abf3-bff163f64eba\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-60a7443a-5467958647-np56w"
Nov 02 13:38:01.363315 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:01.363265    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9mx6\" (UniqueName: \"kubernetes.io/projected/e3014668-082f-41c1-abf3-bff163f64eba-kube-api-access-b9mx6\") pod \"payload-pre-processing-e2e-ait-60a7443a-5467958647-np56w\" (UID: \"e3014668-082f-41c1-abf3-bff163f64eba\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-60a7443a-5467958647-np56w"
Nov 02 13:38:01.363979 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:01.363958    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3014668-082f-41c1-abf3-bff163f64eba-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-60a7443a-5467958647-np56w\" (UID: \"e3014668-082f-41c1-abf3-bff163f64eba\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-60a7443a-5467958647-np56w"
Nov 02 13:38:01.388031 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:01.388004    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9mx6\" (UniqueName: \"kubernetes.io/projected/e3014668-082f-41c1-abf3-bff163f64eba-kube-api-access-b9mx6\") pod \"payload-pre-processing-e2e-ait-60a7443a-5467958647-np56w\" (UID: \"e3014668-082f-41c1-abf3-bff163f64eba\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-60a7443a-5467958647-np56w"
Nov 02 13:38:01.463869 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:01.463827    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db3bac31-5a93-4c18-b8d9-08f28fdd8688-plugins-config-volume\") pod \"payload-processing-e2e-ait-60a7443a-59485bdc55-8jjx5\" (UID: \"db3bac31-5a93-4c18-b8d9-08f28fdd8688\") " pod="openshift-ingress/payload-processing-e2e-ait-60a7443a-59485bdc55-8jjx5"
Nov 02 13:38:01.463869 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:01.463866    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k22xt\" (UniqueName: \"kubernetes.io/projected/db3bac31-5a93-4c18-b8d9-08f28fdd8688-kube-api-access-k22xt\") pod \"payload-processing-e2e-ait-60a7443a-59485bdc55-8jjx5\" (UID: \"db3bac31-5a93-4c18-b8d9-08f28fdd8688\") " pod="openshift-ingress/payload-processing-e2e-ait-60a7443a-59485bdc55-8jjx5"
Nov 02 13:38:01.464538 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:01.464516    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db3bac31-5a93-4c18-b8d9-08f28fdd8688-plugins-config-volume\") pod \"payload-processing-e2e-ait-60a7443a-59485bdc55-8jjx5\" (UID: \"db3bac31-5a93-4c18-b8d9-08f28fdd8688\") " pod="openshift-ingress/payload-processing-e2e-ait-60a7443a-59485bdc55-8jjx5"
Nov 02 13:38:01.487803 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:01.487777    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k22xt\" (UniqueName: \"kubernetes.io/projected/db3bac31-5a93-4c18-b8d9-08f28fdd8688-kube-api-access-k22xt\") pod \"payload-processing-e2e-ait-60a7443a-59485bdc55-8jjx5\" (UID: \"db3bac31-5a93-4c18-b8d9-08f28fdd8688\") " pod="openshift-ingress/payload-processing-e2e-ait-60a7443a-59485bdc55-8jjx5"
Nov 02 13:38:01.501584 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:01.501562    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-60a7443a-5467958647-np56w"
Nov 02 13:38:01.583359 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:01.583279    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-60a7443a-59485bdc55-8jjx5"
Nov 02 13:38:01.631755 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:01.631717    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-60a7443a-788d448c95-8vpr9" event={"ID":"9059b137-efb5-4347-9201-653988b23682","Type":"ContainerStarted","Data":"0834466de2920445a9f9d185bf89690f4e4a72aae89b994d235fb6d742fc0277"}
Nov 02 13:38:01.799319 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:01.799291    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-60a7443a-5467958647-np56w"]
Nov 02 13:38:01.800164 ip-10-0-128-132 kubenswrapper[4559]: W1102 13:38:01.800132    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3014668_082f_41c1_abf3_bff163f64eba.slice/crio-81c0263445e957dc0b29d4d198f8637c04efcf19116c16ac33b7b399a359be89 WatchSource:0}: Error finding container 81c0263445e957dc0b29d4d198f8637c04efcf19116c16ac33b7b399a359be89: Status 404 returned error can't find the container with id 81c0263445e957dc0b29d4d198f8637c04efcf19116c16ac33b7b399a359be89
Nov 02 13:38:01.884108 ip-10-0-128-132 kubenswrapper[4559]: W1102 13:38:01.884077    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb3bac31_5a93_4c18_b8d9_08f28fdd8688.slice/crio-c7b6a8c40ed95290b6319683779a3b12a49178cf778844c84f40f481a808a437 WatchSource:0}: Error finding container c7b6a8c40ed95290b6319683779a3b12a49178cf778844c84f40f481a808a437: Status 404 returned error can't find the container with id c7b6a8c40ed95290b6319683779a3b12a49178cf778844c84f40f481a808a437
Nov 02 13:38:01.890936 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:01.890912    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-60a7443a-59485bdc55-8jjx5"]
Nov 02 13:38:02.636864 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:02.636814    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-60a7443a-59485bdc55-8jjx5" event={"ID":"db3bac31-5a93-4c18-b8d9-08f28fdd8688","Type":"ContainerStarted","Data":"e12b2ccf4eb46bd34b074940241ba35d6aa2332dd204894e9ef5f2a14aaf9982"}
Nov 02 13:38:02.637061 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:02.636882    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-60a7443a-59485bdc55-8jjx5"
Nov 02 13:38:02.637061 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:02.636908    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-60a7443a-59485bdc55-8jjx5" event={"ID":"db3bac31-5a93-4c18-b8d9-08f28fdd8688","Type":"ContainerStarted","Data":"c7b6a8c40ed95290b6319683779a3b12a49178cf778844c84f40f481a808a437"}
Nov 02 13:38:02.638397 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:02.638370    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-60a7443a-5467958647-np56w" event={"ID":"e3014668-082f-41c1-abf3-bff163f64eba","Type":"ContainerStarted","Data":"37bcdfe4533e9be0d7372d60f5da6d23c21f0529e0dd0b3fa3c41f71f6ff6976"}
Nov 02 13:38:02.638519 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:02.638413    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-60a7443a-5467958647-np56w" event={"ID":"e3014668-082f-41c1-abf3-bff163f64eba","Type":"ContainerStarted","Data":"81c0263445e957dc0b29d4d198f8637c04efcf19116c16ac33b7b399a359be89"}
Nov 02 13:38:02.638519 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:02.638456    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-60a7443a-5467958647-np56w"
Nov 02 13:38:02.639975 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:02.639954    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-60a7443a-788d448c95-8vpr9" event={"ID":"9059b137-efb5-4347-9201-653988b23682","Type":"ContainerStarted","Data":"da0e6e54499f32265796ef439e92cfc1b00d8454233b341061a2c6a9c0c3ba13"}
Nov 02 13:38:02.640104 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:02.640091    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-60a7443a-788d448c95-8vpr9"
Nov 02 13:38:02.679009 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:02.678958    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-60a7443a-59485bdc55-8jjx5" podStartSLOduration=1.678944569 podStartE2EDuration="1.678944569s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:38:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:38:02.678031542 +0000 UTC m=+4278.933430651" watchObservedRunningTime="2026-11-02 13:38:02.678944569 +0000 UTC m=+4278.934343663"
Nov 02 13:38:02.722097 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:02.722054    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-60a7443a-788d448c95-8vpr9" podStartSLOduration=2.193124551 podStartE2EDuration="2.722041322s" totalImagesPullingTime="528.916771ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:38:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:38:02.721344889 +0000 UTC m=+4278.976743985" watchObservedRunningTime="2026-11-02 13:38:02.722041322 +0000 UTC m=+4278.977440417"
Nov 02 13:38:02.764074 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:02.764025    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-60a7443a-5467958647-np56w" podStartSLOduration=1.7640126600000001 podStartE2EDuration="1.76401266s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:38:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:38:02.762662863 +0000 UTC m=+4279.018061955" watchObservedRunningTime="2026-11-02 13:38:02.76401266 +0000 UTC m=+4279.019411733"
Nov 02 13:38:04.123283 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:04.123244    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"]
Nov 02 13:38:08.649295 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:08.649266    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-60a7443a-788d448c95-8vpr9"
Nov 02 13:38:13.647459 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:13.647423    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-60a7443a-5467958647-np56w"
Nov 02 13:38:13.647937 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:13.647501    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-60a7443a-59485bdc55-8jjx5"
Nov 02 13:38:14.622275 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:14.622238    4559 patch_prober.go:29] interesting pod/e2e-ait-60a7443a-openshift-default-5f47b4f685-4flqv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.8:15021/healthz/ready\": dial tcp 10.133.1.8:15021: connect: connection refused" start-of-body=
Nov 02 13:38:14.622505 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:14.622289    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-60a7443a-openshift-default-5f47b4f685-4flqv" podUID="93596dfc-4540-4cd5-9c06-d8a790d8c58e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.8:15021/healthz/ready\": dial tcp 10.133.1.8:15021: connect: connection refused"
Nov 02 13:38:14.694627 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:14.694593    4559 generic.go:356] "Generic (PLEG): container finished" podID="93596dfc-4540-4cd5-9c06-d8a790d8c58e" containerID="4f316cbdc38bc2039a6deeac2106aa7e1d6131a4fee60463c5027108f03ee23b" exitCode=0
Nov 02 13:38:14.694995 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:14.694668    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-60a7443a-openshift-default-5f47b4f685-4flqv" event={"ID":"93596dfc-4540-4cd5-9c06-d8a790d8c58e","Type":"ContainerDied","Data":"4f316cbdc38bc2039a6deeac2106aa7e1d6131a4fee60463c5027108f03ee23b"}
Nov 02 13:38:14.695040 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:14.695016    4559 scope.go:122] "RemoveContainer" containerID="4f316cbdc38bc2039a6deeac2106aa7e1d6131a4fee60463c5027108f03ee23b"
Nov 02 13:38:14.696317 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:14.696281    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 13:38:14.696395 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:14.696375    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 13:38:14.696438 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:14.696419    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 13:38:15.117195 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:38:15.117156    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqcmj" containerName="extractor"
Nov 02 13:38:15.715061 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:15.715026    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-60a7443a-openshift-default-5f47b4f685-4flqv" event={"ID":"93596dfc-4540-4cd5-9c06-d8a790d8c58e","Type":"ContainerStarted","Data":"fa414e786f0550f713c4336dc0b575b2ed23f6646ed622e7e3398e19cbf4cec7"}
Nov 02 13:38:15.912832 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:15.912792    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-60a7443a-openshift-default-5f47b4f685-4flqv"
Nov 02 13:38:15.913038 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:15.913014    4559 patch_prober.go:29] interesting pod/e2e-ait-60a7443a-openshift-default-5f47b4f685-4flqv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.8:15021/healthz/ready\": dial tcp 10.133.1.8:15021: connect: connection refused" start-of-body=
Nov 02 13:38:15.913087 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:15.913068    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-60a7443a-openshift-default-5f47b4f685-4flqv" podUID="93596dfc-4540-4cd5-9c06-d8a790d8c58e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.8:15021/healthz/ready\": dial tcp 10.133.1.8:15021: connect: connection refused"
Nov 02 13:38:16.913303 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:16.913265    4559 patch_prober.go:29] interesting pod/e2e-ait-60a7443a-openshift-default-5f47b4f685-4flqv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.8:15021/healthz/ready\": dial tcp 10.133.1.8:15021: connect: connection refused" start-of-body=
Nov 02 13:38:16.913861 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:16.913337    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-60a7443a-openshift-default-5f47b4f685-4flqv" podUID="93596dfc-4540-4cd5-9c06-d8a790d8c58e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.8:15021/healthz/ready\": dial tcp 10.133.1.8:15021: connect: connection refused"
Nov 02 13:38:17.912470 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:17.912436    4559 patch_prober.go:29] interesting pod/e2e-ait-60a7443a-openshift-default-5f47b4f685-4flqv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.8:15021/healthz/ready\": dial tcp 10.133.1.8:15021: connect: connection refused" start-of-body=
Nov 02 13:38:17.912669 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:17.912487    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-60a7443a-openshift-default-5f47b4f685-4flqv" podUID="93596dfc-4540-4cd5-9c06-d8a790d8c58e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.8:15021/healthz/ready\": dial tcp 10.133.1.8:15021: connect: connection refused"
Nov 02 13:38:18.913174 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:18.913129    4559 patch_prober.go:29] interesting pod/e2e-ait-60a7443a-openshift-default-5f47b4f685-4flqv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.8:15021/healthz/ready\": dial tcp 10.133.1.8:15021: connect: connection refused" start-of-body=
Nov 02 13:38:18.913601 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:18.913191    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-60a7443a-openshift-default-5f47b4f685-4flqv" podUID="93596dfc-4540-4cd5-9c06-d8a790d8c58e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.8:15021/healthz/ready\": dial tcp 10.133.1.8:15021: connect: connection refused"
Nov 02 13:38:19.913164 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:19.913124    4559 patch_prober.go:29] interesting pod/e2e-ait-60a7443a-openshift-default-5f47b4f685-4flqv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.8:15021/healthz/ready\": dial tcp 10.133.1.8:15021: connect: connection refused" start-of-body=
Nov 02 13:38:19.913345 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:19.913175    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-60a7443a-openshift-default-5f47b4f685-4flqv" podUID="93596dfc-4540-4cd5-9c06-d8a790d8c58e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.8:15021/healthz/ready\": dial tcp 10.133.1.8:15021: connect: connection refused"
Nov 02 13:38:20.916358 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:20.916329    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-60a7443a-openshift-default-5f47b4f685-4flqv"
Nov 02 13:38:20.916780 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:20.916732    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-60a7443a-openshift-default-5f47b4f685-4flqv"
Nov 02 13:38:20.917290 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:20.917271    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-60a7443a-openshift-default-5f47b4f685-4flqv"
Nov 02 13:38:25.699660 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:25.699610    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-60a7443a-59485bdc55-8jjx5"]
Nov 02 13:38:25.733619 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:25.733582    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-60a7443a-5467958647-np56w"]
Nov 02 13:38:26.264939 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:26.264900    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-60a7443a-788d448c95-8vpr9"]
Nov 02 13:38:26.265175 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:26.265144    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-60a7443a-788d448c95-8vpr9" podUID="9059b137-efb5-4347-9201-653988b23682" containerName="maas-api" containerID="cri-o://da0e6e54499f32265796ef439e92cfc1b00d8454233b341061a2c6a9c0c3ba13" gracePeriod=30
Nov 02 13:38:26.507776 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:26.507748    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-60a7443a-788d448c95-8vpr9"
Nov 02 13:38:26.606020 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:26.605929    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9059b137-efb5-4347-9201-653988b23682-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9059b137-efb5-4347-9201-653988b23682-openshift-service-ca\") pod \"9059b137-efb5-4347-9201-653988b23682\" (UID: \"9059b137-efb5-4347-9201-653988b23682\") "
Nov 02 13:38:26.606237 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:26.606037    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9059b137-efb5-4347-9201-653988b23682-kube-api-access-lth67\" (UniqueName: \"kubernetes.io/projected/9059b137-efb5-4347-9201-653988b23682-kube-api-access-lth67\") pod \"9059b137-efb5-4347-9201-653988b23682\" (UID: \"9059b137-efb5-4347-9201-653988b23682\") "
Nov 02 13:38:26.606237 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:26.606110    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9059b137-efb5-4347-9201-653988b23682-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9059b137-efb5-4347-9201-653988b23682-maas-api-tls\") pod \"9059b137-efb5-4347-9201-653988b23682\" (UID: \"9059b137-efb5-4347-9201-653988b23682\") "
Nov 02 13:38:26.606237 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:26.606159    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9059b137-efb5-4347-9201-653988b23682-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9059b137-efb5-4347-9201-653988b23682-metrics-certs\") pod \"9059b137-efb5-4347-9201-653988b23682\" (UID: \"9059b137-efb5-4347-9201-653988b23682\") "
Nov 02 13:38:26.606447 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:26.606339    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9059b137-efb5-4347-9201-653988b23682-openshift-service-ca" pod "9059b137-efb5-4347-9201-653988b23682" (UID: "9059b137-efb5-4347-9201-653988b23682"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:38:26.606447 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:26.606417    4559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9059b137-efb5-4347-9201-653988b23682-openshift-service-ca\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:38:26.608200 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:26.608165    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9059b137-efb5-4347-9201-653988b23682-maas-api-tls" pod "9059b137-efb5-4347-9201-653988b23682" (UID: "9059b137-efb5-4347-9201-653988b23682"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:38:26.608343 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:26.608267    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9059b137-efb5-4347-9201-653988b23682-metrics-certs" pod "9059b137-efb5-4347-9201-653988b23682" (UID: "9059b137-efb5-4347-9201-653988b23682"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:38:26.608343 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:26.608278    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9059b137-efb5-4347-9201-653988b23682-kube-api-access-lth67" pod "9059b137-efb5-4347-9201-653988b23682" (UID: "9059b137-efb5-4347-9201-653988b23682"). InnerVolumeSpecName "kube-api-access-lth67". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:38:26.707284 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:26.707244    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9059b137-efb5-4347-9201-653988b23682-maas-api-tls\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:38:26.707284 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:26.707274    4559 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9059b137-efb5-4347-9201-653988b23682-metrics-certs\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:38:26.707284 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:26.707284    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lth67\" (UniqueName: \"kubernetes.io/projected/9059b137-efb5-4347-9201-653988b23682-kube-api-access-lth67\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:38:26.763995 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:26.763958    4559 generic.go:356] "Generic (PLEG): container finished" podID="9059b137-efb5-4347-9201-653988b23682" containerID="da0e6e54499f32265796ef439e92cfc1b00d8454233b341061a2c6a9c0c3ba13" exitCode=0
Nov 02 13:38:26.764173 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:26.764007    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-60a7443a-788d448c95-8vpr9"
Nov 02 13:38:26.764173 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:26.764045    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-60a7443a-788d448c95-8vpr9" event={"ID":"9059b137-efb5-4347-9201-653988b23682","Type":"ContainerDied","Data":"da0e6e54499f32265796ef439e92cfc1b00d8454233b341061a2c6a9c0c3ba13"}
Nov 02 13:38:26.764173 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:26.764125    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-60a7443a-788d448c95-8vpr9" event={"ID":"9059b137-efb5-4347-9201-653988b23682","Type":"ContainerDied","Data":"0834466de2920445a9f9d185bf89690f4e4a72aae89b994d235fb6d742fc0277"}
Nov 02 13:38:26.764173 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:26.764146    4559 scope.go:122] "RemoveContainer" containerID="da0e6e54499f32265796ef439e92cfc1b00d8454233b341061a2c6a9c0c3ba13"
Nov 02 13:38:26.773611 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:26.773583    4559 scope.go:122] "RemoveContainer" containerID="da0e6e54499f32265796ef439e92cfc1b00d8454233b341061a2c6a9c0c3ba13"
Nov 02 13:38:26.773891 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:38:26.773870    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da0e6e54499f32265796ef439e92cfc1b00d8454233b341061a2c6a9c0c3ba13\": container with ID starting with da0e6e54499f32265796ef439e92cfc1b00d8454233b341061a2c6a9c0c3ba13 not found: ID does not exist" containerID="da0e6e54499f32265796ef439e92cfc1b00d8454233b341061a2c6a9c0c3ba13"
Nov 02 13:38:26.773976 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:26.773901    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da0e6e54499f32265796ef439e92cfc1b00d8454233b341061a2c6a9c0c3ba13"} err="failed to get container status \"da0e6e54499f32265796ef439e92cfc1b00d8454233b341061a2c6a9c0c3ba13\": rpc error: code = NotFound desc = could not find container \"da0e6e54499f32265796ef439e92cfc1b00d8454233b341061a2c6a9c0c3ba13\": container with ID starting with da0e6e54499f32265796ef439e92cfc1b00d8454233b341061a2c6a9c0c3ba13 not found: ID does not exist"
Nov 02 13:38:26.817089 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:26.817058    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-60a7443a-788d448c95-8vpr9"]
Nov 02 13:38:26.829788 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:26.829765    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-60a7443a-788d448c95-8vpr9"]
Nov 02 13:38:27.890497 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:27.890462    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9059b137-efb5-4347-9201-653988b23682" path="/var/lib/kubelet/pods/9059b137-efb5-4347-9201-653988b23682/volumes"
Nov 02 13:38:28.421447 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:28.421407    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"]
Nov 02 13:38:30.700579 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:30.700537    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-60a7443a-59485bdc55-8jjx5" podUID="db3bac31-5a93-4c18-b8d9-08f28fdd8688" containerName="payload-processing" containerID="cri-o://e12b2ccf4eb46bd34b074940241ba35d6aa2332dd204894e9ef5f2a14aaf9982" gracePeriod=25
Nov 02 13:38:30.734169 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:30.734133    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-60a7443a-5467958647-np56w" podUID="e3014668-082f-41c1-abf3-bff163f64eba" containerName="payload-pre-processing" containerID="cri-o://37bcdfe4533e9be0d7372d60f5da6d23c21f0529e0dd0b3fa3c41f71f6ff6976" gracePeriod=25
Nov 02 13:38:31.000066 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:31.000038    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-60a7443a-59485bdc55-8jjx5"
Nov 02 13:38:31.013058 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:31.013030    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-60a7443a-5467958647-np56w"
Nov 02 13:38:31.043625 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:31.043599    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db3bac31-5a93-4c18-b8d9-08f28fdd8688-kube-api-access-k22xt\" (UniqueName: \"kubernetes.io/projected/db3bac31-5a93-4c18-b8d9-08f28fdd8688-kube-api-access-k22xt\") pod \"db3bac31-5a93-4c18-b8d9-08f28fdd8688\" (UID: \"db3bac31-5a93-4c18-b8d9-08f28fdd8688\") "
Nov 02 13:38:31.043751 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:31.043657    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db3bac31-5a93-4c18-b8d9-08f28fdd8688-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db3bac31-5a93-4c18-b8d9-08f28fdd8688-plugins-config-volume\") pod \"db3bac31-5a93-4c18-b8d9-08f28fdd8688\" (UID: \"db3bac31-5a93-4c18-b8d9-08f28fdd8688\") "
Nov 02 13:38:31.044044 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:31.044022    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db3bac31-5a93-4c18-b8d9-08f28fdd8688-plugins-config-volume" pod "db3bac31-5a93-4c18-b8d9-08f28fdd8688" (UID: "db3bac31-5a93-4c18-b8d9-08f28fdd8688"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:38:31.045711 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:31.045689    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db3bac31-5a93-4c18-b8d9-08f28fdd8688-kube-api-access-k22xt" pod "db3bac31-5a93-4c18-b8d9-08f28fdd8688" (UID: "db3bac31-5a93-4c18-b8d9-08f28fdd8688"). InnerVolumeSpecName "kube-api-access-k22xt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:38:31.144286 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:31.144251    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3014668-082f-41c1-abf3-bff163f64eba-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3014668-082f-41c1-abf3-bff163f64eba-plugins-config-volume\") pod \"e3014668-082f-41c1-abf3-bff163f64eba\" (UID: \"e3014668-082f-41c1-abf3-bff163f64eba\") "
Nov 02 13:38:31.144286 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:31.144295    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3014668-082f-41c1-abf3-bff163f64eba-kube-api-access-b9mx6\" (UniqueName: \"kubernetes.io/projected/e3014668-082f-41c1-abf3-bff163f64eba-kube-api-access-b9mx6\") pod \"e3014668-082f-41c1-abf3-bff163f64eba\" (UID: \"e3014668-082f-41c1-abf3-bff163f64eba\") "
Nov 02 13:38:31.144526 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:31.144478    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k22xt\" (UniqueName: \"kubernetes.io/projected/db3bac31-5a93-4c18-b8d9-08f28fdd8688-kube-api-access-k22xt\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:38:31.144526 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:31.144488    4559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db3bac31-5a93-4c18-b8d9-08f28fdd8688-plugins-config-volume\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:38:31.144701 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:31.144676    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3014668-082f-41c1-abf3-bff163f64eba-plugins-config-volume" pod "e3014668-082f-41c1-abf3-bff163f64eba" (UID: "e3014668-082f-41c1-abf3-bff163f64eba"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:38:31.146477 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:31.146456    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3014668-082f-41c1-abf3-bff163f64eba-kube-api-access-b9mx6" pod "e3014668-082f-41c1-abf3-bff163f64eba" (UID: "e3014668-082f-41c1-abf3-bff163f64eba"). InnerVolumeSpecName "kube-api-access-b9mx6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:38:31.245273 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:31.245180    4559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3014668-082f-41c1-abf3-bff163f64eba-plugins-config-volume\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:38:31.245273 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:31.245215    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b9mx6\" (UniqueName: \"kubernetes.io/projected/e3014668-082f-41c1-abf3-bff163f64eba-kube-api-access-b9mx6\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:38:31.578550 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:31.578457    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-60a7443a-openshift-default-5f47b4f685-4flqv"]
Nov 02 13:38:31.578759 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:31.578730    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-60a7443a-openshift-default-5f47b4f685-4flqv" podUID="93596dfc-4540-4cd5-9c06-d8a790d8c58e" containerName="istio-proxy" containerID="cri-o://fa414e786f0550f713c4336dc0b575b2ed23f6646ed622e7e3398e19cbf4cec7" gracePeriod=30
Nov 02 13:38:31.788561 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:31.788517    4559 generic.go:356] "Generic (PLEG): container finished" podID="e3014668-082f-41c1-abf3-bff163f64eba" containerID="37bcdfe4533e9be0d7372d60f5da6d23c21f0529e0dd0b3fa3c41f71f6ff6976" exitCode=0
Nov 02 13:38:31.789037 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:31.788586    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-60a7443a-5467958647-np56w"
Nov 02 13:38:31.789037 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:31.788599    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-60a7443a-5467958647-np56w" event={"ID":"e3014668-082f-41c1-abf3-bff163f64eba","Type":"ContainerDied","Data":"37bcdfe4533e9be0d7372d60f5da6d23c21f0529e0dd0b3fa3c41f71f6ff6976"}
Nov 02 13:38:31.789037 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:31.788676    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-60a7443a-5467958647-np56w" event={"ID":"e3014668-082f-41c1-abf3-bff163f64eba","Type":"ContainerDied","Data":"81c0263445e957dc0b29d4d198f8637c04efcf19116c16ac33b7b399a359be89"}
Nov 02 13:38:31.789037 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:31.788699    4559 scope.go:122] "RemoveContainer" containerID="37bcdfe4533e9be0d7372d60f5da6d23c21f0529e0dd0b3fa3c41f71f6ff6976"
Nov 02 13:38:31.790027 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:31.789966    4559 generic.go:356] "Generic (PLEG): container finished" podID="db3bac31-5a93-4c18-b8d9-08f28fdd8688" containerID="e12b2ccf4eb46bd34b074940241ba35d6aa2332dd204894e9ef5f2a14aaf9982" exitCode=0
Nov 02 13:38:31.790027 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:31.790003    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-60a7443a-59485bdc55-8jjx5" event={"ID":"db3bac31-5a93-4c18-b8d9-08f28fdd8688","Type":"ContainerDied","Data":"e12b2ccf4eb46bd34b074940241ba35d6aa2332dd204894e9ef5f2a14aaf9982"}
Nov 02 13:38:31.790147 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:31.790034    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-60a7443a-59485bdc55-8jjx5"
Nov 02 13:38:31.790147 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:31.790037    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-60a7443a-59485bdc55-8jjx5" event={"ID":"db3bac31-5a93-4c18-b8d9-08f28fdd8688","Type":"ContainerDied","Data":"c7b6a8c40ed95290b6319683779a3b12a49178cf778844c84f40f481a808a437"}
Nov 02 13:38:31.798596 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:31.798579    4559 scope.go:122] "RemoveContainer" containerID="37bcdfe4533e9be0d7372d60f5da6d23c21f0529e0dd0b3fa3c41f71f6ff6976"
Nov 02 13:38:31.798989 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:38:31.798960    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37bcdfe4533e9be0d7372d60f5da6d23c21f0529e0dd0b3fa3c41f71f6ff6976\": container with ID starting with 37bcdfe4533e9be0d7372d60f5da6d23c21f0529e0dd0b3fa3c41f71f6ff6976 not found: ID does not exist" containerID="37bcdfe4533e9be0d7372d60f5da6d23c21f0529e0dd0b3fa3c41f71f6ff6976"
Nov 02 13:38:31.799094 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:31.798991    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37bcdfe4533e9be0d7372d60f5da6d23c21f0529e0dd0b3fa3c41f71f6ff6976"} err="failed to get container status \"37bcdfe4533e9be0d7372d60f5da6d23c21f0529e0dd0b3fa3c41f71f6ff6976\": rpc error: code = NotFound desc = could not find container \"37bcdfe4533e9be0d7372d60f5da6d23c21f0529e0dd0b3fa3c41f71f6ff6976\": container with ID starting with 37bcdfe4533e9be0d7372d60f5da6d23c21f0529e0dd0b3fa3c41f71f6ff6976 not found: ID does not exist"
Nov 02 13:38:31.799094 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:31.799016    4559 scope.go:122] "RemoveContainer" containerID="e12b2ccf4eb46bd34b074940241ba35d6aa2332dd204894e9ef5f2a14aaf9982"
Nov 02 13:38:31.806581 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:31.806562    4559 scope.go:122] "RemoveContainer" containerID="e12b2ccf4eb46bd34b074940241ba35d6aa2332dd204894e9ef5f2a14aaf9982"
Nov 02 13:38:31.806899 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:38:31.806879    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e12b2ccf4eb46bd34b074940241ba35d6aa2332dd204894e9ef5f2a14aaf9982\": container with ID starting with e12b2ccf4eb46bd34b074940241ba35d6aa2332dd204894e9ef5f2a14aaf9982 not found: ID does not exist" containerID="e12b2ccf4eb46bd34b074940241ba35d6aa2332dd204894e9ef5f2a14aaf9982"
Nov 02 13:38:31.806959 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:31.806906    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e12b2ccf4eb46bd34b074940241ba35d6aa2332dd204894e9ef5f2a14aaf9982"} err="failed to get container status \"e12b2ccf4eb46bd34b074940241ba35d6aa2332dd204894e9ef5f2a14aaf9982\": rpc error: code = NotFound desc = could not find container \"e12b2ccf4eb46bd34b074940241ba35d6aa2332dd204894e9ef5f2a14aaf9982\": container with ID starting with e12b2ccf4eb46bd34b074940241ba35d6aa2332dd204894e9ef5f2a14aaf9982 not found: ID does not exist"
Nov 02 13:38:31.841120 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:31.841054    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-60a7443a-5467958647-np56w"]
Nov 02 13:38:31.843735 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:31.843710    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-60a7443a-5467958647-np56w"]
Nov 02 13:38:31.877464 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:31.877439    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-60a7443a-59485bdc55-8jjx5"]
Nov 02 13:38:31.882408 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:31.882384    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-60a7443a-59485bdc55-8jjx5"]
Nov 02 13:38:31.890594 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:31.890560    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db3bac31-5a93-4c18-b8d9-08f28fdd8688" path="/var/lib/kubelet/pods/db3bac31-5a93-4c18-b8d9-08f28fdd8688/volumes"
Nov 02 13:38:31.891019 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:31.890999    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3014668-082f-41c1-abf3-bff163f64eba" path="/var/lib/kubelet/pods/e3014668-082f-41c1-abf3-bff163f64eba/volumes"
Nov 02 13:38:34.903046 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:34.903014    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"]
Nov 02 13:38:35.918451 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:35.918418    4559 patch_prober.go:29] interesting pod/e2e-ait-60a7443a-openshift-default-5f47b4f685-4flqv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused
Nov 02 13:38:35.918873 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:35.918481    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-60a7443a-openshift-default-5f47b4f685-4flqv" podUID="93596dfc-4540-4cd5-9c06-d8a790d8c58e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 13:38:36.814906 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:36.814880    4559 generic.go:356] "Generic (PLEG): container finished" podID="93596dfc-4540-4cd5-9c06-d8a790d8c58e" containerID="fa414e786f0550f713c4336dc0b575b2ed23f6646ed622e7e3398e19cbf4cec7" exitCode=0
Nov 02 13:38:36.815012 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:36.814939    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-60a7443a-openshift-default-5f47b4f685-4flqv" event={"ID":"93596dfc-4540-4cd5-9c06-d8a790d8c58e","Type":"ContainerDied","Data":"fa414e786f0550f713c4336dc0b575b2ed23f6646ed622e7e3398e19cbf4cec7"}
Nov 02 13:38:36.815012 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:36.814998    4559 scope.go:122] "RemoveContainer" containerID="4f316cbdc38bc2039a6deeac2106aa7e1d6131a4fee60463c5027108f03ee23b"
Nov 02 13:38:36.831783 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:38:36.831758    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f316cbdc38bc2039a6deeac2106aa7e1d6131a4fee60463c5027108f03ee23b\": container with ID starting with 4f316cbdc38bc2039a6deeac2106aa7e1d6131a4fee60463c5027108f03ee23b not found: ID does not exist" containerID="4f316cbdc38bc2039a6deeac2106aa7e1d6131a4fee60463c5027108f03ee23b"
Nov 02 13:38:36.835367 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:36.835349    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-60a7443a-openshift-default-5f47b4f685-4flqv"
Nov 02 13:38:36.895269 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:36.895207    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93596dfc-4540-4cd5-9c06-d8a790d8c58e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93596dfc-4540-4cd5-9c06-d8a790d8c58e-workload-socket\") pod \"93596dfc-4540-4cd5-9c06-d8a790d8c58e\" (UID: \"93596dfc-4540-4cd5-9c06-d8a790d8c58e\") "
Nov 02 13:38:36.895269 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:36.895242    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93596dfc-4540-4cd5-9c06-d8a790d8c58e-kube-api-access-84sfz\" (UniqueName: \"kubernetes.io/projected/93596dfc-4540-4cd5-9c06-d8a790d8c58e-kube-api-access-84sfz\") pod \"93596dfc-4540-4cd5-9c06-d8a790d8c58e\" (UID: \"93596dfc-4540-4cd5-9c06-d8a790d8c58e\") "
Nov 02 13:38:36.895460 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:36.895270    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93596dfc-4540-4cd5-9c06-d8a790d8c58e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93596dfc-4540-4cd5-9c06-d8a790d8c58e-istiod-ca-cert\") pod \"93596dfc-4540-4cd5-9c06-d8a790d8c58e\" (UID: \"93596dfc-4540-4cd5-9c06-d8a790d8c58e\") "
Nov 02 13:38:36.895460 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:36.895313    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/93596dfc-4540-4cd5-9c06-d8a790d8c58e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93596dfc-4540-4cd5-9c06-d8a790d8c58e-istio-podinfo\") pod \"93596dfc-4540-4cd5-9c06-d8a790d8c58e\" (UID: \"93596dfc-4540-4cd5-9c06-d8a790d8c58e\") "
Nov 02 13:38:36.895460 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:36.895334    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93596dfc-4540-4cd5-9c06-d8a790d8c58e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93596dfc-4540-4cd5-9c06-d8a790d8c58e-workload-certs\") pod \"93596dfc-4540-4cd5-9c06-d8a790d8c58e\" (UID: \"93596dfc-4540-4cd5-9c06-d8a790d8c58e\") "
Nov 02 13:38:36.895460 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:36.895374    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93596dfc-4540-4cd5-9c06-d8a790d8c58e-istio-token\" (UniqueName: \"kubernetes.io/projected/93596dfc-4540-4cd5-9c06-d8a790d8c58e-istio-token\") pod \"93596dfc-4540-4cd5-9c06-d8a790d8c58e\" (UID: \"93596dfc-4540-4cd5-9c06-d8a790d8c58e\") "
Nov 02 13:38:36.895460 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:36.895392    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93596dfc-4540-4cd5-9c06-d8a790d8c58e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93596dfc-4540-4cd5-9c06-d8a790d8c58e-istio-envoy\") pod \"93596dfc-4540-4cd5-9c06-d8a790d8c58e\" (UID: \"93596dfc-4540-4cd5-9c06-d8a790d8c58e\") "
Nov 02 13:38:36.895460 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:36.895439    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93596dfc-4540-4cd5-9c06-d8a790d8c58e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93596dfc-4540-4cd5-9c06-d8a790d8c58e-istio-data\") pod \"93596dfc-4540-4cd5-9c06-d8a790d8c58e\" (UID: \"93596dfc-4540-4cd5-9c06-d8a790d8c58e\") "
Nov 02 13:38:36.895840 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:36.895472    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93596dfc-4540-4cd5-9c06-d8a790d8c58e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93596dfc-4540-4cd5-9c06-d8a790d8c58e-credential-socket\") pod \"93596dfc-4540-4cd5-9c06-d8a790d8c58e\" (UID: \"93596dfc-4540-4cd5-9c06-d8a790d8c58e\") "
Nov 02 13:38:36.895840 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:36.895500    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93596dfc-4540-4cd5-9c06-d8a790d8c58e-workload-socket" pod "93596dfc-4540-4cd5-9c06-d8a790d8c58e" (UID: "93596dfc-4540-4cd5-9c06-d8a790d8c58e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:38:36.895840 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:36.895809    4559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93596dfc-4540-4cd5-9c06-d8a790d8c58e-workload-socket\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:38:36.896011 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:36.895839    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93596dfc-4540-4cd5-9c06-d8a790d8c58e-istio-data" pod "93596dfc-4540-4cd5-9c06-d8a790d8c58e" (UID: "93596dfc-4540-4cd5-9c06-d8a790d8c58e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:38:36.896011 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:36.895861    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93596dfc-4540-4cd5-9c06-d8a790d8c58e-istiod-ca-cert" pod "93596dfc-4540-4cd5-9c06-d8a790d8c58e" (UID: "93596dfc-4540-4cd5-9c06-d8a790d8c58e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:38:36.896011 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:36.895942    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93596dfc-4540-4cd5-9c06-d8a790d8c58e-workload-certs" pod "93596dfc-4540-4cd5-9c06-d8a790d8c58e" (UID: "93596dfc-4540-4cd5-9c06-d8a790d8c58e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:38:36.896011 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:36.895979    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93596dfc-4540-4cd5-9c06-d8a790d8c58e-credential-socket" pod "93596dfc-4540-4cd5-9c06-d8a790d8c58e" (UID: "93596dfc-4540-4cd5-9c06-d8a790d8c58e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:38:36.898054 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:36.898018    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93596dfc-4540-4cd5-9c06-d8a790d8c58e-kube-api-access-84sfz" pod "93596dfc-4540-4cd5-9c06-d8a790d8c58e" (UID: "93596dfc-4540-4cd5-9c06-d8a790d8c58e"). InnerVolumeSpecName "kube-api-access-84sfz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:38:36.898054 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:36.898046    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93596dfc-4540-4cd5-9c06-d8a790d8c58e-istio-envoy" pod "93596dfc-4540-4cd5-9c06-d8a790d8c58e" (UID: "93596dfc-4540-4cd5-9c06-d8a790d8c58e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:38:36.898229 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:36.898044    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/93596dfc-4540-4cd5-9c06-d8a790d8c58e-istio-podinfo" pod "93596dfc-4540-4cd5-9c06-d8a790d8c58e" (UID: "93596dfc-4540-4cd5-9c06-d8a790d8c58e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 13:38:36.898229 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:36.898224    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93596dfc-4540-4cd5-9c06-d8a790d8c58e-istio-token" pod "93596dfc-4540-4cd5-9c06-d8a790d8c58e" (UID: "93596dfc-4540-4cd5-9c06-d8a790d8c58e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:38:36.996501 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:36.996470    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-84sfz\" (UniqueName: \"kubernetes.io/projected/93596dfc-4540-4cd5-9c06-d8a790d8c58e-kube-api-access-84sfz\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:38:36.996501 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:36.996496    4559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93596dfc-4540-4cd5-9c06-d8a790d8c58e-istiod-ca-cert\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:38:36.996501 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:36.996506    4559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93596dfc-4540-4cd5-9c06-d8a790d8c58e-istio-podinfo\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:38:36.996946 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:36.996514    4559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93596dfc-4540-4cd5-9c06-d8a790d8c58e-workload-certs\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:38:36.996946 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:36.996523    4559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93596dfc-4540-4cd5-9c06-d8a790d8c58e-istio-token\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:38:36.996946 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:36.996531    4559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93596dfc-4540-4cd5-9c06-d8a790d8c58e-istio-envoy\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:38:36.996946 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:36.996537    4559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93596dfc-4540-4cd5-9c06-d8a790d8c58e-istio-data\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:38:36.996946 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:36.996545    4559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93596dfc-4540-4cd5-9c06-d8a790d8c58e-credential-socket\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:38:37.820204 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:37.820175    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-60a7443a-openshift-default-5f47b4f685-4flqv"
Nov 02 13:38:37.820393 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:37.820176    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-60a7443a-openshift-default-5f47b4f685-4flqv" event={"ID":"93596dfc-4540-4cd5-9c06-d8a790d8c58e","Type":"ContainerDied","Data":"c682234c336d13fdac5bab2b535d33322fef8db3dff63257e4f235c3351cf6e5"}
Nov 02 13:38:37.820393 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:37.820309    4559 scope.go:122] "RemoveContainer" containerID="fa414e786f0550f713c4336dc0b575b2ed23f6646ed622e7e3398e19cbf4cec7"
Nov 02 13:38:37.885432 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:37.885406    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-60a7443a-openshift-default-5f47b4f685-4flqv"]
Nov 02 13:38:37.898169 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:37.898145    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-60a7443a-openshift-default-5f47b4f685-4flqv"]
Nov 02 13:38:38.015333 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:38.015305    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5597f6b9f6-zx498"]
Nov 02 13:38:38.015679 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:38.015608    4559 cpu_manager.go:414] "containerMap: removing container" podUID="db3bac31-5a93-4c18-b8d9-08f28fdd8688" containerName="payload-processing"
Nov 02 13:38:38.015679 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:38.015619    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db3bac31-5a93-4c18-b8d9-08f28fdd8688" containerName="payload-processing"
Nov 02 13:38:38.015679 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:38.015629    4559 cpu_manager.go:414] "containerMap: removing container" podUID="93596dfc-4540-4cd5-9c06-d8a790d8c58e" containerName="istio-proxy"
Nov 02 13:38:38.015679 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:38.015634    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93596dfc-4540-4cd5-9c06-d8a790d8c58e" containerName="istio-proxy"
Nov 02 13:38:38.015679 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:38.015658    4559 cpu_manager.go:414] "containerMap: removing container" podUID="e3014668-082f-41c1-abf3-bff163f64eba" containerName="payload-pre-processing"
Nov 02 13:38:38.015679 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:38.015663    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3014668-082f-41c1-abf3-bff163f64eba" containerName="payload-pre-processing"
Nov 02 13:38:38.015679 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:38.015677    4559 cpu_manager.go:414] "containerMap: removing container" podUID="9059b137-efb5-4347-9201-653988b23682" containerName="maas-api"
Nov 02 13:38:38.015679 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:38.015682    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9059b137-efb5-4347-9201-653988b23682" containerName="maas-api"
Nov 02 13:38:38.016003 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:38.015738    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="db3bac31-5a93-4c18-b8d9-08f28fdd8688" containerName="payload-processing"
Nov 02 13:38:38.016003 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:38.015748    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="93596dfc-4540-4cd5-9c06-d8a790d8c58e" containerName="istio-proxy"
Nov 02 13:38:38.016003 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:38.015755    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3014668-082f-41c1-abf3-bff163f64eba" containerName="payload-pre-processing"
Nov 02 13:38:38.016003 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:38.015763    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="9059b137-efb5-4347-9201-653988b23682" containerName="maas-api"
Nov 02 13:38:38.019925 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:38.019908    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5597f6b9f6-zx498"
Nov 02 13:38:38.041377 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:38.041352    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-crg4p\""
Nov 02 13:38:38.041486 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:38.041428    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Nov 02 13:38:38.041717 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:38.041704    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Nov 02 13:38:38.041766 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:38.041724    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Nov 02 13:38:38.047294 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:38.047272    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5597f6b9f6-zx498"]
Nov 02 13:38:38.104977 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:38.104906    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f2df2e9-089e-48f0-a547-841ca4dc9781-metrics-certs\") pod \"maas-controller-5597f6b9f6-zx498\" (UID: \"1f2df2e9-089e-48f0-a547-841ca4dc9781\") " pod="opendatahub/maas-controller-5597f6b9f6-zx498"
Nov 02 13:38:38.104977 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:38.104943    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1f2df2e9-089e-48f0-a547-841ca4dc9781-webhook-cert\") pod \"maas-controller-5597f6b9f6-zx498\" (UID: \"1f2df2e9-089e-48f0-a547-841ca4dc9781\") " pod="opendatahub/maas-controller-5597f6b9f6-zx498"
Nov 02 13:38:38.105198 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:38.105046    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gmpf\" (UniqueName: \"kubernetes.io/projected/1f2df2e9-089e-48f0-a547-841ca4dc9781-kube-api-access-2gmpf\") pod \"maas-controller-5597f6b9f6-zx498\" (UID: \"1f2df2e9-089e-48f0-a547-841ca4dc9781\") " pod="opendatahub/maas-controller-5597f6b9f6-zx498"
Nov 02 13:38:38.206195 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:38.206159    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gmpf\" (UniqueName: \"kubernetes.io/projected/1f2df2e9-089e-48f0-a547-841ca4dc9781-kube-api-access-2gmpf\") pod \"maas-controller-5597f6b9f6-zx498\" (UID: \"1f2df2e9-089e-48f0-a547-841ca4dc9781\") " pod="opendatahub/maas-controller-5597f6b9f6-zx498"
Nov 02 13:38:38.206382 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:38.206232    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f2df2e9-089e-48f0-a547-841ca4dc9781-metrics-certs\") pod \"maas-controller-5597f6b9f6-zx498\" (UID: \"1f2df2e9-089e-48f0-a547-841ca4dc9781\") " pod="opendatahub/maas-controller-5597f6b9f6-zx498"
Nov 02 13:38:38.206382 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:38.206261    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1f2df2e9-089e-48f0-a547-841ca4dc9781-webhook-cert\") pod \"maas-controller-5597f6b9f6-zx498\" (UID: \"1f2df2e9-089e-48f0-a547-841ca4dc9781\") " pod="opendatahub/maas-controller-5597f6b9f6-zx498"
Nov 02 13:38:38.208894 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:38.208873    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f2df2e9-089e-48f0-a547-841ca4dc9781-metrics-certs\") pod \"maas-controller-5597f6b9f6-zx498\" (UID: \"1f2df2e9-089e-48f0-a547-841ca4dc9781\") " pod="opendatahub/maas-controller-5597f6b9f6-zx498"
Nov 02 13:38:38.208986 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:38.208932    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1f2df2e9-089e-48f0-a547-841ca4dc9781-webhook-cert\") pod \"maas-controller-5597f6b9f6-zx498\" (UID: \"1f2df2e9-089e-48f0-a547-841ca4dc9781\") " pod="opendatahub/maas-controller-5597f6b9f6-zx498"
Nov 02 13:38:38.230394 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:38.230373    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gmpf\" (UniqueName: \"kubernetes.io/projected/1f2df2e9-089e-48f0-a547-841ca4dc9781-kube-api-access-2gmpf\") pod \"maas-controller-5597f6b9f6-zx498\" (UID: \"1f2df2e9-089e-48f0-a547-841ca4dc9781\") " pod="opendatahub/maas-controller-5597f6b9f6-zx498"
Nov 02 13:38:38.332751 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:38.332714    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5597f6b9f6-zx498"
Nov 02 13:38:38.631171 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:38.631139    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5597f6b9f6-zx498"]
Nov 02 13:38:38.632281 ip-10-0-128-132 kubenswrapper[4559]: W1102 13:38:38.632260    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f2df2e9_089e_48f0_a547_841ca4dc9781.slice/crio-b3e065f38906110fd53685a1c0ef4070df239e5c8d97491769a21bc49daba818 WatchSource:0}: Error finding container b3e065f38906110fd53685a1c0ef4070df239e5c8d97491769a21bc49daba818: Status 404 returned error can't find the container with id b3e065f38906110fd53685a1c0ef4070df239e5c8d97491769a21bc49daba818
Nov 02 13:38:38.825397 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:38.825361    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5597f6b9f6-zx498" event={"ID":"1f2df2e9-089e-48f0-a547-841ca4dc9781","Type":"ContainerStarted","Data":"b3e065f38906110fd53685a1c0ef4070df239e5c8d97491769a21bc49daba818"}
Nov 02 13:38:39.892457 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:39.892424    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93596dfc-4540-4cd5-9c06-d8a790d8c58e" path="/var/lib/kubelet/pods/93596dfc-4540-4cd5-9c06-d8a790d8c58e/volumes"
Nov 02 13:38:40.836424 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:40.836392    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5597f6b9f6-zx498" event={"ID":"1f2df2e9-089e-48f0-a547-841ca4dc9781","Type":"ContainerStarted","Data":"3077417e7ae760dbd8e7be7fd3293c34ab64a311436fda06e64da71202cbf774"}
Nov 02 13:38:40.836608 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:40.836509    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5597f6b9f6-zx498"
Nov 02 13:38:40.887237 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:40.887189    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-5597f6b9f6-zx498" podStartSLOduration=2.3673117870000002 podStartE2EDuration="3.887175647s" totalImagesPullingTime="1.51986386s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:38:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:38:40.886482917 +0000 UTC m=+4317.141882014" watchObservedRunningTime="2026-11-02 13:38:40.887175647 +0000 UTC m=+4317.142574742"
Nov 02 13:38:45.085157 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:38:45.085119    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqcmj" containerName="extractor"
Nov 02 13:38:46.099714 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:46.099673    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-gbd89"]
Nov 02 13:38:46.100257 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:46.100176    4559 cpu_manager.go:414] "containerMap: removing container" podUID="93596dfc-4540-4cd5-9c06-d8a790d8c58e" containerName="istio-proxy"
Nov 02 13:38:46.100257 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:46.100196    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93596dfc-4540-4cd5-9c06-d8a790d8c58e" containerName="istio-proxy"
Nov 02 13:38:46.100386 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:46.100288    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="93596dfc-4540-4cd5-9c06-d8a790d8c58e" containerName="istio-proxy"
Nov 02 13:38:46.103488 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:46.103462    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-gbd89"
Nov 02 13:38:46.118977 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:46.118938    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Nov 02 13:38:46.119100 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:46.119072    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-2nvrx\""
Nov 02 13:38:46.133692 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:46.133664    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-gbd89"]
Nov 02 13:38:46.277233 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:46.277185    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f55beaae-7351-41b7-89ca-3142739266cd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-gbd89\" (UID: \"f55beaae-7351-41b7-89ca-3142739266cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-gbd89"
Nov 02 13:38:46.277427 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:46.277303    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f55beaae-7351-41b7-89ca-3142739266cd-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-gbd89\" (UID: \"f55beaae-7351-41b7-89ca-3142739266cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-gbd89"
Nov 02 13:38:46.277427 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:46.277348    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f55beaae-7351-41b7-89ca-3142739266cd-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-gbd89\" (UID: \"f55beaae-7351-41b7-89ca-3142739266cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-gbd89"
Nov 02 13:38:46.277427 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:46.277374    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f55beaae-7351-41b7-89ca-3142739266cd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-gbd89\" (UID: \"f55beaae-7351-41b7-89ca-3142739266cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-gbd89"
Nov 02 13:38:46.277601 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:46.277447    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f55beaae-7351-41b7-89ca-3142739266cd-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-gbd89\" (UID: \"f55beaae-7351-41b7-89ca-3142739266cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-gbd89"
Nov 02 13:38:46.277601 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:46.277475    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f55beaae-7351-41b7-89ca-3142739266cd-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-gbd89\" (UID: \"f55beaae-7351-41b7-89ca-3142739266cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-gbd89"
Nov 02 13:38:46.277601 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:46.277559    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qthz8\" (UniqueName: \"kubernetes.io/projected/f55beaae-7351-41b7-89ca-3142739266cd-kube-api-access-qthz8\") pod \"maas-default-gateway-openshift-default-5bc884dd87-gbd89\" (UID: \"f55beaae-7351-41b7-89ca-3142739266cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-gbd89"
Nov 02 13:38:46.277601 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:46.277592    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f55beaae-7351-41b7-89ca-3142739266cd-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-gbd89\" (UID: \"f55beaae-7351-41b7-89ca-3142739266cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-gbd89"
Nov 02 13:38:46.277826 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:46.277668    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f55beaae-7351-41b7-89ca-3142739266cd-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-gbd89\" (UID: \"f55beaae-7351-41b7-89ca-3142739266cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-gbd89"
Nov 02 13:38:46.378794 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:46.378717    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f55beaae-7351-41b7-89ca-3142739266cd-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-gbd89\" (UID: \"f55beaae-7351-41b7-89ca-3142739266cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-gbd89"
Nov 02 13:38:46.378794 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:46.378761    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f55beaae-7351-41b7-89ca-3142739266cd-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-gbd89\" (UID: \"f55beaae-7351-41b7-89ca-3142739266cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-gbd89"
Nov 02 13:38:46.378794 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:46.378787    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f55beaae-7351-41b7-89ca-3142739266cd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-gbd89\" (UID: \"f55beaae-7351-41b7-89ca-3142739266cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-gbd89"
Nov 02 13:38:46.379078 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:46.378815    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f55beaae-7351-41b7-89ca-3142739266cd-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-gbd89\" (UID: \"f55beaae-7351-41b7-89ca-3142739266cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-gbd89"
Nov 02 13:38:46.379078 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:46.378837    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f55beaae-7351-41b7-89ca-3142739266cd-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-gbd89\" (UID: \"f55beaae-7351-41b7-89ca-3142739266cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-gbd89"
Nov 02 13:38:46.379078 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:46.378860    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qthz8\" (UniqueName: \"kubernetes.io/projected/f55beaae-7351-41b7-89ca-3142739266cd-kube-api-access-qthz8\") pod \"maas-default-gateway-openshift-default-5bc884dd87-gbd89\" (UID: \"f55beaae-7351-41b7-89ca-3142739266cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-gbd89"
Nov 02 13:38:46.379078 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:46.378883    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f55beaae-7351-41b7-89ca-3142739266cd-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-gbd89\" (UID: \"f55beaae-7351-41b7-89ca-3142739266cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-gbd89"
Nov 02 13:38:46.379078 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:46.378907    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f55beaae-7351-41b7-89ca-3142739266cd-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-gbd89\" (UID: \"f55beaae-7351-41b7-89ca-3142739266cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-gbd89"
Nov 02 13:38:46.379078 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:46.378938    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f55beaae-7351-41b7-89ca-3142739266cd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-gbd89\" (UID: \"f55beaae-7351-41b7-89ca-3142739266cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-gbd89"
Nov 02 13:38:46.379418 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:46.379189    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f55beaae-7351-41b7-89ca-3142739266cd-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-gbd89\" (UID: \"f55beaae-7351-41b7-89ca-3142739266cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-gbd89"
Nov 02 13:38:46.379418 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:46.379344    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f55beaae-7351-41b7-89ca-3142739266cd-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-gbd89\" (UID: \"f55beaae-7351-41b7-89ca-3142739266cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-gbd89"
Nov 02 13:38:46.379535 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:46.379452    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f55beaae-7351-41b7-89ca-3142739266cd-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-gbd89\" (UID: \"f55beaae-7351-41b7-89ca-3142739266cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-gbd89"
Nov 02 13:38:46.379535 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:46.379456    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f55beaae-7351-41b7-89ca-3142739266cd-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-gbd89\" (UID: \"f55beaae-7351-41b7-89ca-3142739266cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-gbd89"
Nov 02 13:38:46.379629 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:46.379607    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f55beaae-7351-41b7-89ca-3142739266cd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-gbd89\" (UID: \"f55beaae-7351-41b7-89ca-3142739266cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-gbd89"
Nov 02 13:38:46.381291 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:46.381261    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f55beaae-7351-41b7-89ca-3142739266cd-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-gbd89\" (UID: \"f55beaae-7351-41b7-89ca-3142739266cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-gbd89"
Nov 02 13:38:46.381401 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:46.381388    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f55beaae-7351-41b7-89ca-3142739266cd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-gbd89\" (UID: \"f55beaae-7351-41b7-89ca-3142739266cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-gbd89"
Nov 02 13:38:46.402581 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:46.402558    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f55beaae-7351-41b7-89ca-3142739266cd-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-gbd89\" (UID: \"f55beaae-7351-41b7-89ca-3142739266cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-gbd89"
Nov 02 13:38:46.402913 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:46.402893    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qthz8\" (UniqueName: \"kubernetes.io/projected/f55beaae-7351-41b7-89ca-3142739266cd-kube-api-access-qthz8\") pod \"maas-default-gateway-openshift-default-5bc884dd87-gbd89\" (UID: \"f55beaae-7351-41b7-89ca-3142739266cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-gbd89"
Nov 02 13:38:46.415765 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:46.415745    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-gbd89"
Nov 02 13:38:46.736224 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:46.736198    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-gbd89"]
Nov 02 13:38:46.737569 ip-10-0-128-132 kubenswrapper[4559]: W1102 13:38:46.737541    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf55beaae_7351_41b7_89ca_3142739266cd.slice/crio-421fd58d41ba23d948112a695995e1777b2b521b6f85ba2a0eaff42fdf5394bd WatchSource:0}: Error finding container 421fd58d41ba23d948112a695995e1777b2b521b6f85ba2a0eaff42fdf5394bd: Status 404 returned error can't find the container with id 421fd58d41ba23d948112a695995e1777b2b521b6f85ba2a0eaff42fdf5394bd
Nov 02 13:38:46.740166 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:46.740131    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 13:38:46.740273 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:46.740213    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 13:38:46.740273 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:46.740258    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 13:38:46.866038 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:46.865996    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-gbd89" event={"ID":"f55beaae-7351-41b7-89ca-3142739266cd","Type":"ContainerStarted","Data":"e3779d47ea696015c6efb36968d46ad9532894448bdd395fa2e3217fbe7d7c8f"}
Nov 02 13:38:46.866217 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:46.866063    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-gbd89" event={"ID":"f55beaae-7351-41b7-89ca-3142739266cd","Type":"ContainerStarted","Data":"421fd58d41ba23d948112a695995e1777b2b521b6f85ba2a0eaff42fdf5394bd"}
Nov 02 13:38:46.916231 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:46.916168    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-gbd89" podStartSLOduration=0.916148669 podStartE2EDuration="916.148669ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:38:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:38:46.914133035 +0000 UTC m=+4323.169532131" watchObservedRunningTime="2026-11-02 13:38:46.916148669 +0000 UTC m=+4323.171547765"
Nov 02 13:38:47.416627 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:47.416579    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-gbd89"
Nov 02 13:38:47.418236 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:47.418210    4559 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-gbd89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.13:15021/healthz/ready\": dial tcp 10.133.1.13:15021: connect: connection refused" start-of-body=
Nov 02 13:38:47.418362 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:47.418258    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-gbd89" podUID="f55beaae-7351-41b7-89ca-3142739266cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.13:15021/healthz/ready\": dial tcp 10.133.1.13:15021: connect: connection refused"
Nov 02 13:38:48.416300 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:48.416263    4559 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-gbd89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.13:15021/healthz/ready\": dial tcp 10.133.1.13:15021: connect: connection refused" start-of-body=
Nov 02 13:38:48.416488 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:48.416317    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-gbd89" podUID="f55beaae-7351-41b7-89ca-3142739266cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.13:15021/healthz/ready\": dial tcp 10.133.1.13:15021: connect: connection refused"
Nov 02 13:38:49.416491 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:49.416454    4559 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-gbd89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.13:15021/healthz/ready\": dial tcp 10.133.1.13:15021: connect: connection refused" start-of-body=
Nov 02 13:38:49.416905 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:49.416512    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-gbd89" podUID="f55beaae-7351-41b7-89ca-3142739266cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.13:15021/healthz/ready\": dial tcp 10.133.1.13:15021: connect: connection refused"
Nov 02 13:38:50.416214 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:50.416177    4559 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-gbd89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.13:15021/healthz/ready\": dial tcp 10.133.1.13:15021: connect: connection refused" start-of-body=
Nov 02 13:38:50.416461 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:50.416233    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-gbd89" podUID="f55beaae-7351-41b7-89ca-3142739266cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.13:15021/healthz/ready\": dial tcp 10.133.1.13:15021: connect: connection refused"
Nov 02 13:38:51.416847 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:51.416805    4559 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-gbd89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.13:15021/healthz/ready\": dial tcp 10.133.1.13:15021: connect: connection refused" start-of-body=
Nov 02 13:38:51.417268 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:51.416866    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-gbd89" podUID="f55beaae-7351-41b7-89ca-3142739266cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.13:15021/healthz/ready\": dial tcp 10.133.1.13:15021: connect: connection refused"
Nov 02 13:38:51.843305 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:51.843273    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5597f6b9f6-zx498"
Nov 02 13:38:52.416622 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:52.416577    4559 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-gbd89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.13:15021/healthz/ready\": dial tcp 10.133.1.13:15021: connect: connection refused" start-of-body=
Nov 02 13:38:52.416845 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:52.416663    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-gbd89" podUID="f55beaae-7351-41b7-89ca-3142739266cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.13:15021/healthz/ready\": dial tcp 10.133.1.13:15021: connect: connection refused"
Nov 02 13:38:53.417041 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:53.417004    4559 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-gbd89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.13:15021/healthz/ready\": dial tcp 10.133.1.13:15021: connect: connection refused" start-of-body=
Nov 02 13:38:53.417437 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:53.417058    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-gbd89" podUID="f55beaae-7351-41b7-89ca-3142739266cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.13:15021/healthz/ready\": dial tcp 10.133.1.13:15021: connect: connection refused"
Nov 02 13:38:55.184517 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:55.184481    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-gbd89"
Nov 02 13:38:55.185012 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:55.184930    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-gbd89"
Nov 02 13:38:55.185791 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:38:55.185770    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-gbd89"
Nov 02 13:39:00.666798 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:00.666754    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-fhx8f"]
Nov 02 13:39:00.670343 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:00.670325    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-fhx8f"
Nov 02 13:39:00.686547 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:00.686522    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 13:39:00.699612 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:00.699567    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-45fzn\""
Nov 02 13:39:00.699805 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:00.699674    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 13:39:00.702552 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:00.702518    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 13:39:00.705229 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:00.705187    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-fhx8f"]
Nov 02 13:39:00.707186 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:00.707160    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pjqw\" (UniqueName: \"kubernetes.io/projected/566c809f-50dc-487c-816c-2e7cd62e11c9-kube-api-access-6pjqw\") pod \"maas-api-revoke-sub-85237e45-fhx8f\" (UID: \"566c809f-50dc-487c-816c-2e7cd62e11c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-fhx8f"
Nov 02 13:39:00.707311 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:00.707229    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/566c809f-50dc-487c-816c-2e7cd62e11c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-fhx8f\" (UID: \"566c809f-50dc-487c-816c-2e7cd62e11c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-fhx8f"
Nov 02 13:39:00.707384 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:00.707310    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/566c809f-50dc-487c-816c-2e7cd62e11c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-fhx8f\" (UID: \"566c809f-50dc-487c-816c-2e7cd62e11c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-fhx8f"
Nov 02 13:39:00.807811 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:00.807772    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/566c809f-50dc-487c-816c-2e7cd62e11c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-fhx8f\" (UID: \"566c809f-50dc-487c-816c-2e7cd62e11c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-fhx8f"
Nov 02 13:39:00.808023 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:00.807831    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pjqw\" (UniqueName: \"kubernetes.io/projected/566c809f-50dc-487c-816c-2e7cd62e11c9-kube-api-access-6pjqw\") pod \"maas-api-revoke-sub-85237e45-fhx8f\" (UID: \"566c809f-50dc-487c-816c-2e7cd62e11c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-fhx8f"
Nov 02 13:39:00.808023 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:00.807863    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/566c809f-50dc-487c-816c-2e7cd62e11c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-fhx8f\" (UID: \"566c809f-50dc-487c-816c-2e7cd62e11c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-fhx8f"
Nov 02 13:39:00.808458 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:00.808437    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/566c809f-50dc-487c-816c-2e7cd62e11c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-fhx8f\" (UID: \"566c809f-50dc-487c-816c-2e7cd62e11c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-fhx8f"
Nov 02 13:39:00.810203 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:00.810184    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/566c809f-50dc-487c-816c-2e7cd62e11c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-fhx8f\" (UID: \"566c809f-50dc-487c-816c-2e7cd62e11c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-fhx8f"
Nov 02 13:39:00.832218 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:00.832180    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pjqw\" (UniqueName: \"kubernetes.io/projected/566c809f-50dc-487c-816c-2e7cd62e11c9-kube-api-access-6pjqw\") pod \"maas-api-revoke-sub-85237e45-fhx8f\" (UID: \"566c809f-50dc-487c-816c-2e7cd62e11c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-fhx8f"
Nov 02 13:39:00.983493 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:00.983460    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-fhx8f"
Nov 02 13:39:01.145142 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:01.145105    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qmrsp"]
Nov 02 13:39:01.149788 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:01.149763    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qmrsp"
Nov 02 13:39:01.161000 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:01.160968    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jr7g7"]
Nov 02 13:39:01.164203 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:01.164182    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jr7g7"
Nov 02 13:39:01.173782 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:01.173744    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qmrsp"]
Nov 02 13:39:01.188189 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:01.188097    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jr7g7"]
Nov 02 13:39:01.211340 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:01.211310    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a534baf3-8ff8-4dca-bc27-9022feaaca75-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qmrsp\" (UID: \"a534baf3-8ff8-4dca-bc27-9022feaaca75\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qmrsp"
Nov 02 13:39:01.211340 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:01.211348    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a15c4957-178b-41d5-ae44-e79c37fb55d1-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-jr7g7\" (UID: \"a15c4957-178b-41d5-ae44-e79c37fb55d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jr7g7"
Nov 02 13:39:01.211611 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:01.211374    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjrbv\" (UniqueName: \"kubernetes.io/projected/a534baf3-8ff8-4dca-bc27-9022feaaca75-kube-api-access-qjrbv\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qmrsp\" (UID: \"a534baf3-8ff8-4dca-bc27-9022feaaca75\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qmrsp"
Nov 02 13:39:01.211611 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:01.211455    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a534baf3-8ff8-4dca-bc27-9022feaaca75-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qmrsp\" (UID: \"a534baf3-8ff8-4dca-bc27-9022feaaca75\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qmrsp"
Nov 02 13:39:01.211611 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:01.211510    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a534baf3-8ff8-4dca-bc27-9022feaaca75-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qmrsp\" (UID: \"a534baf3-8ff8-4dca-bc27-9022feaaca75\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qmrsp"
Nov 02 13:39:01.211611 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:01.211553    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a534baf3-8ff8-4dca-bc27-9022feaaca75-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qmrsp\" (UID: \"a534baf3-8ff8-4dca-bc27-9022feaaca75\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qmrsp"
Nov 02 13:39:01.211611 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:01.211584    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a15c4957-178b-41d5-ae44-e79c37fb55d1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-jr7g7\" (UID: \"a15c4957-178b-41d5-ae44-e79c37fb55d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jr7g7"
Nov 02 13:39:01.211900 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:01.211659    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a15c4957-178b-41d5-ae44-e79c37fb55d1-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-jr7g7\" (UID: \"a15c4957-178b-41d5-ae44-e79c37fb55d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jr7g7"
Nov 02 13:39:01.211900 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:01.211686    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cr74r\" (UniqueName: \"kubernetes.io/projected/a15c4957-178b-41d5-ae44-e79c37fb55d1-kube-api-access-cr74r\") pod \"maas-default-gateway-openshift-default-5bc884dd87-jr7g7\" (UID: \"a15c4957-178b-41d5-ae44-e79c37fb55d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jr7g7"
Nov 02 13:39:01.211900 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:01.211715    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a534baf3-8ff8-4dca-bc27-9022feaaca75-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qmrsp\" (UID: \"a534baf3-8ff8-4dca-bc27-9022feaaca75\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qmrsp"
Nov 02 13:39:01.211900 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:01.211782    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a15c4957-178b-41d5-ae44-e79c37fb55d1-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-jr7g7\" (UID: \"a15c4957-178b-41d5-ae44-e79c37fb55d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jr7g7"
Nov 02 13:39:01.211900 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:01.211826    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a15c4957-178b-41d5-ae44-e79c37fb55d1-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-jr7g7\" (UID: \"a15c4957-178b-41d5-ae44-e79c37fb55d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jr7g7"
Nov 02 13:39:01.211900 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:01.211894    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a15c4957-178b-41d5-ae44-e79c37fb55d1-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-jr7g7\" (UID: \"a15c4957-178b-41d5-ae44-e79c37fb55d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jr7g7"
Nov 02 13:39:01.212162 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:01.211929    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a15c4957-178b-41d5-ae44-e79c37fb55d1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-jr7g7\" (UID: \"a15c4957-178b-41d5-ae44-e79c37fb55d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jr7g7"
Nov 02 13:39:01.212162 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:01.211946    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a534baf3-8ff8-4dca-bc27-9022feaaca75-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qmrsp\" (UID: \"a534baf3-8ff8-4dca-bc27-9022feaaca75\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qmrsp"
Nov 02 13:39:01.212162 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:01.212015    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a534baf3-8ff8-4dca-bc27-9022feaaca75-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qmrsp\" (UID: \"a534baf3-8ff8-4dca-bc27-9022feaaca75\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qmrsp"
Nov 02 13:39:01.212162 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:01.212044    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a534baf3-8ff8-4dca-bc27-9022feaaca75-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qmrsp\" (UID: \"a534baf3-8ff8-4dca-bc27-9022feaaca75\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qmrsp"
Nov 02 13:39:01.212162 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:01.212075    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a15c4957-178b-41d5-ae44-e79c37fb55d1-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-jr7g7\" (UID: \"a15c4957-178b-41d5-ae44-e79c37fb55d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jr7g7"
Nov 02 13:39:01.286408 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:01.286372    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-fhx8f"]
Nov 02 13:39:01.287441 ip-10-0-128-132 kubenswrapper[4559]: W1102 13:39:01.287415    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod566c809f_50dc_487c_816c_2e7cd62e11c9.slice/crio-61b54eb781407029b086de85f0ab0b63b23bd9073cb2527316c516c942f1e8a2 WatchSource:0}: Error finding container 61b54eb781407029b086de85f0ab0b63b23bd9073cb2527316c516c942f1e8a2: Status 404 returned error can't find the container with id 61b54eb781407029b086de85f0ab0b63b23bd9073cb2527316c516c942f1e8a2
Nov 02 13:39:01.313005 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:01.312972    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a534baf3-8ff8-4dca-bc27-9022feaaca75-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qmrsp\" (UID: \"a534baf3-8ff8-4dca-bc27-9022feaaca75\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qmrsp"
Nov 02 13:39:01.313005 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:01.313008    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a15c4957-178b-41d5-ae44-e79c37fb55d1-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-jr7g7\" (UID: \"a15c4957-178b-41d5-ae44-e79c37fb55d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jr7g7"
Nov 02 13:39:01.313268 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:01.313028    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjrbv\" (UniqueName: \"kubernetes.io/projected/a534baf3-8ff8-4dca-bc27-9022feaaca75-kube-api-access-qjrbv\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qmrsp\" (UID: \"a534baf3-8ff8-4dca-bc27-9022feaaca75\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qmrsp"
Nov 02 13:39:01.313268 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:01.313054    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a534baf3-8ff8-4dca-bc27-9022feaaca75-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qmrsp\" (UID: \"a534baf3-8ff8-4dca-bc27-9022feaaca75\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qmrsp"
Nov 02 13:39:01.313268 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:01.313081    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a534baf3-8ff8-4dca-bc27-9022feaaca75-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qmrsp\" (UID: \"a534baf3-8ff8-4dca-bc27-9022feaaca75\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qmrsp"
Nov 02 13:39:01.313268 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:01.313120    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a534baf3-8ff8-4dca-bc27-9022feaaca75-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qmrsp\" (UID: \"a534baf3-8ff8-4dca-bc27-9022feaaca75\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qmrsp"
Nov 02 13:39:01.313268 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:01.313144    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a15c4957-178b-41d5-ae44-e79c37fb55d1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-jr7g7\" (UID: \"a15c4957-178b-41d5-ae44-e79c37fb55d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jr7g7"
Nov 02 13:39:01.313268 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:01.313173    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a15c4957-178b-41d5-ae44-e79c37fb55d1-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-jr7g7\" (UID: \"a15c4957-178b-41d5-ae44-e79c37fb55d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jr7g7"
Nov 02 13:39:01.313268 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:01.313197    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cr74r\" (UniqueName: \"kubernetes.io/projected/a15c4957-178b-41d5-ae44-e79c37fb55d1-kube-api-access-cr74r\") pod \"maas-default-gateway-openshift-default-5bc884dd87-jr7g7\" (UID: \"a15c4957-178b-41d5-ae44-e79c37fb55d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jr7g7"
Nov 02 13:39:01.313268 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:01.313234    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a534baf3-8ff8-4dca-bc27-9022feaaca75-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qmrsp\" (UID: \"a534baf3-8ff8-4dca-bc27-9022feaaca75\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qmrsp"
Nov 02 13:39:01.313268 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:01.313263    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a15c4957-178b-41d5-ae44-e79c37fb55d1-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-jr7g7\" (UID: \"a15c4957-178b-41d5-ae44-e79c37fb55d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jr7g7"
Nov 02 13:39:01.313812 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:01.313298    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a15c4957-178b-41d5-ae44-e79c37fb55d1-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-jr7g7\" (UID: \"a15c4957-178b-41d5-ae44-e79c37fb55d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jr7g7"
Nov 02 13:39:01.313812 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:01.313332    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a15c4957-178b-41d5-ae44-e79c37fb55d1-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-jr7g7\" (UID: \"a15c4957-178b-41d5-ae44-e79c37fb55d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jr7g7"
Nov 02 13:39:01.313812 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:01.313372    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a15c4957-178b-41d5-ae44-e79c37fb55d1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-jr7g7\" (UID: \"a15c4957-178b-41d5-ae44-e79c37fb55d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jr7g7"
Nov 02 13:39:01.313812 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:01.313398    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a534baf3-8ff8-4dca-bc27-9022feaaca75-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qmrsp\" (UID: \"a534baf3-8ff8-4dca-bc27-9022feaaca75\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qmrsp"
Nov 02 13:39:01.313812 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:01.313436    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a534baf3-8ff8-4dca-bc27-9022feaaca75-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qmrsp\" (UID: \"a534baf3-8ff8-4dca-bc27-9022feaaca75\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qmrsp"
Nov 02 13:39:01.313812 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:01.313449    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a15c4957-178b-41d5-ae44-e79c37fb55d1-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-jr7g7\" (UID: \"a15c4957-178b-41d5-ae44-e79c37fb55d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jr7g7"
Nov 02 13:39:01.313812 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:01.313465    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a534baf3-8ff8-4dca-bc27-9022feaaca75-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qmrsp\" (UID: \"a534baf3-8ff8-4dca-bc27-9022feaaca75\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qmrsp"
Nov 02 13:39:01.313812 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:01.313493    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a534baf3-8ff8-4dca-bc27-9022feaaca75-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qmrsp\" (UID: \"a534baf3-8ff8-4dca-bc27-9022feaaca75\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qmrsp"
Nov 02 13:39:01.313812 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:01.313535    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a15c4957-178b-41d5-ae44-e79c37fb55d1-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-jr7g7\" (UID: \"a15c4957-178b-41d5-ae44-e79c37fb55d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jr7g7"
Nov 02 13:39:01.313812 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:01.313531    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a534baf3-8ff8-4dca-bc27-9022feaaca75-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qmrsp\" (UID: \"a534baf3-8ff8-4dca-bc27-9022feaaca75\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qmrsp"
Nov 02 13:39:01.313812 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:01.313801    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a15c4957-178b-41d5-ae44-e79c37fb55d1-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-jr7g7\" (UID: \"a15c4957-178b-41d5-ae44-e79c37fb55d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jr7g7"
Nov 02 13:39:01.314397 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:01.313845    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a534baf3-8ff8-4dca-bc27-9022feaaca75-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qmrsp\" (UID: \"a534baf3-8ff8-4dca-bc27-9022feaaca75\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qmrsp"
Nov 02 13:39:01.314397 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:01.314255    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a15c4957-178b-41d5-ae44-e79c37fb55d1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-jr7g7\" (UID: \"a15c4957-178b-41d5-ae44-e79c37fb55d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jr7g7"
Nov 02 13:39:01.314397 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:01.314271    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a534baf3-8ff8-4dca-bc27-9022feaaca75-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qmrsp\" (UID: \"a534baf3-8ff8-4dca-bc27-9022feaaca75\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qmrsp"
Nov 02 13:39:01.314397 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:01.314339    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a15c4957-178b-41d5-ae44-e79c37fb55d1-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-jr7g7\" (UID: \"a15c4957-178b-41d5-ae44-e79c37fb55d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jr7g7"
Nov 02 13:39:01.314633 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:01.314471    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a534baf3-8ff8-4dca-bc27-9022feaaca75-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qmrsp\" (UID: \"a534baf3-8ff8-4dca-bc27-9022feaaca75\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qmrsp"
Nov 02 13:39:01.314715 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:01.314629    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a15c4957-178b-41d5-ae44-e79c37fb55d1-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-jr7g7\" (UID: \"a15c4957-178b-41d5-ae44-e79c37fb55d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jr7g7"
Nov 02 13:39:01.315845 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:01.315815    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a534baf3-8ff8-4dca-bc27-9022feaaca75-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qmrsp\" (UID: \"a534baf3-8ff8-4dca-bc27-9022feaaca75\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qmrsp"
Nov 02 13:39:01.315944 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:01.315919    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a534baf3-8ff8-4dca-bc27-9022feaaca75-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qmrsp\" (UID: \"a534baf3-8ff8-4dca-bc27-9022feaaca75\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qmrsp"
Nov 02 13:39:01.316061 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:01.316042    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a15c4957-178b-41d5-ae44-e79c37fb55d1-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-jr7g7\" (UID: \"a15c4957-178b-41d5-ae44-e79c37fb55d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jr7g7"
Nov 02 13:39:01.316270 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:01.316253    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a15c4957-178b-41d5-ae44-e79c37fb55d1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-jr7g7\" (UID: \"a15c4957-178b-41d5-ae44-e79c37fb55d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jr7g7"
Nov 02 13:39:01.337230 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:01.337198    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a534baf3-8ff8-4dca-bc27-9022feaaca75-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qmrsp\" (UID: \"a534baf3-8ff8-4dca-bc27-9022feaaca75\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qmrsp"
Nov 02 13:39:01.337431 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:01.337278    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjrbv\" (UniqueName: \"kubernetes.io/projected/a534baf3-8ff8-4dca-bc27-9022feaaca75-kube-api-access-qjrbv\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qmrsp\" (UID: \"a534baf3-8ff8-4dca-bc27-9022feaaca75\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qmrsp"
Nov 02 13:39:01.337557 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:01.337537    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a15c4957-178b-41d5-ae44-e79c37fb55d1-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-jr7g7\" (UID: \"a15c4957-178b-41d5-ae44-e79c37fb55d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jr7g7"
Nov 02 13:39:01.337800 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:01.337780    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cr74r\" (UniqueName: \"kubernetes.io/projected/a15c4957-178b-41d5-ae44-e79c37fb55d1-kube-api-access-cr74r\") pod \"maas-default-gateway-openshift-default-5bc884dd87-jr7g7\" (UID: \"a15c4957-178b-41d5-ae44-e79c37fb55d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jr7g7"
Nov 02 13:39:01.463458 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:01.463414    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qmrsp"
Nov 02 13:39:01.477532 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:01.477506    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jr7g7"
Nov 02 13:39:01.799102 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:01.799074    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qmrsp"]
Nov 02 13:39:01.799753 ip-10-0-128-132 kubenswrapper[4559]: W1102 13:39:01.799722    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda534baf3_8ff8_4dca_bc27_9022feaaca75.slice/crio-4bf434995e29863592adff5070e7f12b5591f26f33fab45736562627bda1dc36 WatchSource:0}: Error finding container 4bf434995e29863592adff5070e7f12b5591f26f33fab45736562627bda1dc36: Status 404 returned error can't find the container with id 4bf434995e29863592adff5070e7f12b5591f26f33fab45736562627bda1dc36
Nov 02 13:39:01.802051 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:01.802020    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 13:39:01.802135 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:01.802085    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 13:39:01.802135 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:01.802120    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 13:39:01.813789 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:01.813763    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jr7g7"]
Nov 02 13:39:01.814401 ip-10-0-128-132 kubenswrapper[4559]: W1102 13:39:01.814374    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda15c4957_178b_41d5_ae44_e79c37fb55d1.slice/crio-0533e0846262fbff69d84cf5b3ccf063971cf596fd7b7915ca025942139e4759 WatchSource:0}: Error finding container 0533e0846262fbff69d84cf5b3ccf063971cf596fd7b7915ca025942139e4759: Status 404 returned error can't find the container with id 0533e0846262fbff69d84cf5b3ccf063971cf596fd7b7915ca025942139e4759
Nov 02 13:39:01.821084 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:01.821042    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 13:39:01.821179 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:01.821123    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 13:39:01.821179 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:01.821150    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 13:39:01.954527 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:01.954481    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qmrsp" event={"ID":"a534baf3-8ff8-4dca-bc27-9022feaaca75","Type":"ContainerStarted","Data":"72eec267b29f0aba90679df1419f517e43be4d657397746041852bdc2398f514"}
Nov 02 13:39:01.954758 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:01.954549    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qmrsp" event={"ID":"a534baf3-8ff8-4dca-bc27-9022feaaca75","Type":"ContainerStarted","Data":"4bf434995e29863592adff5070e7f12b5591f26f33fab45736562627bda1dc36"}
Nov 02 13:39:01.957489 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:01.957449    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-fhx8f" event={"ID":"566c809f-50dc-487c-816c-2e7cd62e11c9","Type":"ContainerStarted","Data":"d292a5569e875d1c9e0b65bb06f9283153565f200428eb3270931283cb78f91e"}
Nov 02 13:39:01.957685 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:01.957511    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-fhx8f" event={"ID":"566c809f-50dc-487c-816c-2e7cd62e11c9","Type":"ContainerStarted","Data":"61b54eb781407029b086de85f0ab0b63b23bd9073cb2527316c516c942f1e8a2"}
Nov 02 13:39:01.960167 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:01.960129    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jr7g7" event={"ID":"a15c4957-178b-41d5-ae44-e79c37fb55d1","Type":"ContainerStarted","Data":"ef73f81bd00e178e744dc89e0185e26b2120190df9417d78c2234c11f90caa6a"}
Nov 02 13:39:01.960289 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:01.960192    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jr7g7" event={"ID":"a15c4957-178b-41d5-ae44-e79c37fb55d1","Type":"ContainerStarted","Data":"0533e0846262fbff69d84cf5b3ccf063971cf596fd7b7915ca025942139e4759"}
Nov 02 13:39:02.004079 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:02.004022    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qmrsp" podStartSLOduration=1.004008079 podStartE2EDuration="1.004008079s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:39:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:39:02.002791459 +0000 UTC m=+4338.258190554" watchObservedRunningTime="2026-11-02 13:39:02.004008079 +0000 UTC m=+4338.259407173"
Nov 02 13:39:02.051687 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:02.051552    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jr7g7" podStartSLOduration=1.051536278 podStartE2EDuration="1.051536278s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:39:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:39:02.050665674 +0000 UTC m=+4338.306064763" watchObservedRunningTime="2026-11-02 13:39:02.051536278 +0000 UTC m=+4338.306935374"
Nov 02 13:39:02.093969 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:02.093904    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-fhx8f" podStartSLOduration=2.093883253 podStartE2EDuration="2.093883253s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:39:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:39:02.092997701 +0000 UTC m=+4338.348396797" watchObservedRunningTime="2026-11-02 13:39:02.093883253 +0000 UTC m=+4338.349282349"
Nov 02 13:39:02.464065 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:02.464033    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qmrsp"
Nov 02 13:39:02.465579 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:02.465551    4559 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-qmrsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.15:15021/healthz/ready\": dial tcp 10.133.1.15:15021: connect: connection refused" start-of-body=
Nov 02 13:39:02.465739 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:02.465616    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qmrsp" podUID="a534baf3-8ff8-4dca-bc27-9022feaaca75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.15:15021/healthz/ready\": dial tcp 10.133.1.15:15021: connect: connection refused"
Nov 02 13:39:02.477705 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:02.477677    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jr7g7"
Nov 02 13:39:02.478896 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:02.478876    4559 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-jr7g7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.16:15021/healthz/ready\": dial tcp 10.133.1.16:15021: connect: connection refused" start-of-body=
Nov 02 13:39:02.478994 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:02.478919    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jr7g7" podUID="a15c4957-178b-41d5-ae44-e79c37fb55d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.16:15021/healthz/ready\": dial tcp 10.133.1.16:15021: connect: connection refused"
Nov 02 13:39:02.738028 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:02.737988    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"]
Nov 02 13:39:02.967120 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:02.967075    4559 generic.go:356] "Generic (PLEG): container finished" podID="566c809f-50dc-487c-816c-2e7cd62e11c9" containerID="d292a5569e875d1c9e0b65bb06f9283153565f200428eb3270931283cb78f91e" exitCode=0
Nov 02 13:39:02.967600 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:02.967173    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-fhx8f" event={"ID":"566c809f-50dc-487c-816c-2e7cd62e11c9","Type":"ContainerDied","Data":"d292a5569e875d1c9e0b65bb06f9283153565f200428eb3270931283cb78f91e"}
Nov 02 13:39:03.464370 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:03.464319    4559 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-qmrsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.15:15021/healthz/ready\": dial tcp 10.133.1.15:15021: connect: connection refused" start-of-body=
Nov 02 13:39:03.464605 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:03.464376    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qmrsp" podUID="a534baf3-8ff8-4dca-bc27-9022feaaca75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.15:15021/healthz/ready\": dial tcp 10.133.1.15:15021: connect: connection refused"
Nov 02 13:39:03.478699 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:03.478671    4559 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-jr7g7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.16:15021/healthz/ready\": dial tcp 10.133.1.16:15021: connect: connection refused" start-of-body=
Nov 02 13:39:03.478832 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:03.478710    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jr7g7" podUID="a15c4957-178b-41d5-ae44-e79c37fb55d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.16:15021/healthz/ready\": dial tcp 10.133.1.16:15021: connect: connection refused"
Nov 02 13:39:04.276568 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:04.276539    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-fhx8f"
Nov 02 13:39:04.460544 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:04.460496    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/566c809f-50dc-487c-816c-2e7cd62e11c9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/566c809f-50dc-487c-816c-2e7cd62e11c9-maas-api-xks-ca\") pod \"566c809f-50dc-487c-816c-2e7cd62e11c9\" (UID: \"566c809f-50dc-487c-816c-2e7cd62e11c9\") "
Nov 02 13:39:04.460788 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:04.460555    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/566c809f-50dc-487c-816c-2e7cd62e11c9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/566c809f-50dc-487c-816c-2e7cd62e11c9-maas-api-openshift-ca\") pod \"566c809f-50dc-487c-816c-2e7cd62e11c9\" (UID: \"566c809f-50dc-487c-816c-2e7cd62e11c9\") "
Nov 02 13:39:04.460788 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:04.460592    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/566c809f-50dc-487c-816c-2e7cd62e11c9-kube-api-access-6pjqw\" (UniqueName: \"kubernetes.io/projected/566c809f-50dc-487c-816c-2e7cd62e11c9-kube-api-access-6pjqw\") pod \"566c809f-50dc-487c-816c-2e7cd62e11c9\" (UID: \"566c809f-50dc-487c-816c-2e7cd62e11c9\") "
Nov 02 13:39:04.461623 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:04.461582    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/566c809f-50dc-487c-816c-2e7cd62e11c9-maas-api-openshift-ca" pod "566c809f-50dc-487c-816c-2e7cd62e11c9" (UID: "566c809f-50dc-487c-816c-2e7cd62e11c9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:39:04.463970 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:04.463907    4559 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-qmrsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.15:15021/healthz/ready\": dial tcp 10.133.1.15:15021: connect: connection refused" start-of-body=
Nov 02 13:39:04.464531 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:04.464143    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qmrsp" podUID="a534baf3-8ff8-4dca-bc27-9022feaaca75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.15:15021/healthz/ready\": dial tcp 10.133.1.15:15021: connect: connection refused"
Nov 02 13:39:04.464531 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:04.464365    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/566c809f-50dc-487c-816c-2e7cd62e11c9-maas-api-xks-ca" pod "566c809f-50dc-487c-816c-2e7cd62e11c9" (UID: "566c809f-50dc-487c-816c-2e7cd62e11c9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:39:04.464531 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:04.464506    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/566c809f-50dc-487c-816c-2e7cd62e11c9-kube-api-access-6pjqw" pod "566c809f-50dc-487c-816c-2e7cd62e11c9" (UID: "566c809f-50dc-487c-816c-2e7cd62e11c9"). InnerVolumeSpecName "kube-api-access-6pjqw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:39:04.479207 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:04.479175    4559 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-jr7g7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.16:15021/healthz/ready\": dial tcp 10.133.1.16:15021: connect: connection refused" start-of-body=
Nov 02 13:39:04.479321 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:04.479287    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jr7g7" podUID="a15c4957-178b-41d5-ae44-e79c37fb55d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.16:15021/healthz/ready\": dial tcp 10.133.1.16:15021: connect: connection refused"
Nov 02 13:39:04.561764 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:04.561676    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/566c809f-50dc-487c-816c-2e7cd62e11c9-maas-api-xks-ca\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:39:04.561764 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:04.561711    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/566c809f-50dc-487c-816c-2e7cd62e11c9-maas-api-openshift-ca\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:39:04.561764 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:04.561722    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6pjqw\" (UniqueName: \"kubernetes.io/projected/566c809f-50dc-487c-816c-2e7cd62e11c9-kube-api-access-6pjqw\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:39:04.979369 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:04.979333    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-fhx8f" event={"ID":"566c809f-50dc-487c-816c-2e7cd62e11c9","Type":"ContainerDied","Data":"61b54eb781407029b086de85f0ab0b63b23bd9073cb2527316c516c942f1e8a2"}
Nov 02 13:39:04.979582 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:04.979566    4559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="61b54eb781407029b086de85f0ab0b63b23bd9073cb2527316c516c942f1e8a2"
Nov 02 13:39:04.979721 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:04.979368    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-fhx8f"
Nov 02 13:39:05.464108 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:05.464055    4559 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-qmrsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.15:15021/healthz/ready\": dial tcp 10.133.1.15:15021: connect: connection refused" start-of-body=
Nov 02 13:39:05.464515 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:05.464120    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qmrsp" podUID="a534baf3-8ff8-4dca-bc27-9022feaaca75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.15:15021/healthz/ready\": dial tcp 10.133.1.15:15021: connect: connection refused"
Nov 02 13:39:05.478459 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:05.478426    4559 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-jr7g7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.16:15021/healthz/ready\": dial tcp 10.133.1.16:15021: connect: connection refused" start-of-body=
Nov 02 13:39:05.478554 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:05.478473    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jr7g7" podUID="a15c4957-178b-41d5-ae44-e79c37fb55d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.16:15021/healthz/ready\": dial tcp 10.133.1.16:15021: connect: connection refused"
Nov 02 13:39:06.464858 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:06.464822    4559 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-qmrsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.15:15021/healthz/ready\": dial tcp 10.133.1.15:15021: connect: connection refused" start-of-body=
Nov 02 13:39:06.465263 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:06.464877    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qmrsp" podUID="a534baf3-8ff8-4dca-bc27-9022feaaca75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.15:15021/healthz/ready\": dial tcp 10.133.1.15:15021: connect: connection refused"
Nov 02 13:39:06.478232 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:06.478200    4559 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-jr7g7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.16:15021/healthz/ready\": dial tcp 10.133.1.16:15021: connect: connection refused" start-of-body=
Nov 02 13:39:06.478377 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:06.478245    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jr7g7" podUID="a15c4957-178b-41d5-ae44-e79c37fb55d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.16:15021/healthz/ready\": dial tcp 10.133.1.16:15021: connect: connection refused"
Nov 02 13:39:07.464369 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:07.464331    4559 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-qmrsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.15:15021/healthz/ready\": dial tcp 10.133.1.15:15021: connect: connection refused" start-of-body=
Nov 02 13:39:07.464569 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:07.464388    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qmrsp" podUID="a534baf3-8ff8-4dca-bc27-9022feaaca75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.15:15021/healthz/ready\": dial tcp 10.133.1.15:15021: connect: connection refused"
Nov 02 13:39:07.478716 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:07.478686    4559 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-jr7g7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.16:15021/healthz/ready\": dial tcp 10.133.1.16:15021: connect: connection refused" start-of-body=
Nov 02 13:39:07.479048 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:07.478737    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jr7g7" podUID="a15c4957-178b-41d5-ae44-e79c37fb55d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.16:15021/healthz/ready\": dial tcp 10.133.1.16:15021: connect: connection refused"
Nov 02 13:39:08.464877 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:08.464835    4559 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-qmrsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.15:15021/healthz/ready\": dial tcp 10.133.1.15:15021: connect: connection refused" start-of-body=
Nov 02 13:39:08.465049 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:08.464919    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qmrsp" podUID="a534baf3-8ff8-4dca-bc27-9022feaaca75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.15:15021/healthz/ready\": dial tcp 10.133.1.15:15021: connect: connection refused"
Nov 02 13:39:08.478337 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:08.478301    4559 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-jr7g7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.16:15021/healthz/ready\": dial tcp 10.133.1.16:15021: connect: connection refused" start-of-body=
Nov 02 13:39:08.478477 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:08.478365    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jr7g7" podUID="a15c4957-178b-41d5-ae44-e79c37fb55d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.16:15021/healthz/ready\": dial tcp 10.133.1.16:15021: connect: connection refused"
Nov 02 13:39:09.464267 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:09.464231    4559 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-qmrsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.15:15021/healthz/ready\": dial tcp 10.133.1.15:15021: connect: connection refused" start-of-body=
Nov 02 13:39:09.464686 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:09.464288    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qmrsp" podUID="a534baf3-8ff8-4dca-bc27-9022feaaca75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.15:15021/healthz/ready\": dial tcp 10.133.1.15:15021: connect: connection refused"
Nov 02 13:39:09.478576 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:09.478547    4559 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-jr7g7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.16:15021/healthz/ready\": dial tcp 10.133.1.16:15021: connect: connection refused" start-of-body=
Nov 02 13:39:09.478701 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:09.478590    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jr7g7" podUID="a15c4957-178b-41d5-ae44-e79c37fb55d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.16:15021/healthz/ready\": dial tcp 10.133.1.16:15021: connect: connection refused"
Nov 02 13:39:11.505883 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:11.505831    4559 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-qmrsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.15:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 13:39:11.506537 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:11.505902    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qmrsp" podUID="a534baf3-8ff8-4dca-bc27-9022feaaca75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.15:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:39:11.519856 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:11.519807    4559 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-jr7g7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.16:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 13:39:11.520018 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:11.519865    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jr7g7" podUID="a15c4957-178b-41d5-ae44-e79c37fb55d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.16:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:39:12.520576 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:12.520536    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5597f6b9f6-zx498"]
Nov 02 13:39:12.521138 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:12.520778    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-5597f6b9f6-zx498" podUID="1f2df2e9-089e-48f0-a547-841ca4dc9781" containerName="manager" containerID="cri-o://3077417e7ae760dbd8e7be7fd3293c34ab64a311436fda06e64da71202cbf774" gracePeriod=30
Nov 02 13:39:12.547171 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:12.546766    4559 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-qmrsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.15:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 13:39:12.547171 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:12.546886    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qmrsp" podUID="a534baf3-8ff8-4dca-bc27-9022feaaca75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.15:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:39:12.563754 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:12.563726    4559 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-jr7g7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.16:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 13:39:12.563884 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:12.563766    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jr7g7" podUID="a15c4957-178b-41d5-ae44-e79c37fb55d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.16:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:39:12.714248 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:12.714213    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qmrsp"
Nov 02 13:39:12.716020 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:12.715996    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qmrsp"
Nov 02 13:39:12.730432 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:12.730406    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qmrsp"
Nov 02 13:39:12.794289 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:12.794263    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5597f6b9f6-zx498"
Nov 02 13:39:12.882729 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:12.882693    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f2df2e9-089e-48f0-a547-841ca4dc9781-kube-api-access-2gmpf\" (UniqueName: \"kubernetes.io/projected/1f2df2e9-089e-48f0-a547-841ca4dc9781-kube-api-access-2gmpf\") pod \"1f2df2e9-089e-48f0-a547-841ca4dc9781\" (UID: \"1f2df2e9-089e-48f0-a547-841ca4dc9781\") "
Nov 02 13:39:12.882934 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:12.882762    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1f2df2e9-089e-48f0-a547-841ca4dc9781-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f2df2e9-089e-48f0-a547-841ca4dc9781-metrics-certs\") pod \"1f2df2e9-089e-48f0-a547-841ca4dc9781\" (UID: \"1f2df2e9-089e-48f0-a547-841ca4dc9781\") "
Nov 02 13:39:12.882934 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:12.882789    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1f2df2e9-089e-48f0-a547-841ca4dc9781-webhook-cert\" (UniqueName: \"kubernetes.io/secret/1f2df2e9-089e-48f0-a547-841ca4dc9781-webhook-cert\") pod \"1f2df2e9-089e-48f0-a547-841ca4dc9781\" (UID: \"1f2df2e9-089e-48f0-a547-841ca4dc9781\") "
Nov 02 13:39:12.885519 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:12.885489    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f2df2e9-089e-48f0-a547-841ca4dc9781-webhook-cert" pod "1f2df2e9-089e-48f0-a547-841ca4dc9781" (UID: "1f2df2e9-089e-48f0-a547-841ca4dc9781"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:39:12.885836 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:12.885802    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f2df2e9-089e-48f0-a547-841ca4dc9781-kube-api-access-2gmpf" pod "1f2df2e9-089e-48f0-a547-841ca4dc9781" (UID: "1f2df2e9-089e-48f0-a547-841ca4dc9781"). InnerVolumeSpecName "kube-api-access-2gmpf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:39:12.887039 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:12.887008    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f2df2e9-089e-48f0-a547-841ca4dc9781-metrics-certs" pod "1f2df2e9-089e-48f0-a547-841ca4dc9781" (UID: "1f2df2e9-089e-48f0-a547-841ca4dc9781"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:39:12.961666 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:12.961611    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jr7g7"
Nov 02 13:39:12.961963 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:12.961935    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jr7g7"
Nov 02 13:39:12.962750 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:12.962733    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jr7g7"
Nov 02 13:39:12.984543 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:12.984516    4559 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f2df2e9-089e-48f0-a547-841ca4dc9781-metrics-certs\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:39:12.984543 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:12.984545    4559 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1f2df2e9-089e-48f0-a547-841ca4dc9781-webhook-cert\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:39:12.984792 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:12.984558    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2gmpf\" (UniqueName: \"kubernetes.io/projected/1f2df2e9-089e-48f0-a547-841ca4dc9781-kube-api-access-2gmpf\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:39:13.039075 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:13.039016    4559 generic.go:356] "Generic (PLEG): container finished" podID="1f2df2e9-089e-48f0-a547-841ca4dc9781" containerID="3077417e7ae760dbd8e7be7fd3293c34ab64a311436fda06e64da71202cbf774" exitCode=0
Nov 02 13:39:13.039075 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:13.039062    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5597f6b9f6-zx498"
Nov 02 13:39:13.039227 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:13.039094    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5597f6b9f6-zx498" event={"ID":"1f2df2e9-089e-48f0-a547-841ca4dc9781","Type":"ContainerDied","Data":"3077417e7ae760dbd8e7be7fd3293c34ab64a311436fda06e64da71202cbf774"}
Nov 02 13:39:13.039227 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:13.039143    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5597f6b9f6-zx498" event={"ID":"1f2df2e9-089e-48f0-a547-841ca4dc9781","Type":"ContainerDied","Data":"b3e065f38906110fd53685a1c0ef4070df239e5c8d97491769a21bc49daba818"}
Nov 02 13:39:13.039227 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:13.039165    4559 scope.go:122] "RemoveContainer" containerID="3077417e7ae760dbd8e7be7fd3293c34ab64a311436fda06e64da71202cbf774"
Nov 02 13:39:13.048147 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:13.048132    4559 scope.go:122] "RemoveContainer" containerID="3077417e7ae760dbd8e7be7fd3293c34ab64a311436fda06e64da71202cbf774"
Nov 02 13:39:13.048408 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:39:13.048393    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3077417e7ae760dbd8e7be7fd3293c34ab64a311436fda06e64da71202cbf774\": container with ID starting with 3077417e7ae760dbd8e7be7fd3293c34ab64a311436fda06e64da71202cbf774 not found: ID does not exist" containerID="3077417e7ae760dbd8e7be7fd3293c34ab64a311436fda06e64da71202cbf774"
Nov 02 13:39:13.048452 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:13.048415    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3077417e7ae760dbd8e7be7fd3293c34ab64a311436fda06e64da71202cbf774"} err="failed to get container status \"3077417e7ae760dbd8e7be7fd3293c34ab64a311436fda06e64da71202cbf774\": rpc error: code = NotFound desc = could not find container \"3077417e7ae760dbd8e7be7fd3293c34ab64a311436fda06e64da71202cbf774\": container with ID starting with 3077417e7ae760dbd8e7be7fd3293c34ab64a311436fda06e64da71202cbf774 not found: ID does not exist"
Nov 02 13:39:13.103932 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:13.103900    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5597f6b9f6-zx498"]
Nov 02 13:39:13.110255 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:13.110228    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-5597f6b9f6-zx498"]
Nov 02 13:39:13.891297 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:13.891268    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f2df2e9-089e-48f0-a547-841ca4dc9781" path="/var/lib/kubelet/pods/1f2df2e9-089e-48f0-a547-841ca4dc9781/volumes"
Nov 02 13:39:14.321979 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:14.321946    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5597f6b9f6-cbcqg"]
Nov 02 13:39:14.322320 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:14.322306    4559 cpu_manager.go:414] "containerMap: removing container" podUID="1f2df2e9-089e-48f0-a547-841ca4dc9781" containerName="manager"
Nov 02 13:39:14.322367 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:14.322321    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f2df2e9-089e-48f0-a547-841ca4dc9781" containerName="manager"
Nov 02 13:39:14.322367 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:14.322343    4559 cpu_manager.go:414] "containerMap: removing container" podUID="566c809f-50dc-487c-816c-2e7cd62e11c9" containerName="revoke-keys"
Nov 02 13:39:14.322367 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:14.322348    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="566c809f-50dc-487c-816c-2e7cd62e11c9" containerName="revoke-keys"
Nov 02 13:39:14.322469 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:14.322422    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f2df2e9-089e-48f0-a547-841ca4dc9781" containerName="manager"
Nov 02 13:39:14.322469 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:14.322432    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="566c809f-50dc-487c-816c-2e7cd62e11c9" containerName="revoke-keys"
Nov 02 13:39:14.327833 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:14.327813    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5597f6b9f6-cbcqg"
Nov 02 13:39:14.346911 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:14.346876    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Nov 02 13:39:14.359690 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:14.359670    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-crg4p\""
Nov 02 13:39:14.359828 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:14.359719    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Nov 02 13:39:14.359828 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:14.359761    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Nov 02 13:39:14.361261 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:14.361241    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5597f6b9f6-cbcqg"]
Nov 02 13:39:14.399311 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:14.399285    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b01f338f-5aac-4cbb-9c18-178defd659c1-metrics-certs\") pod \"maas-controller-5597f6b9f6-cbcqg\" (UID: \"b01f338f-5aac-4cbb-9c18-178defd659c1\") " pod="opendatahub/maas-controller-5597f6b9f6-cbcqg"
Nov 02 13:39:14.399418 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:14.399318    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b01f338f-5aac-4cbb-9c18-178defd659c1-webhook-cert\") pod \"maas-controller-5597f6b9f6-cbcqg\" (UID: \"b01f338f-5aac-4cbb-9c18-178defd659c1\") " pod="opendatahub/maas-controller-5597f6b9f6-cbcqg"
Nov 02 13:39:14.399462 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:14.399412    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdntj\" (UniqueName: \"kubernetes.io/projected/b01f338f-5aac-4cbb-9c18-178defd659c1-kube-api-access-pdntj\") pod \"maas-controller-5597f6b9f6-cbcqg\" (UID: \"b01f338f-5aac-4cbb-9c18-178defd659c1\") " pod="opendatahub/maas-controller-5597f6b9f6-cbcqg"
Nov 02 13:39:14.500599 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:14.500572    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b01f338f-5aac-4cbb-9c18-178defd659c1-metrics-certs\") pod \"maas-controller-5597f6b9f6-cbcqg\" (UID: \"b01f338f-5aac-4cbb-9c18-178defd659c1\") " pod="opendatahub/maas-controller-5597f6b9f6-cbcqg"
Nov 02 13:39:14.500726 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:14.500608    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b01f338f-5aac-4cbb-9c18-178defd659c1-webhook-cert\") pod \"maas-controller-5597f6b9f6-cbcqg\" (UID: \"b01f338f-5aac-4cbb-9c18-178defd659c1\") " pod="opendatahub/maas-controller-5597f6b9f6-cbcqg"
Nov 02 13:39:14.500726 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:14.500674    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdntj\" (UniqueName: \"kubernetes.io/projected/b01f338f-5aac-4cbb-9c18-178defd659c1-kube-api-access-pdntj\") pod \"maas-controller-5597f6b9f6-cbcqg\" (UID: \"b01f338f-5aac-4cbb-9c18-178defd659c1\") " pod="opendatahub/maas-controller-5597f6b9f6-cbcqg"
Nov 02 13:39:14.503157 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:14.503128    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b01f338f-5aac-4cbb-9c18-178defd659c1-metrics-certs\") pod \"maas-controller-5597f6b9f6-cbcqg\" (UID: \"b01f338f-5aac-4cbb-9c18-178defd659c1\") " pod="opendatahub/maas-controller-5597f6b9f6-cbcqg"
Nov 02 13:39:14.503239 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:14.503162    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b01f338f-5aac-4cbb-9c18-178defd659c1-webhook-cert\") pod \"maas-controller-5597f6b9f6-cbcqg\" (UID: \"b01f338f-5aac-4cbb-9c18-178defd659c1\") " pod="opendatahub/maas-controller-5597f6b9f6-cbcqg"
Nov 02 13:39:14.524474 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:14.524452    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdntj\" (UniqueName: \"kubernetes.io/projected/b01f338f-5aac-4cbb-9c18-178defd659c1-kube-api-access-pdntj\") pod \"maas-controller-5597f6b9f6-cbcqg\" (UID: \"b01f338f-5aac-4cbb-9c18-178defd659c1\") " pod="opendatahub/maas-controller-5597f6b9f6-cbcqg"
Nov 02 13:39:14.641241 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:14.641180    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5597f6b9f6-cbcqg"
Nov 02 13:39:14.940047 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:14.940019    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5597f6b9f6-cbcqg"]
Nov 02 13:39:14.940788 ip-10-0-128-132 kubenswrapper[4559]: W1102 13:39:14.940760    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb01f338f_5aac_4cbb_9c18_178defd659c1.slice/crio-d837c6ca4b108f574b42a4277799b1f7b9683056a9d4e7f2aebfd2d52c0b430f WatchSource:0}: Error finding container d837c6ca4b108f574b42a4277799b1f7b9683056a9d4e7f2aebfd2d52c0b430f: Status 404 returned error can't find the container with id d837c6ca4b108f574b42a4277799b1f7b9683056a9d4e7f2aebfd2d52c0b430f
Nov 02 13:39:15.049746 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:15.049713    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5597f6b9f6-cbcqg" event={"ID":"b01f338f-5aac-4cbb-9c18-178defd659c1","Type":"ContainerStarted","Data":"d837c6ca4b108f574b42a4277799b1f7b9683056a9d4e7f2aebfd2d52c0b430f"}
Nov 02 13:39:15.085113 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:39:15.085080    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqcmj" containerName="extractor"
Nov 02 13:39:16.056409 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:16.056366    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5597f6b9f6-cbcqg" event={"ID":"b01f338f-5aac-4cbb-9c18-178defd659c1","Type":"ContainerStarted","Data":"09734c99390ef255960fb27482d590cf43a91a911ca7901eab4050ee9aefb1d1"}
Nov 02 13:39:16.056835 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:16.056478    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5597f6b9f6-cbcqg"
Nov 02 13:39:16.113035 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:16.112979    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-5597f6b9f6-cbcqg" podStartSLOduration=1.799182063 podStartE2EDuration="2.112961584s" totalImagesPullingTime="313.779521ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:39:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:39:16.112211416 +0000 UTC m=+4352.367610508" watchObservedRunningTime="2026-11-02 13:39:16.112961584 +0000 UTC m=+4352.368360681"
Nov 02 13:39:27.063555 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:27.063473    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5597f6b9f6-cbcqg"
Nov 02 13:39:34.285396 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:34.285354    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"]
Nov 02 13:39:38.324946 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:38.324909    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-qx7bf"]
Nov 02 13:39:38.328584 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:38.328554    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-qx7bf"
Nov 02 13:39:38.353496 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:38.353466    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkw5l\" (UniqueName: \"kubernetes.io/projected/cc751229-83de-49d7-b3ef-213095467cc6-kube-api-access-fkw5l\") pod \"maas-api-revoke-sub-f904bff9-qx7bf\" (UID: \"cc751229-83de-49d7-b3ef-213095467cc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-qx7bf"
Nov 02 13:39:38.353675 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:38.353523    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc751229-83de-49d7-b3ef-213095467cc6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f904bff9-qx7bf\" (UID: \"cc751229-83de-49d7-b3ef-213095467cc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-qx7bf"
Nov 02 13:39:38.353675 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:38.353635    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc751229-83de-49d7-b3ef-213095467cc6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f904bff9-qx7bf\" (UID: \"cc751229-83de-49d7-b3ef-213095467cc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-qx7bf"
Nov 02 13:39:38.358848 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:38.358822    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 13:39:38.358971 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:38.358877    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 13:39:38.358971 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:38.358901    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 13:39:38.358971 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:38.358961    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-45fzn\""
Nov 02 13:39:38.360708 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:38.360680    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-qx7bf"]
Nov 02 13:39:38.454075 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:38.454047    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc751229-83de-49d7-b3ef-213095467cc6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f904bff9-qx7bf\" (UID: \"cc751229-83de-49d7-b3ef-213095467cc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-qx7bf"
Nov 02 13:39:38.454232 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:38.454107    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc751229-83de-49d7-b3ef-213095467cc6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f904bff9-qx7bf\" (UID: \"cc751229-83de-49d7-b3ef-213095467cc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-qx7bf"
Nov 02 13:39:38.454232 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:38.454144    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkw5l\" (UniqueName: \"kubernetes.io/projected/cc751229-83de-49d7-b3ef-213095467cc6-kube-api-access-fkw5l\") pod \"maas-api-revoke-sub-f904bff9-qx7bf\" (UID: \"cc751229-83de-49d7-b3ef-213095467cc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-qx7bf"
Nov 02 13:39:38.454861 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:38.454837    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc751229-83de-49d7-b3ef-213095467cc6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f904bff9-qx7bf\" (UID: \"cc751229-83de-49d7-b3ef-213095467cc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-qx7bf"
Nov 02 13:39:38.456511 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:38.456488    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc751229-83de-49d7-b3ef-213095467cc6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f904bff9-qx7bf\" (UID: \"cc751229-83de-49d7-b3ef-213095467cc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-qx7bf"
Nov 02 13:39:38.478274 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:38.478254    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkw5l\" (UniqueName: \"kubernetes.io/projected/cc751229-83de-49d7-b3ef-213095467cc6-kube-api-access-fkw5l\") pod \"maas-api-revoke-sub-f904bff9-qx7bf\" (UID: \"cc751229-83de-49d7-b3ef-213095467cc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-qx7bf"
Nov 02 13:39:38.641607 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:38.641546    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-qx7bf"
Nov 02 13:39:38.936209 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:38.936181    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-qx7bf"]
Nov 02 13:39:38.936986 ip-10-0-128-132 kubenswrapper[4559]: W1102 13:39:38.936956    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc751229_83de_49d7_b3ef_213095467cc6.slice/crio-b90888a6de4e84ccdd8d188b90cd1c86c0cb4bc5aeff178a74cb1859172bd804 WatchSource:0}: Error finding container b90888a6de4e84ccdd8d188b90cd1c86c0cb4bc5aeff178a74cb1859172bd804: Status 404 returned error can't find the container with id b90888a6de4e84ccdd8d188b90cd1c86c0cb4bc5aeff178a74cb1859172bd804
Nov 02 13:39:39.170236 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:39.170211    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-qx7bf" event={"ID":"cc751229-83de-49d7-b3ef-213095467cc6","Type":"ContainerStarted","Data":"b90888a6de4e84ccdd8d188b90cd1c86c0cb4bc5aeff178a74cb1859172bd804"}
Nov 02 13:39:40.176867 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:40.176835    4559 generic.go:356] "Generic (PLEG): container finished" podID="cc751229-83de-49d7-b3ef-213095467cc6" containerID="98e673f03124346d78e751009c1a211f5760d689dc8d3a897075a43869783146" exitCode=0
Nov 02 13:39:40.177281 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:40.176917    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-qx7bf" event={"ID":"cc751229-83de-49d7-b3ef-213095467cc6","Type":"ContainerDied","Data":"98e673f03124346d78e751009c1a211f5760d689dc8d3a897075a43869783146"}
Nov 02 13:39:41.314888 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:41.314862    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-qx7bf"
Nov 02 13:39:41.376743 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:41.376716    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc751229-83de-49d7-b3ef-213095467cc6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc751229-83de-49d7-b3ef-213095467cc6-maas-api-openshift-ca\") pod \"cc751229-83de-49d7-b3ef-213095467cc6\" (UID: \"cc751229-83de-49d7-b3ef-213095467cc6\") "
Nov 02 13:39:41.376861 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:41.376768    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cc751229-83de-49d7-b3ef-213095467cc6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc751229-83de-49d7-b3ef-213095467cc6-maas-api-xks-ca\") pod \"cc751229-83de-49d7-b3ef-213095467cc6\" (UID: \"cc751229-83de-49d7-b3ef-213095467cc6\") "
Nov 02 13:39:41.376861 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:41.376804    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc751229-83de-49d7-b3ef-213095467cc6-kube-api-access-fkw5l\" (UniqueName: \"kubernetes.io/projected/cc751229-83de-49d7-b3ef-213095467cc6-kube-api-access-fkw5l\") pod \"cc751229-83de-49d7-b3ef-213095467cc6\" (UID: \"cc751229-83de-49d7-b3ef-213095467cc6\") "
Nov 02 13:39:41.377168 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:41.377139    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc751229-83de-49d7-b3ef-213095467cc6-maas-api-openshift-ca" pod "cc751229-83de-49d7-b3ef-213095467cc6" (UID: "cc751229-83de-49d7-b3ef-213095467cc6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:39:41.378931 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:41.378901    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc751229-83de-49d7-b3ef-213095467cc6-maas-api-xks-ca" pod "cc751229-83de-49d7-b3ef-213095467cc6" (UID: "cc751229-83de-49d7-b3ef-213095467cc6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:39:41.378931 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:41.378912    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc751229-83de-49d7-b3ef-213095467cc6-kube-api-access-fkw5l" pod "cc751229-83de-49d7-b3ef-213095467cc6" (UID: "cc751229-83de-49d7-b3ef-213095467cc6"). InnerVolumeSpecName "kube-api-access-fkw5l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:39:41.477682 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:41.477624    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc751229-83de-49d7-b3ef-213095467cc6-maas-api-xks-ca\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:39:41.477682 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:41.477665    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fkw5l\" (UniqueName: \"kubernetes.io/projected/cc751229-83de-49d7-b3ef-213095467cc6-kube-api-access-fkw5l\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:39:41.477682 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:41.477675    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc751229-83de-49d7-b3ef-213095467cc6-maas-api-openshift-ca\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:39:42.187107 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:42.187076    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-qx7bf" event={"ID":"cc751229-83de-49d7-b3ef-213095467cc6","Type":"ContainerDied","Data":"b90888a6de4e84ccdd8d188b90cd1c86c0cb4bc5aeff178a74cb1859172bd804"}
Nov 02 13:39:42.187272 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:42.187127    4559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b90888a6de4e84ccdd8d188b90cd1c86c0cb4bc5aeff178a74cb1859172bd804"
Nov 02 13:39:42.187272 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:42.187091    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-qx7bf"
Nov 02 13:39:45.087498 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:39:45.087460    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqcmj" containerName="extractor"
Nov 02 13:39:46.337396 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:46.337364    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5wtjm"]
Nov 02 13:39:46.337864 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:46.337781    4559 cpu_manager.go:414] "containerMap: removing container" podUID="cc751229-83de-49d7-b3ef-213095467cc6" containerName="revoke-keys"
Nov 02 13:39:46.337864 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:46.337795    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc751229-83de-49d7-b3ef-213095467cc6" containerName="revoke-keys"
Nov 02 13:39:46.337864 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:46.337861    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc751229-83de-49d7-b3ef-213095467cc6" containerName="revoke-keys"
Nov 02 13:39:46.341263 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:46.341243    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5wtjm"
Nov 02 13:39:46.367300 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:46.367271    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5wtjm"]
Nov 02 13:39:46.420212 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:46.420177    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjjd6\" (UniqueName: \"kubernetes.io/projected/b9661a3b-8a07-4a30-9893-20ab1d95a2a7-kube-api-access-bjjd6\") pod \"maas-default-gateway-openshift-default-5bc884dd87-5wtjm\" (UID: \"b9661a3b-8a07-4a30-9893-20ab1d95a2a7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5wtjm"
Nov 02 13:39:46.420332 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:46.420217    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9661a3b-8a07-4a30-9893-20ab1d95a2a7-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-5wtjm\" (UID: \"b9661a3b-8a07-4a30-9893-20ab1d95a2a7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5wtjm"
Nov 02 13:39:46.420332 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:46.420238    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9661a3b-8a07-4a30-9893-20ab1d95a2a7-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-5wtjm\" (UID: \"b9661a3b-8a07-4a30-9893-20ab1d95a2a7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5wtjm"
Nov 02 13:39:46.420332 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:46.420321    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9661a3b-8a07-4a30-9893-20ab1d95a2a7-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-5wtjm\" (UID: \"b9661a3b-8a07-4a30-9893-20ab1d95a2a7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5wtjm"
Nov 02 13:39:46.420445 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:46.420340    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9661a3b-8a07-4a30-9893-20ab1d95a2a7-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-5wtjm\" (UID: \"b9661a3b-8a07-4a30-9893-20ab1d95a2a7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5wtjm"
Nov 02 13:39:46.420445 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:46.420368    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9661a3b-8a07-4a30-9893-20ab1d95a2a7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-5wtjm\" (UID: \"b9661a3b-8a07-4a30-9893-20ab1d95a2a7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5wtjm"
Nov 02 13:39:46.420445 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:46.420386    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9661a3b-8a07-4a30-9893-20ab1d95a2a7-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-5wtjm\" (UID: \"b9661a3b-8a07-4a30-9893-20ab1d95a2a7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5wtjm"
Nov 02 13:39:46.420552 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:46.420516    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9661a3b-8a07-4a30-9893-20ab1d95a2a7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-5wtjm\" (UID: \"b9661a3b-8a07-4a30-9893-20ab1d95a2a7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5wtjm"
Nov 02 13:39:46.420588 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:46.420553    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9661a3b-8a07-4a30-9893-20ab1d95a2a7-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-5wtjm\" (UID: \"b9661a3b-8a07-4a30-9893-20ab1d95a2a7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5wtjm"
Nov 02 13:39:46.521542 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:46.521515    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjjd6\" (UniqueName: \"kubernetes.io/projected/b9661a3b-8a07-4a30-9893-20ab1d95a2a7-kube-api-access-bjjd6\") pod \"maas-default-gateway-openshift-default-5bc884dd87-5wtjm\" (UID: \"b9661a3b-8a07-4a30-9893-20ab1d95a2a7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5wtjm"
Nov 02 13:39:46.521712 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:46.521551    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9661a3b-8a07-4a30-9893-20ab1d95a2a7-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-5wtjm\" (UID: \"b9661a3b-8a07-4a30-9893-20ab1d95a2a7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5wtjm"
Nov 02 13:39:46.521712 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:46.521568    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9661a3b-8a07-4a30-9893-20ab1d95a2a7-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-5wtjm\" (UID: \"b9661a3b-8a07-4a30-9893-20ab1d95a2a7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5wtjm"
Nov 02 13:39:46.521712 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:46.521591    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9661a3b-8a07-4a30-9893-20ab1d95a2a7-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-5wtjm\" (UID: \"b9661a3b-8a07-4a30-9893-20ab1d95a2a7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5wtjm"
Nov 02 13:39:46.521712 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:46.521614    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9661a3b-8a07-4a30-9893-20ab1d95a2a7-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-5wtjm\" (UID: \"b9661a3b-8a07-4a30-9893-20ab1d95a2a7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5wtjm"
Nov 02 13:39:46.521712 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:46.521668    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9661a3b-8a07-4a30-9893-20ab1d95a2a7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-5wtjm\" (UID: \"b9661a3b-8a07-4a30-9893-20ab1d95a2a7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5wtjm"
Nov 02 13:39:46.521712 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:46.521695    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9661a3b-8a07-4a30-9893-20ab1d95a2a7-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-5wtjm\" (UID: \"b9661a3b-8a07-4a30-9893-20ab1d95a2a7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5wtjm"
Nov 02 13:39:46.522056 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:46.521726    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9661a3b-8a07-4a30-9893-20ab1d95a2a7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-5wtjm\" (UID: \"b9661a3b-8a07-4a30-9893-20ab1d95a2a7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5wtjm"
Nov 02 13:39:46.522056 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:46.521764    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9661a3b-8a07-4a30-9893-20ab1d95a2a7-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-5wtjm\" (UID: \"b9661a3b-8a07-4a30-9893-20ab1d95a2a7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5wtjm"
Nov 02 13:39:46.522163 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:46.522074    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9661a3b-8a07-4a30-9893-20ab1d95a2a7-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-5wtjm\" (UID: \"b9661a3b-8a07-4a30-9893-20ab1d95a2a7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5wtjm"
Nov 02 13:39:46.522213 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:46.522176    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9661a3b-8a07-4a30-9893-20ab1d95a2a7-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-5wtjm\" (UID: \"b9661a3b-8a07-4a30-9893-20ab1d95a2a7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5wtjm"
Nov 02 13:39:46.522274 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:46.522209    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9661a3b-8a07-4a30-9893-20ab1d95a2a7-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-5wtjm\" (UID: \"b9661a3b-8a07-4a30-9893-20ab1d95a2a7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5wtjm"
Nov 02 13:39:46.522416 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:46.522392    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9661a3b-8a07-4a30-9893-20ab1d95a2a7-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-5wtjm\" (UID: \"b9661a3b-8a07-4a30-9893-20ab1d95a2a7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5wtjm"
Nov 02 13:39:46.522557 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:46.522539    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9661a3b-8a07-4a30-9893-20ab1d95a2a7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-5wtjm\" (UID: \"b9661a3b-8a07-4a30-9893-20ab1d95a2a7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5wtjm"
Nov 02 13:39:46.524091 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:46.524069    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9661a3b-8a07-4a30-9893-20ab1d95a2a7-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-5wtjm\" (UID: \"b9661a3b-8a07-4a30-9893-20ab1d95a2a7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5wtjm"
Nov 02 13:39:46.524323 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:46.524303    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9661a3b-8a07-4a30-9893-20ab1d95a2a7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-5wtjm\" (UID: \"b9661a3b-8a07-4a30-9893-20ab1d95a2a7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5wtjm"
Nov 02 13:39:46.545589 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:46.545566    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjjd6\" (UniqueName: \"kubernetes.io/projected/b9661a3b-8a07-4a30-9893-20ab1d95a2a7-kube-api-access-bjjd6\") pod \"maas-default-gateway-openshift-default-5bc884dd87-5wtjm\" (UID: \"b9661a3b-8a07-4a30-9893-20ab1d95a2a7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5wtjm"
Nov 02 13:39:46.545700 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:46.545605    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9661a3b-8a07-4a30-9893-20ab1d95a2a7-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-5wtjm\" (UID: \"b9661a3b-8a07-4a30-9893-20ab1d95a2a7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5wtjm"
Nov 02 13:39:46.654123 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:46.654049    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5wtjm"
Nov 02 13:39:46.968591 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:46.968566    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5wtjm"]
Nov 02 13:39:46.969692 ip-10-0-128-132 kubenswrapper[4559]: W1102 13:39:46.969665    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9661a3b_8a07_4a30_9893_20ab1d95a2a7.slice/crio-d968fa1a6fbfb68fc3658e0ac0929b865029297560cfbd5a917552340eac6f99 WatchSource:0}: Error finding container d968fa1a6fbfb68fc3658e0ac0929b865029297560cfbd5a917552340eac6f99: Status 404 returned error can't find the container with id d968fa1a6fbfb68fc3658e0ac0929b865029297560cfbd5a917552340eac6f99
Nov 02 13:39:46.971860 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:46.971828    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 13:39:46.971973 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:46.971902    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 13:39:46.971973 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:46.971949    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 13:39:47.211327 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:47.211289    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5wtjm" event={"ID":"b9661a3b-8a07-4a30-9893-20ab1d95a2a7","Type":"ContainerStarted","Data":"14854a20e502bacba6cdf8cbfaa1c80dc71b5ca0c1736ea0a06e6de2b9e80bbb"}
Nov 02 13:39:47.211478 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:47.211344    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5wtjm" event={"ID":"b9661a3b-8a07-4a30-9893-20ab1d95a2a7","Type":"ContainerStarted","Data":"d968fa1a6fbfb68fc3658e0ac0929b865029297560cfbd5a917552340eac6f99"}
Nov 02 13:39:47.266811 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:47.266713    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5wtjm" podStartSLOduration=1.266699062 podStartE2EDuration="1.266699062s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:39:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:39:47.266300729 +0000 UTC m=+4383.521699837" watchObservedRunningTime="2026-11-02 13:39:47.266699062 +0000 UTC m=+4383.522098158"
Nov 02 13:39:47.655101 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:47.655062    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5wtjm"
Nov 02 13:39:48.656333 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:48.656297    4559 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-5wtjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.19:15021/healthz/ready\": dial tcp 10.133.1.19:15021: connect: connection refused" start-of-body=
Nov 02 13:39:48.656808 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:48.656355    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5wtjm" podUID="b9661a3b-8a07-4a30-9893-20ab1d95a2a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.19:15021/healthz/ready\": dial tcp 10.133.1.19:15021: connect: connection refused"
Nov 02 13:39:49.613497 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:49.613460    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"]
Nov 02 13:39:49.654611 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:49.654578    4559 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-5wtjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.19:15021/healthz/ready\": dial tcp 10.133.1.19:15021: connect: connection refused" start-of-body=
Nov 02 13:39:49.654796 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:49.654625    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5wtjm" podUID="b9661a3b-8a07-4a30-9893-20ab1d95a2a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.19:15021/healthz/ready\": dial tcp 10.133.1.19:15021: connect: connection refused"
Nov 02 13:39:50.654995 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:50.654957    4559 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-5wtjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.19:15021/healthz/ready\": dial tcp 10.133.1.19:15021: connect: connection refused" start-of-body=
Nov 02 13:39:50.655422 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:50.655011    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5wtjm" podUID="b9661a3b-8a07-4a30-9893-20ab1d95a2a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.19:15021/healthz/ready\": dial tcp 10.133.1.19:15021: connect: connection refused"
Nov 02 13:39:51.655019 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:51.654978    4559 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-5wtjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.19:15021/healthz/ready\": dial tcp 10.133.1.19:15021: connect: connection refused" start-of-body=
Nov 02 13:39:51.655426 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:51.655037    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5wtjm" podUID="b9661a3b-8a07-4a30-9893-20ab1d95a2a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.19:15021/healthz/ready\": dial tcp 10.133.1.19:15021: connect: connection refused"
Nov 02 13:39:52.655353 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:52.655315    4559 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-5wtjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.19:15021/healthz/ready\": dial tcp 10.133.1.19:15021: connect: connection refused" start-of-body=
Nov 02 13:39:52.655859 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:52.655376    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5wtjm" podUID="b9661a3b-8a07-4a30-9893-20ab1d95a2a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.19:15021/healthz/ready\": dial tcp 10.133.1.19:15021: connect: connection refused"
Nov 02 13:39:53.655226 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:53.655171    4559 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-5wtjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.19:15021/healthz/ready\": dial tcp 10.133.1.19:15021: connect: connection refused" start-of-body=
Nov 02 13:39:53.655553 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:53.655506    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5wtjm" podUID="b9661a3b-8a07-4a30-9893-20ab1d95a2a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.19:15021/healthz/ready\": dial tcp 10.133.1.19:15021: connect: connection refused"
Nov 02 13:39:54.654803 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:54.654765    4559 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-5wtjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.19:15021/healthz/ready\": dial tcp 10.133.1.19:15021: connect: connection refused" start-of-body=
Nov 02 13:39:54.655128 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:54.655082    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5wtjm" podUID="b9661a3b-8a07-4a30-9893-20ab1d95a2a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.19:15021/healthz/ready\": dial tcp 10.133.1.19:15021: connect: connection refused"
Nov 02 13:39:54.732029 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:54.731990    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"]
Nov 02 13:39:56.694919 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:56.694871    4559 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-5wtjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.19:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 13:39:56.695473 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:56.694945    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5wtjm" podUID="b9661a3b-8a07-4a30-9893-20ab1d95a2a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.19:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:39:57.736800 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:57.736762    4559 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-5wtjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.19:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 13:39:57.737297 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:57.736833    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5wtjm" podUID="b9661a3b-8a07-4a30-9893-20ab1d95a2a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.19:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:39:58.778801 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:58.778756    4559 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-5wtjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.19:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 13:39:58.779326 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:58.778855    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5wtjm" podUID="b9661a3b-8a07-4a30-9893-20ab1d95a2a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.19:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:39:59.825663 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:59.824728    4559 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-5wtjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.19:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 13:39:59.825663 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:39:59.824810    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5wtjm" podUID="b9661a3b-8a07-4a30-9893-20ab1d95a2a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.19:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:40:00.756664 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:40:00.755887    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"]
Nov 02 13:40:00.834792 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:40:00.834755    4559 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-5wtjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.19:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 13:40:00.835439 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:40:00.835384    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5wtjm" podUID="b9661a3b-8a07-4a30-9893-20ab1d95a2a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.19:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:40:01.654782 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:40:01.654744    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5wtjm"
Nov 02 13:40:01.836782 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:40:01.836745    4559 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-5wtjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.19:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 13:40:01.837210 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:40:01.836819    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5wtjm" podUID="b9661a3b-8a07-4a30-9893-20ab1d95a2a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.19:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:40:02.357915 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:40:02.357886    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-tj5gn"]
Nov 02 13:40:02.361593 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:40:02.361563    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-tj5gn"
Nov 02 13:40:02.391582 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:40:02.391553    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 13:40:02.391758 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:40:02.391580    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 13:40:02.391874 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:40:02.391855    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-45fzn\""
Nov 02 13:40:02.391945 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:40:02.391861    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 13:40:02.393655 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:40:02.393620    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-tj5gn"]
Nov 02 13:40:02.450027 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:40:02.450002    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9449234-e0f9-40ec-95c3-e3875fb23dd8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-tj5gn\" (UID: \"d9449234-e0f9-40ec-95c3-e3875fb23dd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-tj5gn"
Nov 02 13:40:02.450179 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:40:02.450144    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9z9sm\" (UniqueName: \"kubernetes.io/projected/d9449234-e0f9-40ec-95c3-e3875fb23dd8-kube-api-access-9z9sm\") pod \"maas-api-revoke-sub-f03629db-tj5gn\" (UID: \"d9449234-e0f9-40ec-95c3-e3875fb23dd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-tj5gn"
Nov 02 13:40:02.450179 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:40:02.450175    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9449234-e0f9-40ec-95c3-e3875fb23dd8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-tj5gn\" (UID: \"d9449234-e0f9-40ec-95c3-e3875fb23dd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-tj5gn"
Nov 02 13:40:02.551141 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:40:02.551117    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9449234-e0f9-40ec-95c3-e3875fb23dd8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-tj5gn\" (UID: \"d9449234-e0f9-40ec-95c3-e3875fb23dd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-tj5gn"
Nov 02 13:40:02.551291 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:40:02.551197    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9z9sm\" (UniqueName: \"kubernetes.io/projected/d9449234-e0f9-40ec-95c3-e3875fb23dd8-kube-api-access-9z9sm\") pod \"maas-api-revoke-sub-f03629db-tj5gn\" (UID: \"d9449234-e0f9-40ec-95c3-e3875fb23dd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-tj5gn"
Nov 02 13:40:02.551291 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:40:02.551227    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9449234-e0f9-40ec-95c3-e3875fb23dd8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-tj5gn\" (UID: \"d9449234-e0f9-40ec-95c3-e3875fb23dd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-tj5gn"
Nov 02 13:40:02.551855 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:40:02.551832    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9449234-e0f9-40ec-95c3-e3875fb23dd8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-tj5gn\" (UID: \"d9449234-e0f9-40ec-95c3-e3875fb23dd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-tj5gn"
Nov 02 13:40:02.553397 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:40:02.553379    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9449234-e0f9-40ec-95c3-e3875fb23dd8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-tj5gn\" (UID: \"d9449234-e0f9-40ec-95c3-e3875fb23dd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-tj5gn"
Nov 02 13:40:02.575279 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:40:02.575256    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9z9sm\" (UniqueName: \"kubernetes.io/projected/d9449234-e0f9-40ec-95c3-e3875fb23dd8-kube-api-access-9z9sm\") pod \"maas-api-revoke-sub-f03629db-tj5gn\" (UID: \"d9449234-e0f9-40ec-95c3-e3875fb23dd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-tj5gn"
Nov 02 13:40:02.675497 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:40:02.675469    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-tj5gn"
Nov 02 13:40:02.864349 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:40:02.864310    4559 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-5wtjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.19:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 13:40:02.864838 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:40:02.864386    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5wtjm" podUID="b9661a3b-8a07-4a30-9893-20ab1d95a2a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.19:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:40:02.867170 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:40:02.867144    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5wtjm"
Nov 02 13:40:02.867949 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:40:02.867933    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5wtjm"
Nov 02 13:40:02.992762 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:40:02.992730    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-tj5gn"]
Nov 02 13:40:02.993843 ip-10-0-128-132 kubenswrapper[4559]: W1102 13:40:02.993789    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9449234_e0f9_40ec_95c3_e3875fb23dd8.slice/crio-e198fb6df96db8a1f2f7f96e222982fdb7c3c040fe4cda39ac3647fdb47511ec WatchSource:0}: Error finding container e198fb6df96db8a1f2f7f96e222982fdb7c3c040fe4cda39ac3647fdb47511ec: Status 404 returned error can't find the container with id e198fb6df96db8a1f2f7f96e222982fdb7c3c040fe4cda39ac3647fdb47511ec
Nov 02 13:40:03.306127 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:40:03.306034    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-tj5gn" event={"ID":"d9449234-e0f9-40ec-95c3-e3875fb23dd8","Type":"ContainerStarted","Data":"8188f5ac2fda98b8d1a3fef2bc4d63a0cd2c13ca32dfbf1b1b3cedd0ee77624d"}
Nov 02 13:40:03.306127 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:40:03.306088    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-tj5gn" event={"ID":"d9449234-e0f9-40ec-95c3-e3875fb23dd8","Type":"ContainerStarted","Data":"e198fb6df96db8a1f2f7f96e222982fdb7c3c040fe4cda39ac3647fdb47511ec"}
Nov 02 13:40:03.348484 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:40:03.348430    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-tj5gn" podStartSLOduration=1.348413649 podStartE2EDuration="1.348413649s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:40:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:40:03.347764656 +0000 UTC m=+4399.603163751" watchObservedRunningTime="2026-11-02 13:40:03.348413649 +0000 UTC m=+4399.603812746"
Nov 02 13:40:04.312394 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:40:04.312359    4559 generic.go:356] "Generic (PLEG): container finished" podID="d9449234-e0f9-40ec-95c3-e3875fb23dd8" containerID="8188f5ac2fda98b8d1a3fef2bc4d63a0cd2c13ca32dfbf1b1b3cedd0ee77624d" exitCode=0
Nov 02 13:40:04.312864 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:40:04.312404    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-tj5gn" event={"ID":"d9449234-e0f9-40ec-95c3-e3875fb23dd8","Type":"ContainerDied","Data":"8188f5ac2fda98b8d1a3fef2bc4d63a0cd2c13ca32dfbf1b1b3cedd0ee77624d"}
Nov 02 13:40:04.792006 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:40:04.791973    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"]
Nov 02 13:40:05.452151 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:40:05.452124    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-tj5gn"
Nov 02 13:40:05.577811 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:40:05.577747    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9449234-e0f9-40ec-95c3-e3875fb23dd8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9449234-e0f9-40ec-95c3-e3875fb23dd8-maas-api-openshift-ca\") pod \"d9449234-e0f9-40ec-95c3-e3875fb23dd8\" (UID: \"d9449234-e0f9-40ec-95c3-e3875fb23dd8\") "
Nov 02 13:40:05.577944 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:40:05.577819    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9449234-e0f9-40ec-95c3-e3875fb23dd8-kube-api-access-9z9sm\" (UniqueName: \"kubernetes.io/projected/d9449234-e0f9-40ec-95c3-e3875fb23dd8-kube-api-access-9z9sm\") pod \"d9449234-e0f9-40ec-95c3-e3875fb23dd8\" (UID: \"d9449234-e0f9-40ec-95c3-e3875fb23dd8\") "
Nov 02 13:40:05.577944 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:40:05.577869    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d9449234-e0f9-40ec-95c3-e3875fb23dd8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9449234-e0f9-40ec-95c3-e3875fb23dd8-maas-api-xks-ca\") pod \"d9449234-e0f9-40ec-95c3-e3875fb23dd8\" (UID: \"d9449234-e0f9-40ec-95c3-e3875fb23dd8\") "
Nov 02 13:40:05.578081 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:40:05.578059    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9449234-e0f9-40ec-95c3-e3875fb23dd8-maas-api-openshift-ca" pod "d9449234-e0f9-40ec-95c3-e3875fb23dd8" (UID: "d9449234-e0f9-40ec-95c3-e3875fb23dd8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:40:05.579990 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:40:05.579953    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9449234-e0f9-40ec-95c3-e3875fb23dd8-maas-api-xks-ca" pod "d9449234-e0f9-40ec-95c3-e3875fb23dd8" (UID: "d9449234-e0f9-40ec-95c3-e3875fb23dd8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:40:05.580124 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:40:05.580005    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9449234-e0f9-40ec-95c3-e3875fb23dd8-kube-api-access-9z9sm" pod "d9449234-e0f9-40ec-95c3-e3875fb23dd8" (UID: "d9449234-e0f9-40ec-95c3-e3875fb23dd8"). InnerVolumeSpecName "kube-api-access-9z9sm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:40:05.679171 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:40:05.679143    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9449234-e0f9-40ec-95c3-e3875fb23dd8-maas-api-openshift-ca\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:40:05.679171 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:40:05.679168    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9z9sm\" (UniqueName: \"kubernetes.io/projected/d9449234-e0f9-40ec-95c3-e3875fb23dd8-kube-api-access-9z9sm\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:40:05.679351 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:40:05.679182    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9449234-e0f9-40ec-95c3-e3875fb23dd8-maas-api-xks-ca\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:40:06.323605 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:40:06.323577    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-tj5gn"
Nov 02 13:40:06.323807 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:40:06.323576    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-tj5gn" event={"ID":"d9449234-e0f9-40ec-95c3-e3875fb23dd8","Type":"ContainerDied","Data":"e198fb6df96db8a1f2f7f96e222982fdb7c3c040fe4cda39ac3647fdb47511ec"}
Nov 02 13:40:06.323807 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:40:06.323706    4559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e198fb6df96db8a1f2f7f96e222982fdb7c3c040fe4cda39ac3647fdb47511ec"
Nov 02 13:40:15.084322 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:40:15.084286    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqcmj" containerName="extractor"
Nov 02 13:40:18.516505 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:40:18.516474    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-2cp6n"]
Nov 02 13:40:18.517227 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:40:18.516906    4559 cpu_manager.go:414] "containerMap: removing container" podUID="d9449234-e0f9-40ec-95c3-e3875fb23dd8" containerName="revoke-keys"
Nov 02 13:40:18.517227 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:40:18.516918    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9449234-e0f9-40ec-95c3-e3875fb23dd8" containerName="revoke-keys"
Nov 02 13:40:18.517227 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:40:18.516989    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9449234-e0f9-40ec-95c3-e3875fb23dd8" containerName="revoke-keys"
Nov 02 13:40:18.520504 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:40:18.520470    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-2cp6n"
Nov 02 13:40:18.539901 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:40:18.539467    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-2cp6n"]
Nov 02 13:40:18.549862 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:40:18.549838    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 13:40:18.549997 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:40:18.549846    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 13:40:18.549997 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:40:18.549905    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-45fzn\""
Nov 02 13:40:18.550109 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:40:18.550049    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 13:40:18.631001 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:40:18.630975    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66be16cd-39a1-4845-92da-c2585a1777e4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-2cp6n\" (UID: \"66be16cd-39a1-4845-92da-c2585a1777e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-2cp6n"
Nov 02 13:40:18.631143 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:40:18.631021    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jf6xk\" (UniqueName: \"kubernetes.io/projected/66be16cd-39a1-4845-92da-c2585a1777e4-kube-api-access-jf6xk\") pod \"maas-api-revoke-sub-0165d7d2-2cp6n\" (UID: \"66be16cd-39a1-4845-92da-c2585a1777e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-2cp6n"
Nov 02 13:40:18.631143 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:40:18.631043    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66be16cd-39a1-4845-92da-c2585a1777e4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-2cp6n\" (UID: \"66be16cd-39a1-4845-92da-c2585a1777e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-2cp6n"
Nov 02 13:40:18.732596 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:40:18.732571    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66be16cd-39a1-4845-92da-c2585a1777e4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-2cp6n\" (UID: \"66be16cd-39a1-4845-92da-c2585a1777e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-2cp6n"
Nov 02 13:40:18.732767 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:40:18.732609    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jf6xk\" (UniqueName: \"kubernetes.io/projected/66be16cd-39a1-4845-92da-c2585a1777e4-kube-api-access-jf6xk\") pod \"maas-api-revoke-sub-0165d7d2-2cp6n\" (UID: \"66be16cd-39a1-4845-92da-c2585a1777e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-2cp6n"
Nov 02 13:40:18.732767 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:40:18.732634    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66be16cd-39a1-4845-92da-c2585a1777e4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-2cp6n\" (UID: \"66be16cd-39a1-4845-92da-c2585a1777e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-2cp6n"
Nov 02 13:40:18.733490 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:40:18.733470    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66be16cd-39a1-4845-92da-c2585a1777e4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-2cp6n\" (UID: \"66be16cd-39a1-4845-92da-c2585a1777e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-2cp6n"
Nov 02 13:40:18.734999 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:40:18.734977    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66be16cd-39a1-4845-92da-c2585a1777e4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-2cp6n\" (UID: \"66be16cd-39a1-4845-92da-c2585a1777e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-2cp6n"
Nov 02 13:40:18.756438 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:40:18.756407    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jf6xk\" (UniqueName: \"kubernetes.io/projected/66be16cd-39a1-4845-92da-c2585a1777e4-kube-api-access-jf6xk\") pod \"maas-api-revoke-sub-0165d7d2-2cp6n\" (UID: \"66be16cd-39a1-4845-92da-c2585a1777e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-2cp6n"
Nov 02 13:40:18.833957 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:40:18.833907    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-2cp6n"
Nov 02 13:40:19.145092 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:40:19.145060    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-2cp6n"]
Nov 02 13:40:19.145721 ip-10-0-128-132 kubenswrapper[4559]: W1102 13:40:19.145696    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66be16cd_39a1_4845_92da_c2585a1777e4.slice/crio-49fb7433c2d79dd1254d63b1d02000e9ac35d1cc0f32096d989b8186c968e7fe WatchSource:0}: Error finding container 49fb7433c2d79dd1254d63b1d02000e9ac35d1cc0f32096d989b8186c968e7fe: Status 404 returned error can't find the container with id 49fb7433c2d79dd1254d63b1d02000e9ac35d1cc0f32096d989b8186c968e7fe
Nov 02 13:40:19.385205 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:40:19.385168    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-2cp6n" event={"ID":"66be16cd-39a1-4845-92da-c2585a1777e4","Type":"ContainerStarted","Data":"49fb7433c2d79dd1254d63b1d02000e9ac35d1cc0f32096d989b8186c968e7fe"}
Nov 02 13:40:20.391404 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:40:20.391371    4559 generic.go:356] "Generic (PLEG): container finished" podID="66be16cd-39a1-4845-92da-c2585a1777e4" containerID="8022b6826b8b50b4e191176b75d723d30538f52af739d90fab546273494d1b87" exitCode=0
Nov 02 13:40:20.391822 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:40:20.391458    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-2cp6n" event={"ID":"66be16cd-39a1-4845-92da-c2585a1777e4","Type":"ContainerDied","Data":"8022b6826b8b50b4e191176b75d723d30538f52af739d90fab546273494d1b87"}
Nov 02 13:40:21.522714 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:40:21.522686    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-2cp6n"
Nov 02 13:40:21.663424 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:40:21.663329    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66be16cd-39a1-4845-92da-c2585a1777e4-kube-api-access-jf6xk\" (UniqueName: \"kubernetes.io/projected/66be16cd-39a1-4845-92da-c2585a1777e4-kube-api-access-jf6xk\") pod \"66be16cd-39a1-4845-92da-c2585a1777e4\" (UID: \"66be16cd-39a1-4845-92da-c2585a1777e4\") "
Nov 02 13:40:21.663424 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:40:21.663381    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/66be16cd-39a1-4845-92da-c2585a1777e4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66be16cd-39a1-4845-92da-c2585a1777e4-maas-api-xks-ca\") pod \"66be16cd-39a1-4845-92da-c2585a1777e4\" (UID: \"66be16cd-39a1-4845-92da-c2585a1777e4\") "
Nov 02 13:40:21.663679 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:40:21.663489    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66be16cd-39a1-4845-92da-c2585a1777e4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66be16cd-39a1-4845-92da-c2585a1777e4-maas-api-openshift-ca\") pod \"66be16cd-39a1-4845-92da-c2585a1777e4\" (UID: \"66be16cd-39a1-4845-92da-c2585a1777e4\") "
Nov 02 13:40:21.663867 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:40:21.663844    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66be16cd-39a1-4845-92da-c2585a1777e4-maas-api-openshift-ca" pod "66be16cd-39a1-4845-92da-c2585a1777e4" (UID: "66be16cd-39a1-4845-92da-c2585a1777e4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:40:21.665314 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:40:21.665294    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66be16cd-39a1-4845-92da-c2585a1777e4-maas-api-xks-ca" pod "66be16cd-39a1-4845-92da-c2585a1777e4" (UID: "66be16cd-39a1-4845-92da-c2585a1777e4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:40:21.665430 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:40:21.665414    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66be16cd-39a1-4845-92da-c2585a1777e4-kube-api-access-jf6xk" pod "66be16cd-39a1-4845-92da-c2585a1777e4" (UID: "66be16cd-39a1-4845-92da-c2585a1777e4"). InnerVolumeSpecName "kube-api-access-jf6xk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:40:21.764609 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:40:21.764586    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66be16cd-39a1-4845-92da-c2585a1777e4-maas-api-openshift-ca\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:40:21.764609 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:40:21.764608    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jf6xk\" (UniqueName: \"kubernetes.io/projected/66be16cd-39a1-4845-92da-c2585a1777e4-kube-api-access-jf6xk\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:40:21.764609 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:40:21.764618    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66be16cd-39a1-4845-92da-c2585a1777e4-maas-api-xks-ca\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:40:22.401050 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:40:22.401017    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-2cp6n"
Nov 02 13:40:22.401260 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:40:22.401016    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-2cp6n" event={"ID":"66be16cd-39a1-4845-92da-c2585a1777e4","Type":"ContainerDied","Data":"49fb7433c2d79dd1254d63b1d02000e9ac35d1cc0f32096d989b8186c968e7fe"}
Nov 02 13:40:22.401260 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:40:22.401146    4559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="49fb7433c2d79dd1254d63b1d02000e9ac35d1cc0f32096d989b8186c968e7fe"
Nov 02 13:40:29.476453 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:40:29.476421    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-7zsfz"]
Nov 02 13:40:29.476854 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:40:29.476815    4559 cpu_manager.go:414] "containerMap: removing container" podUID="66be16cd-39a1-4845-92da-c2585a1777e4" containerName="revoke-keys"
Nov 02 13:40:29.476854 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:40:29.476827    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66be16cd-39a1-4845-92da-c2585a1777e4" containerName="revoke-keys"
Nov 02 13:40:29.476935 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:40:29.476902    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="66be16cd-39a1-4845-92da-c2585a1777e4" containerName="revoke-keys"
Nov 02 13:40:29.480216 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:40:29.480197    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-7zsfz"
Nov 02 13:40:29.499315 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:40:29.499292    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-7zsfz"]
Nov 02 13:40:29.510229 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:40:29.510206    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 13:40:29.510229 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:40:29.510218    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 13:40:29.510392 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:40:29.510230    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 13:40:29.510392 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:40:29.510224    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-45fzn\""
Nov 02 13:40:29.532804 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:40:29.532780    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d96d563-ada2-46d1-a170-866dc16679ee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-7zsfz\" (UID: \"6d96d563-ada2-46d1-a170-866dc16679ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-7zsfz"
Nov 02 13:40:29.532924 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:40:29.532808    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d96d563-ada2-46d1-a170-866dc16679ee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-7zsfz\" (UID: \"6d96d563-ada2-46d1-a170-866dc16679ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-7zsfz"
Nov 02 13:40:29.533030 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:40:29.533010    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kvm7\" (UniqueName: \"kubernetes.io/projected/6d96d563-ada2-46d1-a170-866dc16679ee-kube-api-access-7kvm7\") pod \"maas-api-revoke-sub-6b19338f-7zsfz\" (UID: \"6d96d563-ada2-46d1-a170-866dc16679ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-7zsfz"
Nov 02 13:40:29.633599 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:40:29.633560    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kvm7\" (UniqueName: \"kubernetes.io/projected/6d96d563-ada2-46d1-a170-866dc16679ee-kube-api-access-7kvm7\") pod \"maas-api-revoke-sub-6b19338f-7zsfz\" (UID: \"6d96d563-ada2-46d1-a170-866dc16679ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-7zsfz"
Nov 02 13:40:29.633753 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:40:29.633666    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d96d563-ada2-46d1-a170-866dc16679ee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-7zsfz\" (UID: \"6d96d563-ada2-46d1-a170-866dc16679ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-7zsfz"
Nov 02 13:40:29.633753 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:40:29.633694    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d96d563-ada2-46d1-a170-866dc16679ee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-7zsfz\" (UID: \"6d96d563-ada2-46d1-a170-866dc16679ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-7zsfz"
Nov 02 13:40:29.634378 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:40:29.634355    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d96d563-ada2-46d1-a170-866dc16679ee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-7zsfz\" (UID: \"6d96d563-ada2-46d1-a170-866dc16679ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-7zsfz"
Nov 02 13:40:29.636181 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:40:29.636157    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d96d563-ada2-46d1-a170-866dc16679ee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-7zsfz\" (UID: \"6d96d563-ada2-46d1-a170-866dc16679ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-7zsfz"
Nov 02 13:40:29.657493 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:40:29.657473    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kvm7\" (UniqueName: \"kubernetes.io/projected/6d96d563-ada2-46d1-a170-866dc16679ee-kube-api-access-7kvm7\") pod \"maas-api-revoke-sub-6b19338f-7zsfz\" (UID: \"6d96d563-ada2-46d1-a170-866dc16679ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-7zsfz"
Nov 02 13:40:29.792673 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:40:29.792586    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-7zsfz"
Nov 02 13:40:30.092592 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:40:30.092561    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-7zsfz"]
Nov 02 13:40:30.093425 ip-10-0-128-132 kubenswrapper[4559]: W1102 13:40:30.093392    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d96d563_ada2_46d1_a170_866dc16679ee.slice/crio-66d255c3a310c9bb08ad213c9cd5b3dae50dae7d30be17a3cfe405855259114d WatchSource:0}: Error finding container 66d255c3a310c9bb08ad213c9cd5b3dae50dae7d30be17a3cfe405855259114d: Status 404 returned error can't find the container with id 66d255c3a310c9bb08ad213c9cd5b3dae50dae7d30be17a3cfe405855259114d
Nov 02 13:40:30.438450 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:40:30.438409    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-7zsfz" event={"ID":"6d96d563-ada2-46d1-a170-866dc16679ee","Type":"ContainerStarted","Data":"a4bd881119ae63c941fe0955d4d3c4a10c1c00c9e9247c1f38c9fb5ae2a05e86"}
Nov 02 13:40:30.438599 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:40:30.438463    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-7zsfz" event={"ID":"6d96d563-ada2-46d1-a170-866dc16679ee","Type":"ContainerStarted","Data":"66d255c3a310c9bb08ad213c9cd5b3dae50dae7d30be17a3cfe405855259114d"}
Nov 02 13:40:30.482618 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:40:30.482577    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-7zsfz" podStartSLOduration=1.482564318 podStartE2EDuration="1.482564318s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:40:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:40:30.480697228 +0000 UTC m=+4426.736096322" watchObservedRunningTime="2026-11-02 13:40:30.482564318 +0000 UTC m=+4426.737963406"
Nov 02 13:40:31.444138 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:40:31.444104    4559 generic.go:356] "Generic (PLEG): container finished" podID="6d96d563-ada2-46d1-a170-866dc16679ee" containerID="a4bd881119ae63c941fe0955d4d3c4a10c1c00c9e9247c1f38c9fb5ae2a05e86" exitCode=0
Nov 02 13:40:31.444323 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:40:31.444191    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-7zsfz" event={"ID":"6d96d563-ada2-46d1-a170-866dc16679ee","Type":"ContainerDied","Data":"a4bd881119ae63c941fe0955d4d3c4a10c1c00c9e9247c1f38c9fb5ae2a05e86"}
Nov 02 13:40:32.591839 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:40:32.591814    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-7zsfz"
Nov 02 13:40:32.657854 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:40:32.657824    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d96d563-ada2-46d1-a170-866dc16679ee-kube-api-access-7kvm7\" (UniqueName: \"kubernetes.io/projected/6d96d563-ada2-46d1-a170-866dc16679ee-kube-api-access-7kvm7\") pod \"6d96d563-ada2-46d1-a170-866dc16679ee\" (UID: \"6d96d563-ada2-46d1-a170-866dc16679ee\") "
Nov 02 13:40:32.657993 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:40:32.657867    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6d96d563-ada2-46d1-a170-866dc16679ee-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d96d563-ada2-46d1-a170-866dc16679ee-maas-api-xks-ca\") pod \"6d96d563-ada2-46d1-a170-866dc16679ee\" (UID: \"6d96d563-ada2-46d1-a170-866dc16679ee\") "
Nov 02 13:40:32.657993 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:40:32.657940    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d96d563-ada2-46d1-a170-866dc16679ee-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d96d563-ada2-46d1-a170-866dc16679ee-maas-api-openshift-ca\") pod \"6d96d563-ada2-46d1-a170-866dc16679ee\" (UID: \"6d96d563-ada2-46d1-a170-866dc16679ee\") "
Nov 02 13:40:32.658381 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:40:32.658349    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d96d563-ada2-46d1-a170-866dc16679ee-maas-api-openshift-ca" pod "6d96d563-ada2-46d1-a170-866dc16679ee" (UID: "6d96d563-ada2-46d1-a170-866dc16679ee"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:40:32.660073 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:40:32.660050    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d96d563-ada2-46d1-a170-866dc16679ee-maas-api-xks-ca" pod "6d96d563-ada2-46d1-a170-866dc16679ee" (UID: "6d96d563-ada2-46d1-a170-866dc16679ee"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:40:32.660717 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:40:32.660691    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d96d563-ada2-46d1-a170-866dc16679ee-kube-api-access-7kvm7" pod "6d96d563-ada2-46d1-a170-866dc16679ee" (UID: "6d96d563-ada2-46d1-a170-866dc16679ee"). InnerVolumeSpecName "kube-api-access-7kvm7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:40:32.759445 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:40:32.759380    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7kvm7\" (UniqueName: \"kubernetes.io/projected/6d96d563-ada2-46d1-a170-866dc16679ee-kube-api-access-7kvm7\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:40:32.759445 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:40:32.759411    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d96d563-ada2-46d1-a170-866dc16679ee-maas-api-xks-ca\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:40:32.759445 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:40:32.759425    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d96d563-ada2-46d1-a170-866dc16679ee-maas-api-openshift-ca\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:40:33.453798 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:40:33.453766    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-7zsfz"
Nov 02 13:40:33.453966 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:40:33.453765    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-7zsfz" event={"ID":"6d96d563-ada2-46d1-a170-866dc16679ee","Type":"ContainerDied","Data":"66d255c3a310c9bb08ad213c9cd5b3dae50dae7d30be17a3cfe405855259114d"}
Nov 02 13:40:33.453966 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:40:33.453890    4559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="66d255c3a310c9bb08ad213c9cd5b3dae50dae7d30be17a3cfe405855259114d"
Nov 02 13:40:40.800186 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:40:40.800150    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"]
Nov 02 13:40:41.866280 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:40:41.866238    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-prio-pool-6b4a85cd-kserve-56f8b8b95b-mljbs"]
Nov 02 13:40:41.866793 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:40:41.866697    4559 cpu_manager.go:414] "containerMap: removing container" podUID="6d96d563-ada2-46d1-a170-866dc16679ee" containerName="revoke-keys"
Nov 02 13:40:41.866793 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:40:41.866714    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d96d563-ada2-46d1-a170-866dc16679ee" containerName="revoke-keys"
Nov 02 13:40:41.866928 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:40:41.866805    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d96d563-ada2-46d1-a170-866dc16679ee" containerName="revoke-keys"
Nov 02 13:40:41.870119 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:40:41.870098    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-6b4a85cd-kserve-56f8b8b95b-mljbs"
Nov 02 13:40:41.905519 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:40:41.905488    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-prio-pool-6b4a85cd-kserve-self-signed-certs\""
Nov 02 13:40:41.970064 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:40:41.970031    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-prio-pool-6b4a85cd-kserve-56f8b8b95b-mljbs"]
Nov 02 13:40:42.061924 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:40:42.061885    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b07cd281-f05d-4df7-a835-d1cc723e12de-home\") pod \"e2e-prio-pool-6b4a85cd-kserve-56f8b8b95b-mljbs\" (UID: \"b07cd281-f05d-4df7-a835-d1cc723e12de\") " pod="llm/e2e-prio-pool-6b4a85cd-kserve-56f8b8b95b-mljbs"
Nov 02 13:40:42.062125 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:40:42.061960    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nk9dz\" (UniqueName: \"kubernetes.io/projected/b07cd281-f05d-4df7-a835-d1cc723e12de-kube-api-access-nk9dz\") pod \"e2e-prio-pool-6b4a85cd-kserve-56f8b8b95b-mljbs\" (UID: \"b07cd281-f05d-4df7-a835-d1cc723e12de\") " pod="llm/e2e-prio-pool-6b4a85cd-kserve-56f8b8b95b-mljbs"
Nov 02 13:40:42.062125 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:40:42.062058    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b07cd281-f05d-4df7-a835-d1cc723e12de-tmp-dir\") pod \"e2e-prio-pool-6b4a85cd-kserve-56f8b8b95b-mljbs\" (UID: \"b07cd281-f05d-4df7-a835-d1cc723e12de\") " pod="llm/e2e-prio-pool-6b4a85cd-kserve-56f8b8b95b-mljbs"
Nov 02 13:40:42.062271 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:40:42.062147    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b07cd281-f05d-4df7-a835-d1cc723e12de-model-cache\") pod \"e2e-prio-pool-6b4a85cd-kserve-56f8b8b95b-mljbs\" (UID: \"b07cd281-f05d-4df7-a835-d1cc723e12de\") " pod="llm/e2e-prio-pool-6b4a85cd-kserve-56f8b8b95b-mljbs"
Nov 02 13:40:42.062271 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:40:42.062253    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b07cd281-f05d-4df7-a835-d1cc723e12de-tls-certs\") pod \"e2e-prio-pool-6b4a85cd-kserve-56f8b8b95b-mljbs\" (UID: \"b07cd281-f05d-4df7-a835-d1cc723e12de\") " pod="llm/e2e-prio-pool-6b4a85cd-kserve-56f8b8b95b-mljbs"
Nov 02 13:40:42.062403 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:40:42.062297    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b07cd281-f05d-4df7-a835-d1cc723e12de-dshm\") pod \"e2e-prio-pool-6b4a85cd-kserve-56f8b8b95b-mljbs\" (UID: \"b07cd281-f05d-4df7-a835-d1cc723e12de\") " pod="llm/e2e-prio-pool-6b4a85cd-kserve-56f8b8b95b-mljbs"
Nov 02 13:40:42.163753 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:40:42.163506    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b07cd281-f05d-4df7-a835-d1cc723e12de-tls-certs\") pod \"e2e-prio-pool-6b4a85cd-kserve-56f8b8b95b-mljbs\" (UID: \"b07cd281-f05d-4df7-a835-d1cc723e12de\") " pod="llm/e2e-prio-pool-6b4a85cd-kserve-56f8b8b95b-mljbs"
Nov 02 13:40:42.163753 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:40:42.163725    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b07cd281-f05d-4df7-a835-d1cc723e12de-dshm\") pod \"e2e-prio-pool-6b4a85cd-kserve-56f8b8b95b-mljbs\" (UID: \"b07cd281-f05d-4df7-a835-d1cc723e12de\") " pod="llm/e2e-prio-pool-6b4a85cd-kserve-56f8b8b95b-mljbs"
Nov 02 13:40:42.163947 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:40:42.163787    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b07cd281-f05d-4df7-a835-d1cc723e12de-home\") pod \"e2e-prio-pool-6b4a85cd-kserve-56f8b8b95b-mljbs\" (UID: \"b07cd281-f05d-4df7-a835-d1cc723e12de\") " pod="llm/e2e-prio-pool-6b4a85cd-kserve-56f8b8b95b-mljbs"
Nov 02 13:40:42.163947 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:40:42.163847    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nk9dz\" (UniqueName: \"kubernetes.io/projected/b07cd281-f05d-4df7-a835-d1cc723e12de-kube-api-access-nk9dz\") pod \"e2e-prio-pool-6b4a85cd-kserve-56f8b8b95b-mljbs\" (UID: \"b07cd281-f05d-4df7-a835-d1cc723e12de\") " pod="llm/e2e-prio-pool-6b4a85cd-kserve-56f8b8b95b-mljbs"
Nov 02 13:40:42.163947 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:40:42.163878    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b07cd281-f05d-4df7-a835-d1cc723e12de-tmp-dir\") pod \"e2e-prio-pool-6b4a85cd-kserve-56f8b8b95b-mljbs\" (UID: \"b07cd281-f05d-4df7-a835-d1cc723e12de\") " pod="llm/e2e-prio-pool-6b4a85cd-kserve-56f8b8b95b-mljbs"
Nov 02 13:40:42.164048 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:40:42.163951    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b07cd281-f05d-4df7-a835-d1cc723e12de-model-cache\") pod \"e2e-prio-pool-6b4a85cd-kserve-56f8b8b95b-mljbs\" (UID: \"b07cd281-f05d-4df7-a835-d1cc723e12de\") " pod="llm/e2e-prio-pool-6b4a85cd-kserve-56f8b8b95b-mljbs"
Nov 02 13:40:42.164253 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:40:42.164227    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b07cd281-f05d-4df7-a835-d1cc723e12de-home\") pod \"e2e-prio-pool-6b4a85cd-kserve-56f8b8b95b-mljbs\" (UID: \"b07cd281-f05d-4df7-a835-d1cc723e12de\") " pod="llm/e2e-prio-pool-6b4a85cd-kserve-56f8b8b95b-mljbs"
Nov 02 13:40:42.164454 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:40:42.164431    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b07cd281-f05d-4df7-a835-d1cc723e12de-tmp-dir\") pod \"e2e-prio-pool-6b4a85cd-kserve-56f8b8b95b-mljbs\" (UID: \"b07cd281-f05d-4df7-a835-d1cc723e12de\") " pod="llm/e2e-prio-pool-6b4a85cd-kserve-56f8b8b95b-mljbs"
Nov 02 13:40:42.164523 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:40:42.164473    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b07cd281-f05d-4df7-a835-d1cc723e12de-model-cache\") pod \"e2e-prio-pool-6b4a85cd-kserve-56f8b8b95b-mljbs\" (UID: \"b07cd281-f05d-4df7-a835-d1cc723e12de\") " pod="llm/e2e-prio-pool-6b4a85cd-kserve-56f8b8b95b-mljbs"
Nov 02 13:40:42.166182 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:40:42.166164    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b07cd281-f05d-4df7-a835-d1cc723e12de-dshm\") pod \"e2e-prio-pool-6b4a85cd-kserve-56f8b8b95b-mljbs\" (UID: \"b07cd281-f05d-4df7-a835-d1cc723e12de\") " pod="llm/e2e-prio-pool-6b4a85cd-kserve-56f8b8b95b-mljbs"
Nov 02 13:40:42.166445 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:40:42.166426    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b07cd281-f05d-4df7-a835-d1cc723e12de-tls-certs\") pod \"e2e-prio-pool-6b4a85cd-kserve-56f8b8b95b-mljbs\" (UID: \"b07cd281-f05d-4df7-a835-d1cc723e12de\") " pod="llm/e2e-prio-pool-6b4a85cd-kserve-56f8b8b95b-mljbs"
Nov 02 13:40:42.218661 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:40:42.216264    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nk9dz\" (UniqueName: \"kubernetes.io/projected/b07cd281-f05d-4df7-a835-d1cc723e12de-kube-api-access-nk9dz\") pod \"e2e-prio-pool-6b4a85cd-kserve-56f8b8b95b-mljbs\" (UID: \"b07cd281-f05d-4df7-a835-d1cc723e12de\") " pod="llm/e2e-prio-pool-6b4a85cd-kserve-56f8b8b95b-mljbs"
Nov 02 13:40:42.264781 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:40:42.264748    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-prio-pool-6b4a85cd-kserve-router-scheduler-7d966f676c-ns2mg"]
Nov 02 13:40:42.269281 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:40:42.269257    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-6b4a85cd-kserve-router-scheduler-7d966f676c-ns2mg"
Nov 02 13:40:42.286956 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:40:42.286929    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-prio-pool-6b4a85cd-kserve-router-scheduler-7d966f676c-ns2mg"]
Nov 02 13:40:42.297035 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:40:42.287945    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-prio-pool-6b4a85cd-epp-sa-dockercfg-lqc56\""
Nov 02 13:40:42.468393 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:40:42.468358    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbn47\" (UniqueName: \"kubernetes.io/projected/28cc88fc-5e5e-4b13-a883-17e70e36e4e9-kube-api-access-nbn47\") pod \"e2e-prio-pool-6b4a85cd-kserve-router-scheduler-7d966f676c-ns2mg\" (UID: \"28cc88fc-5e5e-4b13-a883-17e70e36e4e9\") " pod="llm/e2e-prio-pool-6b4a85cd-kserve-router-scheduler-7d966f676c-ns2mg"
Nov 02 13:40:42.468393 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:40:42.468405    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28cc88fc-5e5e-4b13-a883-17e70e36e4e9-tls-certs\") pod \"e2e-prio-pool-6b4a85cd-kserve-router-scheduler-7d966f676c-ns2mg\" (UID: \"28cc88fc-5e5e-4b13-a883-17e70e36e4e9\") " pod="llm/e2e-prio-pool-6b4a85cd-kserve-router-scheduler-7d966f676c-ns2mg"
Nov 02 13:40:42.482357 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:40:42.482341    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-6b4a85cd-kserve-56f8b8b95b-mljbs"
Nov 02 13:40:42.569089 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:40:42.569052    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbn47\" (UniqueName: \"kubernetes.io/projected/28cc88fc-5e5e-4b13-a883-17e70e36e4e9-kube-api-access-nbn47\") pod \"e2e-prio-pool-6b4a85cd-kserve-router-scheduler-7d966f676c-ns2mg\" (UID: \"28cc88fc-5e5e-4b13-a883-17e70e36e4e9\") " pod="llm/e2e-prio-pool-6b4a85cd-kserve-router-scheduler-7d966f676c-ns2mg"
Nov 02 13:40:42.569314 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:40:42.569289    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28cc88fc-5e5e-4b13-a883-17e70e36e4e9-tls-certs\") pod \"e2e-prio-pool-6b4a85cd-kserve-router-scheduler-7d966f676c-ns2mg\" (UID: \"28cc88fc-5e5e-4b13-a883-17e70e36e4e9\") " pod="llm/e2e-prio-pool-6b4a85cd-kserve-router-scheduler-7d966f676c-ns2mg"
Nov 02 13:40:42.572783 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:40:42.572760    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28cc88fc-5e5e-4b13-a883-17e70e36e4e9-tls-certs\") pod \"e2e-prio-pool-6b4a85cd-kserve-router-scheduler-7d966f676c-ns2mg\" (UID: \"28cc88fc-5e5e-4b13-a883-17e70e36e4e9\") " pod="llm/e2e-prio-pool-6b4a85cd-kserve-router-scheduler-7d966f676c-ns2mg"
Nov 02 13:40:42.606041 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:40:42.605971    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbn47\" (UniqueName: \"kubernetes.io/projected/28cc88fc-5e5e-4b13-a883-17e70e36e4e9-kube-api-access-nbn47\") pod \"e2e-prio-pool-6b4a85cd-kserve-router-scheduler-7d966f676c-ns2mg\" (UID: \"28cc88fc-5e5e-4b13-a883-17e70e36e4e9\") " pod="llm/e2e-prio-pool-6b4a85cd-kserve-router-scheduler-7d966f676c-ns2mg"
Nov 02 13:40:42.898584 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:40:42.898264    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-6b4a85cd-kserve-router-scheduler-7d966f676c-ns2mg"
Nov 02 13:40:42.933154 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:40:42.933088    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-prio-pool-6b4a85cd-kserve-56f8b8b95b-mljbs"]
Nov 02 13:40:42.942298 ip-10-0-128-132 kubenswrapper[4559]: W1102 13:40:42.942246    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb07cd281_f05d_4df7_a835_d1cc723e12de.slice/crio-ccc04248131b8af0fbe5331cae5bc48b3421b8095ab1f66f5be6f911415822da WatchSource:0}: Error finding container ccc04248131b8af0fbe5331cae5bc48b3421b8095ab1f66f5be6f911415822da: Status 404 returned error can't find the container with id ccc04248131b8af0fbe5331cae5bc48b3421b8095ab1f66f5be6f911415822da
Nov 02 13:40:43.221003 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:40:43.220978    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-prio-pool-6b4a85cd-kserve-router-scheduler-7d966f676c-ns2mg"]
Nov 02 13:40:43.222594 ip-10-0-128-132 kubenswrapper[4559]: W1102 13:40:43.222567    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28cc88fc_5e5e_4b13_a883_17e70e36e4e9.slice/crio-37c0cdf05ec007683ca0b1f09f5548f7e7b41cee43fe44d3327527b584b652a2 WatchSource:0}: Error finding container 37c0cdf05ec007683ca0b1f09f5548f7e7b41cee43fe44d3327527b584b652a2: Status 404 returned error can't find the container with id 37c0cdf05ec007683ca0b1f09f5548f7e7b41cee43fe44d3327527b584b652a2
Nov 02 13:40:43.570770 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:40:43.570678    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-6b4a85cd-kserve-router-scheduler-7d966f676c-ns2mg" event={"ID":"28cc88fc-5e5e-4b13-a883-17e70e36e4e9","Type":"ContainerStarted","Data":"37c0cdf05ec007683ca0b1f09f5548f7e7b41cee43fe44d3327527b584b652a2"}
Nov 02 13:40:43.572254 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:40:43.572220    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-6b4a85cd-kserve-56f8b8b95b-mljbs" event={"ID":"b07cd281-f05d-4df7-a835-d1cc723e12de","Type":"ContainerStarted","Data":"8e2f6dc5d37738b762178b6b4fd001a5f8cf2269f21b5c7517620cade9f1336e"}
Nov 02 13:40:43.572254 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:40:43.572253    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-6b4a85cd-kserve-56f8b8b95b-mljbs" event={"ID":"b07cd281-f05d-4df7-a835-d1cc723e12de","Type":"ContainerStarted","Data":"ccc04248131b8af0fbe5331cae5bc48b3421b8095ab1f66f5be6f911415822da"}
Nov 02 13:40:43.616024 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:40:43.615978    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-prio-pool-6b4a85cd-kserve-56f8b8b95b-mljbs" podStartSLOduration=2.615966119 podStartE2EDuration="2.615966119s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:40:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:40:43.614370032 +0000 UTC m=+4439.869769128" watchObservedRunningTime="2026-11-02 13:40:43.615966119 +0000 UTC m=+4439.871365214"
Nov 02 13:40:44.482981 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:40:44.482939    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-prio-pool-6b4a85cd-kserve-56f8b8b95b-mljbs"
Nov 02 13:40:45.110342 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:40:45.110247    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqcmj" containerName="extractor"
Nov 02 13:40:45.589965 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:40:45.589740    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-6b4a85cd-kserve-router-scheduler-7d966f676c-ns2mg" event={"ID":"28cc88fc-5e5e-4b13-a883-17e70e36e4e9","Type":"ContainerStarted","Data":"425d1d421ff793525fdca92849d28810f93351cbf8a832c18977af14ef23486f"}
Nov 02 13:40:45.589965 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:40:45.589840    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-prio-pool-6b4a85cd-kserve-router-scheduler-7d966f676c-ns2mg"
Nov 02 13:40:45.636820 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:40:45.636762    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-prio-pool-6b4a85cd-kserve-router-scheduler-7d966f676c-ns2mg" podStartSLOduration=2.089159905 podStartE2EDuration="3.636742729s" totalImagesPullingTime="1.547582824s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:40:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:40:45.635410065 +0000 UTC m=+4441.890809161" watchObservedRunningTime="2026-11-02 13:40:45.636742729 +0000 UTC m=+4441.892141824"
Nov 02 13:40:48.614156 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:40:48.614122    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"]
Nov 02 13:40:52.483440 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:40:52.483400    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-prio-pool-6b4a85cd-kserve-56f8b8b95b-mljbs"
Nov 02 13:40:52.496014 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:40:52.495989    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-prio-pool-6b4a85cd-kserve-56f8b8b95b-mljbs"
Nov 02 13:40:52.673486 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:40:52.673461    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-prio-pool-6b4a85cd-kserve-56f8b8b95b-mljbs"
Nov 02 13:41:15.085719 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:41:15.085681    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqcmj" containerName="extractor"
Nov 02 13:41:16.608927 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:41:16.608894    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-prio-pool-6b4a85cd-kserve-router-scheduler-7d966f676c-ns2mg"
Nov 02 13:41:25.129098 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:41:25.129062    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"]
Nov 02 13:41:44.104286 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:41:44.104253    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"]
Nov 02 13:41:44.388079 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:41:44.387984    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-prio-pool-6b4a85cd-kserve-56f8b8b95b-mljbs"]
Nov 02 13:41:44.391131 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:41:44.391100    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-prio-pool-6b4a85cd-kserve-router-scheduler-7d966f676c-ns2mg"]
Nov 02 13:41:45.084784 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:41:45.084749    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqcmj" containerName="extractor"
Nov 02 13:41:46.609303 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:41:46.609241    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-6b4a85cd-kserve-router-scheduler-7d966f676c-ns2mg" podUID="28cc88fc-5e5e-4b13-a883-17e70e36e4e9" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 13:41:51.192130 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:41:51.192093    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"]
Nov 02 13:41:55.289463 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:41:55.289426    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"]
Nov 02 13:41:56.620816 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:41:56.620760    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-6b4a85cd-kserve-router-scheduler-7d966f676c-ns2mg" podUID="28cc88fc-5e5e-4b13-a883-17e70e36e4e9" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 13:41:59.409500 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:41:59.409448    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-prio-pool-6b4a85cd-kserve-56f8b8b95b-mljbs" podUID="b07cd281-f05d-4df7-a835-d1cc723e12de" containerName="main" containerID="cri-o://8e2f6dc5d37738b762178b6b4fd001a5f8cf2269f21b5c7517620cade9f1336e" gracePeriod=45
Nov 02 13:41:59.420043 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:41:59.420007    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-prio-pool-6b4a85cd-kserve-router-scheduler-7d966f676c-ns2mg" podUID="28cc88fc-5e5e-4b13-a883-17e70e36e4e9" containerName="main" containerID="cri-o://425d1d421ff793525fdca92849d28810f93351cbf8a832c18977af14ef23486f" gracePeriod=45
Nov 02 13:41:59.675224 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:41:59.675200    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-6b4a85cd-kserve-56f8b8b95b-mljbs"
Nov 02 13:41:59.731050 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:41:59.731021    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b07cd281-f05d-4df7-a835-d1cc723e12de-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b07cd281-f05d-4df7-a835-d1cc723e12de-model-cache\") pod \"b07cd281-f05d-4df7-a835-d1cc723e12de\" (UID: \"b07cd281-f05d-4df7-a835-d1cc723e12de\") "
Nov 02 13:41:59.731204 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:41:59.731059    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b07cd281-f05d-4df7-a835-d1cc723e12de-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b07cd281-f05d-4df7-a835-d1cc723e12de-tmp-dir\") pod \"b07cd281-f05d-4df7-a835-d1cc723e12de\" (UID: \"b07cd281-f05d-4df7-a835-d1cc723e12de\") "
Nov 02 13:41:59.731204 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:41:59.731112    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b07cd281-f05d-4df7-a835-d1cc723e12de-kube-api-access-nk9dz\" (UniqueName: \"kubernetes.io/projected/b07cd281-f05d-4df7-a835-d1cc723e12de-kube-api-access-nk9dz\") pod \"b07cd281-f05d-4df7-a835-d1cc723e12de\" (UID: \"b07cd281-f05d-4df7-a835-d1cc723e12de\") "
Nov 02 13:41:59.731204 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:41:59.731138    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b07cd281-f05d-4df7-a835-d1cc723e12de-tls-certs\" (UniqueName: \"kubernetes.io/secret/b07cd281-f05d-4df7-a835-d1cc723e12de-tls-certs\") pod \"b07cd281-f05d-4df7-a835-d1cc723e12de\" (UID: \"b07cd281-f05d-4df7-a835-d1cc723e12de\") "
Nov 02 13:41:59.731204 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:41:59.731201    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b07cd281-f05d-4df7-a835-d1cc723e12de-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b07cd281-f05d-4df7-a835-d1cc723e12de-dshm\") pod \"b07cd281-f05d-4df7-a835-d1cc723e12de\" (UID: \"b07cd281-f05d-4df7-a835-d1cc723e12de\") "
Nov 02 13:41:59.731372 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:41:59.731254    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b07cd281-f05d-4df7-a835-d1cc723e12de-home\" (UniqueName: \"kubernetes.io/empty-dir/b07cd281-f05d-4df7-a835-d1cc723e12de-home\") pod \"b07cd281-f05d-4df7-a835-d1cc723e12de\" (UID: \"b07cd281-f05d-4df7-a835-d1cc723e12de\") "
Nov 02 13:41:59.731372 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:41:59.731328    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b07cd281-f05d-4df7-a835-d1cc723e12de-model-cache" pod "b07cd281-f05d-4df7-a835-d1cc723e12de" (UID: "b07cd281-f05d-4df7-a835-d1cc723e12de"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:41:59.731448 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:41:59.731385    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b07cd281-f05d-4df7-a835-d1cc723e12de-tmp-dir" pod "b07cd281-f05d-4df7-a835-d1cc723e12de" (UID: "b07cd281-f05d-4df7-a835-d1cc723e12de"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:41:59.731617 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:41:59.731591    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b07cd281-f05d-4df7-a835-d1cc723e12de-home" pod "b07cd281-f05d-4df7-a835-d1cc723e12de" (UID: "b07cd281-f05d-4df7-a835-d1cc723e12de"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:41:59.731793 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:41:59.731631    4559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b07cd281-f05d-4df7-a835-d1cc723e12de-model-cache\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:41:59.731793 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:41:59.731662    4559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b07cd281-f05d-4df7-a835-d1cc723e12de-tmp-dir\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:41:59.733553 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:41:59.733510    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b07cd281-f05d-4df7-a835-d1cc723e12de-tls-certs" pod "b07cd281-f05d-4df7-a835-d1cc723e12de" (UID: "b07cd281-f05d-4df7-a835-d1cc723e12de"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:41:59.733553 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:41:59.733529    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b07cd281-f05d-4df7-a835-d1cc723e12de-dshm" pod "b07cd281-f05d-4df7-a835-d1cc723e12de" (UID: "b07cd281-f05d-4df7-a835-d1cc723e12de"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:41:59.733734 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:41:59.733675    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b07cd281-f05d-4df7-a835-d1cc723e12de-kube-api-access-nk9dz" pod "b07cd281-f05d-4df7-a835-d1cc723e12de" (UID: "b07cd281-f05d-4df7-a835-d1cc723e12de"). InnerVolumeSpecName "kube-api-access-nk9dz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:41:59.832451 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:41:59.832415    4559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b07cd281-f05d-4df7-a835-d1cc723e12de-dshm\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:41:59.832451 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:41:59.832448    4559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b07cd281-f05d-4df7-a835-d1cc723e12de-home\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:41:59.832700 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:41:59.832461    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nk9dz\" (UniqueName: \"kubernetes.io/projected/b07cd281-f05d-4df7-a835-d1cc723e12de-kube-api-access-nk9dz\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:41:59.832700 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:41:59.832471    4559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b07cd281-f05d-4df7-a835-d1cc723e12de-tls-certs\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:42:00.062367 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:42:00.062333    4559 generic.go:356] "Generic (PLEG): container finished" podID="b07cd281-f05d-4df7-a835-d1cc723e12de" containerID="8e2f6dc5d37738b762178b6b4fd001a5f8cf2269f21b5c7517620cade9f1336e" exitCode=0
Nov 02 13:42:00.062547 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:42:00.062381    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-6b4a85cd-kserve-56f8b8b95b-mljbs"
Nov 02 13:42:00.062547 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:42:00.062418    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-6b4a85cd-kserve-56f8b8b95b-mljbs" event={"ID":"b07cd281-f05d-4df7-a835-d1cc723e12de","Type":"ContainerDied","Data":"8e2f6dc5d37738b762178b6b4fd001a5f8cf2269f21b5c7517620cade9f1336e"}
Nov 02 13:42:00.062547 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:42:00.062466    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-6b4a85cd-kserve-56f8b8b95b-mljbs" event={"ID":"b07cd281-f05d-4df7-a835-d1cc723e12de","Type":"ContainerDied","Data":"ccc04248131b8af0fbe5331cae5bc48b3421b8095ab1f66f5be6f911415822da"}
Nov 02 13:42:00.062547 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:42:00.062488    4559 scope.go:122] "RemoveContainer" containerID="8e2f6dc5d37738b762178b6b4fd001a5f8cf2269f21b5c7517620cade9f1336e"
Nov 02 13:42:00.073478 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:42:00.073460    4559 scope.go:122] "RemoveContainer" containerID="8e2f6dc5d37738b762178b6b4fd001a5f8cf2269f21b5c7517620cade9f1336e"
Nov 02 13:42:00.073797 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:42:00.073777    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e2f6dc5d37738b762178b6b4fd001a5f8cf2269f21b5c7517620cade9f1336e\": container with ID starting with 8e2f6dc5d37738b762178b6b4fd001a5f8cf2269f21b5c7517620cade9f1336e not found: ID does not exist" containerID="8e2f6dc5d37738b762178b6b4fd001a5f8cf2269f21b5c7517620cade9f1336e"
Nov 02 13:42:00.073891 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:42:00.073804    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e2f6dc5d37738b762178b6b4fd001a5f8cf2269f21b5c7517620cade9f1336e"} err="failed to get container status \"8e2f6dc5d37738b762178b6b4fd001a5f8cf2269f21b5c7517620cade9f1336e\": rpc error: code = NotFound desc = could not find container \"8e2f6dc5d37738b762178b6b4fd001a5f8cf2269f21b5c7517620cade9f1336e\": container with ID starting with 8e2f6dc5d37738b762178b6b4fd001a5f8cf2269f21b5c7517620cade9f1336e not found: ID does not exist"
Nov 02 13:42:00.115338 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:42:00.115311    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-prio-pool-6b4a85cd-kserve-56f8b8b95b-mljbs"]
Nov 02 13:42:00.119672 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:42:00.119651    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-prio-pool-6b4a85cd-kserve-56f8b8b95b-mljbs"]
Nov 02 13:42:00.663059 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:42:00.663021    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-6b4a85cd-kserve-56f8b8b95b-mljbs" podUID="b07cd281-f05d-4df7-a835-d1cc723e12de" containerName="main" probeResult="failure" output="Get \"https://10.133.1.23:8000/ready\": context deadline exceeded"
Nov 02 13:42:01.891151 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:42:01.891121    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b07cd281-f05d-4df7-a835-d1cc723e12de" path="/var/lib/kubelet/pods/b07cd281-f05d-4df7-a835-d1cc723e12de/volumes"
Nov 02 13:42:05.180221 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:42:05.180188    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-f119d1-848f68677wtcqs"]
Nov 02 13:42:05.180602 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:42:05.180546    4559 cpu_manager.go:414] "containerMap: removing container" podUID="b07cd281-f05d-4df7-a835-d1cc723e12de" containerName="main"
Nov 02 13:42:05.180602 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:42:05.180556    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b07cd281-f05d-4df7-a835-d1cc723e12de" containerName="main"
Nov 02 13:42:05.180716 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:42:05.180652    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="b07cd281-f05d-4df7-a835-d1cc723e12de" containerName="main"
Nov 02 13:42:05.184954 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:42:05.184935    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-f119d1-848f68677wtcqs"
Nov 02 13:42:05.203668 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:42:05.203633    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-f119d1-848f68677wtcqs"]
Nov 02 13:42:05.214059 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:42:05.214029    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-f119d1\""
Nov 02 13:42:05.214183 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:42:05.214093    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-1cf3ed45-dockercfg-2hwc6\""
Nov 02 13:42:05.235555 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:42:05.235531    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-f119d1-7bb5d5p5hj"]
Nov 02 13:42:05.239039 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:42:05.239021    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f119d1-7bb5d5p5hj"
Nov 02 13:42:05.261870 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:42:05.261838    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-f119d1-7bb5d5p5hj"]
Nov 02 13:42:05.280193 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:42:05.280169    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7hgf\" (UniqueName: \"kubernetes.io/projected/b4dc5ddc-6886-42b8-b60a-70a3044f6564-kube-api-access-b7hgf\") pod \"payload-processing-e2e-worker-single-main-f119d1-848f68677wtcqs\" (UID: \"b4dc5ddc-6886-42b8-b60a-70a3044f6564\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-f119d1-848f68677wtcqs"
Nov 02 13:42:05.280314 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:42:05.280240    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4dc5ddc-6886-42b8-b60a-70a3044f6564-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-f119d1-848f68677wtcqs\" (UID: \"b4dc5ddc-6886-42b8-b60a-70a3044f6564\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-f119d1-848f68677wtcqs"
Nov 02 13:42:05.380977 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:42:05.380938    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a30bbf29-6441-467c-abf5-e87a3d680948-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-f119d1-7bb5d5p5hj\" (UID: \"a30bbf29-6441-467c-abf5-e87a3d680948\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f119d1-7bb5d5p5hj"
Nov 02 13:42:05.381145 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:42:05.380995    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7hgf\" (UniqueName: \"kubernetes.io/projected/b4dc5ddc-6886-42b8-b60a-70a3044f6564-kube-api-access-b7hgf\") pod \"payload-processing-e2e-worker-single-main-f119d1-848f68677wtcqs\" (UID: \"b4dc5ddc-6886-42b8-b60a-70a3044f6564\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-f119d1-848f68677wtcqs"
Nov 02 13:42:05.381145 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:42:05.381097    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvzhc\" (UniqueName: \"kubernetes.io/projected/a30bbf29-6441-467c-abf5-e87a3d680948-kube-api-access-cvzhc\") pod \"payload-pre-processing-e2e-worker-single-main-f119d1-7bb5d5p5hj\" (UID: \"a30bbf29-6441-467c-abf5-e87a3d680948\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f119d1-7bb5d5p5hj"
Nov 02 13:42:05.381145 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:42:05.381140    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4dc5ddc-6886-42b8-b60a-70a3044f6564-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-f119d1-848f68677wtcqs\" (UID: \"b4dc5ddc-6886-42b8-b60a-70a3044f6564\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-f119d1-848f68677wtcqs"
Nov 02 13:42:05.381835 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:42:05.381817    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4dc5ddc-6886-42b8-b60a-70a3044f6564-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-f119d1-848f68677wtcqs\" (UID: \"b4dc5ddc-6886-42b8-b60a-70a3044f6564\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-f119d1-848f68677wtcqs"
Nov 02 13:42:05.407798 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:42:05.407765    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7hgf\" (UniqueName: \"kubernetes.io/projected/b4dc5ddc-6886-42b8-b60a-70a3044f6564-kube-api-access-b7hgf\") pod \"payload-processing-e2e-worker-single-main-f119d1-848f68677wtcqs\" (UID: \"b4dc5ddc-6886-42b8-b60a-70a3044f6564\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-f119d1-848f68677wtcqs"
Nov 02 13:42:05.481851 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:42:05.481790    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a30bbf29-6441-467c-abf5-e87a3d680948-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-f119d1-7bb5d5p5hj\" (UID: \"a30bbf29-6441-467c-abf5-e87a3d680948\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f119d1-7bb5d5p5hj"
Nov 02 13:42:05.481982 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:42:05.481860    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvzhc\" (UniqueName: \"kubernetes.io/projected/a30bbf29-6441-467c-abf5-e87a3d680948-kube-api-access-cvzhc\") pod \"payload-pre-processing-e2e-worker-single-main-f119d1-7bb5d5p5hj\" (UID: \"a30bbf29-6441-467c-abf5-e87a3d680948\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f119d1-7bb5d5p5hj"
Nov 02 13:42:05.482447 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:42:05.482428    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a30bbf29-6441-467c-abf5-e87a3d680948-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-f119d1-7bb5d5p5hj\" (UID: \"a30bbf29-6441-467c-abf5-e87a3d680948\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f119d1-7bb5d5p5hj"
Nov 02 13:42:05.497720 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:42:05.497693    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-f119d1-848f68677wtcqs"
Nov 02 13:42:05.507320 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:42:05.507294    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvzhc\" (UniqueName: \"kubernetes.io/projected/a30bbf29-6441-467c-abf5-e87a3d680948-kube-api-access-cvzhc\") pod \"payload-pre-processing-e2e-worker-single-main-f119d1-7bb5d5p5hj\" (UID: \"a30bbf29-6441-467c-abf5-e87a3d680948\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f119d1-7bb5d5p5hj"
Nov 02 13:42:05.552156 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:42:05.551464    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f119d1-7bb5d5p5hj"
Nov 02 13:42:05.799876 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:42:05.799844    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-f119d1-848f68677wtcqs"]
Nov 02 13:42:05.800432 ip-10-0-128-132 kubenswrapper[4559]: W1102 13:42:05.800408    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4dc5ddc_6886_42b8_b60a_70a3044f6564.slice/crio-defd691d5d2495238ae98999a79b9991115c207bd90de8328f91d4634eda3bd8 WatchSource:0}: Error finding container defd691d5d2495238ae98999a79b9991115c207bd90de8328f91d4634eda3bd8: Status 404 returned error can't find the container with id defd691d5d2495238ae98999a79b9991115c207bd90de8328f91d4634eda3bd8
Nov 02 13:42:05.865316 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:42:05.865291    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-f119d1-7bb5d5p5hj"]
Nov 02 13:42:05.867189 ip-10-0-128-132 kubenswrapper[4559]: W1102 13:42:05.867165    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda30bbf29_6441_467c_abf5_e87a3d680948.slice/crio-be539bda389458c32ae95f1037f1e39832debe4893a70146db8290bb5fd6b9bd WatchSource:0}: Error finding container be539bda389458c32ae95f1037f1e39832debe4893a70146db8290bb5fd6b9bd: Status 404 returned error can't find the container with id be539bda389458c32ae95f1037f1e39832debe4893a70146db8290bb5fd6b9bd
Nov 02 13:42:06.094012 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:42:06.093906    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-f119d1-848f68677wtcqs" event={"ID":"b4dc5ddc-6886-42b8-b60a-70a3044f6564","Type":"ContainerStarted","Data":"a59f1c031f97171ab12ca3f47b8a88dde31a03e197e70576f2c83b59f2407203"}
Nov 02 13:42:06.094175 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:42:06.094062    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-f119d1-848f68677wtcqs"
Nov 02 13:42:06.094175 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:42:06.094106    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-f119d1-848f68677wtcqs" event={"ID":"b4dc5ddc-6886-42b8-b60a-70a3044f6564","Type":"ContainerStarted","Data":"defd691d5d2495238ae98999a79b9991115c207bd90de8328f91d4634eda3bd8"}
Nov 02 13:42:06.095495 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:42:06.095473    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f119d1-7bb5d5p5hj" event={"ID":"a30bbf29-6441-467c-abf5-e87a3d680948","Type":"ContainerStarted","Data":"7f1c55f58083e7640f6b7c1acc0e5b195cae92275b18d5fa0c957824290e1504"}
Nov 02 13:42:06.095592 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:42:06.095506    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f119d1-7bb5d5p5hj" event={"ID":"a30bbf29-6441-467c-abf5-e87a3d680948","Type":"ContainerStarted","Data":"be539bda389458c32ae95f1037f1e39832debe4893a70146db8290bb5fd6b9bd"}
Nov 02 13:42:06.095592 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:42:06.095534    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f119d1-7bb5d5p5hj"
Nov 02 13:42:06.136187 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:42:06.136139    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-f119d1-848f68677wtcqs" podStartSLOduration=1.136126437 podStartE2EDuration="1.136126437s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:42:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:42:06.134351017 +0000 UTC m=+4522.389750113" watchObservedRunningTime="2026-11-02 13:42:06.136126437 +0000 UTC m=+4522.391525532"
Nov 02 13:42:06.178687 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:42:06.178619    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f119d1-7bb5d5p5hj" podStartSLOduration=1.178605921 podStartE2EDuration="1.178605921s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:42:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:42:06.176485846 +0000 UTC m=+4522.431884940" watchObservedRunningTime="2026-11-02 13:42:06.178605921 +0000 UTC m=+4522.434005017"
Nov 02 13:42:06.609166 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:42:06.609122    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-6b4a85cd-kserve-router-scheduler-7d966f676c-ns2mg" podUID="28cc88fc-5e5e-4b13-a883-17e70e36e4e9" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 13:42:06.609566 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:42:06.609231    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-prio-pool-6b4a85cd-kserve-router-scheduler-7d966f676c-ns2mg"
Nov 02 13:42:08.336627 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:42:08.336588    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"]
Nov 02 13:42:15.086047 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:42:15.086008    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqcmj" containerName="extractor"
Nov 02 13:42:16.608729 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:42:16.608692    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-6b4a85cd-kserve-router-scheduler-7d966f676c-ns2mg" podUID="28cc88fc-5e5e-4b13-a883-17e70e36e4e9" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 13:42:17.103191 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:42:17.103163    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-f119d1-848f68677wtcqs"
Nov 02 13:42:17.103375 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:42:17.103239    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f119d1-7bb5d5p5hj"
Nov 02 13:42:22.447949 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:42:22.447911    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"]
Nov 02 13:42:26.609273 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:42:26.609191    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-6b4a85cd-kserve-router-scheduler-7d966f676c-ns2mg" podUID="28cc88fc-5e5e-4b13-a883-17e70e36e4e9" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 13:42:29.691203 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:42:29.691148    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-6b4a85cd-kserve-router-scheduler-7d966f676c-ns2mg"
Nov 02 13:42:29.709272 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:42:29.709245    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28cc88fc-5e5e-4b13-a883-17e70e36e4e9-kube-api-access-nbn47\" (UniqueName: \"kubernetes.io/projected/28cc88fc-5e5e-4b13-a883-17e70e36e4e9-kube-api-access-nbn47\") pod \"28cc88fc-5e5e-4b13-a883-17e70e36e4e9\" (UID: \"28cc88fc-5e5e-4b13-a883-17e70e36e4e9\") "
Nov 02 13:42:29.709397 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:42:29.709299    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/28cc88fc-5e5e-4b13-a883-17e70e36e4e9-tls-certs\" (UniqueName: \"kubernetes.io/secret/28cc88fc-5e5e-4b13-a883-17e70e36e4e9-tls-certs\") pod \"28cc88fc-5e5e-4b13-a883-17e70e36e4e9\" (UID: \"28cc88fc-5e5e-4b13-a883-17e70e36e4e9\") "
Nov 02 13:42:29.711375 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:42:29.711347    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28cc88fc-5e5e-4b13-a883-17e70e36e4e9-kube-api-access-nbn47" pod "28cc88fc-5e5e-4b13-a883-17e70e36e4e9" (UID: "28cc88fc-5e5e-4b13-a883-17e70e36e4e9"). InnerVolumeSpecName "kube-api-access-nbn47". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:42:29.711517 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:42:29.711455    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28cc88fc-5e5e-4b13-a883-17e70e36e4e9-tls-certs" pod "28cc88fc-5e5e-4b13-a883-17e70e36e4e9" (UID: "28cc88fc-5e5e-4b13-a883-17e70e36e4e9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:42:29.810393 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:42:29.810363    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nbn47\" (UniqueName: \"kubernetes.io/projected/28cc88fc-5e5e-4b13-a883-17e70e36e4e9-kube-api-access-nbn47\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:42:29.810393 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:42:29.810389    4559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28cc88fc-5e5e-4b13-a883-17e70e36e4e9-tls-certs\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:42:30.209871 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:42:30.209834    4559 generic.go:356] "Generic (PLEG): container finished" podID="28cc88fc-5e5e-4b13-a883-17e70e36e4e9" containerID="425d1d421ff793525fdca92849d28810f93351cbf8a832c18977af14ef23486f" exitCode=0
Nov 02 13:42:30.210060 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:42:30.209900    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-6b4a85cd-kserve-router-scheduler-7d966f676c-ns2mg"
Nov 02 13:42:30.210060 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:42:30.209916    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-6b4a85cd-kserve-router-scheduler-7d966f676c-ns2mg" event={"ID":"28cc88fc-5e5e-4b13-a883-17e70e36e4e9","Type":"ContainerDied","Data":"425d1d421ff793525fdca92849d28810f93351cbf8a832c18977af14ef23486f"}
Nov 02 13:42:30.210060 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:42:30.209996    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-6b4a85cd-kserve-router-scheduler-7d966f676c-ns2mg" event={"ID":"28cc88fc-5e5e-4b13-a883-17e70e36e4e9","Type":"ContainerDied","Data":"37c0cdf05ec007683ca0b1f09f5548f7e7b41cee43fe44d3327527b584b652a2"}
Nov 02 13:42:30.210060 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:42:30.210018    4559 scope.go:122] "RemoveContainer" containerID="425d1d421ff793525fdca92849d28810f93351cbf8a832c18977af14ef23486f"
Nov 02 13:42:30.219077 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:42:30.219058    4559 scope.go:122] "RemoveContainer" containerID="425d1d421ff793525fdca92849d28810f93351cbf8a832c18977af14ef23486f"
Nov 02 13:42:30.219367 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:42:30.219346    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"425d1d421ff793525fdca92849d28810f93351cbf8a832c18977af14ef23486f\": container with ID starting with 425d1d421ff793525fdca92849d28810f93351cbf8a832c18977af14ef23486f not found: ID does not exist" containerID="425d1d421ff793525fdca92849d28810f93351cbf8a832c18977af14ef23486f"
Nov 02 13:42:30.219422 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:42:30.219376    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"425d1d421ff793525fdca92849d28810f93351cbf8a832c18977af14ef23486f"} err="failed to get container status \"425d1d421ff793525fdca92849d28810f93351cbf8a832c18977af14ef23486f\": rpc error: code = NotFound desc = could not find container \"425d1d421ff793525fdca92849d28810f93351cbf8a832c18977af14ef23486f\": container with ID starting with 425d1d421ff793525fdca92849d28810f93351cbf8a832c18977af14ef23486f not found: ID does not exist"
Nov 02 13:42:30.262868 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:42:30.262844    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-prio-pool-6b4a85cd-kserve-router-scheduler-7d966f676c-ns2mg"]
Nov 02 13:42:30.275565 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:42:30.275536    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-prio-pool-6b4a85cd-kserve-router-scheduler-7d966f676c-ns2mg"]
Nov 02 13:42:31.890955 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:42:31.890926    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28cc88fc-5e5e-4b13-a883-17e70e36e4e9" path="/var/lib/kubelet/pods/28cc88fc-5e5e-4b13-a883-17e70e36e4e9/volumes"
Nov 02 13:42:38.510073 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:42:38.510034    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"]
Nov 02 13:42:45.084938 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:42:45.084897    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqcmj" containerName="extractor"
Nov 02 13:42:50.473616 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:42:50.473575    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71066ac4-cbt2l"]
Nov 02 13:42:50.474026 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:42:50.473969    4559 cpu_manager.go:414] "containerMap: removing container" podUID="28cc88fc-5e5e-4b13-a883-17e70e36e4e9" containerName="main"
Nov 02 13:42:50.474026 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:42:50.473980    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28cc88fc-5e5e-4b13-a883-17e70e36e4e9" containerName="main"
Nov 02 13:42:50.474106 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:42:50.474053    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="28cc88fc-5e5e-4b13-a883-17e70e36e4e9" containerName="main"
Nov 02 13:42:50.478779 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:42:50.478762    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71066ac4-cbt2l"
Nov 02 13:42:50.494829 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:42:50.494807    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 13:42:50.497392 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:42:50.497369    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71066ac4-cbt2l"]
Nov 02 13:42:50.507228 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:42:50.507207    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-45fzn\""
Nov 02 13:42:50.507710 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:42:50.507693    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 13:42:50.507775 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:42:50.507732    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 13:42:50.603149 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:42:50.603123    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtdcp\" (UniqueName: \"kubernetes.io/projected/d0aba3dc-1b01-4ef5-9e8b-810523b85fbd-kube-api-access-vtdcp\") pod \"maas-api-revoke-sub-71066ac4-cbt2l\" (UID: \"d0aba3dc-1b01-4ef5-9e8b-810523b85fbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71066ac4-cbt2l"
Nov 02 13:42:50.603255 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:42:50.603168    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0aba3dc-1b01-4ef5-9e8b-810523b85fbd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-71066ac4-cbt2l\" (UID: \"d0aba3dc-1b01-4ef5-9e8b-810523b85fbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71066ac4-cbt2l"
Nov 02 13:42:50.603319 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:42:50.603299    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0aba3dc-1b01-4ef5-9e8b-810523b85fbd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-71066ac4-cbt2l\" (UID: \"d0aba3dc-1b01-4ef5-9e8b-810523b85fbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71066ac4-cbt2l"
Nov 02 13:42:50.703819 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:42:50.703791    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtdcp\" (UniqueName: \"kubernetes.io/projected/d0aba3dc-1b01-4ef5-9e8b-810523b85fbd-kube-api-access-vtdcp\") pod \"maas-api-revoke-sub-71066ac4-cbt2l\" (UID: \"d0aba3dc-1b01-4ef5-9e8b-810523b85fbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71066ac4-cbt2l"
Nov 02 13:42:50.703934 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:42:50.703836    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0aba3dc-1b01-4ef5-9e8b-810523b85fbd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-71066ac4-cbt2l\" (UID: \"d0aba3dc-1b01-4ef5-9e8b-810523b85fbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71066ac4-cbt2l"
Nov 02 13:42:50.703934 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:42:50.703916    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0aba3dc-1b01-4ef5-9e8b-810523b85fbd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-71066ac4-cbt2l\" (UID: \"d0aba3dc-1b01-4ef5-9e8b-810523b85fbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71066ac4-cbt2l"
Nov 02 13:42:50.704479 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:42:50.704454    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0aba3dc-1b01-4ef5-9e8b-810523b85fbd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-71066ac4-cbt2l\" (UID: \"d0aba3dc-1b01-4ef5-9e8b-810523b85fbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71066ac4-cbt2l"
Nov 02 13:42:50.706218 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:42:50.706192    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0aba3dc-1b01-4ef5-9e8b-810523b85fbd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-71066ac4-cbt2l\" (UID: \"d0aba3dc-1b01-4ef5-9e8b-810523b85fbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71066ac4-cbt2l"
Nov 02 13:42:50.727974 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:42:50.727922    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtdcp\" (UniqueName: \"kubernetes.io/projected/d0aba3dc-1b01-4ef5-9e8b-810523b85fbd-kube-api-access-vtdcp\") pod \"maas-api-revoke-sub-71066ac4-cbt2l\" (UID: \"d0aba3dc-1b01-4ef5-9e8b-810523b85fbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71066ac4-cbt2l"
Nov 02 13:42:50.791674 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:42:50.791632    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71066ac4-cbt2l"
Nov 02 13:42:51.091932 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:42:51.091904    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71066ac4-cbt2l"]
Nov 02 13:42:51.092436 ip-10-0-128-132 kubenswrapper[4559]: W1102 13:42:51.092409    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0aba3dc_1b01_4ef5_9e8b_810523b85fbd.slice/crio-919143c94ad4e72b9a19a1d66d3852a87b67c5fe61c363b1544a9f7c30a70852 WatchSource:0}: Error finding container 919143c94ad4e72b9a19a1d66d3852a87b67c5fe61c363b1544a9f7c30a70852: Status 404 returned error can't find the container with id 919143c94ad4e72b9a19a1d66d3852a87b67c5fe61c363b1544a9f7c30a70852
Nov 02 13:42:51.313696 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:42:51.313666    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71066ac4-cbt2l" event={"ID":"d0aba3dc-1b01-4ef5-9e8b-810523b85fbd","Type":"ContainerStarted","Data":"919143c94ad4e72b9a19a1d66d3852a87b67c5fe61c363b1544a9f7c30a70852"}
Nov 02 13:42:52.319831 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:42:52.319747    4559 generic.go:356] "Generic (PLEG): container finished" podID="d0aba3dc-1b01-4ef5-9e8b-810523b85fbd" containerID="9bf53315efae1d5b53ae789e3fb700fb6e8eedfb3a957d5e27d6bca356d5e978" exitCode=0
Nov 02 13:42:52.320242 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:42:52.319833    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71066ac4-cbt2l" event={"ID":"d0aba3dc-1b01-4ef5-9e8b-810523b85fbd","Type":"ContainerDied","Data":"9bf53315efae1d5b53ae789e3fb700fb6e8eedfb3a957d5e27d6bca356d5e978"}
Nov 02 13:42:53.463505 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:42:53.463481    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71066ac4-cbt2l"
Nov 02 13:42:53.538263 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:42:53.538225    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0aba3dc-1b01-4ef5-9e8b-810523b85fbd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0aba3dc-1b01-4ef5-9e8b-810523b85fbd-maas-api-openshift-ca\") pod \"d0aba3dc-1b01-4ef5-9e8b-810523b85fbd\" (UID: \"d0aba3dc-1b01-4ef5-9e8b-810523b85fbd\") "
Nov 02 13:42:53.538615 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:42:53.538588    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d0aba3dc-1b01-4ef5-9e8b-810523b85fbd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0aba3dc-1b01-4ef5-9e8b-810523b85fbd-maas-api-xks-ca\") pod \"d0aba3dc-1b01-4ef5-9e8b-810523b85fbd\" (UID: \"d0aba3dc-1b01-4ef5-9e8b-810523b85fbd\") "
Nov 02 13:42:53.538806 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:42:53.538657    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0aba3dc-1b01-4ef5-9e8b-810523b85fbd-kube-api-access-vtdcp\" (UniqueName: \"kubernetes.io/projected/d0aba3dc-1b01-4ef5-9e8b-810523b85fbd-kube-api-access-vtdcp\") pod \"d0aba3dc-1b01-4ef5-9e8b-810523b85fbd\" (UID: \"d0aba3dc-1b01-4ef5-9e8b-810523b85fbd\") "
Nov 02 13:42:53.538806 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:42:53.538687    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0aba3dc-1b01-4ef5-9e8b-810523b85fbd-maas-api-openshift-ca" pod "d0aba3dc-1b01-4ef5-9e8b-810523b85fbd" (UID: "d0aba3dc-1b01-4ef5-9e8b-810523b85fbd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:42:53.538981 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:42:53.538954    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0aba3dc-1b01-4ef5-9e8b-810523b85fbd-maas-api-openshift-ca\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:42:53.540813 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:42:53.540790    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0aba3dc-1b01-4ef5-9e8b-810523b85fbd-maas-api-xks-ca" pod "d0aba3dc-1b01-4ef5-9e8b-810523b85fbd" (UID: "d0aba3dc-1b01-4ef5-9e8b-810523b85fbd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:42:53.541393 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:42:53.541375    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0aba3dc-1b01-4ef5-9e8b-810523b85fbd-kube-api-access-vtdcp" pod "d0aba3dc-1b01-4ef5-9e8b-810523b85fbd" (UID: "d0aba3dc-1b01-4ef5-9e8b-810523b85fbd"). InnerVolumeSpecName "kube-api-access-vtdcp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:42:53.639778 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:42:53.639698    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0aba3dc-1b01-4ef5-9e8b-810523b85fbd-maas-api-xks-ca\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:42:53.639778 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:42:53.639729    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vtdcp\" (UniqueName: \"kubernetes.io/projected/d0aba3dc-1b01-4ef5-9e8b-810523b85fbd-kube-api-access-vtdcp\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:42:54.331688 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:42:54.331652    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71066ac4-cbt2l" event={"ID":"d0aba3dc-1b01-4ef5-9e8b-810523b85fbd","Type":"ContainerDied","Data":"919143c94ad4e72b9a19a1d66d3852a87b67c5fe61c363b1544a9f7c30a70852"}
Nov 02 13:42:54.331688 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:42:54.331669    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71066ac4-cbt2l"
Nov 02 13:42:54.331905 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:42:54.331703    4559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="919143c94ad4e72b9a19a1d66d3852a87b67c5fe61c363b1544a9f7c30a70852"
Nov 02 13:43:01.990471 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:43:01.990439    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"]
Nov 02 13:43:08.208931 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:43:08.208894    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-f119d1-848f68677wtcqs"]
Nov 02 13:43:08.243756 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:43:08.243712    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-f119d1-7bb5d5p5hj"]
Nov 02 13:43:11.011700 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:43:11.011618    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"]
Nov 02 13:43:13.209671 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:43:13.209614    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-f119d1-848f68677wtcqs" podUID="b4dc5ddc-6886-42b8-b60a-70a3044f6564" containerName="payload-processing" containerID="cri-o://a59f1c031f97171ab12ca3f47b8a88dde31a03e197e70576f2c83b59f2407203" gracePeriod=25
Nov 02 13:43:13.244439 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:43:13.244404    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f119d1-7bb5d5p5hj" podUID="a30bbf29-6441-467c-abf5-e87a3d680948" containerName="payload-pre-processing" containerID="cri-o://7f1c55f58083e7640f6b7c1acc0e5b195cae92275b18d5fa0c957824290e1504" gracePeriod=25
Nov 02 13:43:15.084738 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:43:15.084697    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqcmj" containerName="extractor"
Nov 02 13:43:17.101908 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:43:17.101871    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f119d1-7bb5d5p5hj" podUID="a30bbf29-6441-467c-abf5-e87a3d680948" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 13:43:17.102410 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:43:17.101867    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-f119d1-848f68677wtcqs" podUID="b4dc5ddc-6886-42b8-b60a-70a3044f6564" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 13:43:27.102167 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:43:27.102127    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-f119d1-848f68677wtcqs" podUID="b4dc5ddc-6886-42b8-b60a-70a3044f6564" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 13:43:27.102604 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:43:27.102121    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f119d1-7bb5d5p5hj" podUID="a30bbf29-6441-467c-abf5-e87a3d680948" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 13:43:33.487937 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:43:33.487913    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-f119d1-848f68677wtcqs"
Nov 02 13:43:33.516971 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:43:33.516946    4559 generic.go:356] "Generic (PLEG): container finished" podID="b4dc5ddc-6886-42b8-b60a-70a3044f6564" containerID="a59f1c031f97171ab12ca3f47b8a88dde31a03e197e70576f2c83b59f2407203" exitCode=0
Nov 02 13:43:33.517113 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:43:33.516993    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-f119d1-848f68677wtcqs"
Nov 02 13:43:33.517113 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:43:33.516995    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-f119d1-848f68677wtcqs" event={"ID":"b4dc5ddc-6886-42b8-b60a-70a3044f6564","Type":"ContainerDied","Data":"a59f1c031f97171ab12ca3f47b8a88dde31a03e197e70576f2c83b59f2407203"}
Nov 02 13:43:33.517113 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:43:33.517050    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-f119d1-848f68677wtcqs" event={"ID":"b4dc5ddc-6886-42b8-b60a-70a3044f6564","Type":"ContainerDied","Data":"defd691d5d2495238ae98999a79b9991115c207bd90de8328f91d4634eda3bd8"}
Nov 02 13:43:33.517113 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:43:33.517080    4559 scope.go:122] "RemoveContainer" containerID="a59f1c031f97171ab12ca3f47b8a88dde31a03e197e70576f2c83b59f2407203"
Nov 02 13:43:33.518391 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:43:33.518370    4559 generic.go:356] "Generic (PLEG): container finished" podID="a30bbf29-6441-467c-abf5-e87a3d680948" containerID="7f1c55f58083e7640f6b7c1acc0e5b195cae92275b18d5fa0c957824290e1504" exitCode=0
Nov 02 13:43:33.518494 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:43:33.518453    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f119d1-7bb5d5p5hj" event={"ID":"a30bbf29-6441-467c-abf5-e87a3d680948","Type":"ContainerDied","Data":"7f1c55f58083e7640f6b7c1acc0e5b195cae92275b18d5fa0c957824290e1504"}
Nov 02 13:43:33.518558 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:43:33.518494    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f119d1-7bb5d5p5hj" event={"ID":"a30bbf29-6441-467c-abf5-e87a3d680948","Type":"ContainerDied","Data":"be539bda389458c32ae95f1037f1e39832debe4893a70146db8290bb5fd6b9bd"}
Nov 02 13:43:33.518558 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:43:33.518511    4559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="be539bda389458c32ae95f1037f1e39832debe4893a70146db8290bb5fd6b9bd"
Nov 02 13:43:33.520135 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:43:33.520116    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f119d1-7bb5d5p5hj"
Nov 02 13:43:33.525148 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:43:33.525127    4559 scope.go:122] "RemoveContainer" containerID="a59f1c031f97171ab12ca3f47b8a88dde31a03e197e70576f2c83b59f2407203"
Nov 02 13:43:33.525408 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:43:33.525389    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a59f1c031f97171ab12ca3f47b8a88dde31a03e197e70576f2c83b59f2407203\": container with ID starting with a59f1c031f97171ab12ca3f47b8a88dde31a03e197e70576f2c83b59f2407203 not found: ID does not exist" containerID="a59f1c031f97171ab12ca3f47b8a88dde31a03e197e70576f2c83b59f2407203"
Nov 02 13:43:33.525466 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:43:33.525417    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a59f1c031f97171ab12ca3f47b8a88dde31a03e197e70576f2c83b59f2407203"} err="failed to get container status \"a59f1c031f97171ab12ca3f47b8a88dde31a03e197e70576f2c83b59f2407203\": rpc error: code = NotFound desc = could not find container \"a59f1c031f97171ab12ca3f47b8a88dde31a03e197e70576f2c83b59f2407203\": container with ID starting with a59f1c031f97171ab12ca3f47b8a88dde31a03e197e70576f2c83b59f2407203 not found: ID does not exist"
Nov 02 13:43:33.610843 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:43:33.610813    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4dc5ddc-6886-42b8-b60a-70a3044f6564-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4dc5ddc-6886-42b8-b60a-70a3044f6564-plugins-config-volume\") pod \"b4dc5ddc-6886-42b8-b60a-70a3044f6564\" (UID: \"b4dc5ddc-6886-42b8-b60a-70a3044f6564\") "
Nov 02 13:43:33.611001 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:43:33.610852    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a30bbf29-6441-467c-abf5-e87a3d680948-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a30bbf29-6441-467c-abf5-e87a3d680948-plugins-config-volume\") pod \"a30bbf29-6441-467c-abf5-e87a3d680948\" (UID: \"a30bbf29-6441-467c-abf5-e87a3d680948\") "
Nov 02 13:43:33.611001 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:43:33.610897    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a30bbf29-6441-467c-abf5-e87a3d680948-kube-api-access-cvzhc\" (UniqueName: \"kubernetes.io/projected/a30bbf29-6441-467c-abf5-e87a3d680948-kube-api-access-cvzhc\") pod \"a30bbf29-6441-467c-abf5-e87a3d680948\" (UID: \"a30bbf29-6441-467c-abf5-e87a3d680948\") "
Nov 02 13:43:33.611001 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:43:33.610968    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4dc5ddc-6886-42b8-b60a-70a3044f6564-kube-api-access-b7hgf\" (UniqueName: \"kubernetes.io/projected/b4dc5ddc-6886-42b8-b60a-70a3044f6564-kube-api-access-b7hgf\") pod \"b4dc5ddc-6886-42b8-b60a-70a3044f6564\" (UID: \"b4dc5ddc-6886-42b8-b60a-70a3044f6564\") "
Nov 02 13:43:33.611284 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:43:33.611264    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4dc5ddc-6886-42b8-b60a-70a3044f6564-plugins-config-volume" pod "b4dc5ddc-6886-42b8-b60a-70a3044f6564" (UID: "b4dc5ddc-6886-42b8-b60a-70a3044f6564"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:43:33.611339 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:43:33.611298    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a30bbf29-6441-467c-abf5-e87a3d680948-plugins-config-volume" pod "a30bbf29-6441-467c-abf5-e87a3d680948" (UID: "a30bbf29-6441-467c-abf5-e87a3d680948"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:43:33.612986 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:43:33.612968    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4dc5ddc-6886-42b8-b60a-70a3044f6564-kube-api-access-b7hgf" pod "b4dc5ddc-6886-42b8-b60a-70a3044f6564" (UID: "b4dc5ddc-6886-42b8-b60a-70a3044f6564"). InnerVolumeSpecName "kube-api-access-b7hgf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:43:33.613136 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:43:33.613119    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a30bbf29-6441-467c-abf5-e87a3d680948-kube-api-access-cvzhc" pod "a30bbf29-6441-467c-abf5-e87a3d680948" (UID: "a30bbf29-6441-467c-abf5-e87a3d680948"). InnerVolumeSpecName "kube-api-access-cvzhc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:43:33.712580 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:43:33.712553    4559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4dc5ddc-6886-42b8-b60a-70a3044f6564-plugins-config-volume\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:43:33.712580 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:43:33.712576    4559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a30bbf29-6441-467c-abf5-e87a3d680948-plugins-config-volume\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:43:33.712580 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:43:33.712585    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cvzhc\" (UniqueName: \"kubernetes.io/projected/a30bbf29-6441-467c-abf5-e87a3d680948-kube-api-access-cvzhc\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:43:33.712874 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:43:33.712598    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b7hgf\" (UniqueName: \"kubernetes.io/projected/b4dc5ddc-6886-42b8-b60a-70a3044f6564-kube-api-access-b7hgf\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:43:33.868809 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:43:33.868780    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-f119d1-848f68677wtcqs"]
Nov 02 13:43:33.881619 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:43:33.881594    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-f119d1-848f68677wtcqs"]
Nov 02 13:43:33.892649 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:43:33.892611    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4dc5ddc-6886-42b8-b60a-70a3044f6564" path="/var/lib/kubelet/pods/b4dc5ddc-6886-42b8-b60a-70a3044f6564/volumes"
Nov 02 13:43:34.524583 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:43:34.524513    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f119d1-7bb5d5p5hj"
Nov 02 13:43:34.580025 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:43:34.579982    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-f119d1-7bb5d5p5hj"]
Nov 02 13:43:34.592629 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:43:34.592599    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-f119d1-7bb5d5p5hj"]
Nov 02 13:43:35.890883 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:43:35.890847    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a30bbf29-6441-467c-abf5-e87a3d680948" path="/var/lib/kubelet/pods/a30bbf29-6441-467c-abf5-e87a3d680948/volumes"
Nov 02 13:43:42.701856 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:43:42.701819    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f119d1-667b99d486-mms7p"]
Nov 02 13:43:42.702354 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:43:42.702339    4559 cpu_manager.go:414] "containerMap: removing container" podUID="b4dc5ddc-6886-42b8-b60a-70a3044f6564" containerName="payload-processing"
Nov 02 13:43:42.702398 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:43:42.702356    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4dc5ddc-6886-42b8-b60a-70a3044f6564" containerName="payload-processing"
Nov 02 13:43:42.702398 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:43:42.702381    4559 cpu_manager.go:414] "containerMap: removing container" podUID="a30bbf29-6441-467c-abf5-e87a3d680948" containerName="payload-pre-processing"
Nov 02 13:43:42.702398 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:43:42.702388    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a30bbf29-6441-467c-abf5-e87a3d680948" containerName="payload-pre-processing"
Nov 02 13:43:42.702497 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:43:42.702409    4559 cpu_manager.go:414] "containerMap: removing container" podUID="d0aba3dc-1b01-4ef5-9e8b-810523b85fbd" containerName="revoke-keys"
Nov 02 13:43:42.702497 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:43:42.702417    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0aba3dc-1b01-4ef5-9e8b-810523b85fbd" containerName="revoke-keys"
Nov 02 13:43:42.702562 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:43:42.702498    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4dc5ddc-6886-42b8-b60a-70a3044f6564" containerName="payload-processing"
Nov 02 13:43:42.702562 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:43:42.702512    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="a30bbf29-6441-467c-abf5-e87a3d680948" containerName="payload-pre-processing"
Nov 02 13:43:42.702562 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:43:42.702522    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0aba3dc-1b01-4ef5-9e8b-810523b85fbd" containerName="revoke-keys"
Nov 02 13:43:42.707108 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:43:42.707084    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f119d1-667b99d486-mms7p"
Nov 02 13:43:42.741765 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:43:42.741736    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f119d1-667b99d486-mms7p"]
Nov 02 13:43:42.742862 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:43:42.742836    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-f119d1-serving-cert\""
Nov 02 13:43:42.743226 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:43:42.743181    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Nov 02 13:43:42.743404 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:43:42.743381    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-sr5s9\""
Nov 02 13:43:42.752178 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:43:42.752157    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 13:43:42.752178 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:43:42.752170    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 13:43:42.892588 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:43:42.892556    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gf24r\" (UniqueName: \"kubernetes.io/projected/4791f0c3-afec-48aa-ab44-f538c1f9a0dd-kube-api-access-gf24r\") pod \"maas-api-e2e-worker-single-main-f119d1-667b99d486-mms7p\" (UID: \"4791f0c3-afec-48aa-ab44-f538c1f9a0dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f119d1-667b99d486-mms7p"
Nov 02 13:43:42.892785 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:43:42.892593    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4791f0c3-afec-48aa-ab44-f538c1f9a0dd-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-f119d1-667b99d486-mms7p\" (UID: \"4791f0c3-afec-48aa-ab44-f538c1f9a0dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f119d1-667b99d486-mms7p"
Nov 02 13:43:42.892785 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:43:42.892715    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4791f0c3-afec-48aa-ab44-f538c1f9a0dd-metrics-certs\") pod \"maas-api-e2e-worker-single-main-f119d1-667b99d486-mms7p\" (UID: \"4791f0c3-afec-48aa-ab44-f538c1f9a0dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f119d1-667b99d486-mms7p"
Nov 02 13:43:42.892785 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:43:42.892745    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4791f0c3-afec-48aa-ab44-f538c1f9a0dd-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-f119d1-667b99d486-mms7p\" (UID: \"4791f0c3-afec-48aa-ab44-f538c1f9a0dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f119d1-667b99d486-mms7p"
Nov 02 13:43:42.994384 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:43:42.994235    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gf24r\" (UniqueName: \"kubernetes.io/projected/4791f0c3-afec-48aa-ab44-f538c1f9a0dd-kube-api-access-gf24r\") pod \"maas-api-e2e-worker-single-main-f119d1-667b99d486-mms7p\" (UID: \"4791f0c3-afec-48aa-ab44-f538c1f9a0dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f119d1-667b99d486-mms7p"
Nov 02 13:43:42.994384 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:43:42.994292    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4791f0c3-afec-48aa-ab44-f538c1f9a0dd-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-f119d1-667b99d486-mms7p\" (UID: \"4791f0c3-afec-48aa-ab44-f538c1f9a0dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f119d1-667b99d486-mms7p"
Nov 02 13:43:42.994768 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:43:42.994737    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4791f0c3-afec-48aa-ab44-f538c1f9a0dd-metrics-certs\") pod \"maas-api-e2e-worker-single-main-f119d1-667b99d486-mms7p\" (UID: \"4791f0c3-afec-48aa-ab44-f538c1f9a0dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f119d1-667b99d486-mms7p"
Nov 02 13:43:42.994891 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:43:42.994795    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4791f0c3-afec-48aa-ab44-f538c1f9a0dd-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-f119d1-667b99d486-mms7p\" (UID: \"4791f0c3-afec-48aa-ab44-f538c1f9a0dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f119d1-667b99d486-mms7p"
Nov 02 13:43:42.995537 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:43:42.995510    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4791f0c3-afec-48aa-ab44-f538c1f9a0dd-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-f119d1-667b99d486-mms7p\" (UID: \"4791f0c3-afec-48aa-ab44-f538c1f9a0dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f119d1-667b99d486-mms7p"
Nov 02 13:43:42.997058 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:43:42.997033    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4791f0c3-afec-48aa-ab44-f538c1f9a0dd-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-f119d1-667b99d486-mms7p\" (UID: \"4791f0c3-afec-48aa-ab44-f538c1f9a0dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f119d1-667b99d486-mms7p"
Nov 02 13:43:42.997158 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:43:42.997087    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4791f0c3-afec-48aa-ab44-f538c1f9a0dd-metrics-certs\") pod \"maas-api-e2e-worker-single-main-f119d1-667b99d486-mms7p\" (UID: \"4791f0c3-afec-48aa-ab44-f538c1f9a0dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f119d1-667b99d486-mms7p"
Nov 02 13:43:43.027005 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:43:43.026975    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gf24r\" (UniqueName: \"kubernetes.io/projected/4791f0c3-afec-48aa-ab44-f538c1f9a0dd-kube-api-access-gf24r\") pod \"maas-api-e2e-worker-single-main-f119d1-667b99d486-mms7p\" (UID: \"4791f0c3-afec-48aa-ab44-f538c1f9a0dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f119d1-667b99d486-mms7p"
Nov 02 13:43:43.247383 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:43:43.247304    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-f119d1-7bb5dp7pc2"]
Nov 02 13:43:43.251022 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:43:43.251006    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f119d1-7bb5dp7pc2"
Nov 02 13:43:43.276395 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:43:43.276367    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-f119d1\""
Nov 02 13:43:43.282168 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:43:43.282145    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-1cf3ed45-dockercfg-6rxdt\""
Nov 02 13:43:43.284493 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:43:43.284468    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-f119d1-7bb5dp7pc2"]
Nov 02 13:43:43.313519 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:43:43.313489    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-f119d1-848f68677lv6zp"]
Nov 02 13:43:43.317654 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:43:43.317612    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-f119d1-848f68677lv6zp"
Nov 02 13:43:43.324869 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:43:43.324835    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f119d1-667b99d486-mms7p"
Nov 02 13:43:43.343137 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:43:43.343107    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-f119d1-848f68677lv6zp"]
Nov 02 13:43:43.400128 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:43:43.400090    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfc4d4ea-5b86-4473-be96-16e33bafca2a-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-f119d1-7bb5dp7pc2\" (UID: \"bfc4d4ea-5b86-4473-be96-16e33bafca2a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f119d1-7bb5dp7pc2"
Nov 02 13:43:43.400319 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:43:43.400296    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfkdz\" (UniqueName: \"kubernetes.io/projected/bfc4d4ea-5b86-4473-be96-16e33bafca2a-kube-api-access-mfkdz\") pod \"payload-pre-processing-e2e-worker-single-main-f119d1-7bb5dp7pc2\" (UID: \"bfc4d4ea-5b86-4473-be96-16e33bafca2a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f119d1-7bb5dp7pc2"
Nov 02 13:43:43.501015 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:43:43.500943    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfkdz\" (UniqueName: \"kubernetes.io/projected/bfc4d4ea-5b86-4473-be96-16e33bafca2a-kube-api-access-mfkdz\") pod \"payload-pre-processing-e2e-worker-single-main-f119d1-7bb5dp7pc2\" (UID: \"bfc4d4ea-5b86-4473-be96-16e33bafca2a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f119d1-7bb5dp7pc2"
Nov 02 13:43:43.501015 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:43:43.500985    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfc4d4ea-5b86-4473-be96-16e33bafca2a-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-f119d1-7bb5dp7pc2\" (UID: \"bfc4d4ea-5b86-4473-be96-16e33bafca2a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f119d1-7bb5dp7pc2"
Nov 02 13:43:43.501194 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:43:43.501034    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2whbw\" (UniqueName: \"kubernetes.io/projected/3173428e-13b0-469a-b95e-30cd0be7586a-kube-api-access-2whbw\") pod \"payload-processing-e2e-worker-single-main-f119d1-848f68677lv6zp\" (UID: \"3173428e-13b0-469a-b95e-30cd0be7586a\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-f119d1-848f68677lv6zp"
Nov 02 13:43:43.501194 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:43:43.501070    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3173428e-13b0-469a-b95e-30cd0be7586a-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-f119d1-848f68677lv6zp\" (UID: \"3173428e-13b0-469a-b95e-30cd0be7586a\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-f119d1-848f68677lv6zp"
Nov 02 13:43:43.501774 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:43:43.501749    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfc4d4ea-5b86-4473-be96-16e33bafca2a-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-f119d1-7bb5dp7pc2\" (UID: \"bfc4d4ea-5b86-4473-be96-16e33bafca2a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f119d1-7bb5dp7pc2"
Nov 02 13:43:43.525009 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:43:43.524977    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfkdz\" (UniqueName: \"kubernetes.io/projected/bfc4d4ea-5b86-4473-be96-16e33bafca2a-kube-api-access-mfkdz\") pod \"payload-pre-processing-e2e-worker-single-main-f119d1-7bb5dp7pc2\" (UID: \"bfc4d4ea-5b86-4473-be96-16e33bafca2a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f119d1-7bb5dp7pc2"
Nov 02 13:43:43.563952 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:43:43.563923    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f119d1-7bb5dp7pc2"
Nov 02 13:43:43.602274 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:43:43.602238    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2whbw\" (UniqueName: \"kubernetes.io/projected/3173428e-13b0-469a-b95e-30cd0be7586a-kube-api-access-2whbw\") pod \"payload-processing-e2e-worker-single-main-f119d1-848f68677lv6zp\" (UID: \"3173428e-13b0-469a-b95e-30cd0be7586a\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-f119d1-848f68677lv6zp"
Nov 02 13:43:43.602437 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:43:43.602279    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3173428e-13b0-469a-b95e-30cd0be7586a-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-f119d1-848f68677lv6zp\" (UID: \"3173428e-13b0-469a-b95e-30cd0be7586a\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-f119d1-848f68677lv6zp"
Nov 02 13:43:43.603349 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:43:43.603322    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3173428e-13b0-469a-b95e-30cd0be7586a-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-f119d1-848f68677lv6zp\" (UID: \"3173428e-13b0-469a-b95e-30cd0be7586a\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-f119d1-848f68677lv6zp"
Nov 02 13:43:43.627429 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:43:43.627044    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2whbw\" (UniqueName: \"kubernetes.io/projected/3173428e-13b0-469a-b95e-30cd0be7586a-kube-api-access-2whbw\") pod \"payload-processing-e2e-worker-single-main-f119d1-848f68677lv6zp\" (UID: \"3173428e-13b0-469a-b95e-30cd0be7586a\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-f119d1-848f68677lv6zp"
Nov 02 13:43:43.643611 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:43:43.643584    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-f119d1-848f68677lv6zp"
Nov 02 13:43:43.666371 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:43:43.666344    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f119d1-667b99d486-mms7p"]
Nov 02 13:43:43.670101 ip-10-0-128-132 kubenswrapper[4559]: W1102 13:43:43.670077    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4791f0c3_afec_48aa_ab44_f538c1f9a0dd.slice/crio-264e711956973c91750248470477324b540afd7f20e6403baf25c0796df68dc5 WatchSource:0}: Error finding container 264e711956973c91750248470477324b540afd7f20e6403baf25c0796df68dc5: Status 404 returned error can't find the container with id 264e711956973c91750248470477324b540afd7f20e6403baf25c0796df68dc5
Nov 02 13:43:43.671915 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:43:43.671895    4559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 13:43:43.873954 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:43:43.873911    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-f119d1-7bb5dp7pc2"]
Nov 02 13:43:43.875423 ip-10-0-128-132 kubenswrapper[4559]: W1102 13:43:43.875398    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfc4d4ea_5b86_4473_be96_16e33bafca2a.slice/crio-0ccc3ad675eab9fd867b3f018dab4bd2d5964d6bcbbc2168a490d63feb59bf68 WatchSource:0}: Error finding container 0ccc3ad675eab9fd867b3f018dab4bd2d5964d6bcbbc2168a490d63feb59bf68: Status 404 returned error can't find the container with id 0ccc3ad675eab9fd867b3f018dab4bd2d5964d6bcbbc2168a490d63feb59bf68
Nov 02 13:43:43.986067 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:43:43.986040    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-f119d1-848f68677lv6zp"]
Nov 02 13:43:43.986180 ip-10-0-128-132 kubenswrapper[4559]: W1102 13:43:43.986162    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3173428e_13b0_469a_b95e_30cd0be7586a.slice/crio-07ab1a79f96a6a1e41b125769508e87dad9e5dbce46727804774dc0b67b947d9 WatchSource:0}: Error finding container 07ab1a79f96a6a1e41b125769508e87dad9e5dbce46727804774dc0b67b947d9: Status 404 returned error can't find the container with id 07ab1a79f96a6a1e41b125769508e87dad9e5dbce46727804774dc0b67b947d9
Nov 02 13:43:44.574536 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:43:44.574495    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f119d1-667b99d486-mms7p" event={"ID":"4791f0c3-afec-48aa-ab44-f538c1f9a0dd","Type":"ContainerStarted","Data":"0e1e8267a61a7aa7b929c190da629332e0bb127facef7130697d9b4359f8c6d3"}
Nov 02 13:43:44.574789 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:43:44.574572    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f119d1-667b99d486-mms7p"
Nov 02 13:43:44.574789 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:43:44.574596    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f119d1-667b99d486-mms7p" event={"ID":"4791f0c3-afec-48aa-ab44-f538c1f9a0dd","Type":"ContainerStarted","Data":"264e711956973c91750248470477324b540afd7f20e6403baf25c0796df68dc5"}
Nov 02 13:43:44.576728 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:43:44.576698    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-f119d1-848f68677lv6zp" event={"ID":"3173428e-13b0-469a-b95e-30cd0be7586a","Type":"ContainerStarted","Data":"81a54bef33f34d8eea7084be90cefcc65c5a3f3f04a15ddc89b20d32f7ca20d8"}
Nov 02 13:43:44.576894 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:43:44.576754    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-f119d1-848f68677lv6zp" event={"ID":"3173428e-13b0-469a-b95e-30cd0be7586a","Type":"ContainerStarted","Data":"07ab1a79f96a6a1e41b125769508e87dad9e5dbce46727804774dc0b67b947d9"}
Nov 02 13:43:44.576894 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:43:44.576784    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-f119d1-848f68677lv6zp"
Nov 02 13:43:44.578479 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:43:44.578452    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f119d1-7bb5dp7pc2" event={"ID":"bfc4d4ea-5b86-4473-be96-16e33bafca2a","Type":"ContainerStarted","Data":"1986f012448877c6c45c2cd03f4ba9945198fcc576fc32939ba0a3dfe8e32313"}
Nov 02 13:43:44.578668 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:43:44.578633    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f119d1-7bb5dp7pc2" event={"ID":"bfc4d4ea-5b86-4473-be96-16e33bafca2a","Type":"ContainerStarted","Data":"0ccc3ad675eab9fd867b3f018dab4bd2d5964d6bcbbc2168a490d63feb59bf68"}
Nov 02 13:43:44.578806 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:43:44.578792    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f119d1-7bb5dp7pc2"
Nov 02 13:43:44.629392 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:43:44.629329    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f119d1-667b99d486-mms7p" podStartSLOduration=2.23466476 podStartE2EDuration="2.629309199s" totalImagesPullingTime="394.644439ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:43:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:43:44.627053732 +0000 UTC m=+4620.882452828" watchObservedRunningTime="2026-11-02 13:43:44.629309199 +0000 UTC m=+4620.884708297"
Nov 02 13:43:44.673177 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:43:44.673122    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f119d1-7bb5dp7pc2" podStartSLOduration=1.673107018 podStartE2EDuration="1.673107018s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:43:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:43:44.670497148 +0000 UTC m=+4620.925896243" watchObservedRunningTime="2026-11-02 13:43:44.673107018 +0000 UTC m=+4620.928506151"
Nov 02 13:43:44.721236 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:43:44.721184    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-f119d1-848f68677lv6zp" podStartSLOduration=1.721170282 podStartE2EDuration="1.721170282s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:43:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:43:44.719525414 +0000 UTC m=+4620.974924510" watchObservedRunningTime="2026-11-02 13:43:44.721170282 +0000 UTC m=+4620.976569376"
Nov 02 13:43:45.088772 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:43:45.088736    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqcmj" containerName="extractor"
Nov 02 13:43:47.217424 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:43:47.217386    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"]
Nov 02 13:43:50.589804 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:43:50.589768    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f119d1-667b99d486-mms7p"
Nov 02 13:43:55.587442 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:43:55.587353    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-f119d1-848f68677lv6zp"
Nov 02 13:43:55.587945 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:43:55.587590    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f119d1-7bb5dp7pc2"
Nov 02 13:44:02.682661 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:44:02.682618    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-f119d1-848f68677lv6zp"]
Nov 02 13:44:02.720763 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:44:02.720728    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-f119d1-7bb5dp7pc2"]
Nov 02 13:44:04.740962 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:44:04.740928    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f119d1-667b99d486-mms7p"]
Nov 02 13:44:04.741462 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:44:04.741196    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f119d1-667b99d486-mms7p" podUID="4791f0c3-afec-48aa-ab44-f538c1f9a0dd" containerName="maas-api" containerID="cri-o://0e1e8267a61a7aa7b929c190da629332e0bb127facef7130697d9b4359f8c6d3" gracePeriod=30
Nov 02 13:44:04.995731 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:44:04.995670    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f119d1-667b99d486-mms7p"
Nov 02 13:44:05.031885 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:44:05.031849    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-fhx8f"]
Nov 02 13:44:05.034316 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:44:05.034291    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-fhx8f"]
Nov 02 13:44:05.095433 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:44:05.095403    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4791f0c3-afec-48aa-ab44-f538c1f9a0dd-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4791f0c3-afec-48aa-ab44-f538c1f9a0dd-openshift-service-ca\") pod \"4791f0c3-afec-48aa-ab44-f538c1f9a0dd\" (UID: \"4791f0c3-afec-48aa-ab44-f538c1f9a0dd\") "
Nov 02 13:44:05.095598 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:44:05.095530    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4791f0c3-afec-48aa-ab44-f538c1f9a0dd-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4791f0c3-afec-48aa-ab44-f538c1f9a0dd-metrics-certs\") pod \"4791f0c3-afec-48aa-ab44-f538c1f9a0dd\" (UID: \"4791f0c3-afec-48aa-ab44-f538c1f9a0dd\") "
Nov 02 13:44:05.095598 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:44:05.095568    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4791f0c3-afec-48aa-ab44-f538c1f9a0dd-kube-api-access-gf24r\" (UniqueName: \"kubernetes.io/projected/4791f0c3-afec-48aa-ab44-f538c1f9a0dd-kube-api-access-gf24r\") pod \"4791f0c3-afec-48aa-ab44-f538c1f9a0dd\" (UID: \"4791f0c3-afec-48aa-ab44-f538c1f9a0dd\") "
Nov 02 13:44:05.095598 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:44:05.095595    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4791f0c3-afec-48aa-ab44-f538c1f9a0dd-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4791f0c3-afec-48aa-ab44-f538c1f9a0dd-maas-api-tls\") pod \"4791f0c3-afec-48aa-ab44-f538c1f9a0dd\" (UID: \"4791f0c3-afec-48aa-ab44-f538c1f9a0dd\") "
Nov 02 13:44:05.095837 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:44:05.095808    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4791f0c3-afec-48aa-ab44-f538c1f9a0dd-openshift-service-ca" pod "4791f0c3-afec-48aa-ab44-f538c1f9a0dd" (UID: "4791f0c3-afec-48aa-ab44-f538c1f9a0dd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:44:05.097803 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:44:05.097778    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4791f0c3-afec-48aa-ab44-f538c1f9a0dd-maas-api-tls" pod "4791f0c3-afec-48aa-ab44-f538c1f9a0dd" (UID: "4791f0c3-afec-48aa-ab44-f538c1f9a0dd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:44:05.097930 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:44:05.097819    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4791f0c3-afec-48aa-ab44-f538c1f9a0dd-metrics-certs" pod "4791f0c3-afec-48aa-ab44-f538c1f9a0dd" (UID: "4791f0c3-afec-48aa-ab44-f538c1f9a0dd"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:44:05.097930 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:44:05.097862    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4791f0c3-afec-48aa-ab44-f538c1f9a0dd-kube-api-access-gf24r" pod "4791f0c3-afec-48aa-ab44-f538c1f9a0dd" (UID: "4791f0c3-afec-48aa-ab44-f538c1f9a0dd"). InnerVolumeSpecName "kube-api-access-gf24r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:44:05.196881 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:44:05.196844    4559 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4791f0c3-afec-48aa-ab44-f538c1f9a0dd-metrics-certs\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:44:05.196881 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:44:05.196875    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gf24r\" (UniqueName: \"kubernetes.io/projected/4791f0c3-afec-48aa-ab44-f538c1f9a0dd-kube-api-access-gf24r\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:44:05.197101 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:44:05.196891    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4791f0c3-afec-48aa-ab44-f538c1f9a0dd-maas-api-tls\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:44:05.197101 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:44:05.196904    4559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4791f0c3-afec-48aa-ab44-f538c1f9a0dd-openshift-service-ca\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:44:05.677221 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:44:05.677186    4559 generic.go:356] "Generic (PLEG): container finished" podID="4791f0c3-afec-48aa-ab44-f538c1f9a0dd" containerID="0e1e8267a61a7aa7b929c190da629332e0bb127facef7130697d9b4359f8c6d3" exitCode=0
Nov 02 13:44:05.677409 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:44:05.677235    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f119d1-667b99d486-mms7p"
Nov 02 13:44:05.677409 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:44:05.677278    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f119d1-667b99d486-mms7p" event={"ID":"4791f0c3-afec-48aa-ab44-f538c1f9a0dd","Type":"ContainerDied","Data":"0e1e8267a61a7aa7b929c190da629332e0bb127facef7130697d9b4359f8c6d3"}
Nov 02 13:44:05.677409 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:44:05.677327    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f119d1-667b99d486-mms7p" event={"ID":"4791f0c3-afec-48aa-ab44-f538c1f9a0dd","Type":"ContainerDied","Data":"264e711956973c91750248470477324b540afd7f20e6403baf25c0796df68dc5"}
Nov 02 13:44:05.677409 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:44:05.677349    4559 scope.go:122] "RemoveContainer" containerID="0e1e8267a61a7aa7b929c190da629332e0bb127facef7130697d9b4359f8c6d3"
Nov 02 13:44:05.686501 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:44:05.686482    4559 scope.go:122] "RemoveContainer" containerID="0e1e8267a61a7aa7b929c190da629332e0bb127facef7130697d9b4359f8c6d3"
Nov 02 13:44:05.686800 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:44:05.686777    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e1e8267a61a7aa7b929c190da629332e0bb127facef7130697d9b4359f8c6d3\": container with ID starting with 0e1e8267a61a7aa7b929c190da629332e0bb127facef7130697d9b4359f8c6d3 not found: ID does not exist" containerID="0e1e8267a61a7aa7b929c190da629332e0bb127facef7130697d9b4359f8c6d3"
Nov 02 13:44:05.686917 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:44:05.686807    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e1e8267a61a7aa7b929c190da629332e0bb127facef7130697d9b4359f8c6d3"} err="failed to get container status \"0e1e8267a61a7aa7b929c190da629332e0bb127facef7130697d9b4359f8c6d3\": rpc error: code = NotFound desc = could not find container \"0e1e8267a61a7aa7b929c190da629332e0bb127facef7130697d9b4359f8c6d3\": container with ID starting with 0e1e8267a61a7aa7b929c190da629332e0bb127facef7130697d9b4359f8c6d3 not found: ID does not exist"
Nov 02 13:44:05.730715 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:44:05.730692    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f119d1-667b99d486-mms7p"]
Nov 02 13:44:05.743415 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:44:05.743391    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f119d1-667b99d486-mms7p"]
Nov 02 13:44:05.891248 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:44:05.891213    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4791f0c3-afec-48aa-ab44-f538c1f9a0dd" path="/var/lib/kubelet/pods/4791f0c3-afec-48aa-ab44-f538c1f9a0dd/volumes"
Nov 02 13:44:05.891664 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:44:05.891625    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="566c809f-50dc-487c-816c-2e7cd62e11c9" path="/var/lib/kubelet/pods/566c809f-50dc-487c-816c-2e7cd62e11c9/volumes"
Nov 02 13:44:07.682986 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:44:07.682940    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-f119d1-848f68677lv6zp" podUID="3173428e-13b0-469a-b95e-30cd0be7586a" containerName="payload-processing" containerID="cri-o://81a54bef33f34d8eea7084be90cefcc65c5a3f3f04a15ddc89b20d32f7ca20d8" gracePeriod=25
Nov 02 13:44:07.721075 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:44:07.721028    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f119d1-7bb5dp7pc2" podUID="bfc4d4ea-5b86-4473-be96-16e33bafca2a" containerName="payload-pre-processing" containerID="cri-o://1986f012448877c6c45c2cd03f4ba9945198fcc576fc32939ba0a3dfe8e32313" gracePeriod=25
Nov 02 13:44:07.770549 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:44:07.770525    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"]
Nov 02 13:44:07.967894 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:44:07.967871    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-f119d1-848f68677lv6zp"
Nov 02 13:44:07.989822 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:44:07.989799    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f119d1-7bb5dp7pc2"
Nov 02 13:44:08.025544 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:44:08.025522    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bfc4d4ea-5b86-4473-be96-16e33bafca2a-kube-api-access-mfkdz\" (UniqueName: \"kubernetes.io/projected/bfc4d4ea-5b86-4473-be96-16e33bafca2a-kube-api-access-mfkdz\") pod \"bfc4d4ea-5b86-4473-be96-16e33bafca2a\" (UID: \"bfc4d4ea-5b86-4473-be96-16e33bafca2a\") "
Nov 02 13:44:08.025716 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:44:08.025574    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3173428e-13b0-469a-b95e-30cd0be7586a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3173428e-13b0-469a-b95e-30cd0be7586a-plugins-config-volume\") pod \"3173428e-13b0-469a-b95e-30cd0be7586a\" (UID: \"3173428e-13b0-469a-b95e-30cd0be7586a\") "
Nov 02 13:44:08.025716 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:44:08.025695    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bfc4d4ea-5b86-4473-be96-16e33bafca2a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfc4d4ea-5b86-4473-be96-16e33bafca2a-plugins-config-volume\") pod \"bfc4d4ea-5b86-4473-be96-16e33bafca2a\" (UID: \"bfc4d4ea-5b86-4473-be96-16e33bafca2a\") "
Nov 02 13:44:08.025852 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:44:08.025764    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3173428e-13b0-469a-b95e-30cd0be7586a-kube-api-access-2whbw\" (UniqueName: \"kubernetes.io/projected/3173428e-13b0-469a-b95e-30cd0be7586a-kube-api-access-2whbw\") pod \"3173428e-13b0-469a-b95e-30cd0be7586a\" (UID: \"3173428e-13b0-469a-b95e-30cd0be7586a\") "
Nov 02 13:44:08.026053 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:44:08.026023    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3173428e-13b0-469a-b95e-30cd0be7586a-plugins-config-volume" pod "3173428e-13b0-469a-b95e-30cd0be7586a" (UID: "3173428e-13b0-469a-b95e-30cd0be7586a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:44:08.026181 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:44:08.026106    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfc4d4ea-5b86-4473-be96-16e33bafca2a-plugins-config-volume" pod "bfc4d4ea-5b86-4473-be96-16e33bafca2a" (UID: "bfc4d4ea-5b86-4473-be96-16e33bafca2a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:44:08.027691 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:44:08.027664    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfc4d4ea-5b86-4473-be96-16e33bafca2a-kube-api-access-mfkdz" pod "bfc4d4ea-5b86-4473-be96-16e33bafca2a" (UID: "bfc4d4ea-5b86-4473-be96-16e33bafca2a"). InnerVolumeSpecName "kube-api-access-mfkdz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:44:08.027910 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:44:08.027892    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3173428e-13b0-469a-b95e-30cd0be7586a-kube-api-access-2whbw" pod "3173428e-13b0-469a-b95e-30cd0be7586a" (UID: "3173428e-13b0-469a-b95e-30cd0be7586a"). InnerVolumeSpecName "kube-api-access-2whbw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:44:08.127169 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:44:08.127136    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2whbw\" (UniqueName: \"kubernetes.io/projected/3173428e-13b0-469a-b95e-30cd0be7586a-kube-api-access-2whbw\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:44:08.127169 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:44:08.127164    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mfkdz\" (UniqueName: \"kubernetes.io/projected/bfc4d4ea-5b86-4473-be96-16e33bafca2a-kube-api-access-mfkdz\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:44:08.127169 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:44:08.127175    4559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3173428e-13b0-469a-b95e-30cd0be7586a-plugins-config-volume\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:44:08.127389 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:44:08.127183    4559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfc4d4ea-5b86-4473-be96-16e33bafca2a-plugins-config-volume\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:44:08.693280 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:44:08.693247    4559 generic.go:356] "Generic (PLEG): container finished" podID="bfc4d4ea-5b86-4473-be96-16e33bafca2a" containerID="1986f012448877c6c45c2cd03f4ba9945198fcc576fc32939ba0a3dfe8e32313" exitCode=0
Nov 02 13:44:08.693806 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:44:08.693311    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f119d1-7bb5dp7pc2"
Nov 02 13:44:08.693806 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:44:08.693332    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f119d1-7bb5dp7pc2" event={"ID":"bfc4d4ea-5b86-4473-be96-16e33bafca2a","Type":"ContainerDied","Data":"1986f012448877c6c45c2cd03f4ba9945198fcc576fc32939ba0a3dfe8e32313"}
Nov 02 13:44:08.693806 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:44:08.693384    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f119d1-7bb5dp7pc2" event={"ID":"bfc4d4ea-5b86-4473-be96-16e33bafca2a","Type":"ContainerDied","Data":"0ccc3ad675eab9fd867b3f018dab4bd2d5964d6bcbbc2168a490d63feb59bf68"}
Nov 02 13:44:08.693806 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:44:08.693407    4559 scope.go:122] "RemoveContainer" containerID="1986f012448877c6c45c2cd03f4ba9945198fcc576fc32939ba0a3dfe8e32313"
Nov 02 13:44:08.695025 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:44:08.695002    4559 generic.go:356] "Generic (PLEG): container finished" podID="3173428e-13b0-469a-b95e-30cd0be7586a" containerID="81a54bef33f34d8eea7084be90cefcc65c5a3f3f04a15ddc89b20d32f7ca20d8" exitCode=0
Nov 02 13:44:08.695129 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:44:08.695032    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-f119d1-848f68677lv6zp" event={"ID":"3173428e-13b0-469a-b95e-30cd0be7586a","Type":"ContainerDied","Data":"81a54bef33f34d8eea7084be90cefcc65c5a3f3f04a15ddc89b20d32f7ca20d8"}
Nov 02 13:44:08.695129 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:44:08.695075    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-f119d1-848f68677lv6zp"
Nov 02 13:44:08.695225 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:44:08.695083    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-f119d1-848f68677lv6zp" event={"ID":"3173428e-13b0-469a-b95e-30cd0be7586a","Type":"ContainerDied","Data":"07ab1a79f96a6a1e41b125769508e87dad9e5dbce46727804774dc0b67b947d9"}
Nov 02 13:44:08.705836 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:44:08.705818    4559 scope.go:122] "RemoveContainer" containerID="1986f012448877c6c45c2cd03f4ba9945198fcc576fc32939ba0a3dfe8e32313"
Nov 02 13:44:08.706143 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:44:08.706117    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1986f012448877c6c45c2cd03f4ba9945198fcc576fc32939ba0a3dfe8e32313\": container with ID starting with 1986f012448877c6c45c2cd03f4ba9945198fcc576fc32939ba0a3dfe8e32313 not found: ID does not exist" containerID="1986f012448877c6c45c2cd03f4ba9945198fcc576fc32939ba0a3dfe8e32313"
Nov 02 13:44:08.706264 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:44:08.706150    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1986f012448877c6c45c2cd03f4ba9945198fcc576fc32939ba0a3dfe8e32313"} err="failed to get container status \"1986f012448877c6c45c2cd03f4ba9945198fcc576fc32939ba0a3dfe8e32313\": rpc error: code = NotFound desc = could not find container \"1986f012448877c6c45c2cd03f4ba9945198fcc576fc32939ba0a3dfe8e32313\": container with ID starting with 1986f012448877c6c45c2cd03f4ba9945198fcc576fc32939ba0a3dfe8e32313 not found: ID does not exist"
Nov 02 13:44:08.706264 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:44:08.706169    4559 scope.go:122] "RemoveContainer" containerID="81a54bef33f34d8eea7084be90cefcc65c5a3f3f04a15ddc89b20d32f7ca20d8"
Nov 02 13:44:08.715405 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:44:08.715387    4559 scope.go:122] "RemoveContainer" containerID="81a54bef33f34d8eea7084be90cefcc65c5a3f3f04a15ddc89b20d32f7ca20d8"
Nov 02 13:44:08.715680 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:44:08.715663    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81a54bef33f34d8eea7084be90cefcc65c5a3f3f04a15ddc89b20d32f7ca20d8\": container with ID starting with 81a54bef33f34d8eea7084be90cefcc65c5a3f3f04a15ddc89b20d32f7ca20d8 not found: ID does not exist" containerID="81a54bef33f34d8eea7084be90cefcc65c5a3f3f04a15ddc89b20d32f7ca20d8"
Nov 02 13:44:08.715730 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:44:08.715686    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81a54bef33f34d8eea7084be90cefcc65c5a3f3f04a15ddc89b20d32f7ca20d8"} err="failed to get container status \"81a54bef33f34d8eea7084be90cefcc65c5a3f3f04a15ddc89b20d32f7ca20d8\": rpc error: code = NotFound desc = could not find container \"81a54bef33f34d8eea7084be90cefcc65c5a3f3f04a15ddc89b20d32f7ca20d8\": container with ID starting with 81a54bef33f34d8eea7084be90cefcc65c5a3f3f04a15ddc89b20d32f7ca20d8 not found: ID does not exist"
Nov 02 13:44:08.745987 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:44:08.745968    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-f119d1-848f68677lv6zp"]
Nov 02 13:44:08.759816 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:44:08.759795    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-f119d1-848f68677lv6zp"]
Nov 02 13:44:08.784581 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:44:08.784559    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-f119d1-7bb5dp7pc2"]
Nov 02 13:44:08.788888 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:44:08.788868    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-f119d1-7bb5dp7pc2"]
Nov 02 13:44:09.891562 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:44:09.891531    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3173428e-13b0-469a-b95e-30cd0be7586a" path="/var/lib/kubelet/pods/3173428e-13b0-469a-b95e-30cd0be7586a/volumes"
Nov 02 13:44:09.891954 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:44:09.891909    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bfc4d4ea-5b86-4473-be96-16e33bafca2a" path="/var/lib/kubelet/pods/bfc4d4ea-5b86-4473-be96-16e33bafca2a/volumes"
Nov 02 13:44:11.092339 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:44:11.092305    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-f8e64488-kserve-677b68f4b9-l2qzr"]
Nov 02 13:44:11.092735 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:44:11.092704    4559 cpu_manager.go:414] "containerMap: removing container" podUID="bfc4d4ea-5b86-4473-be96-16e33bafca2a" containerName="payload-pre-processing"
Nov 02 13:44:11.092735 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:44:11.092717    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bfc4d4ea-5b86-4473-be96-16e33bafca2a" containerName="payload-pre-processing"
Nov 02 13:44:11.092735 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:44:11.092729    4559 cpu_manager.go:414] "containerMap: removing container" podUID="3173428e-13b0-469a-b95e-30cd0be7586a" containerName="payload-processing"
Nov 02 13:44:11.092735 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:44:11.092734    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3173428e-13b0-469a-b95e-30cd0be7586a" containerName="payload-processing"
Nov 02 13:44:11.092892 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:44:11.092744    4559 cpu_manager.go:414] "containerMap: removing container" podUID="4791f0c3-afec-48aa-ab44-f538c1f9a0dd" containerName="maas-api"
Nov 02 13:44:11.092892 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:44:11.092749    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4791f0c3-afec-48aa-ab44-f538c1f9a0dd" containerName="maas-api"
Nov 02 13:44:11.092892 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:44:11.092809    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="bfc4d4ea-5b86-4473-be96-16e33bafca2a" containerName="payload-pre-processing"
Nov 02 13:44:11.092892 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:44:11.092817    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="3173428e-13b0-469a-b95e-30cd0be7586a" containerName="payload-processing"
Nov 02 13:44:11.092892 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:44:11.092823    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="4791f0c3-afec-48aa-ab44-f538c1f9a0dd" containerName="maas-api"
Nov 02 13:44:11.097434 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:44:11.097409    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-f8e64488-kserve-677b68f4b9-l2qzr"
Nov 02 13:44:11.126811 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:44:11.126786    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-f8e64488-kserve-self-signed-certs\""
Nov 02 13:44:11.128884 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:44:11.128860    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-f8e64488-kserve-677b68f4b9-l2qzr"]
Nov 02 13:44:11.154259 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:44:11.154225    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43d356ce-f833-4912-b8c4-41c47eb5c798-tmp-dir\") pod \"e2e-same-id-a-f8e64488-kserve-677b68f4b9-l2qzr\" (UID: \"43d356ce-f833-4912-b8c4-41c47eb5c798\") " pod="llm/e2e-same-id-a-f8e64488-kserve-677b68f4b9-l2qzr"
Nov 02 13:44:11.154373 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:44:11.154269    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brtbb\" (UniqueName: \"kubernetes.io/projected/43d356ce-f833-4912-b8c4-41c47eb5c798-kube-api-access-brtbb\") pod \"e2e-same-id-a-f8e64488-kserve-677b68f4b9-l2qzr\" (UID: \"43d356ce-f833-4912-b8c4-41c47eb5c798\") " pod="llm/e2e-same-id-a-f8e64488-kserve-677b68f4b9-l2qzr"
Nov 02 13:44:11.154419 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:44:11.154379    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43d356ce-f833-4912-b8c4-41c47eb5c798-model-cache\") pod \"e2e-same-id-a-f8e64488-kserve-677b68f4b9-l2qzr\" (UID: \"43d356ce-f833-4912-b8c4-41c47eb5c798\") " pod="llm/e2e-same-id-a-f8e64488-kserve-677b68f4b9-l2qzr"
Nov 02 13:44:11.154419 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:44:11.154401    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/43d356ce-f833-4912-b8c4-41c47eb5c798-dshm\") pod \"e2e-same-id-a-f8e64488-kserve-677b68f4b9-l2qzr\" (UID: \"43d356ce-f833-4912-b8c4-41c47eb5c798\") " pod="llm/e2e-same-id-a-f8e64488-kserve-677b68f4b9-l2qzr"
Nov 02 13:44:11.154492 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:44:11.154429    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/43d356ce-f833-4912-b8c4-41c47eb5c798-home\") pod \"e2e-same-id-a-f8e64488-kserve-677b68f4b9-l2qzr\" (UID: \"43d356ce-f833-4912-b8c4-41c47eb5c798\") " pod="llm/e2e-same-id-a-f8e64488-kserve-677b68f4b9-l2qzr"
Nov 02 13:44:11.154492 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:44:11.154447    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43d356ce-f833-4912-b8c4-41c47eb5c798-tls-certs\") pod \"e2e-same-id-a-f8e64488-kserve-677b68f4b9-l2qzr\" (UID: \"43d356ce-f833-4912-b8c4-41c47eb5c798\") " pod="llm/e2e-same-id-a-f8e64488-kserve-677b68f4b9-l2qzr"
Nov 02 13:44:11.255216 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:44:11.255179    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43d356ce-f833-4912-b8c4-41c47eb5c798-model-cache\") pod \"e2e-same-id-a-f8e64488-kserve-677b68f4b9-l2qzr\" (UID: \"43d356ce-f833-4912-b8c4-41c47eb5c798\") " pod="llm/e2e-same-id-a-f8e64488-kserve-677b68f4b9-l2qzr"
Nov 02 13:44:11.255216 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:44:11.255220    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/43d356ce-f833-4912-b8c4-41c47eb5c798-dshm\") pod \"e2e-same-id-a-f8e64488-kserve-677b68f4b9-l2qzr\" (UID: \"43d356ce-f833-4912-b8c4-41c47eb5c798\") " pod="llm/e2e-same-id-a-f8e64488-kserve-677b68f4b9-l2qzr"
Nov 02 13:44:11.255457 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:44:11.255251    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/43d356ce-f833-4912-b8c4-41c47eb5c798-home\") pod \"e2e-same-id-a-f8e64488-kserve-677b68f4b9-l2qzr\" (UID: \"43d356ce-f833-4912-b8c4-41c47eb5c798\") " pod="llm/e2e-same-id-a-f8e64488-kserve-677b68f4b9-l2qzr"
Nov 02 13:44:11.255457 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:44:11.255277    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43d356ce-f833-4912-b8c4-41c47eb5c798-tls-certs\") pod \"e2e-same-id-a-f8e64488-kserve-677b68f4b9-l2qzr\" (UID: \"43d356ce-f833-4912-b8c4-41c47eb5c798\") " pod="llm/e2e-same-id-a-f8e64488-kserve-677b68f4b9-l2qzr"
Nov 02 13:44:11.255457 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:44:11.255333    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43d356ce-f833-4912-b8c4-41c47eb5c798-tmp-dir\") pod \"e2e-same-id-a-f8e64488-kserve-677b68f4b9-l2qzr\" (UID: \"43d356ce-f833-4912-b8c4-41c47eb5c798\") " pod="llm/e2e-same-id-a-f8e64488-kserve-677b68f4b9-l2qzr"
Nov 02 13:44:11.255457 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:44:11.255369    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brtbb\" (UniqueName: \"kubernetes.io/projected/43d356ce-f833-4912-b8c4-41c47eb5c798-kube-api-access-brtbb\") pod \"e2e-same-id-a-f8e64488-kserve-677b68f4b9-l2qzr\" (UID: \"43d356ce-f833-4912-b8c4-41c47eb5c798\") " pod="llm/e2e-same-id-a-f8e64488-kserve-677b68f4b9-l2qzr"
Nov 02 13:44:11.255678 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:44:11.255625    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43d356ce-f833-4912-b8c4-41c47eb5c798-model-cache\") pod \"e2e-same-id-a-f8e64488-kserve-677b68f4b9-l2qzr\" (UID: \"43d356ce-f833-4912-b8c4-41c47eb5c798\") " pod="llm/e2e-same-id-a-f8e64488-kserve-677b68f4b9-l2qzr"
Nov 02 13:44:11.255756 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:44:11.255736    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/43d356ce-f833-4912-b8c4-41c47eb5c798-home\") pod \"e2e-same-id-a-f8e64488-kserve-677b68f4b9-l2qzr\" (UID: \"43d356ce-f833-4912-b8c4-41c47eb5c798\") " pod="llm/e2e-same-id-a-f8e64488-kserve-677b68f4b9-l2qzr"
Nov 02 13:44:11.255816 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:44:11.255784    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43d356ce-f833-4912-b8c4-41c47eb5c798-tmp-dir\") pod \"e2e-same-id-a-f8e64488-kserve-677b68f4b9-l2qzr\" (UID: \"43d356ce-f833-4912-b8c4-41c47eb5c798\") " pod="llm/e2e-same-id-a-f8e64488-kserve-677b68f4b9-l2qzr"
Nov 02 13:44:11.257599 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:44:11.257567    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/43d356ce-f833-4912-b8c4-41c47eb5c798-dshm\") pod \"e2e-same-id-a-f8e64488-kserve-677b68f4b9-l2qzr\" (UID: \"43d356ce-f833-4912-b8c4-41c47eb5c798\") " pod="llm/e2e-same-id-a-f8e64488-kserve-677b68f4b9-l2qzr"
Nov 02 13:44:11.258063 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:44:11.258040    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43d356ce-f833-4912-b8c4-41c47eb5c798-tls-certs\") pod \"e2e-same-id-a-f8e64488-kserve-677b68f4b9-l2qzr\" (UID: \"43d356ce-f833-4912-b8c4-41c47eb5c798\") " pod="llm/e2e-same-id-a-f8e64488-kserve-677b68f4b9-l2qzr"
Nov 02 13:44:11.279971 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:44:11.279940    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-brtbb\" (UniqueName: \"kubernetes.io/projected/43d356ce-f833-4912-b8c4-41c47eb5c798-kube-api-access-brtbb\") pod \"e2e-same-id-a-f8e64488-kserve-677b68f4b9-l2qzr\" (UID: \"43d356ce-f833-4912-b8c4-41c47eb5c798\") " pod="llm/e2e-same-id-a-f8e64488-kserve-677b68f4b9-l2qzr"
Nov 02 13:44:11.409972 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:44:11.409880    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-f8e64488-kserve-677b68f4b9-l2qzr"
Nov 02 13:44:11.710374 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:44:11.710342    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-f8e64488-kserve-677b68f4b9-l2qzr"]
Nov 02 13:44:11.710691 ip-10-0-128-132 kubenswrapper[4559]: W1102 13:44:11.710665    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod43d356ce_f833_4912_b8c4_41c47eb5c798.slice/crio-701c01ca8045d0026fe7e6595261295845ad6fee75bea2cb08693604106e677b WatchSource:0}: Error finding container 701c01ca8045d0026fe7e6595261295845ad6fee75bea2cb08693604106e677b: Status 404 returned error can't find the container with id 701c01ca8045d0026fe7e6595261295845ad6fee75bea2cb08693604106e677b
Nov 02 13:44:12.612153 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:44:12.612117    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"]
Nov 02 13:44:12.718664 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:44:12.718604    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-f8e64488-kserve-677b68f4b9-l2qzr" event={"ID":"43d356ce-f833-4912-b8c4-41c47eb5c798","Type":"ContainerStarted","Data":"d42de798c0e3bf288810a11ca56dda9a3cef522a37c28cc5416d5684f24b0849"}
Nov 02 13:44:12.718863 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:44:12.718675    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-f8e64488-kserve-677b68f4b9-l2qzr" event={"ID":"43d356ce-f833-4912-b8c4-41c47eb5c798","Type":"ContainerStarted","Data":"701c01ca8045d0026fe7e6595261295845ad6fee75bea2cb08693604106e677b"}
Nov 02 13:44:12.761464 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:44:12.761414    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-a-f8e64488-kserve-677b68f4b9-l2qzr" podStartSLOduration=1.7613981650000001 podStartE2EDuration="1.761398165s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:44:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:44:12.761351266 +0000 UTC m=+4649.016750360" watchObservedRunningTime="2026-11-02 13:44:12.761398165 +0000 UTC m=+4649.016797264"
Nov 02 13:44:13.410436 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:44:13.410399    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-f8e64488-kserve-677b68f4b9-l2qzr"
Nov 02 13:44:15.209662 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:44:15.204930    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqcmj" containerName="extractor"
Nov 02 13:44:21.410855 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:44:21.410816    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-f8e64488-kserve-677b68f4b9-l2qzr"
Nov 02 13:44:21.423493 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:44:21.423470    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-f8e64488-kserve-677b68f4b9-l2qzr"
Nov 02 13:44:21.847419 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:44:21.847383    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-f8e64488-kserve-677b68f4b9-l2qzr"
Nov 02 13:44:27.017326 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:44:27.017276    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-f8e64488-kserve-d789bf9fc-8chn5"]
Nov 02 13:44:27.021460 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:44:27.021439    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-f8e64488-kserve-d789bf9fc-8chn5"
Nov 02 13:44:27.049577 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:44:27.049551    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-f8e64488-kserve-self-signed-certs\""
Nov 02 13:44:27.051538 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:44:27.051511    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-f8e64488-kserve-d789bf9fc-8chn5"]
Nov 02 13:44:27.158566 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:44:27.158530    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/687b51ca-91c0-4510-a3b4-bb903cba96df-model-cache\") pod \"e2e-same-id-b-f8e64488-kserve-d789bf9fc-8chn5\" (UID: \"687b51ca-91c0-4510-a3b4-bb903cba96df\") " pod="llm/e2e-same-id-b-f8e64488-kserve-d789bf9fc-8chn5"
Nov 02 13:44:27.158756 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:44:27.158667    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/687b51ca-91c0-4510-a3b4-bb903cba96df-tls-certs\") pod \"e2e-same-id-b-f8e64488-kserve-d789bf9fc-8chn5\" (UID: \"687b51ca-91c0-4510-a3b4-bb903cba96df\") " pod="llm/e2e-same-id-b-f8e64488-kserve-d789bf9fc-8chn5"
Nov 02 13:44:27.158817 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:44:27.158756    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/687b51ca-91c0-4510-a3b4-bb903cba96df-tmp-dir\") pod \"e2e-same-id-b-f8e64488-kserve-d789bf9fc-8chn5\" (UID: \"687b51ca-91c0-4510-a3b4-bb903cba96df\") " pod="llm/e2e-same-id-b-f8e64488-kserve-d789bf9fc-8chn5"
Nov 02 13:44:27.158817 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:44:27.158778    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnfr6\" (UniqueName: \"kubernetes.io/projected/687b51ca-91c0-4510-a3b4-bb903cba96df-kube-api-access-fnfr6\") pod \"e2e-same-id-b-f8e64488-kserve-d789bf9fc-8chn5\" (UID: \"687b51ca-91c0-4510-a3b4-bb903cba96df\") " pod="llm/e2e-same-id-b-f8e64488-kserve-d789bf9fc-8chn5"
Nov 02 13:44:27.158936 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:44:27.158869    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/687b51ca-91c0-4510-a3b4-bb903cba96df-home\") pod \"e2e-same-id-b-f8e64488-kserve-d789bf9fc-8chn5\" (UID: \"687b51ca-91c0-4510-a3b4-bb903cba96df\") " pod="llm/e2e-same-id-b-f8e64488-kserve-d789bf9fc-8chn5"
Nov 02 13:44:27.158936 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:44:27.158894    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/687b51ca-91c0-4510-a3b4-bb903cba96df-dshm\") pod \"e2e-same-id-b-f8e64488-kserve-d789bf9fc-8chn5\" (UID: \"687b51ca-91c0-4510-a3b4-bb903cba96df\") " pod="llm/e2e-same-id-b-f8e64488-kserve-d789bf9fc-8chn5"
Nov 02 13:44:27.259575 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:44:27.259537    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/687b51ca-91c0-4510-a3b4-bb903cba96df-tmp-dir\") pod \"e2e-same-id-b-f8e64488-kserve-d789bf9fc-8chn5\" (UID: \"687b51ca-91c0-4510-a3b4-bb903cba96df\") " pod="llm/e2e-same-id-b-f8e64488-kserve-d789bf9fc-8chn5"
Nov 02 13:44:27.259575 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:44:27.259575    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnfr6\" (UniqueName: \"kubernetes.io/projected/687b51ca-91c0-4510-a3b4-bb903cba96df-kube-api-access-fnfr6\") pod \"e2e-same-id-b-f8e64488-kserve-d789bf9fc-8chn5\" (UID: \"687b51ca-91c0-4510-a3b4-bb903cba96df\") " pod="llm/e2e-same-id-b-f8e64488-kserve-d789bf9fc-8chn5"
Nov 02 13:44:27.259879 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:44:27.259597    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/687b51ca-91c0-4510-a3b4-bb903cba96df-home\") pod \"e2e-same-id-b-f8e64488-kserve-d789bf9fc-8chn5\" (UID: \"687b51ca-91c0-4510-a3b4-bb903cba96df\") " pod="llm/e2e-same-id-b-f8e64488-kserve-d789bf9fc-8chn5"
Nov 02 13:44:27.259879 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:44:27.259611    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/687b51ca-91c0-4510-a3b4-bb903cba96df-dshm\") pod \"e2e-same-id-b-f8e64488-kserve-d789bf9fc-8chn5\" (UID: \"687b51ca-91c0-4510-a3b4-bb903cba96df\") " pod="llm/e2e-same-id-b-f8e64488-kserve-d789bf9fc-8chn5"
Nov 02 13:44:27.259879 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:44:27.259670    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/687b51ca-91c0-4510-a3b4-bb903cba96df-model-cache\") pod \"e2e-same-id-b-f8e64488-kserve-d789bf9fc-8chn5\" (UID: \"687b51ca-91c0-4510-a3b4-bb903cba96df\") " pod="llm/e2e-same-id-b-f8e64488-kserve-d789bf9fc-8chn5"
Nov 02 13:44:27.259879 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:44:27.259706    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/687b51ca-91c0-4510-a3b4-bb903cba96df-tls-certs\") pod \"e2e-same-id-b-f8e64488-kserve-d789bf9fc-8chn5\" (UID: \"687b51ca-91c0-4510-a3b4-bb903cba96df\") " pod="llm/e2e-same-id-b-f8e64488-kserve-d789bf9fc-8chn5"
Nov 02 13:44:27.260114 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:44:27.260008    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/687b51ca-91c0-4510-a3b4-bb903cba96df-home\") pod \"e2e-same-id-b-f8e64488-kserve-d789bf9fc-8chn5\" (UID: \"687b51ca-91c0-4510-a3b4-bb903cba96df\") " pod="llm/e2e-same-id-b-f8e64488-kserve-d789bf9fc-8chn5"
Nov 02 13:44:27.260114 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:44:27.260040    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/687b51ca-91c0-4510-a3b4-bb903cba96df-tmp-dir\") pod \"e2e-same-id-b-f8e64488-kserve-d789bf9fc-8chn5\" (UID: \"687b51ca-91c0-4510-a3b4-bb903cba96df\") " pod="llm/e2e-same-id-b-f8e64488-kserve-d789bf9fc-8chn5"
Nov 02 13:44:27.260194 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:44:27.260116    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/687b51ca-91c0-4510-a3b4-bb903cba96df-model-cache\") pod \"e2e-same-id-b-f8e64488-kserve-d789bf9fc-8chn5\" (UID: \"687b51ca-91c0-4510-a3b4-bb903cba96df\") " pod="llm/e2e-same-id-b-f8e64488-kserve-d789bf9fc-8chn5"
Nov 02 13:44:27.262013 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:44:27.261991    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/687b51ca-91c0-4510-a3b4-bb903cba96df-dshm\") pod \"e2e-same-id-b-f8e64488-kserve-d789bf9fc-8chn5\" (UID: \"687b51ca-91c0-4510-a3b4-bb903cba96df\") " pod="llm/e2e-same-id-b-f8e64488-kserve-d789bf9fc-8chn5"
Nov 02 13:44:27.262271 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:44:27.262254    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/687b51ca-91c0-4510-a3b4-bb903cba96df-tls-certs\") pod \"e2e-same-id-b-f8e64488-kserve-d789bf9fc-8chn5\" (UID: \"687b51ca-91c0-4510-a3b4-bb903cba96df\") " pod="llm/e2e-same-id-b-f8e64488-kserve-d789bf9fc-8chn5"
Nov 02 13:44:27.283730 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:44:27.283680    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnfr6\" (UniqueName: \"kubernetes.io/projected/687b51ca-91c0-4510-a3b4-bb903cba96df-kube-api-access-fnfr6\") pod \"e2e-same-id-b-f8e64488-kserve-d789bf9fc-8chn5\" (UID: \"687b51ca-91c0-4510-a3b4-bb903cba96df\") " pod="llm/e2e-same-id-b-f8e64488-kserve-d789bf9fc-8chn5"
Nov 02 13:44:27.332961 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:44:27.332933    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-f8e64488-kserve-d789bf9fc-8chn5"
Nov 02 13:44:27.633350 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:44:27.633323    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-f8e64488-kserve-d789bf9fc-8chn5"]
Nov 02 13:44:27.633816 ip-10-0-128-132 kubenswrapper[4559]: W1102 13:44:27.633788    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod687b51ca_91c0_4510_a3b4_bb903cba96df.slice/crio-85c39c9e1fc405f30de113600105585710e3cd1c5c02178bf9675d9057980682 WatchSource:0}: Error finding container 85c39c9e1fc405f30de113600105585710e3cd1c5c02178bf9675d9057980682: Status 404 returned error can't find the container with id 85c39c9e1fc405f30de113600105585710e3cd1c5c02178bf9675d9057980682
Nov 02 13:44:27.864827 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:44:27.864785    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-f8e64488-kserve-d789bf9fc-8chn5" event={"ID":"687b51ca-91c0-4510-a3b4-bb903cba96df","Type":"ContainerStarted","Data":"aca170e322797b07712b3ff4792f1006f640ff52122f0393d3d6180610bf0636"}
Nov 02 13:44:27.864827 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:44:27.864831    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-f8e64488-kserve-d789bf9fc-8chn5" event={"ID":"687b51ca-91c0-4510-a3b4-bb903cba96df","Type":"ContainerStarted","Data":"85c39c9e1fc405f30de113600105585710e3cd1c5c02178bf9675d9057980682"}
Nov 02 13:44:27.910063 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:44:27.909969    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-b-f8e64488-kserve-d789bf9fc-8chn5" podStartSLOduration=1.909955826 podStartE2EDuration="1.909955826s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:44:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:44:27.908887389 +0000 UTC m=+4664.164286485" watchObservedRunningTime="2026-11-02 13:44:27.909955826 +0000 UTC m=+4664.165354920"
Nov 02 13:44:28.333551 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:44:28.333518    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-f8e64488-kserve-d789bf9fc-8chn5"
Nov 02 13:44:29.272826 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:44:29.272785    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"]
Nov 02 13:44:37.334055 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:44:37.334018    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-f8e64488-kserve-d789bf9fc-8chn5"
Nov 02 13:44:37.346882 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:44:37.346857    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-f8e64488-kserve-d789bf9fc-8chn5"
Nov 02 13:44:37.961512 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:44:37.961486    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-f8e64488-kserve-d789bf9fc-8chn5"
Nov 02 13:44:42.031109 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:44:42.031073    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-qx7bf"]
Nov 02 13:44:42.047429 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:44:42.047393    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-qx7bf"]
Nov 02 13:44:43.891661 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:44:43.891615    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc751229-83de-49d7-b3ef-213095467cc6" path="/var/lib/kubelet/pods/cc751229-83de-49d7-b3ef-213095467cc6/volumes"
Nov 02 13:44:44.080614 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:44:44.080587    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-5r225"]
Nov 02 13:44:44.085264 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:44:44.085248    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-5r225"
Nov 02 13:44:44.103807 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:44:44.103785    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-5r225"]
Nov 02 13:44:44.114449 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:44:44.114428    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 13:44:44.114737 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:44:44.114720    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 13:44:44.114825 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:44:44.114745    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 13:44:44.114825 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:44:44.114781    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-45fzn\""
Nov 02 13:44:44.156998 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:44:44.156942    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3357a4d-f86a-40f7-9117-a87fd501e540-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-5r225\" (UID: \"d3357a4d-f86a-40f7-9117-a87fd501e540\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-5r225"
Nov 02 13:44:44.157088 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:44:44.157002    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3357a4d-f86a-40f7-9117-a87fd501e540-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-5r225\" (UID: \"d3357a4d-f86a-40f7-9117-a87fd501e540\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-5r225"
Nov 02 13:44:44.157131 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:44:44.157101    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6l2r\" (UniqueName: \"kubernetes.io/projected/d3357a4d-f86a-40f7-9117-a87fd501e540-kube-api-access-r6l2r\") pod \"maas-api-revoke-sub-452b6d7b-5r225\" (UID: \"d3357a4d-f86a-40f7-9117-a87fd501e540\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-5r225"
Nov 02 13:44:44.257882 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:44:44.257855    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3357a4d-f86a-40f7-9117-a87fd501e540-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-5r225\" (UID: \"d3357a4d-f86a-40f7-9117-a87fd501e540\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-5r225"
Nov 02 13:44:44.258009 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:44:44.257906    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6l2r\" (UniqueName: \"kubernetes.io/projected/d3357a4d-f86a-40f7-9117-a87fd501e540-kube-api-access-r6l2r\") pod \"maas-api-revoke-sub-452b6d7b-5r225\" (UID: \"d3357a4d-f86a-40f7-9117-a87fd501e540\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-5r225"
Nov 02 13:44:44.258009 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:44:44.257981    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3357a4d-f86a-40f7-9117-a87fd501e540-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-5r225\" (UID: \"d3357a4d-f86a-40f7-9117-a87fd501e540\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-5r225"
Nov 02 13:44:44.258675 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:44:44.258630    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3357a4d-f86a-40f7-9117-a87fd501e540-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-5r225\" (UID: \"d3357a4d-f86a-40f7-9117-a87fd501e540\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-5r225"
Nov 02 13:44:44.260280 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:44:44.260240    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3357a4d-f86a-40f7-9117-a87fd501e540-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-5r225\" (UID: \"d3357a4d-f86a-40f7-9117-a87fd501e540\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-5r225"
Nov 02 13:44:44.281553 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:44:44.281531    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6l2r\" (UniqueName: \"kubernetes.io/projected/d3357a4d-f86a-40f7-9117-a87fd501e540-kube-api-access-r6l2r\") pod \"maas-api-revoke-sub-452b6d7b-5r225\" (UID: \"d3357a4d-f86a-40f7-9117-a87fd501e540\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-5r225"
Nov 02 13:44:44.398885 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:44:44.398860    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-5r225"
Nov 02 13:44:44.698977 ip-10-0-128-132 kubenswrapper[4559]: W1102 13:44:44.698943    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3357a4d_f86a_40f7_9117_a87fd501e540.slice/crio-4a59bedb261efff723369da55b091d0ac882d83ba1730f6aff957b521f7ff719 WatchSource:0}: Error finding container 4a59bedb261efff723369da55b091d0ac882d83ba1730f6aff957b521f7ff719: Status 404 returned error can't find the container with id 4a59bedb261efff723369da55b091d0ac882d83ba1730f6aff957b521f7ff719
Nov 02 13:44:44.699116 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:44:44.699100    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-5r225"]
Nov 02 13:44:44.984458 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:44:44.984430    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-5r225" event={"ID":"d3357a4d-f86a-40f7-9117-a87fd501e540","Type":"ContainerStarted","Data":"4a59bedb261efff723369da55b091d0ac882d83ba1730f6aff957b521f7ff719"}
Nov 02 13:44:45.085150 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:44:45.085117    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqcmj" containerName="extractor"
Nov 02 13:44:45.313731 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:44:45.313661    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"]
Nov 02 13:44:45.993430 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:44:45.993390    4559 generic.go:356] "Generic (PLEG): container finished" podID="d3357a4d-f86a-40f7-9117-a87fd501e540" containerID="8ed795599c105564e7654815f1f5f65b0e8e7da9e7c8ca78866fc1252f145367" exitCode=0
Nov 02 13:44:45.994006 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:44:45.993543    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-5r225" event={"ID":"d3357a4d-f86a-40f7-9117-a87fd501e540","Type":"ContainerDied","Data":"8ed795599c105564e7654815f1f5f65b0e8e7da9e7c8ca78866fc1252f145367"}
Nov 02 13:44:47.737829 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:44:47.737796    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-5r225"
Nov 02 13:44:47.925662 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:44:47.922362    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3357a4d-f86a-40f7-9117-a87fd501e540-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3357a4d-f86a-40f7-9117-a87fd501e540-maas-api-openshift-ca\") pod \"d3357a4d-f86a-40f7-9117-a87fd501e540\" (UID: \"d3357a4d-f86a-40f7-9117-a87fd501e540\") "
Nov 02 13:44:47.925662 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:44:47.922436    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3357a4d-f86a-40f7-9117-a87fd501e540-kube-api-access-r6l2r\" (UniqueName: \"kubernetes.io/projected/d3357a4d-f86a-40f7-9117-a87fd501e540-kube-api-access-r6l2r\") pod \"d3357a4d-f86a-40f7-9117-a87fd501e540\" (UID: \"d3357a4d-f86a-40f7-9117-a87fd501e540\") "
Nov 02 13:44:47.925662 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:44:47.922516    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d3357a4d-f86a-40f7-9117-a87fd501e540-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3357a4d-f86a-40f7-9117-a87fd501e540-maas-api-xks-ca\") pod \"d3357a4d-f86a-40f7-9117-a87fd501e540\" (UID: \"d3357a4d-f86a-40f7-9117-a87fd501e540\") "
Nov 02 13:44:47.934669 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:44:47.933961    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3357a4d-f86a-40f7-9117-a87fd501e540-maas-api-openshift-ca" pod "d3357a4d-f86a-40f7-9117-a87fd501e540" (UID: "d3357a4d-f86a-40f7-9117-a87fd501e540"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:44:48.032663 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:44:48.032080    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3357a4d-f86a-40f7-9117-a87fd501e540-maas-api-openshift-ca\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:44:48.057335 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:44:48.057301    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3357a4d-f86a-40f7-9117-a87fd501e540-maas-api-xks-ca" pod "d3357a4d-f86a-40f7-9117-a87fd501e540" (UID: "d3357a4d-f86a-40f7-9117-a87fd501e540"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:44:48.070455 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:44:48.070180    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-5r225" event={"ID":"d3357a4d-f86a-40f7-9117-a87fd501e540","Type":"ContainerDied","Data":"4a59bedb261efff723369da55b091d0ac882d83ba1730f6aff957b521f7ff719"}
Nov 02 13:44:48.070455 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:44:48.070251    4559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4a59bedb261efff723369da55b091d0ac882d83ba1730f6aff957b521f7ff719"
Nov 02 13:44:48.070455 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:44:48.070322    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-5r225"
Nov 02 13:44:48.073557 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:44:48.073529    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3357a4d-f86a-40f7-9117-a87fd501e540-kube-api-access-r6l2r" pod "d3357a4d-f86a-40f7-9117-a87fd501e540" (UID: "d3357a4d-f86a-40f7-9117-a87fd501e540"). InnerVolumeSpecName "kube-api-access-r6l2r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:44:48.146209 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:44:48.146179    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3357a4d-f86a-40f7-9117-a87fd501e540-maas-api-xks-ca\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:44:48.146403 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:44:48.146386    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r6l2r\" (UniqueName: \"kubernetes.io/projected/d3357a4d-f86a-40f7-9117-a87fd501e540-kube-api-access-r6l2r\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:44:48.918673 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:44:48.917783    4559 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-5wtjm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.19:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 13:44:48.918673 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:44:48.917864    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5wtjm" podUID="b9661a3b-8a07-4a30-9893-20ab1d95a2a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.19:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:44:55.312230 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:44:55.312195    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-f8e64488-kserve-677b68f4b9-l2qzr"]
Nov 02 13:44:57.274512 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:44:57.274471    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"]
Nov 02 13:44:59.483033 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:44:59.482995    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-f8e64488-kserve-d789bf9fc-8chn5"]
Nov 02 13:45:00.159963 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:00.159916    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893785-fv7tz"]
Nov 02 13:45:00.160834 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:00.160806    4559 cpu_manager.go:414] "containerMap: removing container" podUID="d3357a4d-f86a-40f7-9117-a87fd501e540" containerName="revoke-keys"
Nov 02 13:45:00.161019 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:00.161002    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3357a4d-f86a-40f7-9117-a87fd501e540" containerName="revoke-keys"
Nov 02 13:45:00.161229 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:00.161213    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3357a4d-f86a-40f7-9117-a87fd501e540" containerName="revoke-keys"
Nov 02 13:45:00.201989 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:00.201908    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893785-fv7tz"
Nov 02 13:45:00.223664 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:00.202277    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893785-fv7tz"]
Nov 02 13:45:00.236942 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:00.236911    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 13:45:00.237272 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:00.237251    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-45fzn\""
Nov 02 13:45:00.249659 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:00.237455    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 13:45:00.412244 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:00.412202    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/59f18952-4e48-4885-979d-6f3cfee55f3e-maas-api-service-ca\") pod \"maas-api-key-cleanup-29893785-fv7tz\" (UID: \"59f18952-4e48-4885-979d-6f3cfee55f3e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893785-fv7tz"
Nov 02 13:45:00.415791 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:00.415753    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5bl9\" (UniqueName: \"kubernetes.io/projected/59f18952-4e48-4885-979d-6f3cfee55f3e-kube-api-access-z5bl9\") pod \"maas-api-key-cleanup-29893785-fv7tz\" (UID: \"59f18952-4e48-4885-979d-6f3cfee55f3e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893785-fv7tz"
Nov 02 13:45:00.526710 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:00.526336    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/59f18952-4e48-4885-979d-6f3cfee55f3e-maas-api-service-ca\") pod \"maas-api-key-cleanup-29893785-fv7tz\" (UID: \"59f18952-4e48-4885-979d-6f3cfee55f3e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893785-fv7tz"
Nov 02 13:45:00.526710 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:00.526428    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5bl9\" (UniqueName: \"kubernetes.io/projected/59f18952-4e48-4885-979d-6f3cfee55f3e-kube-api-access-z5bl9\") pod \"maas-api-key-cleanup-29893785-fv7tz\" (UID: \"59f18952-4e48-4885-979d-6f3cfee55f3e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893785-fv7tz"
Nov 02 13:45:00.538673 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:00.527580    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/59f18952-4e48-4885-979d-6f3cfee55f3e-maas-api-service-ca\") pod \"maas-api-key-cleanup-29893785-fv7tz\" (UID: \"59f18952-4e48-4885-979d-6f3cfee55f3e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893785-fv7tz"
Nov 02 13:45:00.567703 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:00.567608    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5bl9\" (UniqueName: \"kubernetes.io/projected/59f18952-4e48-4885-979d-6f3cfee55f3e-kube-api-access-z5bl9\") pod \"maas-api-key-cleanup-29893785-fv7tz\" (UID: \"59f18952-4e48-4885-979d-6f3cfee55f3e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893785-fv7tz"
Nov 02 13:45:00.838666 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:00.838567    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893785-fv7tz"
Nov 02 13:45:01.168322 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:01.168289    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893785-fv7tz"]
Nov 02 13:45:01.168803 ip-10-0-128-132 kubenswrapper[4559]: W1102 13:45:01.168773    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59f18952_4e48_4885_979d_6f3cfee55f3e.slice/crio-6454554547b5500672c2b7b352758a9addfce986a14d2b80158a52974d854d98 WatchSource:0}: Error finding container 6454554547b5500672c2b7b352758a9addfce986a14d2b80158a52974d854d98: Status 404 returned error can't find the container with id 6454554547b5500672c2b7b352758a9addfce986a14d2b80158a52974d854d98
Nov 02 13:45:01.206152 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:01.206126    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893785-fv7tz" event={"ID":"59f18952-4e48-4885-979d-6f3cfee55f3e","Type":"ContainerStarted","Data":"6454554547b5500672c2b7b352758a9addfce986a14d2b80158a52974d854d98"}
Nov 02 13:45:02.212032 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:02.211995    4559 generic.go:356] "Generic (PLEG): container finished" podID="59f18952-4e48-4885-979d-6f3cfee55f3e" containerID="40ef17f1a6e13f0f98adcfac3e6d252100036fda8eeca4959857a8bbb23406ca" exitCode=60
Nov 02 13:45:02.212449 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:02.212050    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893785-fv7tz" event={"ID":"59f18952-4e48-4885-979d-6f3cfee55f3e","Type":"ContainerDied","Data":"40ef17f1a6e13f0f98adcfac3e6d252100036fda8eeca4959857a8bbb23406ca"}
Nov 02 13:45:02.212449 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:02.212345    4559 scope.go:122] "RemoveContainer" containerID="40ef17f1a6e13f0f98adcfac3e6d252100036fda8eeca4959857a8bbb23406ca"
Nov 02 13:45:03.226321 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:03.226284    4559 generic.go:356] "Generic (PLEG): container finished" podID="59f18952-4e48-4885-979d-6f3cfee55f3e" containerID="9133eb9e0476a829d283646ea19ee88ed18ec268924f7a95b61d5ef4eb2a3046" exitCode=60
Nov 02 13:45:03.226785 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:03.226382    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893785-fv7tz" event={"ID":"59f18952-4e48-4885-979d-6f3cfee55f3e","Type":"ContainerDied","Data":"9133eb9e0476a829d283646ea19ee88ed18ec268924f7a95b61d5ef4eb2a3046"}
Nov 02 13:45:03.226785 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:03.226438    4559 scope.go:122] "RemoveContainer" containerID="40ef17f1a6e13f0f98adcfac3e6d252100036fda8eeca4959857a8bbb23406ca"
Nov 02 13:45:03.226918 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:03.226834    4559 scope.go:122] "RemoveContainer" containerID="9133eb9e0476a829d283646ea19ee88ed18ec268924f7a95b61d5ef4eb2a3046"
Nov 02 13:45:03.227155 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:45:03.227126    4559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29893785-fv7tz_odh-ai-gateway-infra(59f18952-4e48-4885-979d-6f3cfee55f3e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893785-fv7tz" podUID="59f18952-4e48-4885-979d-6f3cfee55f3e"
Nov 02 13:45:04.267676 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:04.260837    4559 scope.go:122] "RemoveContainer" containerID="9133eb9e0476a829d283646ea19ee88ed18ec268924f7a95b61d5ef4eb2a3046"
Nov 02 13:45:04.267676 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:45:04.261113    4559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29893785-fv7tz_odh-ai-gateway-infra(59f18952-4e48-4885-979d-6f3cfee55f3e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893785-fv7tz" podUID="59f18952-4e48-4885-979d-6f3cfee55f3e"
Nov 02 13:45:04.928663 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:04.928170    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"]
Nov 02 13:45:06.029915 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:06.029882    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-tj5gn"]
Nov 02 13:45:06.042578 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:06.042556    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-tj5gn"]
Nov 02 13:45:07.891239 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:07.891205    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9449234-e0f9-40ec-95c3-e3875fb23dd8" path="/var/lib/kubelet/pods/d9449234-e0f9-40ec-95c3-e3875fb23dd8/volumes"
Nov 02 13:45:10.332355 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:10.332317    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-a-f8e64488-kserve-677b68f4b9-l2qzr" podUID="43d356ce-f833-4912-b8c4-41c47eb5c798" containerName="main" containerID="cri-o://d42de798c0e3bf288810a11ca56dda9a3cef522a37c28cc5416d5684f24b0849" gracePeriod=45
Nov 02 13:45:10.583915 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:10.583860    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-f8e64488-kserve-677b68f4b9-l2qzr"
Nov 02 13:45:10.689168 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:10.689136    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43d356ce-f833-4912-b8c4-41c47eb5c798-kube-api-access-brtbb\" (UniqueName: \"kubernetes.io/projected/43d356ce-f833-4912-b8c4-41c47eb5c798-kube-api-access-brtbb\") pod \"43d356ce-f833-4912-b8c4-41c47eb5c798\" (UID: \"43d356ce-f833-4912-b8c4-41c47eb5c798\") "
Nov 02 13:45:10.689307 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:10.689231    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43d356ce-f833-4912-b8c4-41c47eb5c798-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43d356ce-f833-4912-b8c4-41c47eb5c798-model-cache\") pod \"43d356ce-f833-4912-b8c4-41c47eb5c798\" (UID: \"43d356ce-f833-4912-b8c4-41c47eb5c798\") "
Nov 02 13:45:10.689307 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:10.689261    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43d356ce-f833-4912-b8c4-41c47eb5c798-dshm\" (UniqueName: \"kubernetes.io/empty-dir/43d356ce-f833-4912-b8c4-41c47eb5c798-dshm\") pod \"43d356ce-f833-4912-b8c4-41c47eb5c798\" (UID: \"43d356ce-f833-4912-b8c4-41c47eb5c798\") "
Nov 02 13:45:10.689307 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:10.689285    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43d356ce-f833-4912-b8c4-41c47eb5c798-home\" (UniqueName: \"kubernetes.io/empty-dir/43d356ce-f833-4912-b8c4-41c47eb5c798-home\") pod \"43d356ce-f833-4912-b8c4-41c47eb5c798\" (UID: \"43d356ce-f833-4912-b8c4-41c47eb5c798\") "
Nov 02 13:45:10.689513 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:10.689313    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43d356ce-f833-4912-b8c4-41c47eb5c798-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43d356ce-f833-4912-b8c4-41c47eb5c798-tmp-dir\") pod \"43d356ce-f833-4912-b8c4-41c47eb5c798\" (UID: \"43d356ce-f833-4912-b8c4-41c47eb5c798\") "
Nov 02 13:45:10.689513 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:10.689362    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/43d356ce-f833-4912-b8c4-41c47eb5c798-tls-certs\" (UniqueName: \"kubernetes.io/secret/43d356ce-f833-4912-b8c4-41c47eb5c798-tls-certs\") pod \"43d356ce-f833-4912-b8c4-41c47eb5c798\" (UID: \"43d356ce-f833-4912-b8c4-41c47eb5c798\") "
Nov 02 13:45:10.689668 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:10.689544    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43d356ce-f833-4912-b8c4-41c47eb5c798-model-cache" pod "43d356ce-f833-4912-b8c4-41c47eb5c798" (UID: "43d356ce-f833-4912-b8c4-41c47eb5c798"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:45:10.689668 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:10.689595    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43d356ce-f833-4912-b8c4-41c47eb5c798-home" pod "43d356ce-f833-4912-b8c4-41c47eb5c798" (UID: "43d356ce-f833-4912-b8c4-41c47eb5c798"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:45:10.689668 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:10.689652    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43d356ce-f833-4912-b8c4-41c47eb5c798-tmp-dir" pod "43d356ce-f833-4912-b8c4-41c47eb5c798" (UID: "43d356ce-f833-4912-b8c4-41c47eb5c798"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:45:10.691270 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:10.691246    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43d356ce-f833-4912-b8c4-41c47eb5c798-dshm" pod "43d356ce-f833-4912-b8c4-41c47eb5c798" (UID: "43d356ce-f833-4912-b8c4-41c47eb5c798"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:45:10.691395 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:10.691327    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43d356ce-f833-4912-b8c4-41c47eb5c798-kube-api-access-brtbb" pod "43d356ce-f833-4912-b8c4-41c47eb5c798" (UID: "43d356ce-f833-4912-b8c4-41c47eb5c798"). InnerVolumeSpecName "kube-api-access-brtbb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:45:10.691443 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:10.691386    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43d356ce-f833-4912-b8c4-41c47eb5c798-tls-certs" pod "43d356ce-f833-4912-b8c4-41c47eb5c798" (UID: "43d356ce-f833-4912-b8c4-41c47eb5c798"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:45:10.790618 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:10.790591    4559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43d356ce-f833-4912-b8c4-41c47eb5c798-model-cache\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:45:10.790618 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:10.790615    4559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/43d356ce-f833-4912-b8c4-41c47eb5c798-dshm\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:45:10.790618 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:10.790623    4559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/43d356ce-f833-4912-b8c4-41c47eb5c798-home\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:45:10.790861 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:10.790630    4559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43d356ce-f833-4912-b8c4-41c47eb5c798-tmp-dir\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:45:10.790861 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:10.790656    4559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43d356ce-f833-4912-b8c4-41c47eb5c798-tls-certs\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:45:10.790861 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:10.790664    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-brtbb\" (UniqueName: \"kubernetes.io/projected/43d356ce-f833-4912-b8c4-41c47eb5c798-kube-api-access-brtbb\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:45:11.300955 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:11.300918    4559 generic.go:356] "Generic (PLEG): container finished" podID="43d356ce-f833-4912-b8c4-41c47eb5c798" containerID="d42de798c0e3bf288810a11ca56dda9a3cef522a37c28cc5416d5684f24b0849" exitCode=0
Nov 02 13:45:11.301145 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:11.300975    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-f8e64488-kserve-677b68f4b9-l2qzr"
Nov 02 13:45:11.301145 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:11.301000    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-f8e64488-kserve-677b68f4b9-l2qzr" event={"ID":"43d356ce-f833-4912-b8c4-41c47eb5c798","Type":"ContainerDied","Data":"d42de798c0e3bf288810a11ca56dda9a3cef522a37c28cc5416d5684f24b0849"}
Nov 02 13:45:11.301145 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:11.301051    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-f8e64488-kserve-677b68f4b9-l2qzr" event={"ID":"43d356ce-f833-4912-b8c4-41c47eb5c798","Type":"ContainerDied","Data":"701c01ca8045d0026fe7e6595261295845ad6fee75bea2cb08693604106e677b"}
Nov 02 13:45:11.301145 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:11.301073    4559 scope.go:122] "RemoveContainer" containerID="d42de798c0e3bf288810a11ca56dda9a3cef522a37c28cc5416d5684f24b0849"
Nov 02 13:45:11.310577 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:11.310556    4559 scope.go:122] "RemoveContainer" containerID="d42de798c0e3bf288810a11ca56dda9a3cef522a37c28cc5416d5684f24b0849"
Nov 02 13:45:11.310872 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:45:11.310853    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d42de798c0e3bf288810a11ca56dda9a3cef522a37c28cc5416d5684f24b0849\": container with ID starting with d42de798c0e3bf288810a11ca56dda9a3cef522a37c28cc5416d5684f24b0849 not found: ID does not exist" containerID="d42de798c0e3bf288810a11ca56dda9a3cef522a37c28cc5416d5684f24b0849"
Nov 02 13:45:11.310934 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:11.310881    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d42de798c0e3bf288810a11ca56dda9a3cef522a37c28cc5416d5684f24b0849"} err="failed to get container status \"d42de798c0e3bf288810a11ca56dda9a3cef522a37c28cc5416d5684f24b0849\": rpc error: code = NotFound desc = could not find container \"d42de798c0e3bf288810a11ca56dda9a3cef522a37c28cc5416d5684f24b0849\": container with ID starting with d42de798c0e3bf288810a11ca56dda9a3cef522a37c28cc5416d5684f24b0849 not found: ID does not exist"
Nov 02 13:45:11.353774 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:11.353753    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-f8e64488-kserve-677b68f4b9-l2qzr"]
Nov 02 13:45:11.366193 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:11.366171    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-f8e64488-kserve-677b68f4b9-l2qzr"]
Nov 02 13:45:11.890957 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:11.890928    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43d356ce-f833-4912-b8c4-41c47eb5c798" path="/var/lib/kubelet/pods/43d356ce-f833-4912-b8c4-41c47eb5c798/volumes"
Nov 02 13:45:14.499603 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:14.499567    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"]
Nov 02 13:45:14.886370 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:14.886155    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-b-f8e64488-kserve-d789bf9fc-8chn5" podUID="687b51ca-91c0-4510-a3b4-bb903cba96df" containerName="main" containerID="cri-o://aca170e322797b07712b3ff4792f1006f640ff52122f0393d3d6180610bf0636" gracePeriod=45
Nov 02 13:45:14.951785 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:14.951733    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-same-id-b-f8e64488-kserve-d789bf9fc-8chn5" podUID="687b51ca-91c0-4510-a3b4-bb903cba96df" containerName="main" probeResult="failure" output="Get \"https://10.133.1.32:8000/ready\": dial tcp 10.133.1.32:8000: connect: connection refused"
Nov 02 13:45:15.243160 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:45:15.243088    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqcmj" containerName="extractor"
Nov 02 13:45:15.343581 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:15.343351    4559 generic.go:356] "Generic (PLEG): container finished" podID="687b51ca-91c0-4510-a3b4-bb903cba96df" containerID="aca170e322797b07712b3ff4792f1006f640ff52122f0393d3d6180610bf0636" exitCode=0
Nov 02 13:45:15.343581 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:15.343514    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-f8e64488-kserve-d789bf9fc-8chn5" event={"ID":"687b51ca-91c0-4510-a3b4-bb903cba96df","Type":"ContainerDied","Data":"aca170e322797b07712b3ff4792f1006f640ff52122f0393d3d6180610bf0636"}
Nov 02 13:45:15.834686 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:15.830834    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-f8e64488-kserve-d789bf9fc-8chn5"
Nov 02 13:45:15.970725 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:15.967107    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/687b51ca-91c0-4510-a3b4-bb903cba96df-dshm\" (UniqueName: \"kubernetes.io/empty-dir/687b51ca-91c0-4510-a3b4-bb903cba96df-dshm\") pod \"687b51ca-91c0-4510-a3b4-bb903cba96df\" (UID: \"687b51ca-91c0-4510-a3b4-bb903cba96df\") "
Nov 02 13:45:15.970725 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:15.967192    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/687b51ca-91c0-4510-a3b4-bb903cba96df-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/687b51ca-91c0-4510-a3b4-bb903cba96df-model-cache\") pod \"687b51ca-91c0-4510-a3b4-bb903cba96df\" (UID: \"687b51ca-91c0-4510-a3b4-bb903cba96df\") "
Nov 02 13:45:15.970725 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:15.967277    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/687b51ca-91c0-4510-a3b4-bb903cba96df-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/687b51ca-91c0-4510-a3b4-bb903cba96df-tmp-dir\") pod \"687b51ca-91c0-4510-a3b4-bb903cba96df\" (UID: \"687b51ca-91c0-4510-a3b4-bb903cba96df\") "
Nov 02 13:45:15.970725 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:15.967315    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/687b51ca-91c0-4510-a3b4-bb903cba96df-tls-certs\" (UniqueName: \"kubernetes.io/secret/687b51ca-91c0-4510-a3b4-bb903cba96df-tls-certs\") pod \"687b51ca-91c0-4510-a3b4-bb903cba96df\" (UID: \"687b51ca-91c0-4510-a3b4-bb903cba96df\") "
Nov 02 13:45:15.970725 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:15.967344    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/687b51ca-91c0-4510-a3b4-bb903cba96df-home\" (UniqueName: \"kubernetes.io/empty-dir/687b51ca-91c0-4510-a3b4-bb903cba96df-home\") pod \"687b51ca-91c0-4510-a3b4-bb903cba96df\" (UID: \"687b51ca-91c0-4510-a3b4-bb903cba96df\") "
Nov 02 13:45:15.970725 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:15.967380    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/687b51ca-91c0-4510-a3b4-bb903cba96df-kube-api-access-fnfr6\" (UniqueName: \"kubernetes.io/projected/687b51ca-91c0-4510-a3b4-bb903cba96df-kube-api-access-fnfr6\") pod \"687b51ca-91c0-4510-a3b4-bb903cba96df\" (UID: \"687b51ca-91c0-4510-a3b4-bb903cba96df\") "
Nov 02 13:45:15.970725 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:15.968629    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/687b51ca-91c0-4510-a3b4-bb903cba96df-model-cache" pod "687b51ca-91c0-4510-a3b4-bb903cba96df" (UID: "687b51ca-91c0-4510-a3b4-bb903cba96df"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:45:15.970725 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:15.969209    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/687b51ca-91c0-4510-a3b4-bb903cba96df-tmp-dir" pod "687b51ca-91c0-4510-a3b4-bb903cba96df" (UID: "687b51ca-91c0-4510-a3b4-bb903cba96df"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:45:15.970725 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:15.969411    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/687b51ca-91c0-4510-a3b4-bb903cba96df-home" pod "687b51ca-91c0-4510-a3b4-bb903cba96df" (UID: "687b51ca-91c0-4510-a3b4-bb903cba96df"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:45:16.023042 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:16.023007    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/687b51ca-91c0-4510-a3b4-bb903cba96df-dshm" pod "687b51ca-91c0-4510-a3b4-bb903cba96df" (UID: "687b51ca-91c0-4510-a3b4-bb903cba96df"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:45:16.052658 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:16.044364    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/687b51ca-91c0-4510-a3b4-bb903cba96df-tls-certs" pod "687b51ca-91c0-4510-a3b4-bb903cba96df" (UID: "687b51ca-91c0-4510-a3b4-bb903cba96df"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:45:16.058884 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:16.058851    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/687b51ca-91c0-4510-a3b4-bb903cba96df-kube-api-access-fnfr6" pod "687b51ca-91c0-4510-a3b4-bb903cba96df" (UID: "687b51ca-91c0-4510-a3b4-bb903cba96df"). InnerVolumeSpecName "kube-api-access-fnfr6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:45:16.080178 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:16.080144    4559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/687b51ca-91c0-4510-a3b4-bb903cba96df-dshm\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:45:16.080373 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:16.080354    4559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/687b51ca-91c0-4510-a3b4-bb903cba96df-model-cache\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:45:16.087706 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:16.087683    4559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/687b51ca-91c0-4510-a3b4-bb903cba96df-tmp-dir\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:45:16.087870 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:16.087855    4559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/687b51ca-91c0-4510-a3b4-bb903cba96df-tls-certs\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:45:16.087987 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:16.087974    4559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/687b51ca-91c0-4510-a3b4-bb903cba96df-home\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:45:16.088088 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:16.088074    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fnfr6\" (UniqueName: \"kubernetes.io/projected/687b51ca-91c0-4510-a3b4-bb903cba96df-kube-api-access-fnfr6\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:45:16.360423 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:16.360328    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-f8e64488-kserve-d789bf9fc-8chn5" event={"ID":"687b51ca-91c0-4510-a3b4-bb903cba96df","Type":"ContainerDied","Data":"85c39c9e1fc405f30de113600105585710e3cd1c5c02178bf9675d9057980682"}
Nov 02 13:45:16.360697 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:16.360674    4559 scope.go:122] "RemoveContainer" containerID="aca170e322797b07712b3ff4792f1006f640ff52122f0393d3d6180610bf0636"
Nov 02 13:45:16.360968 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:16.360950    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-f8e64488-kserve-d789bf9fc-8chn5"
Nov 02 13:45:16.435805 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:16.435748    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-f8e64488-kserve-d789bf9fc-8chn5"]
Nov 02 13:45:16.448061 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:16.447989    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-f8e64488-kserve-d789bf9fc-8chn5"]
Nov 02 13:45:17.885878 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:17.885850    4559 scope.go:122] "RemoveContainer" containerID="9133eb9e0476a829d283646ea19ee88ed18ec268924f7a95b61d5ef4eb2a3046"
Nov 02 13:45:17.891377 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:17.891353    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="687b51ca-91c0-4510-a3b4-bb903cba96df" path="/var/lib/kubelet/pods/687b51ca-91c0-4510-a3b4-bb903cba96df/volumes"
Nov 02 13:45:18.372707 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:18.372678    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893785-fv7tz" event={"ID":"59f18952-4e48-4885-979d-6f3cfee55f3e","Type":"ContainerStarted","Data":"4458731035d491feb09dbbb314b03bae67c2df90d35d2bd573add0866b8d25d7"}
Nov 02 13:45:18.415083 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:18.415040    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893785-fv7tz" podStartSLOduration=18.415026321 podStartE2EDuration="18.415026321s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:45:18.413935727 +0000 UTC m=+4714.669334822" watchObservedRunningTime="2026-11-02 13:45:18.415026321 +0000 UTC m=+4714.670425410"
Nov 02 13:45:19.308986 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:19.308953    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"]
Nov 02 13:45:19.379151 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:19.379120    4559 generic.go:356] "Generic (PLEG): container finished" podID="59f18952-4e48-4885-979d-6f3cfee55f3e" containerID="4458731035d491feb09dbbb314b03bae67c2df90d35d2bd573add0866b8d25d7" exitCode=60
Nov 02 13:45:19.379338 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:19.379191    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893785-fv7tz" event={"ID":"59f18952-4e48-4885-979d-6f3cfee55f3e","Type":"ContainerDied","Data":"4458731035d491feb09dbbb314b03bae67c2df90d35d2bd573add0866b8d25d7"}
Nov 02 13:45:19.379338 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:19.379260    4559 scope.go:122] "RemoveContainer" containerID="9133eb9e0476a829d283646ea19ee88ed18ec268924f7a95b61d5ef4eb2a3046"
Nov 02 13:45:19.379601 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:19.379581    4559 scope.go:122] "RemoveContainer" containerID="4458731035d491feb09dbbb314b03bae67c2df90d35d2bd573add0866b8d25d7"
Nov 02 13:45:19.379831 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:45:19.379812    4559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29893785-fv7tz_odh-ai-gateway-infra(59f18952-4e48-4885-979d-6f3cfee55f3e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893785-fv7tz" podUID="59f18952-4e48-4885-979d-6f3cfee55f3e"
Nov 02 13:45:19.428087 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:19.428058    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893785-fv7tz"]
Nov 02 13:45:20.706899 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:20.706582    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893785-fv7tz"
Nov 02 13:45:20.779748 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:20.778964    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59f18952-4e48-4885-979d-6f3cfee55f3e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/59f18952-4e48-4885-979d-6f3cfee55f3e-maas-api-service-ca\") pod \"59f18952-4e48-4885-979d-6f3cfee55f3e\" (UID: \"59f18952-4e48-4885-979d-6f3cfee55f3e\") "
Nov 02 13:45:20.779748 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:20.779073    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59f18952-4e48-4885-979d-6f3cfee55f3e-kube-api-access-z5bl9\" (UniqueName: \"kubernetes.io/projected/59f18952-4e48-4885-979d-6f3cfee55f3e-kube-api-access-z5bl9\") pod \"59f18952-4e48-4885-979d-6f3cfee55f3e\" (UID: \"59f18952-4e48-4885-979d-6f3cfee55f3e\") "
Nov 02 13:45:20.780383 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:20.780342    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59f18952-4e48-4885-979d-6f3cfee55f3e-maas-api-service-ca" pod "59f18952-4e48-4885-979d-6f3cfee55f3e" (UID: "59f18952-4e48-4885-979d-6f3cfee55f3e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:45:20.795174 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:20.795128    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59f18952-4e48-4885-979d-6f3cfee55f3e-kube-api-access-z5bl9" pod "59f18952-4e48-4885-979d-6f3cfee55f3e" (UID: "59f18952-4e48-4885-979d-6f3cfee55f3e"). InnerVolumeSpecName "kube-api-access-z5bl9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:45:20.887672 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:20.881231    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/59f18952-4e48-4885-979d-6f3cfee55f3e-maas-api-service-ca\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:45:20.911799 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:20.911713    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5bl9\" (UniqueName: \"kubernetes.io/projected/59f18952-4e48-4885-979d-6f3cfee55f3e-kube-api-access-z5bl9\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:45:21.416540 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:21.416501    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893785-fv7tz" event={"ID":"59f18952-4e48-4885-979d-6f3cfee55f3e","Type":"ContainerDied","Data":"6454554547b5500672c2b7b352758a9addfce986a14d2b80158a52974d854d98"}
Nov 02 13:45:21.416856 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:21.416833    4559 scope.go:122] "RemoveContainer" containerID="4458731035d491feb09dbbb314b03bae67c2df90d35d2bd573add0866b8d25d7"
Nov 02 13:45:21.417061 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:21.416767    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893785-fv7tz"
Nov 02 13:45:21.546708 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:21.546676    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893785-fv7tz"]
Nov 02 13:45:21.566473 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:21.566431    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893785-fv7tz"]
Nov 02 13:45:21.905672 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:21.900560    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59f18952-4e48-4885-979d-6f3cfee55f3e" path="/var/lib/kubelet/pods/59f18952-4e48-4885-979d-6f3cfee55f3e/volumes"
Nov 02 13:45:22.035284 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:22.035250    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-2cp6n"]
Nov 02 13:45:22.047948 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:22.047879    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-2cp6n"]
Nov 02 13:45:23.404652 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:23.404617    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4de99d0f-p5wjr"]
Nov 02 13:45:23.405049 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:23.405005    4559 cpu_manager.go:414] "containerMap: removing container" podUID="59f18952-4e48-4885-979d-6f3cfee55f3e" containerName="cleanup"
Nov 02 13:45:23.405049 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:23.405017    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59f18952-4e48-4885-979d-6f3cfee55f3e" containerName="cleanup"
Nov 02 13:45:23.405049 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:23.405036    4559 cpu_manager.go:414] "containerMap: removing container" podUID="59f18952-4e48-4885-979d-6f3cfee55f3e" containerName="cleanup"
Nov 02 13:45:23.405049 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:23.405045    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59f18952-4e48-4885-979d-6f3cfee55f3e" containerName="cleanup"
Nov 02 13:45:23.405193 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:23.405066    4559 cpu_manager.go:414] "containerMap: removing container" podUID="687b51ca-91c0-4510-a3b4-bb903cba96df" containerName="main"
Nov 02 13:45:23.405193 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:23.405072    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="687b51ca-91c0-4510-a3b4-bb903cba96df" containerName="main"
Nov 02 13:45:23.405193 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:23.405080    4559 cpu_manager.go:414] "containerMap: removing container" podUID="43d356ce-f833-4912-b8c4-41c47eb5c798" containerName="main"
Nov 02 13:45:23.405193 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:23.405085    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43d356ce-f833-4912-b8c4-41c47eb5c798" containerName="main"
Nov 02 13:45:23.405193 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:23.405140    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="59f18952-4e48-4885-979d-6f3cfee55f3e" containerName="cleanup"
Nov 02 13:45:23.405193 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:23.405152    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="687b51ca-91c0-4510-a3b4-bb903cba96df" containerName="main"
Nov 02 13:45:23.405193 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:23.405159    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="43d356ce-f833-4912-b8c4-41c47eb5c798" containerName="main"
Nov 02 13:45:23.405193 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:23.405166    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="59f18952-4e48-4885-979d-6f3cfee55f3e" containerName="cleanup"
Nov 02 13:45:23.405193 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:23.405173    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="59f18952-4e48-4885-979d-6f3cfee55f3e" containerName="cleanup"
Nov 02 13:45:23.409574 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:23.409556    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4de99d0f-p5wjr"
Nov 02 13:45:23.439870 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:23.439845    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4de99d0f-p5wjr"]
Nov 02 13:45:23.450261 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:23.450234    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 13:45:23.450425 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:23.450250    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-45fzn\""
Nov 02 13:45:23.453803 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:23.453782    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 13:45:23.454017 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:23.454000    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 13:45:23.498435 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:23.498403    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r59cq\" (UniqueName: \"kubernetes.io/projected/a993a691-6869-4702-9873-1b4a1616ae92-kube-api-access-r59cq\") pod \"maas-api-revoke-sub-4de99d0f-p5wjr\" (UID: \"a993a691-6869-4702-9873-1b4a1616ae92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4de99d0f-p5wjr"
Nov 02 13:45:23.498588 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:23.498569    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a993a691-6869-4702-9873-1b4a1616ae92-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4de99d0f-p5wjr\" (UID: \"a993a691-6869-4702-9873-1b4a1616ae92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4de99d0f-p5wjr"
Nov 02 13:45:23.498705 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:23.498677    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a993a691-6869-4702-9873-1b4a1616ae92-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4de99d0f-p5wjr\" (UID: \"a993a691-6869-4702-9873-1b4a1616ae92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4de99d0f-p5wjr"
Nov 02 13:45:23.600023 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:23.599995    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a993a691-6869-4702-9873-1b4a1616ae92-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4de99d0f-p5wjr\" (UID: \"a993a691-6869-4702-9873-1b4a1616ae92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4de99d0f-p5wjr"
Nov 02 13:45:23.600160 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:23.600038    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a993a691-6869-4702-9873-1b4a1616ae92-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4de99d0f-p5wjr\" (UID: \"a993a691-6869-4702-9873-1b4a1616ae92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4de99d0f-p5wjr"
Nov 02 13:45:23.600160 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:23.600062    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r59cq\" (UniqueName: \"kubernetes.io/projected/a993a691-6869-4702-9873-1b4a1616ae92-kube-api-access-r59cq\") pod \"maas-api-revoke-sub-4de99d0f-p5wjr\" (UID: \"a993a691-6869-4702-9873-1b4a1616ae92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4de99d0f-p5wjr"
Nov 02 13:45:23.600616 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:23.600596    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a993a691-6869-4702-9873-1b4a1616ae92-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4de99d0f-p5wjr\" (UID: \"a993a691-6869-4702-9873-1b4a1616ae92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4de99d0f-p5wjr"
Nov 02 13:45:23.602507 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:23.602488    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a993a691-6869-4702-9873-1b4a1616ae92-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4de99d0f-p5wjr\" (UID: \"a993a691-6869-4702-9873-1b4a1616ae92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4de99d0f-p5wjr"
Nov 02 13:45:23.624115 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:23.624087    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r59cq\" (UniqueName: \"kubernetes.io/projected/a993a691-6869-4702-9873-1b4a1616ae92-kube-api-access-r59cq\") pod \"maas-api-revoke-sub-4de99d0f-p5wjr\" (UID: \"a993a691-6869-4702-9873-1b4a1616ae92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4de99d0f-p5wjr"
Nov 02 13:45:23.722313 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:23.722293    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4de99d0f-p5wjr"
Nov 02 13:45:23.892173 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:23.892135    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66be16cd-39a1-4845-92da-c2585a1777e4" path="/var/lib/kubelet/pods/66be16cd-39a1-4845-92da-c2585a1777e4/volumes"
Nov 02 13:45:24.020105 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:24.020070    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4de99d0f-p5wjr"]
Nov 02 13:45:24.020361 ip-10-0-128-132 kubenswrapper[4559]: W1102 13:45:24.020339    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda993a691_6869_4702_9873_1b4a1616ae92.slice/crio-d9dc832523d0a8688314f176a2b0880e0030c37a49fa1ccfd293745f29bd9f9a WatchSource:0}: Error finding container d9dc832523d0a8688314f176a2b0880e0030c37a49fa1ccfd293745f29bd9f9a: Status 404 returned error can't find the container with id d9dc832523d0a8688314f176a2b0880e0030c37a49fa1ccfd293745f29bd9f9a
Nov 02 13:45:24.458571 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:24.458531    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4de99d0f-p5wjr" event={"ID":"a993a691-6869-4702-9873-1b4a1616ae92","Type":"ContainerStarted","Data":"8e91d4488d114218e9666f89ae316055530e5644d351d02a2d174a4dbbdfb4df"}
Nov 02 13:45:24.459045 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:24.458612    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4de99d0f-p5wjr" event={"ID":"a993a691-6869-4702-9873-1b4a1616ae92","Type":"ContainerStarted","Data":"d9dc832523d0a8688314f176a2b0880e0030c37a49fa1ccfd293745f29bd9f9a"}
Nov 02 13:45:24.501739 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:24.501685    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4de99d0f-p5wjr" podStartSLOduration=1.5016716369999998 podStartE2EDuration="1.501671637s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:45:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:45:24.500369467 +0000 UTC m=+4720.755768567" watchObservedRunningTime="2026-11-02 13:45:24.501671637 +0000 UTC m=+4720.757070766"
Nov 02 13:45:25.465481 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:25.465447    4559 generic.go:356] "Generic (PLEG): container finished" podID="a993a691-6869-4702-9873-1b4a1616ae92" containerID="8e91d4488d114218e9666f89ae316055530e5644d351d02a2d174a4dbbdfb4df" exitCode=0
Nov 02 13:45:25.465879 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:25.465499    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4de99d0f-p5wjr" event={"ID":"a993a691-6869-4702-9873-1b4a1616ae92","Type":"ContainerDied","Data":"8e91d4488d114218e9666f89ae316055530e5644d351d02a2d174a4dbbdfb4df"}
Nov 02 13:45:26.599070 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:26.599044    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4de99d0f-p5wjr"
Nov 02 13:45:26.729081 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:26.729001    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a993a691-6869-4702-9873-1b4a1616ae92-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a993a691-6869-4702-9873-1b4a1616ae92-maas-api-xks-ca\") pod \"a993a691-6869-4702-9873-1b4a1616ae92\" (UID: \"a993a691-6869-4702-9873-1b4a1616ae92\") "
Nov 02 13:45:26.729244 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:26.729089    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a993a691-6869-4702-9873-1b4a1616ae92-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a993a691-6869-4702-9873-1b4a1616ae92-maas-api-openshift-ca\") pod \"a993a691-6869-4702-9873-1b4a1616ae92\" (UID: \"a993a691-6869-4702-9873-1b4a1616ae92\") "
Nov 02 13:45:26.729244 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:26.729116    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a993a691-6869-4702-9873-1b4a1616ae92-kube-api-access-r59cq\" (UniqueName: \"kubernetes.io/projected/a993a691-6869-4702-9873-1b4a1616ae92-kube-api-access-r59cq\") pod \"a993a691-6869-4702-9873-1b4a1616ae92\" (UID: \"a993a691-6869-4702-9873-1b4a1616ae92\") "
Nov 02 13:45:26.729467 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:26.729440    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a993a691-6869-4702-9873-1b4a1616ae92-maas-api-openshift-ca" pod "a993a691-6869-4702-9873-1b4a1616ae92" (UID: "a993a691-6869-4702-9873-1b4a1616ae92"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:45:26.731119 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:26.731085    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a993a691-6869-4702-9873-1b4a1616ae92-maas-api-xks-ca" pod "a993a691-6869-4702-9873-1b4a1616ae92" (UID: "a993a691-6869-4702-9873-1b4a1616ae92"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:45:26.731222 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:26.731187    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a993a691-6869-4702-9873-1b4a1616ae92-kube-api-access-r59cq" pod "a993a691-6869-4702-9873-1b4a1616ae92" (UID: "a993a691-6869-4702-9873-1b4a1616ae92"). InnerVolumeSpecName "kube-api-access-r59cq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:45:26.830461 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:26.830425    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a993a691-6869-4702-9873-1b4a1616ae92-maas-api-xks-ca\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:45:26.830461 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:26.830459    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a993a691-6869-4702-9873-1b4a1616ae92-maas-api-openshift-ca\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:45:26.830697 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:26.830473    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r59cq\" (UniqueName: \"kubernetes.io/projected/a993a691-6869-4702-9873-1b4a1616ae92-kube-api-access-r59cq\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:45:27.476019 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:27.475988    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4de99d0f-p5wjr"
Nov 02 13:45:27.476205 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:27.475993    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4de99d0f-p5wjr" event={"ID":"a993a691-6869-4702-9873-1b4a1616ae92","Type":"ContainerDied","Data":"d9dc832523d0a8688314f176a2b0880e0030c37a49fa1ccfd293745f29bd9f9a"}
Nov 02 13:45:27.476205 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:27.476108    4559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d9dc832523d0a8688314f176a2b0880e0030c37a49fa1ccfd293745f29bd9f9a"
Nov 02 13:45:33.032283 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:33.032241    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-7zsfz"]
Nov 02 13:45:33.045039 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:33.045007    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-7zsfz"]
Nov 02 13:45:33.891151 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:33.891119    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d96d563-ada2-46d1-a170-866dc16679ee" path="/var/lib/kubelet/pods/6d96d563-ada2-46d1-a170-866dc16679ee/volumes"
Nov 02 13:45:34.686564 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:34.686530    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"]
Nov 02 13:45:39.022091 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:39.022050    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-6f7bc57488-96bzn"]
Nov 02 13:45:39.022519 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:39.022426    4559 cpu_manager.go:414] "containerMap: removing container" podUID="59f18952-4e48-4885-979d-6f3cfee55f3e" containerName="cleanup"
Nov 02 13:45:39.022519 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:39.022436    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59f18952-4e48-4885-979d-6f3cfee55f3e" containerName="cleanup"
Nov 02 13:45:39.022519 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:39.022449    4559 cpu_manager.go:414] "containerMap: removing container" podUID="a993a691-6869-4702-9873-1b4a1616ae92" containerName="revoke-keys"
Nov 02 13:45:39.022519 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:39.022454    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a993a691-6869-4702-9873-1b4a1616ae92" containerName="revoke-keys"
Nov 02 13:45:39.022519 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:39.022514    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="a993a691-6869-4702-9873-1b4a1616ae92" containerName="revoke-keys"
Nov 02 13:45:39.026026 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:39.026005    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-6f7bc57488-96bzn"
Nov 02 13:45:39.045130 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:39.045108    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-6f7bc57488-96bzn"]
Nov 02 13:45:39.145756 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:39.145727    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9wv4\" (UniqueName: \"kubernetes.io/projected/7217353f-4f97-4760-9b66-16689ced7e9c-kube-api-access-p9wv4\") pod \"payload-pre-processing-6f7bc57488-96bzn\" (UID: \"7217353f-4f97-4760-9b66-16689ced7e9c\") " pod="openshift-ingress/payload-pre-processing-6f7bc57488-96bzn"
Nov 02 13:45:39.145919 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:39.145763    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7217353f-4f97-4760-9b66-16689ced7e9c-plugins-config-volume\") pod \"payload-pre-processing-6f7bc57488-96bzn\" (UID: \"7217353f-4f97-4760-9b66-16689ced7e9c\") " pod="openshift-ingress/payload-pre-processing-6f7bc57488-96bzn"
Nov 02 13:45:39.246426 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:39.246396    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9wv4\" (UniqueName: \"kubernetes.io/projected/7217353f-4f97-4760-9b66-16689ced7e9c-kube-api-access-p9wv4\") pod \"payload-pre-processing-6f7bc57488-96bzn\" (UID: \"7217353f-4f97-4760-9b66-16689ced7e9c\") " pod="openshift-ingress/payload-pre-processing-6f7bc57488-96bzn"
Nov 02 13:45:39.246426 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:39.246429    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7217353f-4f97-4760-9b66-16689ced7e9c-plugins-config-volume\") pod \"payload-pre-processing-6f7bc57488-96bzn\" (UID: \"7217353f-4f97-4760-9b66-16689ced7e9c\") " pod="openshift-ingress/payload-pre-processing-6f7bc57488-96bzn"
Nov 02 13:45:39.247176 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:39.247156    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7217353f-4f97-4760-9b66-16689ced7e9c-plugins-config-volume\") pod \"payload-pre-processing-6f7bc57488-96bzn\" (UID: \"7217353f-4f97-4760-9b66-16689ced7e9c\") " pod="openshift-ingress/payload-pre-processing-6f7bc57488-96bzn"
Nov 02 13:45:39.270672 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:39.270629    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9wv4\" (UniqueName: \"kubernetes.io/projected/7217353f-4f97-4760-9b66-16689ced7e9c-kube-api-access-p9wv4\") pod \"payload-pre-processing-6f7bc57488-96bzn\" (UID: \"7217353f-4f97-4760-9b66-16689ced7e9c\") " pod="openshift-ingress/payload-pre-processing-6f7bc57488-96bzn"
Nov 02 13:45:39.338208 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:39.338150    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-6f7bc57488-96bzn"
Nov 02 13:45:39.635030 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:39.635003    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-6f7bc57488-96bzn"]
Nov 02 13:45:39.635774 ip-10-0-128-132 kubenswrapper[4559]: W1102 13:45:39.635744    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7217353f_4f97_4760_9b66_16689ced7e9c.slice/crio-7eb5a31b08f5708207480f107203125d20ef2f108e054c21aa39542ca8208288 WatchSource:0}: Error finding container 7eb5a31b08f5708207480f107203125d20ef2f108e054c21aa39542ca8208288: Status 404 returned error can't find the container with id 7eb5a31b08f5708207480f107203125d20ef2f108e054c21aa39542ca8208288
Nov 02 13:45:40.540713 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:40.540677    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-6f7bc57488-96bzn" event={"ID":"7217353f-4f97-4760-9b66-16689ced7e9c","Type":"ContainerStarted","Data":"30f841c6b8595f31cffb13aaf33eb3f1cbe5cc3eb24c9e8cab9eebd1e3a9b9cd"}
Nov 02 13:45:40.541119 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:40.540732    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-6f7bc57488-96bzn" event={"ID":"7217353f-4f97-4760-9b66-16689ced7e9c","Type":"ContainerStarted","Data":"7eb5a31b08f5708207480f107203125d20ef2f108e054c21aa39542ca8208288"}
Nov 02 13:45:40.541119 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:40.540759    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-6f7bc57488-96bzn"
Nov 02 13:45:40.583356 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:40.583312    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-6f7bc57488-96bzn" podStartSLOduration=2.583298649 podStartE2EDuration="2.583298649s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:45:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:45:40.582083381 +0000 UTC m=+4736.837482477" watchObservedRunningTime="2026-11-02 13:45:40.583298649 +0000 UTC m=+4736.838697744"
Nov 02 13:45:45.084509 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:45:45.084472    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqcmj" containerName="extractor"
Nov 02 13:45:47.803776 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:47.803744    4559 scope.go:122] "RemoveContainer" containerID="98e673f03124346d78e751009c1a211f5760d689dc8d3a897075a43869783146"
Nov 02 13:45:47.812187 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:47.812168    4559 scope.go:122] "RemoveContainer" containerID="d292a5569e875d1c9e0b65bb06f9283153565f200428eb3270931283cb78f91e"
Nov 02 13:45:51.548691 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:51.548633    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-6f7bc57488-96bzn"
Nov 02 13:45:51.619118 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:51.619076    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-qg68n"]
Nov 02 13:45:53.342926 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:53.342886    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-7b49d5f75d-mlqv7"]
Nov 02 13:45:53.348288 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:53.348261    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-7b49d5f75d-mlqv7"
Nov 02 13:45:53.366093 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:53.366067    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-7b49d5f75d-mlqv7"]
Nov 02 13:45:53.370733 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:53.370705    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e13b7fbf-86c5-44e9-b610-b35ddf12da45-plugins-config-volume\") pod \"payload-processing-7b49d5f75d-mlqv7\" (UID: \"e13b7fbf-86c5-44e9-b610-b35ddf12da45\") " pod="openshift-ingress/payload-processing-7b49d5f75d-mlqv7"
Nov 02 13:45:53.370869 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:53.370753    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qc9n2\" (UniqueName: \"kubernetes.io/projected/e13b7fbf-86c5-44e9-b610-b35ddf12da45-kube-api-access-qc9n2\") pod \"payload-processing-7b49d5f75d-mlqv7\" (UID: \"e13b7fbf-86c5-44e9-b610-b35ddf12da45\") " pod="openshift-ingress/payload-processing-7b49d5f75d-mlqv7"
Nov 02 13:45:53.471498 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:53.471466    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qc9n2\" (UniqueName: \"kubernetes.io/projected/e13b7fbf-86c5-44e9-b610-b35ddf12da45-kube-api-access-qc9n2\") pod \"payload-processing-7b49d5f75d-mlqv7\" (UID: \"e13b7fbf-86c5-44e9-b610-b35ddf12da45\") " pod="openshift-ingress/payload-processing-7b49d5f75d-mlqv7"
Nov 02 13:45:53.471670 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:53.471553    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e13b7fbf-86c5-44e9-b610-b35ddf12da45-plugins-config-volume\") pod \"payload-processing-7b49d5f75d-mlqv7\" (UID: \"e13b7fbf-86c5-44e9-b610-b35ddf12da45\") " pod="openshift-ingress/payload-processing-7b49d5f75d-mlqv7"
Nov 02 13:45:53.472219 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:53.472199    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e13b7fbf-86c5-44e9-b610-b35ddf12da45-plugins-config-volume\") pod \"payload-processing-7b49d5f75d-mlqv7\" (UID: \"e13b7fbf-86c5-44e9-b610-b35ddf12da45\") " pod="openshift-ingress/payload-processing-7b49d5f75d-mlqv7"
Nov 02 13:45:53.496140 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:53.496120    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qc9n2\" (UniqueName: \"kubernetes.io/projected/e13b7fbf-86c5-44e9-b610-b35ddf12da45-kube-api-access-qc9n2\") pod \"payload-processing-7b49d5f75d-mlqv7\" (UID: \"e13b7fbf-86c5-44e9-b610-b35ddf12da45\") " pod="openshift-ingress/payload-processing-7b49d5f75d-mlqv7"
Nov 02 13:45:53.661411 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:53.661338    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-7b49d5f75d-mlqv7"
Nov 02 13:45:53.957667 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:53.957624    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-7b49d5f75d-mlqv7"]
Nov 02 13:45:53.958688 ip-10-0-128-132 kubenswrapper[4559]: W1102 13:45:53.957876    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode13b7fbf_86c5_44e9_b610_b35ddf12da45.slice/crio-29ebd988fa123054c94cebad3deb080ec1744bd3a48a5ebe60da35e04dfcfdde WatchSource:0}: Error finding container 29ebd988fa123054c94cebad3deb080ec1744bd3a48a5ebe60da35e04dfcfdde: Status 404 returned error can't find the container with id 29ebd988fa123054c94cebad3deb080ec1744bd3a48a5ebe60da35e04dfcfdde
Nov 02 13:45:54.610006 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:54.609963    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-7b49d5f75d-mlqv7" event={"ID":"e13b7fbf-86c5-44e9-b610-b35ddf12da45","Type":"ContainerStarted","Data":"1932183e06670c715b8ae06ec82a28d5ed6f9ea28de4fd2af6cf28eedbfdc6b3"}
Nov 02 13:45:54.610402 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:54.610027    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-7b49d5f75d-mlqv7" event={"ID":"e13b7fbf-86c5-44e9-b610-b35ddf12da45","Type":"ContainerStarted","Data":"29ebd988fa123054c94cebad3deb080ec1744bd3a48a5ebe60da35e04dfcfdde"}
Nov 02 13:45:54.610402 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:54.610075    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-7b49d5f75d-mlqv7"
Nov 02 13:45:54.653518 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:54.653465    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-7b49d5f75d-mlqv7" podStartSLOduration=1.653450358 podStartE2EDuration="1.653450358s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:45:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:45:54.65231006 +0000 UTC m=+4750.907709155" watchObservedRunningTime="2026-11-02 13:45:54.653450358 +0000 UTC m=+4750.908849454"
Nov 02 13:45:56.620155 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:56.620097    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-7b98bb678c-qg68n" podUID="15512035-7903-48f9-ba69-2b53ef921f6e" containerName="payload-pre-processing" containerID="cri-o://3cdfa9ed6a91a8f8825d974d47d3e9669ead0e68707c70255d75a0b52947f004" gracePeriod=25
Nov 02 13:45:56.871581 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:56.871526    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-qg68n"
Nov 02 13:45:56.903788 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:56.903764    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15512035-7903-48f9-ba69-2b53ef921f6e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15512035-7903-48f9-ba69-2b53ef921f6e-plugins-config-volume\") pod \"15512035-7903-48f9-ba69-2b53ef921f6e\" (UID: \"15512035-7903-48f9-ba69-2b53ef921f6e\") "
Nov 02 13:45:56.903915 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:56.903850    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15512035-7903-48f9-ba69-2b53ef921f6e-kube-api-access-ngqtz\" (UniqueName: \"kubernetes.io/projected/15512035-7903-48f9-ba69-2b53ef921f6e-kube-api-access-ngqtz\") pod \"15512035-7903-48f9-ba69-2b53ef921f6e\" (UID: \"15512035-7903-48f9-ba69-2b53ef921f6e\") "
Nov 02 13:45:56.904165 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:56.904141    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15512035-7903-48f9-ba69-2b53ef921f6e-plugins-config-volume" pod "15512035-7903-48f9-ba69-2b53ef921f6e" (UID: "15512035-7903-48f9-ba69-2b53ef921f6e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:45:56.905793 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:56.905768    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15512035-7903-48f9-ba69-2b53ef921f6e-kube-api-access-ngqtz" pod "15512035-7903-48f9-ba69-2b53ef921f6e" (UID: "15512035-7903-48f9-ba69-2b53ef921f6e"). InnerVolumeSpecName "kube-api-access-ngqtz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:45:57.004826 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:57.004796    4559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15512035-7903-48f9-ba69-2b53ef921f6e-plugins-config-volume\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:45:57.004826 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:57.004821    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ngqtz\" (UniqueName: \"kubernetes.io/projected/15512035-7903-48f9-ba69-2b53ef921f6e-kube-api-access-ngqtz\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:45:57.624084 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:57.624046    4559 generic.go:356] "Generic (PLEG): container finished" podID="15512035-7903-48f9-ba69-2b53ef921f6e" containerID="3cdfa9ed6a91a8f8825d974d47d3e9669ead0e68707c70255d75a0b52947f004" exitCode=0
Nov 02 13:45:57.624622 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:57.624097    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-qg68n"
Nov 02 13:45:57.624622 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:57.624133    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-qg68n" event={"ID":"15512035-7903-48f9-ba69-2b53ef921f6e","Type":"ContainerDied","Data":"3cdfa9ed6a91a8f8825d974d47d3e9669ead0e68707c70255d75a0b52947f004"}
Nov 02 13:45:57.624622 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:57.624192    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-qg68n" event={"ID":"15512035-7903-48f9-ba69-2b53ef921f6e","Type":"ContainerDied","Data":"c81e10686211ac4db3fd0e51faa969d57cab203d7d35079d2056570e3d690858"}
Nov 02 13:45:57.624622 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:57.624223    4559 scope.go:122] "RemoveContainer" containerID="3cdfa9ed6a91a8f8825d974d47d3e9669ead0e68707c70255d75a0b52947f004"
Nov 02 13:45:57.635832 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:57.635811    4559 scope.go:122] "RemoveContainer" containerID="3cdfa9ed6a91a8f8825d974d47d3e9669ead0e68707c70255d75a0b52947f004"
Nov 02 13:45:57.636108 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:45:57.636089    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3cdfa9ed6a91a8f8825d974d47d3e9669ead0e68707c70255d75a0b52947f004\": container with ID starting with 3cdfa9ed6a91a8f8825d974d47d3e9669ead0e68707c70255d75a0b52947f004 not found: ID does not exist" containerID="3cdfa9ed6a91a8f8825d974d47d3e9669ead0e68707c70255d75a0b52947f004"
Nov 02 13:45:57.636177 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:57.636118    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3cdfa9ed6a91a8f8825d974d47d3e9669ead0e68707c70255d75a0b52947f004"} err="failed to get container status \"3cdfa9ed6a91a8f8825d974d47d3e9669ead0e68707c70255d75a0b52947f004\": rpc error: code = NotFound desc = could not find container \"3cdfa9ed6a91a8f8825d974d47d3e9669ead0e68707c70255d75a0b52947f004\": container with ID starting with 3cdfa9ed6a91a8f8825d974d47d3e9669ead0e68707c70255d75a0b52947f004 not found: ID does not exist"
Nov 02 13:45:57.680300 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:57.680275    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-qg68n"]
Nov 02 13:45:57.692872 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:57.692851    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-qg68n"]
Nov 02 13:45:57.891722 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:45:57.891631    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15512035-7903-48f9-ba69-2b53ef921f6e" path="/var/lib/kubelet/pods/15512035-7903-48f9-ba69-2b53ef921f6e/volumes"
Nov 02 13:46:05.617175 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:46:05.617144    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-7b49d5f75d-mlqv7"
Nov 02 13:46:05.690581 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:46:05.690546    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-r6rdp"]
Nov 02 13:46:07.600392 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:46:07.600355    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-7dbf4f4996-66pcj"]
Nov 02 13:46:07.600807 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:46:07.600791    4559 cpu_manager.go:414] "containerMap: removing container" podUID="15512035-7903-48f9-ba69-2b53ef921f6e" containerName="payload-pre-processing"
Nov 02 13:46:07.600859 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:46:07.600809    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15512035-7903-48f9-ba69-2b53ef921f6e" containerName="payload-pre-processing"
Nov 02 13:46:07.600901 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:46:07.600889    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="15512035-7903-48f9-ba69-2b53ef921f6e" containerName="payload-pre-processing"
Nov 02 13:46:07.605517 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:46:07.605496    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-7dbf4f4996-66pcj"
Nov 02 13:46:07.623427 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:46:07.623393    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-7dbf4f4996-66pcj"]
Nov 02 13:46:07.704444 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:46:07.704410    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6925\" (UniqueName: \"kubernetes.io/projected/d6b257bf-cee7-4330-91f7-74b8ba1f1a1f-kube-api-access-h6925\") pod \"payload-processing-7dbf4f4996-66pcj\" (UID: \"d6b257bf-cee7-4330-91f7-74b8ba1f1a1f\") " pod="openshift-ingress/payload-processing-7dbf4f4996-66pcj"
Nov 02 13:46:07.704632 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:46:07.704562    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6b257bf-cee7-4330-91f7-74b8ba1f1a1f-plugins-config-volume\") pod \"payload-processing-7dbf4f4996-66pcj\" (UID: \"d6b257bf-cee7-4330-91f7-74b8ba1f1a1f\") " pod="openshift-ingress/payload-processing-7dbf4f4996-66pcj"
Nov 02 13:46:07.766527 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:46:07.766493    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-86f4845867-n579j"]
Nov 02 13:46:07.770429 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:46:07.770407    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-86f4845867-n579j"
Nov 02 13:46:07.790311 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:46:07.790286    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-86f4845867-n579j"]
Nov 02 13:46:07.805556 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:46:07.805528    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vw9jh\" (UniqueName: \"kubernetes.io/projected/d6760973-a200-4c89-9304-103bcb3d846e-kube-api-access-vw9jh\") pod \"payload-pre-processing-86f4845867-n579j\" (UID: \"d6760973-a200-4c89-9304-103bcb3d846e\") " pod="openshift-ingress/payload-pre-processing-86f4845867-n579j"
Nov 02 13:46:07.805731 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:46:07.805568    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6b257bf-cee7-4330-91f7-74b8ba1f1a1f-plugins-config-volume\") pod \"payload-processing-7dbf4f4996-66pcj\" (UID: \"d6b257bf-cee7-4330-91f7-74b8ba1f1a1f\") " pod="openshift-ingress/payload-processing-7dbf4f4996-66pcj"
Nov 02 13:46:07.805731 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:46:07.805660    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6925\" (UniqueName: \"kubernetes.io/projected/d6b257bf-cee7-4330-91f7-74b8ba1f1a1f-kube-api-access-h6925\") pod \"payload-processing-7dbf4f4996-66pcj\" (UID: \"d6b257bf-cee7-4330-91f7-74b8ba1f1a1f\") " pod="openshift-ingress/payload-processing-7dbf4f4996-66pcj"
Nov 02 13:46:07.805833 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:46:07.805752    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6760973-a200-4c89-9304-103bcb3d846e-plugins-config-volume\") pod \"payload-pre-processing-86f4845867-n579j\" (UID: \"d6760973-a200-4c89-9304-103bcb3d846e\") " pod="openshift-ingress/payload-pre-processing-86f4845867-n579j"
Nov 02 13:46:07.806337 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:46:07.806318    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6b257bf-cee7-4330-91f7-74b8ba1f1a1f-plugins-config-volume\") pod \"payload-processing-7dbf4f4996-66pcj\" (UID: \"d6b257bf-cee7-4330-91f7-74b8ba1f1a1f\") " pod="openshift-ingress/payload-processing-7dbf4f4996-66pcj"
Nov 02 13:46:07.830205 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:46:07.830177    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6925\" (UniqueName: \"kubernetes.io/projected/d6b257bf-cee7-4330-91f7-74b8ba1f1a1f-kube-api-access-h6925\") pod \"payload-processing-7dbf4f4996-66pcj\" (UID: \"d6b257bf-cee7-4330-91f7-74b8ba1f1a1f\") " pod="openshift-ingress/payload-processing-7dbf4f4996-66pcj"
Nov 02 13:46:07.906460 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:46:07.906380    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vw9jh\" (UniqueName: \"kubernetes.io/projected/d6760973-a200-4c89-9304-103bcb3d846e-kube-api-access-vw9jh\") pod \"payload-pre-processing-86f4845867-n579j\" (UID: \"d6760973-a200-4c89-9304-103bcb3d846e\") " pod="openshift-ingress/payload-pre-processing-86f4845867-n579j"
Nov 02 13:46:07.906460 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:46:07.906442    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6760973-a200-4c89-9304-103bcb3d846e-plugins-config-volume\") pod \"payload-pre-processing-86f4845867-n579j\" (UID: \"d6760973-a200-4c89-9304-103bcb3d846e\") " pod="openshift-ingress/payload-pre-processing-86f4845867-n579j"
Nov 02 13:46:07.907115 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:46:07.907093    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6760973-a200-4c89-9304-103bcb3d846e-plugins-config-volume\") pod \"payload-pre-processing-86f4845867-n579j\" (UID: \"d6760973-a200-4c89-9304-103bcb3d846e\") " pod="openshift-ingress/payload-pre-processing-86f4845867-n579j"
Nov 02 13:46:07.918302 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:46:07.918279    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-7dbf4f4996-66pcj"
Nov 02 13:46:07.930755 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:46:07.930728    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vw9jh\" (UniqueName: \"kubernetes.io/projected/d6760973-a200-4c89-9304-103bcb3d846e-kube-api-access-vw9jh\") pod \"payload-pre-processing-86f4845867-n579j\" (UID: \"d6760973-a200-4c89-9304-103bcb3d846e\") " pod="openshift-ingress/payload-pre-processing-86f4845867-n579j"
Nov 02 13:46:08.082996 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:46:08.082956    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-86f4845867-n579j"
Nov 02 13:46:08.217050 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:46:08.217018    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-7dbf4f4996-66pcj"]
Nov 02 13:46:08.217462 ip-10-0-128-132 kubenswrapper[4559]: W1102 13:46:08.217440    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6b257bf_cee7_4330_91f7_74b8ba1f1a1f.slice/crio-f11d31841432e5621575720b16a3182f3683d440f9aa4145687b59b82bb08597 WatchSource:0}: Error finding container f11d31841432e5621575720b16a3182f3683d440f9aa4145687b59b82bb08597: Status 404 returned error can't find the container with id f11d31841432e5621575720b16a3182f3683d440f9aa4145687b59b82bb08597
Nov 02 13:46:08.382805 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:46:08.382762    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-86f4845867-n579j"]
Nov 02 13:46:08.383267 ip-10-0-128-132 kubenswrapper[4559]: W1102 13:46:08.383229    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6760973_a200_4c89_9304_103bcb3d846e.slice/crio-3b4ee7b0e840416d6292a12f661cf15eac032356e6e0d38b0e75f53de9c633d8 WatchSource:0}: Error finding container 3b4ee7b0e840416d6292a12f661cf15eac032356e6e0d38b0e75f53de9c633d8: Status 404 returned error can't find the container with id 3b4ee7b0e840416d6292a12f661cf15eac032356e6e0d38b0e75f53de9c633d8
Nov 02 13:46:08.676970 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:46:08.676935    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-86f4845867-n579j" event={"ID":"d6760973-a200-4c89-9304-103bcb3d846e","Type":"ContainerStarted","Data":"2ee931d5b0b2a76e1092bd228711ab1b1f4eea84256c13aeca67b50a2f8432f8"}
Nov 02 13:46:08.677486 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:46:08.676995    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-86f4845867-n579j" event={"ID":"d6760973-a200-4c89-9304-103bcb3d846e","Type":"ContainerStarted","Data":"3b4ee7b0e840416d6292a12f661cf15eac032356e6e0d38b0e75f53de9c633d8"}
Nov 02 13:46:08.677486 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:46:08.677047    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-86f4845867-n579j"
Nov 02 13:46:08.678590 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:46:08.678563    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-7dbf4f4996-66pcj" event={"ID":"d6b257bf-cee7-4330-91f7-74b8ba1f1a1f","Type":"ContainerStarted","Data":"88ffdbace12e7f5ecf3dd90623e33f48a8416c2723a1bbbd09eafa443f143860"}
Nov 02 13:46:08.678714 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:46:08.678609    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-7dbf4f4996-66pcj" event={"ID":"d6b257bf-cee7-4330-91f7-74b8ba1f1a1f","Type":"ContainerStarted","Data":"f11d31841432e5621575720b16a3182f3683d440f9aa4145687b59b82bb08597"}
Nov 02 13:46:08.678714 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:46:08.678669    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-7dbf4f4996-66pcj"
Nov 02 13:46:08.720285 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:46:08.720236    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-86f4845867-n579j" podStartSLOduration=1.720222666 podStartE2EDuration="1.720222666s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:46:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:46:08.719051003 +0000 UTC m=+4764.974450106" watchObservedRunningTime="2026-11-02 13:46:08.720222666 +0000 UTC m=+4764.975621761"
Nov 02 13:46:08.762458 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:46:08.762407    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-7dbf4f4996-66pcj" podStartSLOduration=1.762391546 podStartE2EDuration="1.762391546s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:46:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:46:08.760715926 +0000 UTC m=+4765.016115021" watchObservedRunningTime="2026-11-02 13:46:08.762391546 +0000 UTC m=+4765.017790642"
Nov 02 13:46:10.691409 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:46:10.691373    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-55fbbbf9dc-r6rdp" podUID="2fef2bd6-5793-4c34-a390-37c907bcfd19" containerName="payload-processing" containerID="cri-o://7d9057ec8b704504a7349a97dbd26808c8e4eb6943f94c2d75c3295fd2b72b2d" gracePeriod=25
Nov 02 13:46:10.940513 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:46:10.940484    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-r6rdp"
Nov 02 13:46:11.035792 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:46:11.035718    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2fef2bd6-5793-4c34-a390-37c907bcfd19-kube-api-access-f8ntk\" (UniqueName: \"kubernetes.io/projected/2fef2bd6-5793-4c34-a390-37c907bcfd19-kube-api-access-f8ntk\") pod \"2fef2bd6-5793-4c34-a390-37c907bcfd19\" (UID: \"2fef2bd6-5793-4c34-a390-37c907bcfd19\") "
Nov 02 13:46:11.035968 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:46:11.035800    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2fef2bd6-5793-4c34-a390-37c907bcfd19-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2fef2bd6-5793-4c34-a390-37c907bcfd19-plugins-config-volume\") pod \"2fef2bd6-5793-4c34-a390-37c907bcfd19\" (UID: \"2fef2bd6-5793-4c34-a390-37c907bcfd19\") "
Nov 02 13:46:11.036178 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:46:11.036147    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2fef2bd6-5793-4c34-a390-37c907bcfd19-plugins-config-volume" pod "2fef2bd6-5793-4c34-a390-37c907bcfd19" (UID: "2fef2bd6-5793-4c34-a390-37c907bcfd19"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:46:11.037789 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:46:11.037769    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fef2bd6-5793-4c34-a390-37c907bcfd19-kube-api-access-f8ntk" pod "2fef2bd6-5793-4c34-a390-37c907bcfd19" (UID: "2fef2bd6-5793-4c34-a390-37c907bcfd19"). InnerVolumeSpecName "kube-api-access-f8ntk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:46:11.136929 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:46:11.136898    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f8ntk\" (UniqueName: \"kubernetes.io/projected/2fef2bd6-5793-4c34-a390-37c907bcfd19-kube-api-access-f8ntk\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:46:11.136929 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:46:11.136925    4559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2fef2bd6-5793-4c34-a390-37c907bcfd19-plugins-config-volume\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:46:11.694610 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:46:11.694578    4559 generic.go:356] "Generic (PLEG): container finished" podID="2fef2bd6-5793-4c34-a390-37c907bcfd19" containerID="7d9057ec8b704504a7349a97dbd26808c8e4eb6943f94c2d75c3295fd2b72b2d" exitCode=0
Nov 02 13:46:11.695072 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:46:11.694652    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-r6rdp"
Nov 02 13:46:11.695072 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:46:11.694671    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-r6rdp" event={"ID":"2fef2bd6-5793-4c34-a390-37c907bcfd19","Type":"ContainerDied","Data":"7d9057ec8b704504a7349a97dbd26808c8e4eb6943f94c2d75c3295fd2b72b2d"}
Nov 02 13:46:11.695072 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:46:11.694749    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-r6rdp" event={"ID":"2fef2bd6-5793-4c34-a390-37c907bcfd19","Type":"ContainerDied","Data":"4e85fcce870daf8e3f7085e77f6ac429898acbaca12fd8096256c0dd25d56641"}
Nov 02 13:46:11.695072 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:46:11.694772    4559 scope.go:122] "RemoveContainer" containerID="7d9057ec8b704504a7349a97dbd26808c8e4eb6943f94c2d75c3295fd2b72b2d"
Nov 02 13:46:11.705011 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:46:11.704997    4559 scope.go:122] "RemoveContainer" containerID="7d9057ec8b704504a7349a97dbd26808c8e4eb6943f94c2d75c3295fd2b72b2d"
Nov 02 13:46:11.705260 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:46:11.705240    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d9057ec8b704504a7349a97dbd26808c8e4eb6943f94c2d75c3295fd2b72b2d\": container with ID starting with 7d9057ec8b704504a7349a97dbd26808c8e4eb6943f94c2d75c3295fd2b72b2d not found: ID does not exist" containerID="7d9057ec8b704504a7349a97dbd26808c8e4eb6943f94c2d75c3295fd2b72b2d"
Nov 02 13:46:11.705335 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:46:11.705272    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d9057ec8b704504a7349a97dbd26808c8e4eb6943f94c2d75c3295fd2b72b2d"} err="failed to get container status \"7d9057ec8b704504a7349a97dbd26808c8e4eb6943f94c2d75c3295fd2b72b2d\": rpc error: code = NotFound desc = could not find container \"7d9057ec8b704504a7349a97dbd26808c8e4eb6943f94c2d75c3295fd2b72b2d\": container with ID starting with 7d9057ec8b704504a7349a97dbd26808c8e4eb6943f94c2d75c3295fd2b72b2d not found: ID does not exist"
Nov 02 13:46:11.747301 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:46:11.747280    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-r6rdp"]
Nov 02 13:46:11.760003 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:46:11.759979    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-r6rdp"]
Nov 02 13:46:11.890920 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:46:11.890889    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2fef2bd6-5793-4c34-a390-37c907bcfd19" path="/var/lib/kubelet/pods/2fef2bd6-5793-4c34-a390-37c907bcfd19/volumes"
Nov 02 13:46:15.085002 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:46:15.084960    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqcmj" containerName="extractor"
Nov 02 13:46:19.687295 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:46:19.687266    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-7dbf4f4996-66pcj"
Nov 02 13:46:19.687714 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:46:19.687344    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-86f4845867-n579j"
Nov 02 13:46:19.759376 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:46:19.757996    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-7b49d5f75d-mlqv7"]
Nov 02 13:46:19.798477 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:46:19.798446    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-6f7bc57488-96bzn"]
Nov 02 13:46:23.236290 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:46:23.236257    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-59ffk"]
Nov 02 13:46:23.236704 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:46:23.236621    4559 cpu_manager.go:414] "containerMap: removing container" podUID="2fef2bd6-5793-4c34-a390-37c907bcfd19" containerName="payload-processing"
Nov 02 13:46:23.236704 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:46:23.236632    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2fef2bd6-5793-4c34-a390-37c907bcfd19" containerName="payload-processing"
Nov 02 13:46:23.236804 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:46:23.236724    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="2fef2bd6-5793-4c34-a390-37c907bcfd19" containerName="payload-processing"
Nov 02 13:46:23.241103 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:46:23.241087    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5bcd7997-59ffk"
Nov 02 13:46:23.260458 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:46:23.260432    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-59ffk"]
Nov 02 13:46:23.270942 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:46:23.270917    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\""
Nov 02 13:46:23.271127 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:46:23.270941    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"e2e-rotation-external-model-api-key\""
Nov 02 13:46:23.271127 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:46:23.270975    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\""
Nov 02 13:46:23.271280 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:46:23.270970    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-scg5m\""
Nov 02 13:46:23.271280 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:46:23.271001    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\""
Nov 02 13:46:23.271280 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:46:23.271072    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\""
Nov 02 13:46:23.346438 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:46:23.346403    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1ba2645c-9413-4125-9dac-3528fe6d3c00-routing-overlay-volume\") pod \"payload-processing-external-model-5bcd7997-59ffk\" (UID: \"1ba2645c-9413-4125-9dac-3528fe6d3c00\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-59ffk"
Nov 02 13:46:23.346610 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:46:23.346457    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z25z7\" (UniqueName: \"kubernetes.io/projected/1ba2645c-9413-4125-9dac-3528fe6d3c00-kube-api-access-z25z7\") pod \"payload-processing-external-model-5bcd7997-59ffk\" (UID: \"1ba2645c-9413-4125-9dac-3528fe6d3c00\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-59ffk"
Nov 02 13:46:23.346610 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:46:23.346570    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ba2645c-9413-4125-9dac-3528fe6d3c00-plugins-config-volume\") pod \"payload-processing-external-model-5bcd7997-59ffk\" (UID: \"1ba2645c-9413-4125-9dac-3528fe6d3c00\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-59ffk"
Nov 02 13:46:23.346770 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:46:23.346752    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1ba2645c-9413-4125-9dac-3528fe6d3c00-provider-credentials\") pod \"payload-processing-external-model-5bcd7997-59ffk\" (UID: \"1ba2645c-9413-4125-9dac-3528fe6d3c00\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-59ffk"
Nov 02 13:46:23.447757 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:46:23.447720    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1ba2645c-9413-4125-9dac-3528fe6d3c00-provider-credentials\") pod \"payload-processing-external-model-5bcd7997-59ffk\" (UID: \"1ba2645c-9413-4125-9dac-3528fe6d3c00\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-59ffk"
Nov 02 13:46:23.447969 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:46:23.447775    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1ba2645c-9413-4125-9dac-3528fe6d3c00-routing-overlay-volume\") pod \"payload-processing-external-model-5bcd7997-59ffk\" (UID: \"1ba2645c-9413-4125-9dac-3528fe6d3c00\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-59ffk"
Nov 02 13:46:23.447969 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:46:23.447806    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z25z7\" (UniqueName: \"kubernetes.io/projected/1ba2645c-9413-4125-9dac-3528fe6d3c00-kube-api-access-z25z7\") pod \"payload-processing-external-model-5bcd7997-59ffk\" (UID: \"1ba2645c-9413-4125-9dac-3528fe6d3c00\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-59ffk"
Nov 02 13:46:23.447969 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:46:23.447837    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ba2645c-9413-4125-9dac-3528fe6d3c00-plugins-config-volume\") pod \"payload-processing-external-model-5bcd7997-59ffk\" (UID: \"1ba2645c-9413-4125-9dac-3528fe6d3c00\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-59ffk"
Nov 02 13:46:23.448564 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:46:23.448541    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1ba2645c-9413-4125-9dac-3528fe6d3c00-routing-overlay-volume\") pod \"payload-processing-external-model-5bcd7997-59ffk\" (UID: \"1ba2645c-9413-4125-9dac-3528fe6d3c00\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-59ffk"
Nov 02 13:46:23.448634 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:46:23.448565    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ba2645c-9413-4125-9dac-3528fe6d3c00-plugins-config-volume\") pod \"payload-processing-external-model-5bcd7997-59ffk\" (UID: \"1ba2645c-9413-4125-9dac-3528fe6d3c00\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-59ffk"
Nov 02 13:46:23.450359 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:46:23.450329    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1ba2645c-9413-4125-9dac-3528fe6d3c00-provider-credentials\") pod \"payload-processing-external-model-5bcd7997-59ffk\" (UID: \"1ba2645c-9413-4125-9dac-3528fe6d3c00\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-59ffk"
Nov 02 13:46:23.473656 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:46:23.473612    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z25z7\" (UniqueName: \"kubernetes.io/projected/1ba2645c-9413-4125-9dac-3528fe6d3c00-kube-api-access-z25z7\") pod \"payload-processing-external-model-5bcd7997-59ffk\" (UID: \"1ba2645c-9413-4125-9dac-3528fe6d3c00\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-59ffk"
Nov 02 13:46:23.554467 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:46:23.554411    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5bcd7997-59ffk"
Nov 02 13:46:23.851834 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:46:23.851806    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-59ffk"]
Nov 02 13:46:23.852322 ip-10-0-128-132 kubenswrapper[4559]: W1102 13:46:23.852296    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ba2645c_9413_4125_9dac_3528fe6d3c00.slice/crio-d45464c46dc5b6b526d4ac215e76a323cebcad14c7c003524bf60a5202bea07b WatchSource:0}: Error finding container d45464c46dc5b6b526d4ac215e76a323cebcad14c7c003524bf60a5202bea07b: Status 404 returned error can't find the container with id d45464c46dc5b6b526d4ac215e76a323cebcad14c7c003524bf60a5202bea07b
Nov 02 13:46:24.484609 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:46:24.484575    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"]
Nov 02 13:46:24.759296 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:46:24.758760    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-7b49d5f75d-mlqv7" podUID="e13b7fbf-86c5-44e9-b610-b35ddf12da45" containerName="payload-processing" containerID="cri-o://1932183e06670c715b8ae06ec82a28d5ed6f9ea28de4fd2af6cf28eedbfdc6b3" gracePeriod=25
Nov 02 13:46:24.773953 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:46:24.773905    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5bcd7997-59ffk" event={"ID":"1ba2645c-9413-4125-9dac-3528fe6d3c00","Type":"ContainerStarted","Data":"b6789cf5ca9bc51169154b84ca1bf2678204d5bcbdc41fc97185193b55b1e7f5"}
Nov 02 13:46:24.774189 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:46:24.774161    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-5bcd7997-59ffk"
Nov 02 13:46:24.774310 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:46:24.774214    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5bcd7997-59ffk" event={"ID":"1ba2645c-9413-4125-9dac-3528fe6d3c00","Type":"ContainerStarted","Data":"d45464c46dc5b6b526d4ac215e76a323cebcad14c7c003524bf60a5202bea07b"}
Nov 02 13:46:24.799737 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:46:24.799688    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-6f7bc57488-96bzn" podUID="7217353f-4f97-4760-9b66-16689ced7e9c" containerName="payload-pre-processing" containerID="cri-o://30f841c6b8595f31cffb13aaf33eb3f1cbe5cc3eb24c9e8cab9eebd1e3a9b9cd" gracePeriod=25
Nov 02 13:46:24.821987 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:46:24.821943    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-5bcd7997-59ffk" podStartSLOduration=1.821929172 podStartE2EDuration="1.821929172s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:46:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:46:24.82025025 +0000 UTC m=+4781.075649351" watchObservedRunningTime="2026-11-02 13:46:24.821929172 +0000 UTC m=+4781.077328267"
Nov 02 13:46:25.122092 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:46:25.122066    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-7b49d5f75d-mlqv7"
Nov 02 13:46:25.124199 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:46:25.124175    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-6f7bc57488-96bzn"
Nov 02 13:46:25.267116 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:46:25.267055    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e13b7fbf-86c5-44e9-b610-b35ddf12da45-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e13b7fbf-86c5-44e9-b610-b35ddf12da45-plugins-config-volume\") pod \"e13b7fbf-86c5-44e9-b610-b35ddf12da45\" (UID: \"e13b7fbf-86c5-44e9-b610-b35ddf12da45\") "
Nov 02 13:46:25.267327 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:46:25.267144    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7217353f-4f97-4760-9b66-16689ced7e9c-kube-api-access-p9wv4\" (UniqueName: \"kubernetes.io/projected/7217353f-4f97-4760-9b66-16689ced7e9c-kube-api-access-p9wv4\") pod \"7217353f-4f97-4760-9b66-16689ced7e9c\" (UID: \"7217353f-4f97-4760-9b66-16689ced7e9c\") "
Nov 02 13:46:25.267327 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:46:25.267173    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e13b7fbf-86c5-44e9-b610-b35ddf12da45-kube-api-access-qc9n2\" (UniqueName: \"kubernetes.io/projected/e13b7fbf-86c5-44e9-b610-b35ddf12da45-kube-api-access-qc9n2\") pod \"e13b7fbf-86c5-44e9-b610-b35ddf12da45\" (UID: \"e13b7fbf-86c5-44e9-b610-b35ddf12da45\") "
Nov 02 13:46:25.267327 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:46:25.267218    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7217353f-4f97-4760-9b66-16689ced7e9c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7217353f-4f97-4760-9b66-16689ced7e9c-plugins-config-volume\") pod \"7217353f-4f97-4760-9b66-16689ced7e9c\" (UID: \"7217353f-4f97-4760-9b66-16689ced7e9c\") "
Nov 02 13:46:25.267934 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:46:25.267899    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e13b7fbf-86c5-44e9-b610-b35ddf12da45-plugins-config-volume" pod "e13b7fbf-86c5-44e9-b610-b35ddf12da45" (UID: "e13b7fbf-86c5-44e9-b610-b35ddf12da45"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:46:25.268132 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:46:25.267912    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7217353f-4f97-4760-9b66-16689ced7e9c-plugins-config-volume" pod "7217353f-4f97-4760-9b66-16689ced7e9c" (UID: "7217353f-4f97-4760-9b66-16689ced7e9c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:46:25.269734 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:46:25.269706    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e13b7fbf-86c5-44e9-b610-b35ddf12da45-kube-api-access-qc9n2" pod "e13b7fbf-86c5-44e9-b610-b35ddf12da45" (UID: "e13b7fbf-86c5-44e9-b610-b35ddf12da45"). InnerVolumeSpecName "kube-api-access-qc9n2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:46:25.269970 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:46:25.269950    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7217353f-4f97-4760-9b66-16689ced7e9c-kube-api-access-p9wv4" pod "7217353f-4f97-4760-9b66-16689ced7e9c" (UID: "7217353f-4f97-4760-9b66-16689ced7e9c"). InnerVolumeSpecName "kube-api-access-p9wv4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:46:25.368572 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:46:25.368490    4559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e13b7fbf-86c5-44e9-b610-b35ddf12da45-plugins-config-volume\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:46:25.368572 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:46:25.368543    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9wv4\" (UniqueName: \"kubernetes.io/projected/7217353f-4f97-4760-9b66-16689ced7e9c-kube-api-access-p9wv4\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:46:25.368572 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:46:25.368560    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qc9n2\" (UniqueName: \"kubernetes.io/projected/e13b7fbf-86c5-44e9-b610-b35ddf12da45-kube-api-access-qc9n2\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:46:25.368901 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:46:25.368574    4559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7217353f-4f97-4760-9b66-16689ced7e9c-plugins-config-volume\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:46:25.783332 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:46:25.783282    4559 generic.go:356] "Generic (PLEG): container finished" podID="7217353f-4f97-4760-9b66-16689ced7e9c" containerID="30f841c6b8595f31cffb13aaf33eb3f1cbe5cc3eb24c9e8cab9eebd1e3a9b9cd" exitCode=0
Nov 02 13:46:25.783984 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:46:25.783338    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-6f7bc57488-96bzn"
Nov 02 13:46:25.784144 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:46:25.783361    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-6f7bc57488-96bzn" event={"ID":"7217353f-4f97-4760-9b66-16689ced7e9c","Type":"ContainerDied","Data":"30f841c6b8595f31cffb13aaf33eb3f1cbe5cc3eb24c9e8cab9eebd1e3a9b9cd"}
Nov 02 13:46:25.784303 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:46:25.784285    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-6f7bc57488-96bzn" event={"ID":"7217353f-4f97-4760-9b66-16689ced7e9c","Type":"ContainerDied","Data":"7eb5a31b08f5708207480f107203125d20ef2f108e054c21aa39542ca8208288"}
Nov 02 13:46:25.784478 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:46:25.784463    4559 scope.go:122] "RemoveContainer" containerID="30f841c6b8595f31cffb13aaf33eb3f1cbe5cc3eb24c9e8cab9eebd1e3a9b9cd"
Nov 02 13:46:25.787108 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:46:25.786295    4559 generic.go:356] "Generic (PLEG): container finished" podID="e13b7fbf-86c5-44e9-b610-b35ddf12da45" containerID="1932183e06670c715b8ae06ec82a28d5ed6f9ea28de4fd2af6cf28eedbfdc6b3" exitCode=0
Nov 02 13:46:25.787108 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:46:25.786543    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-7b49d5f75d-mlqv7" event={"ID":"e13b7fbf-86c5-44e9-b610-b35ddf12da45","Type":"ContainerDied","Data":"1932183e06670c715b8ae06ec82a28d5ed6f9ea28de4fd2af6cf28eedbfdc6b3"}
Nov 02 13:46:25.787108 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:46:25.786608    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-7b49d5f75d-mlqv7" event={"ID":"e13b7fbf-86c5-44e9-b610-b35ddf12da45","Type":"ContainerDied","Data":"29ebd988fa123054c94cebad3deb080ec1744bd3a48a5ebe60da35e04dfcfdde"}
Nov 02 13:46:25.787108 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:46:25.786614    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-7b49d5f75d-mlqv7"
Nov 02 13:46:25.799325 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:46:25.799297    4559 scope.go:122] "RemoveContainer" containerID="30f841c6b8595f31cffb13aaf33eb3f1cbe5cc3eb24c9e8cab9eebd1e3a9b9cd"
Nov 02 13:46:25.800550 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:46:25.800524    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30f841c6b8595f31cffb13aaf33eb3f1cbe5cc3eb24c9e8cab9eebd1e3a9b9cd\": container with ID starting with 30f841c6b8595f31cffb13aaf33eb3f1cbe5cc3eb24c9e8cab9eebd1e3a9b9cd not found: ID does not exist" containerID="30f841c6b8595f31cffb13aaf33eb3f1cbe5cc3eb24c9e8cab9eebd1e3a9b9cd"
Nov 02 13:46:25.800780 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:46:25.800565    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30f841c6b8595f31cffb13aaf33eb3f1cbe5cc3eb24c9e8cab9eebd1e3a9b9cd"} err="failed to get container status \"30f841c6b8595f31cffb13aaf33eb3f1cbe5cc3eb24c9e8cab9eebd1e3a9b9cd\": rpc error: code = NotFound desc = could not find container \"30f841c6b8595f31cffb13aaf33eb3f1cbe5cc3eb24c9e8cab9eebd1e3a9b9cd\": container with ID starting with 30f841c6b8595f31cffb13aaf33eb3f1cbe5cc3eb24c9e8cab9eebd1e3a9b9cd not found: ID does not exist"
Nov 02 13:46:25.800780 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:46:25.800587    4559 scope.go:122] "RemoveContainer" containerID="1932183e06670c715b8ae06ec82a28d5ed6f9ea28de4fd2af6cf28eedbfdc6b3"
Nov 02 13:46:25.809452 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:46:25.809425    4559 scope.go:122] "RemoveContainer" containerID="1932183e06670c715b8ae06ec82a28d5ed6f9ea28de4fd2af6cf28eedbfdc6b3"
Nov 02 13:46:25.809794 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:46:25.809739    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1932183e06670c715b8ae06ec82a28d5ed6f9ea28de4fd2af6cf28eedbfdc6b3\": container with ID starting with 1932183e06670c715b8ae06ec82a28d5ed6f9ea28de4fd2af6cf28eedbfdc6b3 not found: ID does not exist" containerID="1932183e06670c715b8ae06ec82a28d5ed6f9ea28de4fd2af6cf28eedbfdc6b3"
Nov 02 13:46:25.809794 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:46:25.809779    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1932183e06670c715b8ae06ec82a28d5ed6f9ea28de4fd2af6cf28eedbfdc6b3"} err="failed to get container status \"1932183e06670c715b8ae06ec82a28d5ed6f9ea28de4fd2af6cf28eedbfdc6b3\": rpc error: code = NotFound desc = could not find container \"1932183e06670c715b8ae06ec82a28d5ed6f9ea28de4fd2af6cf28eedbfdc6b3\": container with ID starting with 1932183e06670c715b8ae06ec82a28d5ed6f9ea28de4fd2af6cf28eedbfdc6b3 not found: ID does not exist"
Nov 02 13:46:25.854196 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:46:25.854169    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-7b49d5f75d-mlqv7"]
Nov 02 13:46:25.861428 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:46:25.861396    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-7b49d5f75d-mlqv7"]
Nov 02 13:46:25.897366 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:46:25.897333    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e13b7fbf-86c5-44e9-b610-b35ddf12da45" path="/var/lib/kubelet/pods/e13b7fbf-86c5-44e9-b610-b35ddf12da45/volumes"
Nov 02 13:46:25.904835 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:46:25.904459    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-6f7bc57488-96bzn"]
Nov 02 13:46:25.914051 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:46:25.914014    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-6f7bc57488-96bzn"]
Nov 02 13:46:27.891503 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:46:27.891472    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7217353f-4f97-4760-9b66-16689ced7e9c" path="/var/lib/kubelet/pods/7217353f-4f97-4760-9b66-16689ced7e9c/volumes"
Nov 02 13:46:35.789721 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:46:35.789689    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-5bcd7997-59ffk"
Nov 02 13:46:45.084454 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:46:45.084419    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqcmj" containerName="extractor"
Nov 02 13:46:47.851924 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:46:47.851897    4559 scope.go:122] "RemoveContainer" containerID="a4bd881119ae63c941fe0955d4d3c4a10c1c00c9e9247c1f38c9fb5ae2a05e86"
Nov 02 13:46:47.860208 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:46:47.860190    4559 scope.go:122] "RemoveContainer" containerID="8188f5ac2fda98b8d1a3fef2bc4d63a0cd2c13ca32dfbf1b1b3cedd0ee77624d"
Nov 02 13:46:47.868121 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:46:47.868098    4559 scope.go:122] "RemoveContainer" containerID="8022b6826b8b50b4e191176b75d723d30538f52af739d90fab546273494d1b87"
Nov 02 13:47:15.084984 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:47:15.084949    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqcmj" containerName="extractor"
Nov 02 13:47:45.085030 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:47:45.084994    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqcmj" containerName="extractor"
Nov 02 13:47:54.032608 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:47:54.032575    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71066ac4-cbt2l"]
Nov 02 13:47:54.044025 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:47:54.043986    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71066ac4-cbt2l"]
Nov 02 13:47:54.132278 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:47:54.132244    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-hmknb"]
Nov 02 13:47:54.132618 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:47:54.132606    4559 cpu_manager.go:414] "containerMap: removing container" podUID="e13b7fbf-86c5-44e9-b610-b35ddf12da45" containerName="payload-processing"
Nov 02 13:47:54.132684 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:47:54.132619    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e13b7fbf-86c5-44e9-b610-b35ddf12da45" containerName="payload-processing"
Nov 02 13:47:54.132684 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:47:54.132659    4559 cpu_manager.go:414] "containerMap: removing container" podUID="7217353f-4f97-4760-9b66-16689ced7e9c" containerName="payload-pre-processing"
Nov 02 13:47:54.132684 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:47:54.132664    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7217353f-4f97-4760-9b66-16689ced7e9c" containerName="payload-pre-processing"
Nov 02 13:47:54.132793 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:47:54.132727    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="7217353f-4f97-4760-9b66-16689ced7e9c" containerName="payload-pre-processing"
Nov 02 13:47:54.132793 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:47:54.132739    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="e13b7fbf-86c5-44e9-b610-b35ddf12da45" containerName="payload-processing"
Nov 02 13:47:54.135941 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:47:54.135924    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-hmknb"
Nov 02 13:47:54.154225 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:47:54.154201    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 13:47:54.154421 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:47:54.154310    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-45fzn\""
Nov 02 13:47:54.155017 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:47:54.154995    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 13:47:54.155307 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:47:54.155290    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 13:47:54.156416 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:47:54.156392    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-hmknb"]
Nov 02 13:47:54.182161 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:47:54.182126    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqgkr\" (UniqueName: \"kubernetes.io/projected/6974e987-7d18-48d9-88df-a35017f430e6-kube-api-access-rqgkr\") pod \"maas-api-revoke-sub-e2547622-hmknb\" (UID: \"6974e987-7d18-48d9-88df-a35017f430e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-hmknb"
Nov 02 13:47:54.182300 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:47:54.182172    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6974e987-7d18-48d9-88df-a35017f430e6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2547622-hmknb\" (UID: \"6974e987-7d18-48d9-88df-a35017f430e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-hmknb"
Nov 02 13:47:54.182372 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:47:54.182295    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6974e987-7d18-48d9-88df-a35017f430e6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2547622-hmknb\" (UID: \"6974e987-7d18-48d9-88df-a35017f430e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-hmknb"
Nov 02 13:47:54.283060 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:47:54.282992    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqgkr\" (UniqueName: \"kubernetes.io/projected/6974e987-7d18-48d9-88df-a35017f430e6-kube-api-access-rqgkr\") pod \"maas-api-revoke-sub-e2547622-hmknb\" (UID: \"6974e987-7d18-48d9-88df-a35017f430e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-hmknb"
Nov 02 13:47:54.283060 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:47:54.283028    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6974e987-7d18-48d9-88df-a35017f430e6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2547622-hmknb\" (UID: \"6974e987-7d18-48d9-88df-a35017f430e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-hmknb"
Nov 02 13:47:54.283253 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:47:54.283147    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6974e987-7d18-48d9-88df-a35017f430e6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2547622-hmknb\" (UID: \"6974e987-7d18-48d9-88df-a35017f430e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-hmknb"
Nov 02 13:47:54.283634 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:47:54.283612    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6974e987-7d18-48d9-88df-a35017f430e6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2547622-hmknb\" (UID: \"6974e987-7d18-48d9-88df-a35017f430e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-hmknb"
Nov 02 13:47:54.285469 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:47:54.285446    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6974e987-7d18-48d9-88df-a35017f430e6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2547622-hmknb\" (UID: \"6974e987-7d18-48d9-88df-a35017f430e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-hmknb"
Nov 02 13:47:54.306711 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:47:54.306688    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqgkr\" (UniqueName: \"kubernetes.io/projected/6974e987-7d18-48d9-88df-a35017f430e6-kube-api-access-rqgkr\") pod \"maas-api-revoke-sub-e2547622-hmknb\" (UID: \"6974e987-7d18-48d9-88df-a35017f430e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-hmknb"
Nov 02 13:47:54.448713 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:47:54.448684    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-hmknb"
Nov 02 13:47:54.751344 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:47:54.751320    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-hmknb"]
Nov 02 13:47:54.752450 ip-10-0-128-132 kubenswrapper[4559]: W1102 13:47:54.752427    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6974e987_7d18_48d9_88df_a35017f430e6.slice/crio-0633b8f65026657bf99be72168bf79cdd6d895d20281f963ac6276ca4a36d68f WatchSource:0}: Error finding container 0633b8f65026657bf99be72168bf79cdd6d895d20281f963ac6276ca4a36d68f: Status 404 returned error can't find the container with id 0633b8f65026657bf99be72168bf79cdd6d895d20281f963ac6276ca4a36d68f
Nov 02 13:47:55.226122 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:47:55.226073    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-hmknb" event={"ID":"6974e987-7d18-48d9-88df-a35017f430e6","Type":"ContainerStarted","Data":"4075b58eeda95097fcabadd34fb6456670b3b3d385a354900ac086dac006a748"}
Nov 02 13:47:55.226633 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:47:55.226138    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-hmknb" event={"ID":"6974e987-7d18-48d9-88df-a35017f430e6","Type":"ContainerStarted","Data":"0633b8f65026657bf99be72168bf79cdd6d895d20281f963ac6276ca4a36d68f"}
Nov 02 13:47:55.268906 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:47:55.268824    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-hmknb" podStartSLOduration=1.268812002 podStartE2EDuration="1.268812002s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:47:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:47:55.267226583 +0000 UTC m=+4871.522625681" watchObservedRunningTime="2026-11-02 13:47:55.268812002 +0000 UTC m=+4871.524211096"
Nov 02 13:47:55.890728 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:47:55.890627    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0aba3dc-1b01-4ef5-9e8b-810523b85fbd" path="/var/lib/kubelet/pods/d0aba3dc-1b01-4ef5-9e8b-810523b85fbd/volumes"
Nov 02 13:47:56.232339 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:47:56.232307    4559 generic.go:356] "Generic (PLEG): container finished" podID="6974e987-7d18-48d9-88df-a35017f430e6" containerID="4075b58eeda95097fcabadd34fb6456670b3b3d385a354900ac086dac006a748" exitCode=0
Nov 02 13:47:56.232762 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:47:56.232390    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-hmknb" event={"ID":"6974e987-7d18-48d9-88df-a35017f430e6","Type":"ContainerDied","Data":"4075b58eeda95097fcabadd34fb6456670b3b3d385a354900ac086dac006a748"}
Nov 02 13:47:57.364280 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:47:57.364254    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-hmknb"
Nov 02 13:47:57.413307 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:47:57.413273    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6974e987-7d18-48d9-88df-a35017f430e6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6974e987-7d18-48d9-88df-a35017f430e6-maas-api-xks-ca\") pod \"6974e987-7d18-48d9-88df-a35017f430e6\" (UID: \"6974e987-7d18-48d9-88df-a35017f430e6\") "
Nov 02 13:47:57.413486 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:47:57.413410    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6974e987-7d18-48d9-88df-a35017f430e6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6974e987-7d18-48d9-88df-a35017f430e6-maas-api-openshift-ca\") pod \"6974e987-7d18-48d9-88df-a35017f430e6\" (UID: \"6974e987-7d18-48d9-88df-a35017f430e6\") "
Nov 02 13:47:57.413486 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:47:57.413473    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6974e987-7d18-48d9-88df-a35017f430e6-kube-api-access-rqgkr\" (UniqueName: \"kubernetes.io/projected/6974e987-7d18-48d9-88df-a35017f430e6-kube-api-access-rqgkr\") pod \"6974e987-7d18-48d9-88df-a35017f430e6\" (UID: \"6974e987-7d18-48d9-88df-a35017f430e6\") "
Nov 02 13:47:57.414230 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:47:57.414199    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6974e987-7d18-48d9-88df-a35017f430e6-maas-api-openshift-ca" pod "6974e987-7d18-48d9-88df-a35017f430e6" (UID: "6974e987-7d18-48d9-88df-a35017f430e6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:47:57.415680 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:47:57.415656    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6974e987-7d18-48d9-88df-a35017f430e6-kube-api-access-rqgkr" pod "6974e987-7d18-48d9-88df-a35017f430e6" (UID: "6974e987-7d18-48d9-88df-a35017f430e6"). InnerVolumeSpecName "kube-api-access-rqgkr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:47:57.415988 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:47:57.415963    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6974e987-7d18-48d9-88df-a35017f430e6-maas-api-xks-ca" pod "6974e987-7d18-48d9-88df-a35017f430e6" (UID: "6974e987-7d18-48d9-88df-a35017f430e6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:47:57.515106 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:47:57.515027    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6974e987-7d18-48d9-88df-a35017f430e6-maas-api-openshift-ca\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:47:57.515106 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:47:57.515067    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rqgkr\" (UniqueName: \"kubernetes.io/projected/6974e987-7d18-48d9-88df-a35017f430e6-kube-api-access-rqgkr\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:47:57.515106 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:47:57.515077    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6974e987-7d18-48d9-88df-a35017f430e6-maas-api-xks-ca\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:47:58.242385 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:47:58.242351    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-hmknb"
Nov 02 13:47:58.242573 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:47:58.242380    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-hmknb" event={"ID":"6974e987-7d18-48d9-88df-a35017f430e6","Type":"ContainerDied","Data":"0633b8f65026657bf99be72168bf79cdd6d895d20281f963ac6276ca4a36d68f"}
Nov 02 13:47:58.242573 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:47:58.242434    4559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0633b8f65026657bf99be72168bf79cdd6d895d20281f963ac6276ca4a36d68f"
Nov 02 13:48:04.947944 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:48:04.947908    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-59ffk"]
Nov 02 13:48:06.375988 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:48:06.375955    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"]
Nov 02 13:48:09.948208 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:48:09.948137    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-5bcd7997-59ffk" podUID="1ba2645c-9413-4125-9dac-3528fe6d3c00" containerName="payload-processing" containerID="cri-o://b6789cf5ca9bc51169154b84ca1bf2678204d5bcbdc41fc97185193b55b1e7f5" gracePeriod=25
Nov 02 13:48:10.199433 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:48:10.199371    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5bcd7997-59ffk"
Nov 02 13:48:10.321458 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:48:10.321421    4559 generic.go:356] "Generic (PLEG): container finished" podID="1ba2645c-9413-4125-9dac-3528fe6d3c00" containerID="b6789cf5ca9bc51169154b84ca1bf2678204d5bcbdc41fc97185193b55b1e7f5" exitCode=0
Nov 02 13:48:10.321698 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:48:10.321474    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5bcd7997-59ffk"
Nov 02 13:48:10.321698 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:48:10.321509    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5bcd7997-59ffk" event={"ID":"1ba2645c-9413-4125-9dac-3528fe6d3c00","Type":"ContainerDied","Data":"b6789cf5ca9bc51169154b84ca1bf2678204d5bcbdc41fc97185193b55b1e7f5"}
Nov 02 13:48:10.321698 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:48:10.321567    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5bcd7997-59ffk" event={"ID":"1ba2645c-9413-4125-9dac-3528fe6d3c00","Type":"ContainerDied","Data":"d45464c46dc5b6b526d4ac215e76a323cebcad14c7c003524bf60a5202bea07b"}
Nov 02 13:48:10.321698 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:48:10.321609    4559 scope.go:122] "RemoveContainer" containerID="b6789cf5ca9bc51169154b84ca1bf2678204d5bcbdc41fc97185193b55b1e7f5"
Nov 02 13:48:10.329874 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:48:10.329857    4559 scope.go:122] "RemoveContainer" containerID="b6789cf5ca9bc51169154b84ca1bf2678204d5bcbdc41fc97185193b55b1e7f5"
Nov 02 13:48:10.330151 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:48:10.330133    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6789cf5ca9bc51169154b84ca1bf2678204d5bcbdc41fc97185193b55b1e7f5\": container with ID starting with b6789cf5ca9bc51169154b84ca1bf2678204d5bcbdc41fc97185193b55b1e7f5 not found: ID does not exist" containerID="b6789cf5ca9bc51169154b84ca1bf2678204d5bcbdc41fc97185193b55b1e7f5"
Nov 02 13:48:10.330222 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:48:10.330163    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6789cf5ca9bc51169154b84ca1bf2678204d5bcbdc41fc97185193b55b1e7f5"} err="failed to get container status \"b6789cf5ca9bc51169154b84ca1bf2678204d5bcbdc41fc97185193b55b1e7f5\": rpc error: code = NotFound desc = could not find container \"b6789cf5ca9bc51169154b84ca1bf2678204d5bcbdc41fc97185193b55b1e7f5\": container with ID starting with b6789cf5ca9bc51169154b84ca1bf2678204d5bcbdc41fc97185193b55b1e7f5 not found: ID does not exist"
Nov 02 13:48:10.367303 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:48:10.367275    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ba2645c-9413-4125-9dac-3528fe6d3c00-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ba2645c-9413-4125-9dac-3528fe6d3c00-plugins-config-volume\") pod \"1ba2645c-9413-4125-9dac-3528fe6d3c00\" (UID: \"1ba2645c-9413-4125-9dac-3528fe6d3c00\") "
Nov 02 13:48:10.367415 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:48:10.367338    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ba2645c-9413-4125-9dac-3528fe6d3c00-provider-credentials\" (UniqueName: \"kubernetes.io/projected/1ba2645c-9413-4125-9dac-3528fe6d3c00-provider-credentials\") pod \"1ba2645c-9413-4125-9dac-3528fe6d3c00\" (UID: \"1ba2645c-9413-4125-9dac-3528fe6d3c00\") "
Nov 02 13:48:10.367488 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:48:10.367417    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ba2645c-9413-4125-9dac-3528fe6d3c00-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1ba2645c-9413-4125-9dac-3528fe6d3c00-routing-overlay-volume\") pod \"1ba2645c-9413-4125-9dac-3528fe6d3c00\" (UID: \"1ba2645c-9413-4125-9dac-3528fe6d3c00\") "
Nov 02 13:48:10.367488 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:48:10.367444    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ba2645c-9413-4125-9dac-3528fe6d3c00-kube-api-access-z25z7\" (UniqueName: \"kubernetes.io/projected/1ba2645c-9413-4125-9dac-3528fe6d3c00-kube-api-access-z25z7\") pod \"1ba2645c-9413-4125-9dac-3528fe6d3c00\" (UID: \"1ba2645c-9413-4125-9dac-3528fe6d3c00\") "
Nov 02 13:48:10.367792 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:48:10.367764    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ba2645c-9413-4125-9dac-3528fe6d3c00-plugins-config-volume" pod "1ba2645c-9413-4125-9dac-3528fe6d3c00" (UID: "1ba2645c-9413-4125-9dac-3528fe6d3c00"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:48:10.367891 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:48:10.367836    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ba2645c-9413-4125-9dac-3528fe6d3c00-routing-overlay-volume" pod "1ba2645c-9413-4125-9dac-3528fe6d3c00" (UID: "1ba2645c-9413-4125-9dac-3528fe6d3c00"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:48:10.369443 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:48:10.369426    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ba2645c-9413-4125-9dac-3528fe6d3c00-kube-api-access-z25z7" pod "1ba2645c-9413-4125-9dac-3528fe6d3c00" (UID: "1ba2645c-9413-4125-9dac-3528fe6d3c00"). InnerVolumeSpecName "kube-api-access-z25z7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:48:10.369551 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:48:10.369532    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ba2645c-9413-4125-9dac-3528fe6d3c00-provider-credentials" pod "1ba2645c-9413-4125-9dac-3528fe6d3c00" (UID: "1ba2645c-9413-4125-9dac-3528fe6d3c00"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:48:10.468930 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:48:10.468878    4559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ba2645c-9413-4125-9dac-3528fe6d3c00-plugins-config-volume\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:48:10.468930 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:48:10.468899    4559 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1ba2645c-9413-4125-9dac-3528fe6d3c00-provider-credentials\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:48:10.468930 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:48:10.468911    4559 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1ba2645c-9413-4125-9dac-3528fe6d3c00-routing-overlay-volume\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:48:10.468930 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:48:10.468920    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z25z7\" (UniqueName: \"kubernetes.io/projected/1ba2645c-9413-4125-9dac-3528fe6d3c00-kube-api-access-z25z7\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:48:10.667232 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:48:10.667204    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-59ffk"]
Nov 02 13:48:10.679964 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:48:10.679940    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-59ffk"]
Nov 02 13:48:11.891336 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:48:11.891307    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ba2645c-9413-4125-9dac-3528fe6d3c00" path="/var/lib/kubelet/pods/1ba2645c-9413-4125-9dac-3528fe6d3c00/volumes"
Nov 02 13:48:13.217067 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:48:13.217031    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"]
Nov 02 13:48:15.278021 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:48:15.277940    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqcmj" containerName="extractor"
Nov 02 13:48:17.843960 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:48:17.843925    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-9wzwg"]
Nov 02 13:48:17.844404 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:48:17.844362    4559 cpu_manager.go:414] "containerMap: removing container" podUID="1ba2645c-9413-4125-9dac-3528fe6d3c00" containerName="payload-processing"
Nov 02 13:48:17.844404 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:48:17.844379    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ba2645c-9413-4125-9dac-3528fe6d3c00" containerName="payload-processing"
Nov 02 13:48:17.844404 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:48:17.844392    4559 cpu_manager.go:414] "containerMap: removing container" podUID="6974e987-7d18-48d9-88df-a35017f430e6" containerName="revoke-keys"
Nov 02 13:48:17.844404 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:48:17.844398    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6974e987-7d18-48d9-88df-a35017f430e6" containerName="revoke-keys"
Nov 02 13:48:17.844572 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:48:17.844465    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ba2645c-9413-4125-9dac-3528fe6d3c00" containerName="payload-processing"
Nov 02 13:48:17.844572 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:48:17.844476    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="6974e987-7d18-48d9-88df-a35017f430e6" containerName="revoke-keys"
Nov 02 13:48:17.847546 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:48:17.847529    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-9wzwg"
Nov 02 13:48:17.867770 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:48:17.867594    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-9wzwg"]
Nov 02 13:48:17.878190 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:48:17.878163    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 13:48:17.878375 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:48:17.878176    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 13:48:17.878465 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:48:17.878252    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 13:48:17.878546 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:48:17.878367    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-45fzn\""
Nov 02 13:48:17.973029 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:48:17.972998    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvnhp\" (UniqueName: \"kubernetes.io/projected/3b18195c-ef6c-41ba-83fb-bc46f1cb0a75-kube-api-access-xvnhp\") pod \"maas-api-revoke-sub-c1e271a9-9wzwg\" (UID: \"3b18195c-ef6c-41ba-83fb-bc46f1cb0a75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-9wzwg"
Nov 02 13:48:17.973188 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:48:17.973065    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b18195c-ef6c-41ba-83fb-bc46f1cb0a75-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-9wzwg\" (UID: \"3b18195c-ef6c-41ba-83fb-bc46f1cb0a75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-9wzwg"
Nov 02 13:48:17.973188 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:48:17.973144    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b18195c-ef6c-41ba-83fb-bc46f1cb0a75-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-9wzwg\" (UID: \"3b18195c-ef6c-41ba-83fb-bc46f1cb0a75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-9wzwg"
Nov 02 13:48:18.073809 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:48:18.073770    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b18195c-ef6c-41ba-83fb-bc46f1cb0a75-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-9wzwg\" (UID: \"3b18195c-ef6c-41ba-83fb-bc46f1cb0a75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-9wzwg"
Nov 02 13:48:18.073809 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:48:18.073812    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b18195c-ef6c-41ba-83fb-bc46f1cb0a75-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-9wzwg\" (UID: \"3b18195c-ef6c-41ba-83fb-bc46f1cb0a75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-9wzwg"
Nov 02 13:48:18.074058 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:48:18.073898    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvnhp\" (UniqueName: \"kubernetes.io/projected/3b18195c-ef6c-41ba-83fb-bc46f1cb0a75-kube-api-access-xvnhp\") pod \"maas-api-revoke-sub-c1e271a9-9wzwg\" (UID: \"3b18195c-ef6c-41ba-83fb-bc46f1cb0a75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-9wzwg"
Nov 02 13:48:18.074602 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:48:18.074571    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b18195c-ef6c-41ba-83fb-bc46f1cb0a75-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-9wzwg\" (UID: \"3b18195c-ef6c-41ba-83fb-bc46f1cb0a75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-9wzwg"
Nov 02 13:48:18.076277 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:48:18.076249    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b18195c-ef6c-41ba-83fb-bc46f1cb0a75-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-9wzwg\" (UID: \"3b18195c-ef6c-41ba-83fb-bc46f1cb0a75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-9wzwg"
Nov 02 13:48:18.098338 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:48:18.098275    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvnhp\" (UniqueName: \"kubernetes.io/projected/3b18195c-ef6c-41ba-83fb-bc46f1cb0a75-kube-api-access-xvnhp\") pod \"maas-api-revoke-sub-c1e271a9-9wzwg\" (UID: \"3b18195c-ef6c-41ba-83fb-bc46f1cb0a75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-9wzwg"
Nov 02 13:48:18.164892 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:48:18.164868    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-9wzwg"
Nov 02 13:48:18.462442 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:48:18.462416    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-9wzwg"]
Nov 02 13:48:18.463129 ip-10-0-128-132 kubenswrapper[4559]: W1102 13:48:18.463101    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b18195c_ef6c_41ba_83fb_bc46f1cb0a75.slice/crio-dd28ac58efc9a2e7dc36dad1f46bf5f661f3b7def9088cef1f79591149b7ba20 WatchSource:0}: Error finding container dd28ac58efc9a2e7dc36dad1f46bf5f661f3b7def9088cef1f79591149b7ba20: Status 404 returned error can't find the container with id dd28ac58efc9a2e7dc36dad1f46bf5f661f3b7def9088cef1f79591149b7ba20
Nov 02 13:48:19.376872 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:48:19.376834    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-9wzwg" event={"ID":"3b18195c-ef6c-41ba-83fb-bc46f1cb0a75","Type":"ContainerStarted","Data":"2a32d24005943611c69297614643660dbb12fcf2393e4c2aeda277d553a06130"}
Nov 02 13:48:19.377291 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:48:19.376895    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-9wzwg" event={"ID":"3b18195c-ef6c-41ba-83fb-bc46f1cb0a75","Type":"ContainerStarted","Data":"dd28ac58efc9a2e7dc36dad1f46bf5f661f3b7def9088cef1f79591149b7ba20"}
Nov 02 13:48:19.417994 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:48:19.417944    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-9wzwg" podStartSLOduration=2.417931044 podStartE2EDuration="2.417931044s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:48:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:48:19.417826206 +0000 UTC m=+4895.673225301" watchObservedRunningTime="2026-11-02 13:48:19.417931044 +0000 UTC m=+4895.673330139"
Nov 02 13:48:20.383139 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:48:20.383105    4559 generic.go:356] "Generic (PLEG): container finished" podID="3b18195c-ef6c-41ba-83fb-bc46f1cb0a75" containerID="2a32d24005943611c69297614643660dbb12fcf2393e4c2aeda277d553a06130" exitCode=0
Nov 02 13:48:20.383612 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:48:20.383194    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-9wzwg" event={"ID":"3b18195c-ef6c-41ba-83fb-bc46f1cb0a75","Type":"ContainerDied","Data":"2a32d24005943611c69297614643660dbb12fcf2393e4c2aeda277d553a06130"}
Nov 02 13:48:21.518332 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:48:21.518307    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-9wzwg"
Nov 02 13:48:21.711289 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:48:21.711255    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3b18195c-ef6c-41ba-83fb-bc46f1cb0a75-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b18195c-ef6c-41ba-83fb-bc46f1cb0a75-maas-api-xks-ca\") pod \"3b18195c-ef6c-41ba-83fb-bc46f1cb0a75\" (UID: \"3b18195c-ef6c-41ba-83fb-bc46f1cb0a75\") "
Nov 02 13:48:21.711485 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:48:21.711303    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b18195c-ef6c-41ba-83fb-bc46f1cb0a75-kube-api-access-xvnhp\" (UniqueName: \"kubernetes.io/projected/3b18195c-ef6c-41ba-83fb-bc46f1cb0a75-kube-api-access-xvnhp\") pod \"3b18195c-ef6c-41ba-83fb-bc46f1cb0a75\" (UID: \"3b18195c-ef6c-41ba-83fb-bc46f1cb0a75\") "
Nov 02 13:48:21.711485 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:48:21.711347    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b18195c-ef6c-41ba-83fb-bc46f1cb0a75-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b18195c-ef6c-41ba-83fb-bc46f1cb0a75-maas-api-openshift-ca\") pod \"3b18195c-ef6c-41ba-83fb-bc46f1cb0a75\" (UID: \"3b18195c-ef6c-41ba-83fb-bc46f1cb0a75\") "
Nov 02 13:48:21.711819 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:48:21.711791    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b18195c-ef6c-41ba-83fb-bc46f1cb0a75-maas-api-openshift-ca" pod "3b18195c-ef6c-41ba-83fb-bc46f1cb0a75" (UID: "3b18195c-ef6c-41ba-83fb-bc46f1cb0a75"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:48:21.713448 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:48:21.713420    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b18195c-ef6c-41ba-83fb-bc46f1cb0a75-maas-api-xks-ca" pod "3b18195c-ef6c-41ba-83fb-bc46f1cb0a75" (UID: "3b18195c-ef6c-41ba-83fb-bc46f1cb0a75"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:48:21.713448 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:48:21.713437    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b18195c-ef6c-41ba-83fb-bc46f1cb0a75-kube-api-access-xvnhp" pod "3b18195c-ef6c-41ba-83fb-bc46f1cb0a75" (UID: "3b18195c-ef6c-41ba-83fb-bc46f1cb0a75"). InnerVolumeSpecName "kube-api-access-xvnhp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:48:21.812783 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:48:21.812749    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b18195c-ef6c-41ba-83fb-bc46f1cb0a75-maas-api-xks-ca\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:48:21.812783 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:48:21.812777    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvnhp\" (UniqueName: \"kubernetes.io/projected/3b18195c-ef6c-41ba-83fb-bc46f1cb0a75-kube-api-access-xvnhp\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:48:21.812783 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:48:21.812788    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b18195c-ef6c-41ba-83fb-bc46f1cb0a75-maas-api-openshift-ca\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:48:22.392924 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:48:22.392884    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-9wzwg" event={"ID":"3b18195c-ef6c-41ba-83fb-bc46f1cb0a75","Type":"ContainerDied","Data":"dd28ac58efc9a2e7dc36dad1f46bf5f661f3b7def9088cef1f79591149b7ba20"}
Nov 02 13:48:22.392924 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:48:22.392920    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-9wzwg"
Nov 02 13:48:22.393150 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:48:22.392947    4559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dd28ac58efc9a2e7dc36dad1f46bf5f661f3b7def9088cef1f79591149b7ba20"
Nov 02 13:48:29.123324 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:48:29.123238    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"]
Nov 02 13:48:33.397409 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:48:33.397372    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"]
Nov 02 13:48:37.143298 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:48:37.143255    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-dpt95"]
Nov 02 13:48:37.143842 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:48:37.143732    4559 cpu_manager.go:414] "containerMap: removing container" podUID="3b18195c-ef6c-41ba-83fb-bc46f1cb0a75" containerName="revoke-keys"
Nov 02 13:48:37.143842 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:48:37.143748    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b18195c-ef6c-41ba-83fb-bc46f1cb0a75" containerName="revoke-keys"
Nov 02 13:48:37.144041 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:48:37.143846    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b18195c-ef6c-41ba-83fb-bc46f1cb0a75" containerName="revoke-keys"
Nov 02 13:48:37.146716 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:48:37.146695    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-dpt95"
Nov 02 13:48:37.166781 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:48:37.166750    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-dpt95"]
Nov 02 13:48:37.176567 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:48:37.176537    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 13:48:37.176771 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:48:37.176626    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 13:48:37.176771 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:48:37.176735    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-45fzn\""
Nov 02 13:48:37.176923 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:48:37.176807    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 13:48:37.182096 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:48:37.182065    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f8f5849-a389-40e9-86e3-1697da21291d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-dpt95\" (UID: \"6f8f5849-a389-40e9-86e3-1697da21291d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-dpt95"
Nov 02 13:48:37.182211 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:48:37.182117    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f8f5849-a389-40e9-86e3-1697da21291d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-dpt95\" (UID: \"6f8f5849-a389-40e9-86e3-1697da21291d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-dpt95"
Nov 02 13:48:37.182269 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:48:37.182237    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bsbxv\" (UniqueName: \"kubernetes.io/projected/6f8f5849-a389-40e9-86e3-1697da21291d-kube-api-access-bsbxv\") pod \"maas-api-revoke-sub-2e1f5b5f-dpt95\" (UID: \"6f8f5849-a389-40e9-86e3-1697da21291d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-dpt95"
Nov 02 13:48:37.282891 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:48:37.282858    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f8f5849-a389-40e9-86e3-1697da21291d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-dpt95\" (UID: \"6f8f5849-a389-40e9-86e3-1697da21291d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-dpt95"
Nov 02 13:48:37.283066 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:48:37.282920    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bsbxv\" (UniqueName: \"kubernetes.io/projected/6f8f5849-a389-40e9-86e3-1697da21291d-kube-api-access-bsbxv\") pod \"maas-api-revoke-sub-2e1f5b5f-dpt95\" (UID: \"6f8f5849-a389-40e9-86e3-1697da21291d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-dpt95"
Nov 02 13:48:37.283066 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:48:37.283032    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f8f5849-a389-40e9-86e3-1697da21291d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-dpt95\" (UID: \"6f8f5849-a389-40e9-86e3-1697da21291d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-dpt95"
Nov 02 13:48:37.283558 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:48:37.283532    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f8f5849-a389-40e9-86e3-1697da21291d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-dpt95\" (UID: \"6f8f5849-a389-40e9-86e3-1697da21291d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-dpt95"
Nov 02 13:48:37.285297 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:48:37.285269    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f8f5849-a389-40e9-86e3-1697da21291d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-dpt95\" (UID: \"6f8f5849-a389-40e9-86e3-1697da21291d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-dpt95"
Nov 02 13:48:37.306969 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:48:37.306936    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bsbxv\" (UniqueName: \"kubernetes.io/projected/6f8f5849-a389-40e9-86e3-1697da21291d-kube-api-access-bsbxv\") pod \"maas-api-revoke-sub-2e1f5b5f-dpt95\" (UID: \"6f8f5849-a389-40e9-86e3-1697da21291d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-dpt95"
Nov 02 13:48:37.464674 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:48:37.464616    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-dpt95"
Nov 02 13:48:37.761064 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:48:37.761035    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-dpt95"]
Nov 02 13:48:37.761831 ip-10-0-128-132 kubenswrapper[4559]: W1102 13:48:37.761805    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f8f5849_a389_40e9_86e3_1697da21291d.slice/crio-fc5e33173fc1617b62e4b3405f43605ed34e1e01a45e787653bef20b3b93a81e WatchSource:0}: Error finding container fc5e33173fc1617b62e4b3405f43605ed34e1e01a45e787653bef20b3b93a81e: Status 404 returned error can't find the container with id fc5e33173fc1617b62e4b3405f43605ed34e1e01a45e787653bef20b3b93a81e
Nov 02 13:48:38.483715 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:48:38.483680    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-dpt95" event={"ID":"6f8f5849-a389-40e9-86e3-1697da21291d","Type":"ContainerStarted","Data":"464df2712fa14316ea20741c1247e65907196e8fde6e4c6b7f0d6048ebf9c774"}
Nov 02 13:48:38.484156 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:48:38.483741    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-dpt95" event={"ID":"6f8f5849-a389-40e9-86e3-1697da21291d","Type":"ContainerStarted","Data":"fc5e33173fc1617b62e4b3405f43605ed34e1e01a45e787653bef20b3b93a81e"}
Nov 02 13:48:38.525858 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:48:38.525814    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-dpt95" podStartSLOduration=1.525802338 podStartE2EDuration="1.525802338s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:48:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:48:38.524621855 +0000 UTC m=+4914.780020949" watchObservedRunningTime="2026-11-02 13:48:38.525802338 +0000 UTC m=+4914.781201429"
Nov 02 13:48:39.489433 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:48:39.489397    4559 generic.go:356] "Generic (PLEG): container finished" podID="6f8f5849-a389-40e9-86e3-1697da21291d" containerID="464df2712fa14316ea20741c1247e65907196e8fde6e4c6b7f0d6048ebf9c774" exitCode=0
Nov 02 13:48:39.489868 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:48:39.489488    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-dpt95" event={"ID":"6f8f5849-a389-40e9-86e3-1697da21291d","Type":"ContainerDied","Data":"464df2712fa14316ea20741c1247e65907196e8fde6e4c6b7f0d6048ebf9c774"}
Nov 02 13:48:40.622605 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:48:40.622577    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-dpt95"
Nov 02 13:48:40.706208 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:48:40.706165    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f8f5849-a389-40e9-86e3-1697da21291d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f8f5849-a389-40e9-86e3-1697da21291d-maas-api-openshift-ca\") pod \"6f8f5849-a389-40e9-86e3-1697da21291d\" (UID: \"6f8f5849-a389-40e9-86e3-1697da21291d\") "
Nov 02 13:48:40.706384 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:48:40.706243    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f8f5849-a389-40e9-86e3-1697da21291d-kube-api-access-bsbxv\" (UniqueName: \"kubernetes.io/projected/6f8f5849-a389-40e9-86e3-1697da21291d-kube-api-access-bsbxv\") pod \"6f8f5849-a389-40e9-86e3-1697da21291d\" (UID: \"6f8f5849-a389-40e9-86e3-1697da21291d\") "
Nov 02 13:48:40.706384 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:48:40.706336    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6f8f5849-a389-40e9-86e3-1697da21291d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f8f5849-a389-40e9-86e3-1697da21291d-maas-api-xks-ca\") pod \"6f8f5849-a389-40e9-86e3-1697da21291d\" (UID: \"6f8f5849-a389-40e9-86e3-1697da21291d\") "
Nov 02 13:48:40.706552 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:48:40.706527    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f8f5849-a389-40e9-86e3-1697da21291d-maas-api-openshift-ca" pod "6f8f5849-a389-40e9-86e3-1697da21291d" (UID: "6f8f5849-a389-40e9-86e3-1697da21291d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:48:40.706744 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:48:40.706716    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f8f5849-a389-40e9-86e3-1697da21291d-maas-api-openshift-ca\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:48:40.708386 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:48:40.708362    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f8f5849-a389-40e9-86e3-1697da21291d-maas-api-xks-ca" pod "6f8f5849-a389-40e9-86e3-1697da21291d" (UID: "6f8f5849-a389-40e9-86e3-1697da21291d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:48:40.708710 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:48:40.708693    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f8f5849-a389-40e9-86e3-1697da21291d-kube-api-access-bsbxv" pod "6f8f5849-a389-40e9-86e3-1697da21291d" (UID: "6f8f5849-a389-40e9-86e3-1697da21291d"). InnerVolumeSpecName "kube-api-access-bsbxv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:48:40.807447 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:48:40.807384    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bsbxv\" (UniqueName: \"kubernetes.io/projected/6f8f5849-a389-40e9-86e3-1697da21291d-kube-api-access-bsbxv\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:48:40.807447 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:48:40.807409    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f8f5849-a389-40e9-86e3-1697da21291d-maas-api-xks-ca\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:48:41.500262 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:48:41.500234    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-dpt95"
Nov 02 13:48:41.500447 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:48:41.500228    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-dpt95" event={"ID":"6f8f5849-a389-40e9-86e3-1697da21291d","Type":"ContainerDied","Data":"fc5e33173fc1617b62e4b3405f43605ed34e1e01a45e787653bef20b3b93a81e"}
Nov 02 13:48:41.500447 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:48:41.500360    4559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fc5e33173fc1617b62e4b3405f43605ed34e1e01a45e787653bef20b3b93a81e"
Nov 02 13:48:45.085285 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:48:45.085244    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqcmj" containerName="extractor"
Nov 02 13:48:47.915204 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:48:47.915172    4559 scope.go:122] "RemoveContainer" containerID="7f1c55f58083e7640f6b7c1acc0e5b195cae92275b18d5fa0c957824290e1504"
Nov 02 13:48:48.330432 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:48:48.330399    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-2hnrw"]
Nov 02 13:48:48.330817 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:48:48.330803    4559 cpu_manager.go:414] "containerMap: removing container" podUID="6f8f5849-a389-40e9-86e3-1697da21291d" containerName="revoke-keys"
Nov 02 13:48:48.330887 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:48:48.330818    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f8f5849-a389-40e9-86e3-1697da21291d" containerName="revoke-keys"
Nov 02 13:48:48.330887 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:48:48.330873    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f8f5849-a389-40e9-86e3-1697da21291d" containerName="revoke-keys"
Nov 02 13:48:48.334800 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:48:48.334781    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-2hnrw"
Nov 02 13:48:48.360542 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:48:48.360506    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-2hnrw"]
Nov 02 13:48:48.370414 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:48:48.370387    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 13:48:48.370592 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:48:48.370417    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 13:48:48.370592 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:48:48.370441    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 13:48:48.370592 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:48:48.370489    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-45fzn\""
Nov 02 13:48:48.372941 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:48:48.372915    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce8b223b-090f-4bb7-ad38-deddcc9f6dd6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-2hnrw\" (UID: \"ce8b223b-090f-4bb7-ad38-deddcc9f6dd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-2hnrw"
Nov 02 13:48:48.373046 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:48:48.372998    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvddp\" (UniqueName: \"kubernetes.io/projected/ce8b223b-090f-4bb7-ad38-deddcc9f6dd6-kube-api-access-mvddp\") pod \"maas-api-revoke-sub-ec843018-2hnrw\" (UID: \"ce8b223b-090f-4bb7-ad38-deddcc9f6dd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-2hnrw"
Nov 02 13:48:48.373107 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:48:48.373058    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce8b223b-090f-4bb7-ad38-deddcc9f6dd6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-2hnrw\" (UID: \"ce8b223b-090f-4bb7-ad38-deddcc9f6dd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-2hnrw"
Nov 02 13:48:48.473793 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:48:48.473760    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce8b223b-090f-4bb7-ad38-deddcc9f6dd6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-2hnrw\" (UID: \"ce8b223b-090f-4bb7-ad38-deddcc9f6dd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-2hnrw"
Nov 02 13:48:48.473964 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:48:48.473804    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce8b223b-090f-4bb7-ad38-deddcc9f6dd6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-2hnrw\" (UID: \"ce8b223b-090f-4bb7-ad38-deddcc9f6dd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-2hnrw"
Nov 02 13:48:48.473964 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:48:48.473875    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvddp\" (UniqueName: \"kubernetes.io/projected/ce8b223b-090f-4bb7-ad38-deddcc9f6dd6-kube-api-access-mvddp\") pod \"maas-api-revoke-sub-ec843018-2hnrw\" (UID: \"ce8b223b-090f-4bb7-ad38-deddcc9f6dd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-2hnrw"
Nov 02 13:48:48.474345 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:48:48.474323    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce8b223b-090f-4bb7-ad38-deddcc9f6dd6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-2hnrw\" (UID: \"ce8b223b-090f-4bb7-ad38-deddcc9f6dd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-2hnrw"
Nov 02 13:48:48.476165 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:48:48.476139    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce8b223b-090f-4bb7-ad38-deddcc9f6dd6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-2hnrw\" (UID: \"ce8b223b-090f-4bb7-ad38-deddcc9f6dd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-2hnrw"
Nov 02 13:48:48.497679 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:48:48.497650    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"]
Nov 02 13:48:48.497788 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:48:48.497684    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvddp\" (UniqueName: \"kubernetes.io/projected/ce8b223b-090f-4bb7-ad38-deddcc9f6dd6-kube-api-access-mvddp\") pod \"maas-api-revoke-sub-ec843018-2hnrw\" (UID: \"ce8b223b-090f-4bb7-ad38-deddcc9f6dd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-2hnrw"
Nov 02 13:48:48.647242 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:48:48.647172    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-2hnrw"
Nov 02 13:48:48.949358 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:48:48.949326    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-2hnrw"]
Nov 02 13:48:48.949849 ip-10-0-128-132 kubenswrapper[4559]: W1102 13:48:48.949826    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce8b223b_090f_4bb7_ad38_deddcc9f6dd6.slice/crio-69fe7d9489bd9c7b493ef7ac5d6d60d117d7729d0849811076f5b1d29d5f4cfe WatchSource:0}: Error finding container 69fe7d9489bd9c7b493ef7ac5d6d60d117d7729d0849811076f5b1d29d5f4cfe: Status 404 returned error can't find the container with id 69fe7d9489bd9c7b493ef7ac5d6d60d117d7729d0849811076f5b1d29d5f4cfe
Nov 02 13:48:49.538119 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:48:49.538085    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-2hnrw" event={"ID":"ce8b223b-090f-4bb7-ad38-deddcc9f6dd6","Type":"ContainerStarted","Data":"a6fab3a2b765ee10ffa2711fac46ddf53736a54049d117e170bcb13cdcad4ed5"}
Nov 02 13:48:49.538284 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:48:49.538135    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-2hnrw" event={"ID":"ce8b223b-090f-4bb7-ad38-deddcc9f6dd6","Type":"ContainerStarted","Data":"69fe7d9489bd9c7b493ef7ac5d6d60d117d7729d0849811076f5b1d29d5f4cfe"}
Nov 02 13:48:49.581689 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:48:49.581617    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-2hnrw" podStartSLOduration=1.5815965300000001 podStartE2EDuration="1.58159653s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:48:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:48:49.581084943 +0000 UTC m=+4925.836484037" watchObservedRunningTime="2026-11-02 13:48:49.58159653 +0000 UTC m=+4925.836995625"
Nov 02 13:48:50.546962 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:48:50.546928    4559 generic.go:356] "Generic (PLEG): container finished" podID="ce8b223b-090f-4bb7-ad38-deddcc9f6dd6" containerID="a6fab3a2b765ee10ffa2711fac46ddf53736a54049d117e170bcb13cdcad4ed5" exitCode=0
Nov 02 13:48:50.547384 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:48:50.546981    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-2hnrw" event={"ID":"ce8b223b-090f-4bb7-ad38-deddcc9f6dd6","Type":"ContainerDied","Data":"a6fab3a2b765ee10ffa2711fac46ddf53736a54049d117e170bcb13cdcad4ed5"}
Nov 02 13:48:51.753027 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:48:51.752742    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-2hnrw"
Nov 02 13:48:51.806262 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:48:51.805767    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce8b223b-090f-4bb7-ad38-deddcc9f6dd6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce8b223b-090f-4bb7-ad38-deddcc9f6dd6-maas-api-openshift-ca\") pod \"ce8b223b-090f-4bb7-ad38-deddcc9f6dd6\" (UID: \"ce8b223b-090f-4bb7-ad38-deddcc9f6dd6\") "
Nov 02 13:48:51.806262 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:48:51.805860    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce8b223b-090f-4bb7-ad38-deddcc9f6dd6-kube-api-access-mvddp\" (UniqueName: \"kubernetes.io/projected/ce8b223b-090f-4bb7-ad38-deddcc9f6dd6-kube-api-access-mvddp\") pod \"ce8b223b-090f-4bb7-ad38-deddcc9f6dd6\" (UID: \"ce8b223b-090f-4bb7-ad38-deddcc9f6dd6\") "
Nov 02 13:48:51.806262 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:48:51.805915    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ce8b223b-090f-4bb7-ad38-deddcc9f6dd6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce8b223b-090f-4bb7-ad38-deddcc9f6dd6-maas-api-xks-ca\") pod \"ce8b223b-090f-4bb7-ad38-deddcc9f6dd6\" (UID: \"ce8b223b-090f-4bb7-ad38-deddcc9f6dd6\") "
Nov 02 13:48:51.807387 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:48:51.807340    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce8b223b-090f-4bb7-ad38-deddcc9f6dd6-maas-api-openshift-ca" pod "ce8b223b-090f-4bb7-ad38-deddcc9f6dd6" (UID: "ce8b223b-090f-4bb7-ad38-deddcc9f6dd6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:48:51.809857 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:48:51.809827    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce8b223b-090f-4bb7-ad38-deddcc9f6dd6-maas-api-xks-ca" pod "ce8b223b-090f-4bb7-ad38-deddcc9f6dd6" (UID: "ce8b223b-090f-4bb7-ad38-deddcc9f6dd6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:48:51.815109 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:48:51.815082    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce8b223b-090f-4bb7-ad38-deddcc9f6dd6-kube-api-access-mvddp" pod "ce8b223b-090f-4bb7-ad38-deddcc9f6dd6" (UID: "ce8b223b-090f-4bb7-ad38-deddcc9f6dd6"). InnerVolumeSpecName "kube-api-access-mvddp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:48:51.907742 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:48:51.907666    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce8b223b-090f-4bb7-ad38-deddcc9f6dd6-maas-api-openshift-ca\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:48:51.907742 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:48:51.907699    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mvddp\" (UniqueName: \"kubernetes.io/projected/ce8b223b-090f-4bb7-ad38-deddcc9f6dd6-kube-api-access-mvddp\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:48:51.907742 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:48:51.907716    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce8b223b-090f-4bb7-ad38-deddcc9f6dd6-maas-api-xks-ca\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:48:52.609663 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:48:52.606611    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-2hnrw" event={"ID":"ce8b223b-090f-4bb7-ad38-deddcc9f6dd6","Type":"ContainerDied","Data":"69fe7d9489bd9c7b493ef7ac5d6d60d117d7729d0849811076f5b1d29d5f4cfe"}
Nov 02 13:48:52.609663 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:48:52.608781    4559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="69fe7d9489bd9c7b493ef7ac5d6d60d117d7729d0849811076f5b1d29d5f4cfe"
Nov 02 13:48:52.609663 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:48:52.608880    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-2hnrw"
Nov 02 13:49:04.019027 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:49:04.018989    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"]
Nov 02 13:49:15.085117 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:49:15.085079    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqcmj" containerName="extractor"
Nov 02 13:49:16.012360 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:49:16.012325    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-qzrs5"]
Nov 02 13:49:16.012744 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:49:16.012730    4559 cpu_manager.go:414] "containerMap: removing container" podUID="ce8b223b-090f-4bb7-ad38-deddcc9f6dd6" containerName="revoke-keys"
Nov 02 13:49:16.012799 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:49:16.012745    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce8b223b-090f-4bb7-ad38-deddcc9f6dd6" containerName="revoke-keys"
Nov 02 13:49:16.012843 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:49:16.012833    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce8b223b-090f-4bb7-ad38-deddcc9f6dd6" containerName="revoke-keys"
Nov 02 13:49:16.016357 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:49:16.016340    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-qzrs5"
Nov 02 13:49:16.036501 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:49:16.036472    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-qzrs5"]
Nov 02 13:49:16.046291 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:49:16.046265    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 13:49:16.046425 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:49:16.046406    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-45fzn\""
Nov 02 13:49:16.046495 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:49:16.046453    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 13:49:16.046603 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:49:16.046573    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 13:49:16.085747 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:49:16.085710    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb8ac514-d0c1-4520-a68c-d1559d32d439-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-qzrs5\" (UID: \"bb8ac514-d0c1-4520-a68c-d1559d32d439\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-qzrs5"
Nov 02 13:49:16.086200 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:49:16.085759    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mglr6\" (UniqueName: \"kubernetes.io/projected/bb8ac514-d0c1-4520-a68c-d1559d32d439-kube-api-access-mglr6\") pod \"maas-api-revoke-sub-046e1343-qzrs5\" (UID: \"bb8ac514-d0c1-4520-a68c-d1559d32d439\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-qzrs5"
Nov 02 13:49:16.086200 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:49:16.085907    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb8ac514-d0c1-4520-a68c-d1559d32d439-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-qzrs5\" (UID: \"bb8ac514-d0c1-4520-a68c-d1559d32d439\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-qzrs5"
Nov 02 13:49:16.186783 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:49:16.186751    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb8ac514-d0c1-4520-a68c-d1559d32d439-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-qzrs5\" (UID: \"bb8ac514-d0c1-4520-a68c-d1559d32d439\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-qzrs5"
Nov 02 13:49:16.186987 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:49:16.186819    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb8ac514-d0c1-4520-a68c-d1559d32d439-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-qzrs5\" (UID: \"bb8ac514-d0c1-4520-a68c-d1559d32d439\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-qzrs5"
Nov 02 13:49:16.186987 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:49:16.186856    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mglr6\" (UniqueName: \"kubernetes.io/projected/bb8ac514-d0c1-4520-a68c-d1559d32d439-kube-api-access-mglr6\") pod \"maas-api-revoke-sub-046e1343-qzrs5\" (UID: \"bb8ac514-d0c1-4520-a68c-d1559d32d439\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-qzrs5"
Nov 02 13:49:16.187484 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:49:16.187456    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb8ac514-d0c1-4520-a68c-d1559d32d439-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-qzrs5\" (UID: \"bb8ac514-d0c1-4520-a68c-d1559d32d439\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-qzrs5"
Nov 02 13:49:16.189122 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:49:16.189103    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb8ac514-d0c1-4520-a68c-d1559d32d439-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-qzrs5\" (UID: \"bb8ac514-d0c1-4520-a68c-d1559d32d439\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-qzrs5"
Nov 02 13:49:16.212307 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:49:16.212284    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mglr6\" (UniqueName: \"kubernetes.io/projected/bb8ac514-d0c1-4520-a68c-d1559d32d439-kube-api-access-mglr6\") pod \"maas-api-revoke-sub-046e1343-qzrs5\" (UID: \"bb8ac514-d0c1-4520-a68c-d1559d32d439\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-qzrs5"
Nov 02 13:49:16.330823 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:49:16.330727    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-qzrs5"
Nov 02 13:49:16.658741 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:49:16.658709    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-qzrs5"]
Nov 02 13:49:16.659608 ip-10-0-128-132 kubenswrapper[4559]: W1102 13:49:16.659580    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb8ac514_d0c1_4520_a68c_d1559d32d439.slice/crio-b398c7ca505a7b4fce88c7bd8926da698a3f82b9c5c959468ddf9c9526fe3e86 WatchSource:0}: Error finding container b398c7ca505a7b4fce88c7bd8926da698a3f82b9c5c959468ddf9c9526fe3e86: Status 404 returned error can't find the container with id b398c7ca505a7b4fce88c7bd8926da698a3f82b9c5c959468ddf9c9526fe3e86
Nov 02 13:49:16.744813 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:49:16.744781    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-qzrs5" event={"ID":"bb8ac514-d0c1-4520-a68c-d1559d32d439","Type":"ContainerStarted","Data":"b398c7ca505a7b4fce88c7bd8926da698a3f82b9c5c959468ddf9c9526fe3e86"}
Nov 02 13:49:17.750903 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:49:17.750873    4559 generic.go:356] "Generic (PLEG): container finished" podID="bb8ac514-d0c1-4520-a68c-d1559d32d439" containerID="44e3bae57c20d4380447b1ec09653bd48e2530f4139e366629870f4c6ebcee60" exitCode=0
Nov 02 13:49:17.751300 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:49:17.750960    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-qzrs5" event={"ID":"bb8ac514-d0c1-4520-a68c-d1559d32d439","Type":"ContainerDied","Data":"44e3bae57c20d4380447b1ec09653bd48e2530f4139e366629870f4c6ebcee60"}
Nov 02 13:49:18.892805 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:49:18.892779    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-qzrs5"
Nov 02 13:49:19.016432 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:49:19.016361    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bb8ac514-d0c1-4520-a68c-d1559d32d439-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb8ac514-d0c1-4520-a68c-d1559d32d439-maas-api-xks-ca\") pod \"bb8ac514-d0c1-4520-a68c-d1559d32d439\" (UID: \"bb8ac514-d0c1-4520-a68c-d1559d32d439\") "
Nov 02 13:49:19.016432 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:49:19.016405    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb8ac514-d0c1-4520-a68c-d1559d32d439-kube-api-access-mglr6\" (UniqueName: \"kubernetes.io/projected/bb8ac514-d0c1-4520-a68c-d1559d32d439-kube-api-access-mglr6\") pod \"bb8ac514-d0c1-4520-a68c-d1559d32d439\" (UID: \"bb8ac514-d0c1-4520-a68c-d1559d32d439\") "
Nov 02 13:49:19.016678 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:49:19.016515    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb8ac514-d0c1-4520-a68c-d1559d32d439-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb8ac514-d0c1-4520-a68c-d1559d32d439-maas-api-openshift-ca\") pod \"bb8ac514-d0c1-4520-a68c-d1559d32d439\" (UID: \"bb8ac514-d0c1-4520-a68c-d1559d32d439\") "
Nov 02 13:49:19.016887 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:49:19.016865    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb8ac514-d0c1-4520-a68c-d1559d32d439-maas-api-openshift-ca" pod "bb8ac514-d0c1-4520-a68c-d1559d32d439" (UID: "bb8ac514-d0c1-4520-a68c-d1559d32d439"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:49:19.019092 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:49:19.019073    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb8ac514-d0c1-4520-a68c-d1559d32d439-maas-api-xks-ca" pod "bb8ac514-d0c1-4520-a68c-d1559d32d439" (UID: "bb8ac514-d0c1-4520-a68c-d1559d32d439"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:49:19.019167 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:49:19.019144    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb8ac514-d0c1-4520-a68c-d1559d32d439-kube-api-access-mglr6" pod "bb8ac514-d0c1-4520-a68c-d1559d32d439" (UID: "bb8ac514-d0c1-4520-a68c-d1559d32d439"). InnerVolumeSpecName "kube-api-access-mglr6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:49:19.118219 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:49:19.118188    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb8ac514-d0c1-4520-a68c-d1559d32d439-maas-api-openshift-ca\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:49:19.118219 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:49:19.118216    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb8ac514-d0c1-4520-a68c-d1559d32d439-maas-api-xks-ca\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:49:19.118219 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:49:19.118227    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mglr6\" (UniqueName: \"kubernetes.io/projected/bb8ac514-d0c1-4520-a68c-d1559d32d439-kube-api-access-mglr6\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:49:19.760752 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:49:19.760722    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-qzrs5"
Nov 02 13:49:19.760752 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:49:19.760741    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-qzrs5" event={"ID":"bb8ac514-d0c1-4520-a68c-d1559d32d439","Type":"ContainerDied","Data":"b398c7ca505a7b4fce88c7bd8926da698a3f82b9c5c959468ddf9c9526fe3e86"}
Nov 02 13:49:19.760971 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:49:19.760828    4559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b398c7ca505a7b4fce88c7bd8926da698a3f82b9c5c959468ddf9c9526fe3e86"
Nov 02 13:49:26.373540 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:49:26.373502    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-mj4v4"]
Nov 02 13:49:26.374115 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:49:26.374090    4559 cpu_manager.go:414] "containerMap: removing container" podUID="bb8ac514-d0c1-4520-a68c-d1559d32d439" containerName="revoke-keys"
Nov 02 13:49:26.374163 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:49:26.374118    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb8ac514-d0c1-4520-a68c-d1559d32d439" containerName="revoke-keys"
Nov 02 13:49:26.374249 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:49:26.374236    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb8ac514-d0c1-4520-a68c-d1559d32d439" containerName="revoke-keys"
Nov 02 13:49:26.377598 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:49:26.377577    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-mj4v4"
Nov 02 13:49:26.398603 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:49:26.398568    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-mj4v4"]
Nov 02 13:49:26.408801 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:49:26.408771    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 13:49:26.408971 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:49:26.408807    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 13:49:26.408971 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:49:26.408813    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 13:49:26.408971 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:49:26.408871    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-45fzn\""
Nov 02 13:49:26.482926 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:49:26.482896    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfjtq\" (UniqueName: \"kubernetes.io/projected/7ead5881-7f3d-401a-95c9-e3466dd4cfc2-kube-api-access-kfjtq\") pod \"maas-api-revoke-sub-d3fba056-mj4v4\" (UID: \"7ead5881-7f3d-401a-95c9-e3466dd4cfc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-mj4v4"
Nov 02 13:49:26.483112 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:49:26.482933    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ead5881-7f3d-401a-95c9-e3466dd4cfc2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-mj4v4\" (UID: \"7ead5881-7f3d-401a-95c9-e3466dd4cfc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-mj4v4"
Nov 02 13:49:26.483112 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:49:26.482965    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ead5881-7f3d-401a-95c9-e3466dd4cfc2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-mj4v4\" (UID: \"7ead5881-7f3d-401a-95c9-e3466dd4cfc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-mj4v4"
Nov 02 13:49:26.584121 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:49:26.584089    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ead5881-7f3d-401a-95c9-e3466dd4cfc2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-mj4v4\" (UID: \"7ead5881-7f3d-401a-95c9-e3466dd4cfc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-mj4v4"
Nov 02 13:49:26.584278 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:49:26.584172    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfjtq\" (UniqueName: \"kubernetes.io/projected/7ead5881-7f3d-401a-95c9-e3466dd4cfc2-kube-api-access-kfjtq\") pod \"maas-api-revoke-sub-d3fba056-mj4v4\" (UID: \"7ead5881-7f3d-401a-95c9-e3466dd4cfc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-mj4v4"
Nov 02 13:49:26.584278 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:49:26.584193    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ead5881-7f3d-401a-95c9-e3466dd4cfc2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-mj4v4\" (UID: \"7ead5881-7f3d-401a-95c9-e3466dd4cfc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-mj4v4"
Nov 02 13:49:26.584834 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:49:26.584808    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ead5881-7f3d-401a-95c9-e3466dd4cfc2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-mj4v4\" (UID: \"7ead5881-7f3d-401a-95c9-e3466dd4cfc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-mj4v4"
Nov 02 13:49:26.586410 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:49:26.586389    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ead5881-7f3d-401a-95c9-e3466dd4cfc2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-mj4v4\" (UID: \"7ead5881-7f3d-401a-95c9-e3466dd4cfc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-mj4v4"
Nov 02 13:49:26.608330 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:49:26.608301    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfjtq\" (UniqueName: \"kubernetes.io/projected/7ead5881-7f3d-401a-95c9-e3466dd4cfc2-kube-api-access-kfjtq\") pod \"maas-api-revoke-sub-d3fba056-mj4v4\" (UID: \"7ead5881-7f3d-401a-95c9-e3466dd4cfc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-mj4v4"
Nov 02 13:49:26.690743 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:49:26.690720    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-mj4v4"
Nov 02 13:49:26.992611 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:49:26.992574    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-mj4v4"]
Nov 02 13:49:26.993005 ip-10-0-128-132 kubenswrapper[4559]: W1102 13:49:26.992970    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ead5881_7f3d_401a_95c9_e3466dd4cfc2.slice/crio-87722e9509c3123f8ff1728f2d786c48805971ad538724fb29cea3a2652c2017 WatchSource:0}: Error finding container 87722e9509c3123f8ff1728f2d786c48805971ad538724fb29cea3a2652c2017: Status 404 returned error can't find the container with id 87722e9509c3123f8ff1728f2d786c48805971ad538724fb29cea3a2652c2017
Nov 02 13:49:27.312519 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:49:27.312434    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"]
Nov 02 13:49:27.800304 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:49:27.800274    4559 generic.go:356] "Generic (PLEG): container finished" podID="7ead5881-7f3d-401a-95c9-e3466dd4cfc2" containerID="56ada7fdcb518c82330803dc59d83103843e0901647c5d97444e9ba51573d8f1" exitCode=0
Nov 02 13:49:27.800722 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:49:27.800360    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-mj4v4" event={"ID":"7ead5881-7f3d-401a-95c9-e3466dd4cfc2","Type":"ContainerDied","Data":"56ada7fdcb518c82330803dc59d83103843e0901647c5d97444e9ba51573d8f1"}
Nov 02 13:49:27.800722 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:49:27.800430    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-mj4v4" event={"ID":"7ead5881-7f3d-401a-95c9-e3466dd4cfc2","Type":"ContainerStarted","Data":"87722e9509c3123f8ff1728f2d786c48805971ad538724fb29cea3a2652c2017"}
Nov 02 13:49:29.457414 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:49:29.457065    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-mj4v4"
Nov 02 13:49:29.527008 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:49:29.526955    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7ead5881-7f3d-401a-95c9-e3466dd4cfc2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ead5881-7f3d-401a-95c9-e3466dd4cfc2-maas-api-xks-ca\") pod \"7ead5881-7f3d-401a-95c9-e3466dd4cfc2\" (UID: \"7ead5881-7f3d-401a-95c9-e3466dd4cfc2\") "
Nov 02 13:49:29.542696 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:49:29.541980    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ead5881-7f3d-401a-95c9-e3466dd4cfc2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ead5881-7f3d-401a-95c9-e3466dd4cfc2-maas-api-openshift-ca\") pod \"7ead5881-7f3d-401a-95c9-e3466dd4cfc2\" (UID: \"7ead5881-7f3d-401a-95c9-e3466dd4cfc2\") "
Nov 02 13:49:29.542696 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:49:29.542029    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ead5881-7f3d-401a-95c9-e3466dd4cfc2-kube-api-access-kfjtq\" (UniqueName: \"kubernetes.io/projected/7ead5881-7f3d-401a-95c9-e3466dd4cfc2-kube-api-access-kfjtq\") pod \"7ead5881-7f3d-401a-95c9-e3466dd4cfc2\" (UID: \"7ead5881-7f3d-401a-95c9-e3466dd4cfc2\") "
Nov 02 13:49:29.571665 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:49:29.551837    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ead5881-7f3d-401a-95c9-e3466dd4cfc2-maas-api-openshift-ca" pod "7ead5881-7f3d-401a-95c9-e3466dd4cfc2" (UID: "7ead5881-7f3d-401a-95c9-e3466dd4cfc2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:49:29.583659 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:49:29.576220    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ead5881-7f3d-401a-95c9-e3466dd4cfc2-maas-api-xks-ca" pod "7ead5881-7f3d-401a-95c9-e3466dd4cfc2" (UID: "7ead5881-7f3d-401a-95c9-e3466dd4cfc2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:49:29.583659 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:49:29.578811    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ead5881-7f3d-401a-95c9-e3466dd4cfc2-kube-api-access-kfjtq" pod "7ead5881-7f3d-401a-95c9-e3466dd4cfc2" (UID: "7ead5881-7f3d-401a-95c9-e3466dd4cfc2"). InnerVolumeSpecName "kube-api-access-kfjtq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:49:29.649659 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:49:29.649178    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ead5881-7f3d-401a-95c9-e3466dd4cfc2-maas-api-openshift-ca\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:49:29.649659 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:49:29.649212    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kfjtq\" (UniqueName: \"kubernetes.io/projected/7ead5881-7f3d-401a-95c9-e3466dd4cfc2-kube-api-access-kfjtq\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:49:29.649659 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:49:29.649228    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ead5881-7f3d-401a-95c9-e3466dd4cfc2-maas-api-xks-ca\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:49:29.841513 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:49:29.841420    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-mj4v4" event={"ID":"7ead5881-7f3d-401a-95c9-e3466dd4cfc2","Type":"ContainerDied","Data":"87722e9509c3123f8ff1728f2d786c48805971ad538724fb29cea3a2652c2017"}
Nov 02 13:49:29.841818 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:49:29.841797    4559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="87722e9509c3123f8ff1728f2d786c48805971ad538724fb29cea3a2652c2017"
Nov 02 13:49:29.842011 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:49:29.841995    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-mj4v4"
Nov 02 13:49:37.626560 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:49:37.626522    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"]
Nov 02 13:49:42.103616 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:49:42.103582    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"]
Nov 02 13:49:45.093742 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:49:45.093685    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqcmj" containerName="extractor"
Nov 02 13:49:47.915188 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:49:47.915155    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-7xwkv"]
Nov 02 13:49:47.915692 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:49:47.915518    4559 cpu_manager.go:414] "containerMap: removing container" podUID="7ead5881-7f3d-401a-95c9-e3466dd4cfc2" containerName="revoke-keys"
Nov 02 13:49:47.915692 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:49:47.915529    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ead5881-7f3d-401a-95c9-e3466dd4cfc2" containerName="revoke-keys"
Nov 02 13:49:47.915692 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:49:47.915589    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ead5881-7f3d-401a-95c9-e3466dd4cfc2" containerName="revoke-keys"
Nov 02 13:49:47.919988 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:49:47.919964    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-7xwkv"
Nov 02 13:49:47.937100 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:49:47.937076    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 13:49:47.937286 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:49:47.937175    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-45fzn\""
Nov 02 13:49:47.938024 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:49:47.938001    4559 scope.go:122] "RemoveContainer" containerID="9bf53315efae1d5b53ae789e3fb700fb6e8eedfb3a957d5e27d6bca356d5e978"
Nov 02 13:49:47.939402 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:49:47.939163    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-7xwkv"]
Nov 02 13:49:47.949215 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:49:47.949198    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 13:49:47.949392 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:49:47.949313    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 13:49:48.031110 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:49:48.031074    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-5r225"]
Nov 02 13:49:48.034772 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:49:48.034746    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-5r225"]
Nov 02 13:49:48.077856 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:49:48.077818    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffc78c45-ff69-42fd-ab06-8c8cf4623bf3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-7xwkv\" (UID: \"ffc78c45-ff69-42fd-ab06-8c8cf4623bf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-7xwkv"
Nov 02 13:49:48.077856 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:49:48.077855    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49w26\" (UniqueName: \"kubernetes.io/projected/ffc78c45-ff69-42fd-ab06-8c8cf4623bf3-kube-api-access-49w26\") pod \"maas-api-revoke-sub-ae594979-7xwkv\" (UID: \"ffc78c45-ff69-42fd-ab06-8c8cf4623bf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-7xwkv"
Nov 02 13:49:48.078074 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:49:48.077881    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffc78c45-ff69-42fd-ab06-8c8cf4623bf3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-7xwkv\" (UID: \"ffc78c45-ff69-42fd-ab06-8c8cf4623bf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-7xwkv"
Nov 02 13:49:48.179114 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:49:48.179072    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffc78c45-ff69-42fd-ab06-8c8cf4623bf3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-7xwkv\" (UID: \"ffc78c45-ff69-42fd-ab06-8c8cf4623bf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-7xwkv"
Nov 02 13:49:48.179114 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:49:48.179112    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49w26\" (UniqueName: \"kubernetes.io/projected/ffc78c45-ff69-42fd-ab06-8c8cf4623bf3-kube-api-access-49w26\") pod \"maas-api-revoke-sub-ae594979-7xwkv\" (UID: \"ffc78c45-ff69-42fd-ab06-8c8cf4623bf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-7xwkv"
Nov 02 13:49:48.179373 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:49:48.179139    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffc78c45-ff69-42fd-ab06-8c8cf4623bf3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-7xwkv\" (UID: \"ffc78c45-ff69-42fd-ab06-8c8cf4623bf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-7xwkv"
Nov 02 13:49:48.179795 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:49:48.179774    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffc78c45-ff69-42fd-ab06-8c8cf4623bf3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-7xwkv\" (UID: \"ffc78c45-ff69-42fd-ab06-8c8cf4623bf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-7xwkv"
Nov 02 13:49:48.181457 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:49:48.181432    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffc78c45-ff69-42fd-ab06-8c8cf4623bf3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-7xwkv\" (UID: \"ffc78c45-ff69-42fd-ab06-8c8cf4623bf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-7xwkv"
Nov 02 13:49:48.203219 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:49:48.203193    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-49w26\" (UniqueName: \"kubernetes.io/projected/ffc78c45-ff69-42fd-ab06-8c8cf4623bf3-kube-api-access-49w26\") pod \"maas-api-revoke-sub-ae594979-7xwkv\" (UID: \"ffc78c45-ff69-42fd-ab06-8c8cf4623bf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-7xwkv"
Nov 02 13:49:48.233194 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:49:48.233161    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-7xwkv"
Nov 02 13:49:48.536041 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:49:48.536005    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-7xwkv"]
Nov 02 13:49:48.536974 ip-10-0-128-132 kubenswrapper[4559]: W1102 13:49:48.536939    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffc78c45_ff69_42fd_ab06_8c8cf4623bf3.slice/crio-c5538e616ad0a76eca61f8a383ad67c47bbffeb9a492ec81da00deba4c55e493 WatchSource:0}: Error finding container c5538e616ad0a76eca61f8a383ad67c47bbffeb9a492ec81da00deba4c55e493: Status 404 returned error can't find the container with id c5538e616ad0a76eca61f8a383ad67c47bbffeb9a492ec81da00deba4c55e493
Nov 02 13:49:48.964232 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:49:48.964190    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-7xwkv" event={"ID":"ffc78c45-ff69-42fd-ab06-8c8cf4623bf3","Type":"ContainerStarted","Data":"fd9c2b45dce73cad68b17ac3ab5c789d5a400e606ec1ae1b5490e796e52b32dd"}
Nov 02 13:49:48.964753 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:49:48.964242    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-7xwkv" event={"ID":"ffc78c45-ff69-42fd-ab06-8c8cf4623bf3","Type":"ContainerStarted","Data":"c5538e616ad0a76eca61f8a383ad67c47bbffeb9a492ec81da00deba4c55e493"}
Nov 02 13:49:49.009582 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:49:49.009525    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-7xwkv" podStartSLOduration=2.009508978 podStartE2EDuration="2.009508978s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:49:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:49:49.008152066 +0000 UTC m=+4985.263551171" watchObservedRunningTime="2026-11-02 13:49:49.009508978 +0000 UTC m=+4985.264908072"
Nov 02 13:49:49.890914 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:49:49.890880    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3357a4d-f86a-40f7-9117-a87fd501e540" path="/var/lib/kubelet/pods/d3357a4d-f86a-40f7-9117-a87fd501e540/volumes"
Nov 02 13:49:49.970371 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:49:49.970338    4559 generic.go:356] "Generic (PLEG): container finished" podID="ffc78c45-ff69-42fd-ab06-8c8cf4623bf3" containerID="fd9c2b45dce73cad68b17ac3ab5c789d5a400e606ec1ae1b5490e796e52b32dd" exitCode=0
Nov 02 13:49:49.970800 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:49:49.970422    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-7xwkv" event={"ID":"ffc78c45-ff69-42fd-ab06-8c8cf4623bf3","Type":"ContainerDied","Data":"fd9c2b45dce73cad68b17ac3ab5c789d5a400e606ec1ae1b5490e796e52b32dd"}
Nov 02 13:49:51.101749 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:49:51.101728    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-7xwkv"
Nov 02 13:49:51.211002 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:49:51.210969    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ffc78c45-ff69-42fd-ab06-8c8cf4623bf3-kube-api-access-49w26\" (UniqueName: \"kubernetes.io/projected/ffc78c45-ff69-42fd-ab06-8c8cf4623bf3-kube-api-access-49w26\") pod \"ffc78c45-ff69-42fd-ab06-8c8cf4623bf3\" (UID: \"ffc78c45-ff69-42fd-ab06-8c8cf4623bf3\") "
Nov 02 13:49:51.211211 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:49:51.211049    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ffc78c45-ff69-42fd-ab06-8c8cf4623bf3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffc78c45-ff69-42fd-ab06-8c8cf4623bf3-maas-api-openshift-ca\") pod \"ffc78c45-ff69-42fd-ab06-8c8cf4623bf3\" (UID: \"ffc78c45-ff69-42fd-ab06-8c8cf4623bf3\") "
Nov 02 13:49:51.211211 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:49:51.211094    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ffc78c45-ff69-42fd-ab06-8c8cf4623bf3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffc78c45-ff69-42fd-ab06-8c8cf4623bf3-maas-api-xks-ca\") pod \"ffc78c45-ff69-42fd-ab06-8c8cf4623bf3\" (UID: \"ffc78c45-ff69-42fd-ab06-8c8cf4623bf3\") "
Nov 02 13:49:51.211456 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:49:51.211432    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffc78c45-ff69-42fd-ab06-8c8cf4623bf3-maas-api-openshift-ca" pod "ffc78c45-ff69-42fd-ab06-8c8cf4623bf3" (UID: "ffc78c45-ff69-42fd-ab06-8c8cf4623bf3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:49:51.213067 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:49:51.213047    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffc78c45-ff69-42fd-ab06-8c8cf4623bf3-kube-api-access-49w26" pod "ffc78c45-ff69-42fd-ab06-8c8cf4623bf3" (UID: "ffc78c45-ff69-42fd-ab06-8c8cf4623bf3"). InnerVolumeSpecName "kube-api-access-49w26". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:49:51.213167 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:49:51.213150    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ffc78c45-ff69-42fd-ab06-8c8cf4623bf3-maas-api-xks-ca" pod "ffc78c45-ff69-42fd-ab06-8c8cf4623bf3" (UID: "ffc78c45-ff69-42fd-ab06-8c8cf4623bf3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:49:51.311820 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:49:51.311738    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-49w26\" (UniqueName: \"kubernetes.io/projected/ffc78c45-ff69-42fd-ab06-8c8cf4623bf3-kube-api-access-49w26\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:49:51.311820 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:49:51.311780    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffc78c45-ff69-42fd-ab06-8c8cf4623bf3-maas-api-openshift-ca\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:49:51.311820 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:49:51.311791    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffc78c45-ff69-42fd-ab06-8c8cf4623bf3-maas-api-xks-ca\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:49:51.981050 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:49:51.981025    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-7xwkv"
Nov 02 13:49:51.981203 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:49:51.981022    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-7xwkv" event={"ID":"ffc78c45-ff69-42fd-ab06-8c8cf4623bf3","Type":"ContainerDied","Data":"c5538e616ad0a76eca61f8a383ad67c47bbffeb9a492ec81da00deba4c55e493"}
Nov 02 13:49:51.981203 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:49:51.981145    4559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c5538e616ad0a76eca61f8a383ad67c47bbffeb9a492ec81da00deba4c55e493"
Nov 02 13:49:59.218911 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:49:59.218835    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"]
Nov 02 13:50:03.271746 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:50:03.271694    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-9nglq"]
Nov 02 13:50:03.292677 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:50:03.272933    4559 cpu_manager.go:414] "containerMap: removing container" podUID="ffc78c45-ff69-42fd-ab06-8c8cf4623bf3" containerName="revoke-keys"
Nov 02 13:50:03.292677 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:50:03.272963    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffc78c45-ff69-42fd-ab06-8c8cf4623bf3" containerName="revoke-keys"
Nov 02 13:50:03.292677 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:50:03.273067    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffc78c45-ff69-42fd-ab06-8c8cf4623bf3" containerName="revoke-keys"
Nov 02 13:50:03.324295 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:50:03.313867    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-9nglq"]
Nov 02 13:50:03.324295 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:50:03.314004    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-9nglq"
Nov 02 13:50:03.355778 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:50:03.348394    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-45fzn\""
Nov 02 13:50:03.453082 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:50:03.356758    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 13:50:03.453082 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:50:03.356841    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 13:50:03.453082 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:50:03.356895    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 13:50:03.453082 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:50:03.440269    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c3598e7-1cb1-42b0-a291-656fc5e14505-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-9nglq\" (UID: \"4c3598e7-1cb1-42b0-a291-656fc5e14505\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-9nglq"
Nov 02 13:50:03.453082 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:50:03.452922    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c3598e7-1cb1-42b0-a291-656fc5e14505-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-9nglq\" (UID: \"4c3598e7-1cb1-42b0-a291-656fc5e14505\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-9nglq"
Nov 02 13:50:03.453082 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:50:03.452977    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfjp4\" (UniqueName: \"kubernetes.io/projected/4c3598e7-1cb1-42b0-a291-656fc5e14505-kube-api-access-sfjp4\") pod \"maas-api-revoke-sub-a4aef7d7-9nglq\" (UID: \"4c3598e7-1cb1-42b0-a291-656fc5e14505\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-9nglq"
Nov 02 13:50:03.559721 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:50:03.557094    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c3598e7-1cb1-42b0-a291-656fc5e14505-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-9nglq\" (UID: \"4c3598e7-1cb1-42b0-a291-656fc5e14505\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-9nglq"
Nov 02 13:50:03.559721 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:50:03.557181    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c3598e7-1cb1-42b0-a291-656fc5e14505-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-9nglq\" (UID: \"4c3598e7-1cb1-42b0-a291-656fc5e14505\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-9nglq"
Nov 02 13:50:03.559721 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:50:03.557225    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfjp4\" (UniqueName: \"kubernetes.io/projected/4c3598e7-1cb1-42b0-a291-656fc5e14505-kube-api-access-sfjp4\") pod \"maas-api-revoke-sub-a4aef7d7-9nglq\" (UID: \"4c3598e7-1cb1-42b0-a291-656fc5e14505\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-9nglq"
Nov 02 13:50:03.560762 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:50:03.560733    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c3598e7-1cb1-42b0-a291-656fc5e14505-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-9nglq\" (UID: \"4c3598e7-1cb1-42b0-a291-656fc5e14505\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-9nglq"
Nov 02 13:50:03.574340 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:50:03.574316    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c3598e7-1cb1-42b0-a291-656fc5e14505-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-9nglq\" (UID: \"4c3598e7-1cb1-42b0-a291-656fc5e14505\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-9nglq"
Nov 02 13:50:03.618353 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:50:03.618295    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfjp4\" (UniqueName: \"kubernetes.io/projected/4c3598e7-1cb1-42b0-a291-656fc5e14505-kube-api-access-sfjp4\") pod \"maas-api-revoke-sub-a4aef7d7-9nglq\" (UID: \"4c3598e7-1cb1-42b0-a291-656fc5e14505\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-9nglq"
Nov 02 13:50:03.690046 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:50:03.689767    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-9nglq"
Nov 02 13:50:04.523664 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:50:04.512396    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-9nglq"]
Nov 02 13:50:04.669681 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:50:04.651207    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"]
Nov 02 13:50:05.153145 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:50:05.153067    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-9nglq" event={"ID":"4c3598e7-1cb1-42b0-a291-656fc5e14505","Type":"ContainerStarted","Data":"32dc46760cd32df531b5f9b8fee582eab3b32522e6c2880e9723499d9e124f6d"}
Nov 02 13:50:05.153145 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:50:05.153125    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-9nglq" event={"ID":"4c3598e7-1cb1-42b0-a291-656fc5e14505","Type":"ContainerStarted","Data":"a1d256732b126848f7ce6f70da17bcf9784b49ca792c5b268ffdf1177b4e0790"}
Nov 02 13:50:05.195041 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:50:05.194988    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-9nglq" podStartSLOduration=2.194973773 podStartE2EDuration="2.194973773s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:50:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:50:05.194710078 +0000 UTC m=+5001.450109173" watchObservedRunningTime="2026-11-02 13:50:05.194973773 +0000 UTC m=+5001.450372870"
Nov 02 13:50:06.159475 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:50:06.159438    4559 generic.go:356] "Generic (PLEG): container finished" podID="4c3598e7-1cb1-42b0-a291-656fc5e14505" containerID="32dc46760cd32df531b5f9b8fee582eab3b32522e6c2880e9723499d9e124f6d" exitCode=0
Nov 02 13:50:06.159889 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:50:06.159476    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-9nglq" event={"ID":"4c3598e7-1cb1-42b0-a291-656fc5e14505","Type":"ContainerDied","Data":"32dc46760cd32df531b5f9b8fee582eab3b32522e6c2880e9723499d9e124f6d"}
Nov 02 13:50:07.303823 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:50:07.303797    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-9nglq"
Nov 02 13:50:07.482296 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:50:07.482262    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c3598e7-1cb1-42b0-a291-656fc5e14505-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c3598e7-1cb1-42b0-a291-656fc5e14505-maas-api-openshift-ca\") pod \"4c3598e7-1cb1-42b0-a291-656fc5e14505\" (UID: \"4c3598e7-1cb1-42b0-a291-656fc5e14505\") "
Nov 02 13:50:07.482493 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:50:07.482359    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c3598e7-1cb1-42b0-a291-656fc5e14505-kube-api-access-sfjp4\" (UniqueName: \"kubernetes.io/projected/4c3598e7-1cb1-42b0-a291-656fc5e14505-kube-api-access-sfjp4\") pod \"4c3598e7-1cb1-42b0-a291-656fc5e14505\" (UID: \"4c3598e7-1cb1-42b0-a291-656fc5e14505\") "
Nov 02 13:50:07.482493 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:50:07.482480    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4c3598e7-1cb1-42b0-a291-656fc5e14505-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c3598e7-1cb1-42b0-a291-656fc5e14505-maas-api-xks-ca\") pod \"4c3598e7-1cb1-42b0-a291-656fc5e14505\" (UID: \"4c3598e7-1cb1-42b0-a291-656fc5e14505\") "
Nov 02 13:50:07.482668 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:50:07.482617    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c3598e7-1cb1-42b0-a291-656fc5e14505-maas-api-openshift-ca" pod "4c3598e7-1cb1-42b0-a291-656fc5e14505" (UID: "4c3598e7-1cb1-42b0-a291-656fc5e14505"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:50:07.482793 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:50:07.482775    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c3598e7-1cb1-42b0-a291-656fc5e14505-maas-api-openshift-ca\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:50:07.484521 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:50:07.484502    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c3598e7-1cb1-42b0-a291-656fc5e14505-maas-api-xks-ca" pod "4c3598e7-1cb1-42b0-a291-656fc5e14505" (UID: "4c3598e7-1cb1-42b0-a291-656fc5e14505"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:50:07.484630 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:50:07.484610    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c3598e7-1cb1-42b0-a291-656fc5e14505-kube-api-access-sfjp4" pod "4c3598e7-1cb1-42b0-a291-656fc5e14505" (UID: "4c3598e7-1cb1-42b0-a291-656fc5e14505"). InnerVolumeSpecName "kube-api-access-sfjp4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:50:07.583628 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:50:07.583601    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sfjp4\" (UniqueName: \"kubernetes.io/projected/4c3598e7-1cb1-42b0-a291-656fc5e14505-kube-api-access-sfjp4\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:50:07.583628 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:50:07.583630    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c3598e7-1cb1-42b0-a291-656fc5e14505-maas-api-xks-ca\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:50:08.169317 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:50:08.169289    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-9nglq"
Nov 02 13:50:08.169482 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:50:08.169284    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-9nglq" event={"ID":"4c3598e7-1cb1-42b0-a291-656fc5e14505","Type":"ContainerDied","Data":"a1d256732b126848f7ce6f70da17bcf9784b49ca792c5b268ffdf1177b4e0790"}
Nov 02 13:50:08.169482 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:50:08.169406    4559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a1d256732b126848f7ce6f70da17bcf9784b49ca792c5b268ffdf1177b4e0790"
Nov 02 13:50:08.716353 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:50:08.716317    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"]
Nov 02 13:50:15.086136 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:50:15.086094    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqcmj" containerName="extractor"
Nov 02 13:50:18.721832 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:50:18.721800    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"]
Nov 02 13:50:20.179381 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:50:20.176251    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-qnvts"]
Nov 02 13:50:20.179381 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:50:20.176862    4559 cpu_manager.go:414] "containerMap: removing container" podUID="4c3598e7-1cb1-42b0-a291-656fc5e14505" containerName="revoke-keys"
Nov 02 13:50:20.179381 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:50:20.176883    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c3598e7-1cb1-42b0-a291-656fc5e14505" containerName="revoke-keys"
Nov 02 13:50:20.179381 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:50:20.177030    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c3598e7-1cb1-42b0-a291-656fc5e14505" containerName="revoke-keys"
Nov 02 13:50:20.182217 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:50:20.182193    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-qnvts"
Nov 02 13:50:20.235678 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:50:20.213381    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5ff9c16-f825-418d-ba32-ce9faaa39351-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-qnvts\" (UID: \"c5ff9c16-f825-418d-ba32-ce9faaa39351\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-qnvts"
Nov 02 13:50:20.235678 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:50:20.213427    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5ff9c16-f825-418d-ba32-ce9faaa39351-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-qnvts\" (UID: \"c5ff9c16-f825-418d-ba32-ce9faaa39351\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-qnvts"
Nov 02 13:50:20.235678 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:50:20.213519    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lds9k\" (UniqueName: \"kubernetes.io/projected/c5ff9c16-f825-418d-ba32-ce9faaa39351-kube-api-access-lds9k\") pod \"maas-api-revoke-sub-79805ca9-qnvts\" (UID: \"c5ff9c16-f825-418d-ba32-ce9faaa39351\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-qnvts"
Nov 02 13:50:20.235678 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:50:20.234930    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 13:50:20.235678 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:50:20.235073    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-45fzn\""
Nov 02 13:50:20.235678 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:50:20.235149    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 13:50:20.235678 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:50:20.235215    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 13:50:20.237153 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:50:20.237124    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-qnvts"]
Nov 02 13:50:20.325663 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:50:20.321083    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lds9k\" (UniqueName: \"kubernetes.io/projected/c5ff9c16-f825-418d-ba32-ce9faaa39351-kube-api-access-lds9k\") pod \"maas-api-revoke-sub-79805ca9-qnvts\" (UID: \"c5ff9c16-f825-418d-ba32-ce9faaa39351\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-qnvts"
Nov 02 13:50:20.329806 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:50:20.329771    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5ff9c16-f825-418d-ba32-ce9faaa39351-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-qnvts\" (UID: \"c5ff9c16-f825-418d-ba32-ce9faaa39351\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-qnvts"
Nov 02 13:50:20.358725 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:50:20.358699    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5ff9c16-f825-418d-ba32-ce9faaa39351-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-qnvts\" (UID: \"c5ff9c16-f825-418d-ba32-ce9faaa39351\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-qnvts"
Nov 02 13:50:20.360746 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:50:20.335380    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5ff9c16-f825-418d-ba32-ce9faaa39351-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-qnvts\" (UID: \"c5ff9c16-f825-418d-ba32-ce9faaa39351\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-qnvts"
Nov 02 13:50:20.403659 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:50:20.400067    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5ff9c16-f825-418d-ba32-ce9faaa39351-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-qnvts\" (UID: \"c5ff9c16-f825-418d-ba32-ce9faaa39351\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-qnvts"
Nov 02 13:50:20.463937 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:50:20.463860    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lds9k\" (UniqueName: \"kubernetes.io/projected/c5ff9c16-f825-418d-ba32-ce9faaa39351-kube-api-access-lds9k\") pod \"maas-api-revoke-sub-79805ca9-qnvts\" (UID: \"c5ff9c16-f825-418d-ba32-ce9faaa39351\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-qnvts"
Nov 02 13:50:20.548797 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:50:20.548765    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-qnvts"
Nov 02 13:50:21.109026 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:50:21.102407    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-qnvts"]
Nov 02 13:50:21.402045 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:50:21.401947    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-qnvts" event={"ID":"c5ff9c16-f825-418d-ba32-ce9faaa39351","Type":"ContainerStarted","Data":"c637881d6c826eff5adc7f19ed83ec11ee33cb7d2306d585d93de04f721592bb"}
Nov 02 13:50:22.408266 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:50:22.408234    4559 generic.go:356] "Generic (PLEG): container finished" podID="c5ff9c16-f825-418d-ba32-ce9faaa39351" containerID="e560ea6bf1cc319621214e3a56c604fb17847c8910919741840d1d5aa2e2eddc" exitCode=0
Nov 02 13:50:22.408608 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:50:22.408306    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-qnvts" event={"ID":"c5ff9c16-f825-418d-ba32-ce9faaa39351","Type":"ContainerDied","Data":"e560ea6bf1cc319621214e3a56c604fb17847c8910919741840d1d5aa2e2eddc"}
Nov 02 13:50:23.545090 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:50:23.545066    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-qnvts"
Nov 02 13:50:23.549131 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:50:23.549111    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5ff9c16-f825-418d-ba32-ce9faaa39351-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5ff9c16-f825-418d-ba32-ce9faaa39351-maas-api-openshift-ca\") pod \"c5ff9c16-f825-418d-ba32-ce9faaa39351\" (UID: \"c5ff9c16-f825-418d-ba32-ce9faaa39351\") "
Nov 02 13:50:23.549211 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:50:23.549155    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5ff9c16-f825-418d-ba32-ce9faaa39351-kube-api-access-lds9k\" (UniqueName: \"kubernetes.io/projected/c5ff9c16-f825-418d-ba32-ce9faaa39351-kube-api-access-lds9k\") pod \"c5ff9c16-f825-418d-ba32-ce9faaa39351\" (UID: \"c5ff9c16-f825-418d-ba32-ce9faaa39351\") "
Nov 02 13:50:23.549211 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:50:23.549179    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c5ff9c16-f825-418d-ba32-ce9faaa39351-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5ff9c16-f825-418d-ba32-ce9faaa39351-maas-api-xks-ca\") pod \"c5ff9c16-f825-418d-ba32-ce9faaa39351\" (UID: \"c5ff9c16-f825-418d-ba32-ce9faaa39351\") "
Nov 02 13:50:23.549552 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:50:23.549525    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5ff9c16-f825-418d-ba32-ce9faaa39351-maas-api-openshift-ca" pod "c5ff9c16-f825-418d-ba32-ce9faaa39351" (UID: "c5ff9c16-f825-418d-ba32-ce9faaa39351"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:50:23.551123 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:50:23.551104    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5ff9c16-f825-418d-ba32-ce9faaa39351-maas-api-xks-ca" pod "c5ff9c16-f825-418d-ba32-ce9faaa39351" (UID: "c5ff9c16-f825-418d-ba32-ce9faaa39351"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:50:23.551212 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:50:23.551137    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5ff9c16-f825-418d-ba32-ce9faaa39351-kube-api-access-lds9k" pod "c5ff9c16-f825-418d-ba32-ce9faaa39351" (UID: "c5ff9c16-f825-418d-ba32-ce9faaa39351"). InnerVolumeSpecName "kube-api-access-lds9k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:50:23.650288 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:50:23.650257    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5ff9c16-f825-418d-ba32-ce9faaa39351-maas-api-openshift-ca\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:50:23.650288 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:50:23.650286    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lds9k\" (UniqueName: \"kubernetes.io/projected/c5ff9c16-f825-418d-ba32-ce9faaa39351-kube-api-access-lds9k\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:50:23.650470 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:50:23.650296    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5ff9c16-f825-418d-ba32-ce9faaa39351-maas-api-xks-ca\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:50:23.876295 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:50:23.876263    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"]
Nov 02 13:50:24.418378 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:50:24.418348    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-qnvts"
Nov 02 13:50:24.418558 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:50:24.418389    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-qnvts" event={"ID":"c5ff9c16-f825-418d-ba32-ce9faaa39351","Type":"ContainerDied","Data":"c637881d6c826eff5adc7f19ed83ec11ee33cb7d2306d585d93de04f721592bb"}
Nov 02 13:50:24.418558 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:50:24.418437    4559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c637881d6c826eff5adc7f19ed83ec11ee33cb7d2306d585d93de04f721592bb"
Nov 02 13:50:27.031032 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:50:27.030996    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4de99d0f-p5wjr"]
Nov 02 13:50:27.044080 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:50:27.044041    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4de99d0f-p5wjr"]
Nov 02 13:50:27.891004 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:50:27.890959    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a993a691-6869-4702-9873-1b4a1616ae92" path="/var/lib/kubelet/pods/a993a691-6869-4702-9873-1b4a1616ae92/volumes"
Nov 02 13:50:32.420676 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:50:32.420625    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"]
Nov 02 13:50:33.875980 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:50:33.875945    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-qz8b5"]
Nov 02 13:50:33.877118 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:50:33.877089    4559 cpu_manager.go:414] "containerMap: removing container" podUID="c5ff9c16-f825-418d-ba32-ce9faaa39351" containerName="revoke-keys"
Nov 02 13:50:33.877281 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:50:33.877265    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5ff9c16-f825-418d-ba32-ce9faaa39351" containerName="revoke-keys"
Nov 02 13:50:33.877510 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:50:33.877495    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5ff9c16-f825-418d-ba32-ce9faaa39351" containerName="revoke-keys"
Nov 02 13:50:33.881216 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:50:33.881197    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-qz8b5"
Nov 02 13:50:33.930968 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:50:33.930449    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-45fzn\""
Nov 02 13:50:33.930968 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:50:33.930545    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 13:50:33.930968 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:50:33.930585    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 13:50:34.040663 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:50:33.981347    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 13:50:34.271665 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:50:34.259205    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-qz8b5"]
Nov 02 13:50:34.281542 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:50:34.281503    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wp59b\" (UniqueName: \"kubernetes.io/projected/804809f5-f9da-436a-9800-e9866dd8d586-kube-api-access-wp59b\") pod \"maas-api-revoke-sub-0b7e8edb-qz8b5\" (UID: \"804809f5-f9da-436a-9800-e9866dd8d586\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-qz8b5"
Nov 02 13:50:34.282915 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:50:34.282888    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/804809f5-f9da-436a-9800-e9866dd8d586-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-qz8b5\" (UID: \"804809f5-f9da-436a-9800-e9866dd8d586\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-qz8b5"
Nov 02 13:50:34.283183 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:50:34.283163    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/804809f5-f9da-436a-9800-e9866dd8d586-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-qz8b5\" (UID: \"804809f5-f9da-436a-9800-e9866dd8d586\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-qz8b5"
Nov 02 13:50:34.401264 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:50:34.401231    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wp59b\" (UniqueName: \"kubernetes.io/projected/804809f5-f9da-436a-9800-e9866dd8d586-kube-api-access-wp59b\") pod \"maas-api-revoke-sub-0b7e8edb-qz8b5\" (UID: \"804809f5-f9da-436a-9800-e9866dd8d586\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-qz8b5"
Nov 02 13:50:34.401665 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:50:34.401613    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/804809f5-f9da-436a-9800-e9866dd8d586-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-qz8b5\" (UID: \"804809f5-f9da-436a-9800-e9866dd8d586\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-qz8b5"
Nov 02 13:50:34.401903 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:50:34.401865    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/804809f5-f9da-436a-9800-e9866dd8d586-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-qz8b5\" (UID: \"804809f5-f9da-436a-9800-e9866dd8d586\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-qz8b5"
Nov 02 13:50:34.403582 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:50:34.403561    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/804809f5-f9da-436a-9800-e9866dd8d586-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-qz8b5\" (UID: \"804809f5-f9da-436a-9800-e9866dd8d586\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-qz8b5"
Nov 02 13:50:34.412186 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:50:34.412165    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/804809f5-f9da-436a-9800-e9866dd8d586-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-qz8b5\" (UID: \"804809f5-f9da-436a-9800-e9866dd8d586\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-qz8b5"
Nov 02 13:50:34.458877 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:50:34.458848    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wp59b\" (UniqueName: \"kubernetes.io/projected/804809f5-f9da-436a-9800-e9866dd8d586-kube-api-access-wp59b\") pod \"maas-api-revoke-sub-0b7e8edb-qz8b5\" (UID: \"804809f5-f9da-436a-9800-e9866dd8d586\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-qz8b5"
Nov 02 13:50:34.600133 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:50:34.599779    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-qz8b5"
Nov 02 13:50:35.363664 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:50:35.363520    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-qz8b5"]
Nov 02 13:50:35.396330 ip-10-0-128-132 kubenswrapper[4559]: W1102 13:50:35.396297    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod804809f5_f9da_436a_9800_e9866dd8d586.slice/crio-609bdd93e9c3fc5b04d45c3643b7673a530b2b91b9eb1f5bd93ed8b5b6694618 WatchSource:0}: Error finding container 609bdd93e9c3fc5b04d45c3643b7673a530b2b91b9eb1f5bd93ed8b5b6694618: Status 404 returned error can't find the container with id 609bdd93e9c3fc5b04d45c3643b7673a530b2b91b9eb1f5bd93ed8b5b6694618
Nov 02 13:50:35.520663 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:50:35.516608    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-qz8b5" event={"ID":"804809f5-f9da-436a-9800-e9866dd8d586","Type":"ContainerStarted","Data":"609bdd93e9c3fc5b04d45c3643b7673a530b2b91b9eb1f5bd93ed8b5b6694618"}
Nov 02 13:50:36.522880 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:50:36.522845    4559 generic.go:356] "Generic (PLEG): container finished" podID="804809f5-f9da-436a-9800-e9866dd8d586" containerID="ccab5b95f29bdc19dcf86ad102d16e5a8ed709afe347d32fe162214310396021" exitCode=0
Nov 02 13:50:36.523325 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:50:36.522892    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-qz8b5" event={"ID":"804809f5-f9da-436a-9800-e9866dd8d586","Type":"ContainerDied","Data":"ccab5b95f29bdc19dcf86ad102d16e5a8ed709afe347d32fe162214310396021"}
Nov 02 13:50:37.615970 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:50:37.615935    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"]
Nov 02 13:50:37.655345 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:50:37.655318    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-qz8b5"
Nov 02 13:50:37.702284 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:50:37.702255    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/804809f5-f9da-436a-9800-e9866dd8d586-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/804809f5-f9da-436a-9800-e9866dd8d586-maas-api-openshift-ca\") pod \"804809f5-f9da-436a-9800-e9866dd8d586\" (UID: \"804809f5-f9da-436a-9800-e9866dd8d586\") "
Nov 02 13:50:37.702439 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:50:37.702303    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/804809f5-f9da-436a-9800-e9866dd8d586-kube-api-access-wp59b\" (UniqueName: \"kubernetes.io/projected/804809f5-f9da-436a-9800-e9866dd8d586-kube-api-access-wp59b\") pod \"804809f5-f9da-436a-9800-e9866dd8d586\" (UID: \"804809f5-f9da-436a-9800-e9866dd8d586\") "
Nov 02 13:50:37.702439 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:50:37.702326    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/804809f5-f9da-436a-9800-e9866dd8d586-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/804809f5-f9da-436a-9800-e9866dd8d586-maas-api-xks-ca\") pod \"804809f5-f9da-436a-9800-e9866dd8d586\" (UID: \"804809f5-f9da-436a-9800-e9866dd8d586\") "
Nov 02 13:50:37.702695 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:50:37.702667    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/804809f5-f9da-436a-9800-e9866dd8d586-maas-api-openshift-ca" pod "804809f5-f9da-436a-9800-e9866dd8d586" (UID: "804809f5-f9da-436a-9800-e9866dd8d586"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:50:37.704500 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:50:37.704476    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/804809f5-f9da-436a-9800-e9866dd8d586-kube-api-access-wp59b" pod "804809f5-f9da-436a-9800-e9866dd8d586" (UID: "804809f5-f9da-436a-9800-e9866dd8d586"). InnerVolumeSpecName "kube-api-access-wp59b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:50:37.704715 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:50:37.704697    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/804809f5-f9da-436a-9800-e9866dd8d586-maas-api-xks-ca" pod "804809f5-f9da-436a-9800-e9866dd8d586" (UID: "804809f5-f9da-436a-9800-e9866dd8d586"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:50:37.803145 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:50:37.803082    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/804809f5-f9da-436a-9800-e9866dd8d586-maas-api-openshift-ca\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:50:37.803145 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:50:37.803107    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wp59b\" (UniqueName: \"kubernetes.io/projected/804809f5-f9da-436a-9800-e9866dd8d586-kube-api-access-wp59b\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:50:37.803145 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:50:37.803117    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/804809f5-f9da-436a-9800-e9866dd8d586-maas-api-xks-ca\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:50:38.533961 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:50:38.533931    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-qz8b5"
Nov 02 13:50:38.534147 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:50:38.533960    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-qz8b5" event={"ID":"804809f5-f9da-436a-9800-e9866dd8d586","Type":"ContainerDied","Data":"609bdd93e9c3fc5b04d45c3643b7673a530b2b91b9eb1f5bd93ed8b5b6694618"}
Nov 02 13:50:38.534147 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:50:38.534002    4559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="609bdd93e9c3fc5b04d45c3643b7673a530b2b91b9eb1f5bd93ed8b5b6694618"
Nov 02 13:50:44.608827 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:50:44.608793    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-n8b8l"]
Nov 02 13:50:44.609208 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:50:44.609170    4559 cpu_manager.go:414] "containerMap: removing container" podUID="804809f5-f9da-436a-9800-e9866dd8d586" containerName="revoke-keys"
Nov 02 13:50:44.609208 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:50:44.609182    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="804809f5-f9da-436a-9800-e9866dd8d586" containerName="revoke-keys"
Nov 02 13:50:44.609301 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:50:44.609268    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="804809f5-f9da-436a-9800-e9866dd8d586" containerName="revoke-keys"
Nov 02 13:50:44.612764 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:50:44.612744    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-n8b8l"
Nov 02 13:50:44.632075 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:50:44.632047    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-45fzn\""
Nov 02 13:50:44.632183 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:50:44.632111    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 13:50:44.645425 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:50:44.645393    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 13:50:44.645610 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:50:44.645396    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 13:50:44.648182 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:50:44.648158    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-n8b8l"]
Nov 02 13:50:44.768507 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:50:44.768471    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9db51790-83e0-44a2-ac63-6b87ed8eeb08-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-n8b8l\" (UID: \"9db51790-83e0-44a2-ac63-6b87ed8eeb08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-n8b8l"
Nov 02 13:50:44.768690 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:50:44.768584    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9db51790-83e0-44a2-ac63-6b87ed8eeb08-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-n8b8l\" (UID: \"9db51790-83e0-44a2-ac63-6b87ed8eeb08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-n8b8l"
Nov 02 13:50:44.768690 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:50:44.768665    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fcsh\" (UniqueName: \"kubernetes.io/projected/9db51790-83e0-44a2-ac63-6b87ed8eeb08-kube-api-access-5fcsh\") pod \"maas-api-revoke-sub-3d4156f1-n8b8l\" (UID: \"9db51790-83e0-44a2-ac63-6b87ed8eeb08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-n8b8l"
Nov 02 13:50:44.869689 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:50:44.869600    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9db51790-83e0-44a2-ac63-6b87ed8eeb08-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-n8b8l\" (UID: \"9db51790-83e0-44a2-ac63-6b87ed8eeb08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-n8b8l"
Nov 02 13:50:44.869829 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:50:44.869695    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9db51790-83e0-44a2-ac63-6b87ed8eeb08-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-n8b8l\" (UID: \"9db51790-83e0-44a2-ac63-6b87ed8eeb08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-n8b8l"
Nov 02 13:50:44.869829 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:50:44.869735    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fcsh\" (UniqueName: \"kubernetes.io/projected/9db51790-83e0-44a2-ac63-6b87ed8eeb08-kube-api-access-5fcsh\") pod \"maas-api-revoke-sub-3d4156f1-n8b8l\" (UID: \"9db51790-83e0-44a2-ac63-6b87ed8eeb08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-n8b8l"
Nov 02 13:50:44.870346 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:50:44.870325    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9db51790-83e0-44a2-ac63-6b87ed8eeb08-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-n8b8l\" (UID: \"9db51790-83e0-44a2-ac63-6b87ed8eeb08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-n8b8l"
Nov 02 13:50:44.871976 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:50:44.871953    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9db51790-83e0-44a2-ac63-6b87ed8eeb08-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-n8b8l\" (UID: \"9db51790-83e0-44a2-ac63-6b87ed8eeb08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-n8b8l"
Nov 02 13:50:44.893939 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:50:44.893912    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fcsh\" (UniqueName: \"kubernetes.io/projected/9db51790-83e0-44a2-ac63-6b87ed8eeb08-kube-api-access-5fcsh\") pod \"maas-api-revoke-sub-3d4156f1-n8b8l\" (UID: \"9db51790-83e0-44a2-ac63-6b87ed8eeb08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-n8b8l"
Nov 02 13:50:44.929914 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:50:44.929890    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-n8b8l"
Nov 02 13:50:45.084626 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:50:45.084586    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqcmj" containerName="extractor"
Nov 02 13:50:45.226985 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:50:45.226956    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-n8b8l"]
Nov 02 13:50:45.227127 ip-10-0-128-132 kubenswrapper[4559]: W1102 13:50:45.227032    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9db51790_83e0_44a2_ac63_6b87ed8eeb08.slice/crio-dec0bbac30abb44315fd0beab5f6812cbf41838d5b3b9e97efa43264deec8baa WatchSource:0}: Error finding container dec0bbac30abb44315fd0beab5f6812cbf41838d5b3b9e97efa43264deec8baa: Status 404 returned error can't find the container with id dec0bbac30abb44315fd0beab5f6812cbf41838d5b3b9e97efa43264deec8baa
Nov 02 13:50:45.569714 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:50:45.569684    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-n8b8l" event={"ID":"9db51790-83e0-44a2-ac63-6b87ed8eeb08","Type":"ContainerStarted","Data":"2fd96168dc310ca0bf1cb443d3017ad839de0881c7f6742c95c60f6c8be18a28"}
Nov 02 13:50:45.569821 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:50:45.569732    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-n8b8l" event={"ID":"9db51790-83e0-44a2-ac63-6b87ed8eeb08","Type":"ContainerStarted","Data":"dec0bbac30abb44315fd0beab5f6812cbf41838d5b3b9e97efa43264deec8baa"}
Nov 02 13:50:45.612430 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:50:45.612387    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-n8b8l" podStartSLOduration=1.6123735300000002 podStartE2EDuration="1.61237353s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:50:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:50:45.611301555 +0000 UTC m=+5041.866700651" watchObservedRunningTime="2026-11-02 13:50:45.61237353 +0000 UTC m=+5041.867772625"
Nov 02 13:50:46.577327 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:50:46.577292    4559 generic.go:356] "Generic (PLEG): container finished" podID="9db51790-83e0-44a2-ac63-6b87ed8eeb08" containerID="2fd96168dc310ca0bf1cb443d3017ad839de0881c7f6742c95c60f6c8be18a28" exitCode=0
Nov 02 13:50:46.577514 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:50:46.577371    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-n8b8l" event={"ID":"9db51790-83e0-44a2-ac63-6b87ed8eeb08","Type":"ContainerDied","Data":"2fd96168dc310ca0bf1cb443d3017ad839de0881c7f6742c95c60f6c8be18a28"}
Nov 02 13:50:47.710339 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:50:47.710313    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-n8b8l"
Nov 02 13:50:47.797523 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:50:47.797497    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9db51790-83e0-44a2-ac63-6b87ed8eeb08-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9db51790-83e0-44a2-ac63-6b87ed8eeb08-maas-api-openshift-ca\") pod \"9db51790-83e0-44a2-ac63-6b87ed8eeb08\" (UID: \"9db51790-83e0-44a2-ac63-6b87ed8eeb08\") "
Nov 02 13:50:47.797672 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:50:47.797541    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9db51790-83e0-44a2-ac63-6b87ed8eeb08-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9db51790-83e0-44a2-ac63-6b87ed8eeb08-maas-api-xks-ca\") pod \"9db51790-83e0-44a2-ac63-6b87ed8eeb08\" (UID: \"9db51790-83e0-44a2-ac63-6b87ed8eeb08\") "
Nov 02 13:50:47.797672 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:50:47.797628    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9db51790-83e0-44a2-ac63-6b87ed8eeb08-kube-api-access-5fcsh\" (UniqueName: \"kubernetes.io/projected/9db51790-83e0-44a2-ac63-6b87ed8eeb08-kube-api-access-5fcsh\") pod \"9db51790-83e0-44a2-ac63-6b87ed8eeb08\" (UID: \"9db51790-83e0-44a2-ac63-6b87ed8eeb08\") "
Nov 02 13:50:47.797861 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:50:47.797840    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9db51790-83e0-44a2-ac63-6b87ed8eeb08-maas-api-openshift-ca" pod "9db51790-83e0-44a2-ac63-6b87ed8eeb08" (UID: "9db51790-83e0-44a2-ac63-6b87ed8eeb08"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:50:47.797996 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:50:47.797982    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9db51790-83e0-44a2-ac63-6b87ed8eeb08-maas-api-openshift-ca\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:50:47.799620 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:50:47.799602    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9db51790-83e0-44a2-ac63-6b87ed8eeb08-maas-api-xks-ca" pod "9db51790-83e0-44a2-ac63-6b87ed8eeb08" (UID: "9db51790-83e0-44a2-ac63-6b87ed8eeb08"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:50:47.799719 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:50:47.799601    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9db51790-83e0-44a2-ac63-6b87ed8eeb08-kube-api-access-5fcsh" pod "9db51790-83e0-44a2-ac63-6b87ed8eeb08" (UID: "9db51790-83e0-44a2-ac63-6b87ed8eeb08"). InnerVolumeSpecName "kube-api-access-5fcsh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:50:47.898477 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:50:47.898449    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5fcsh\" (UniqueName: \"kubernetes.io/projected/9db51790-83e0-44a2-ac63-6b87ed8eeb08-kube-api-access-5fcsh\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:50:47.898599 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:50:47.898478    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9db51790-83e0-44a2-ac63-6b87ed8eeb08-maas-api-xks-ca\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:50:47.961563 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:50:47.961535    4559 scope.go:122] "RemoveContainer" containerID="8ed795599c105564e7654815f1f5f65b0e8e7da9e7c8ca78866fc1252f145367"
Nov 02 13:50:48.588666 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:50:48.588619    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-n8b8l" event={"ID":"9db51790-83e0-44a2-ac63-6b87ed8eeb08","Type":"ContainerDied","Data":"dec0bbac30abb44315fd0beab5f6812cbf41838d5b3b9e97efa43264deec8baa"}
Nov 02 13:50:48.588814 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:50:48.588684    4559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dec0bbac30abb44315fd0beab5f6812cbf41838d5b3b9e97efa43264deec8baa"
Nov 02 13:50:48.588814 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:50:48.588628    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-n8b8l"
Nov 02 13:50:55.904766 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:50:55.904727    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"]
Nov 02 13:50:58.761664 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:50:58.760774    4559 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-qmrsp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.15:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 13:50:58.761664 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:50:58.760864    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qmrsp" podUID="a534baf3-8ff8-4dca-bc27-9022feaaca75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.15:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:51:01.127182 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:51:01.127140    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"]
Nov 02 13:51:01.821662 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:51:01.821596    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-n8b8l"]
Nov 02 13:51:01.834121 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:51:01.834088    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-n8b8l"]
Nov 02 13:51:01.875964 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:51:01.875933    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2jwl6"]
Nov 02 13:51:01.876306 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:51:01.876294    4559 cpu_manager.go:414] "containerMap: removing container" podUID="9db51790-83e0-44a2-ac63-6b87ed8eeb08" containerName="revoke-keys"
Nov 02 13:51:01.876350 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:51:01.876307    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9db51790-83e0-44a2-ac63-6b87ed8eeb08" containerName="revoke-keys"
Nov 02 13:51:01.876386 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:51:01.876378    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="9db51790-83e0-44a2-ac63-6b87ed8eeb08" containerName="revoke-keys"
Nov 02 13:51:01.879729 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:51:01.879707    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2jwl6"
Nov 02 13:51:01.892314 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:51:01.892288    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9db51790-83e0-44a2-ac63-6b87ed8eeb08" path="/var/lib/kubelet/pods/9db51790-83e0-44a2-ac63-6b87ed8eeb08/volumes"
Nov 02 13:51:01.898001 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:51:01.897981    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 13:51:01.898156 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:51:01.898141    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-45fzn\""
Nov 02 13:51:01.898573 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:51:01.898553    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 13:51:01.898676 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:51:01.898614    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 13:51:01.900211 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:51:01.900186    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2jwl6"]
Nov 02 13:51:01.977971 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:51:01.977939    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhqx4\" (UniqueName: \"kubernetes.io/projected/81cb4194-b6ef-43f0-aed2-58146101fa75-kube-api-access-jhqx4\") pod \"maas-api-revoke-sub-3d4156f1-2jwl6\" (UID: \"81cb4194-b6ef-43f0-aed2-58146101fa75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2jwl6"
Nov 02 13:51:01.978109 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:51:01.977988    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81cb4194-b6ef-43f0-aed2-58146101fa75-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-2jwl6\" (UID: \"81cb4194-b6ef-43f0-aed2-58146101fa75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2jwl6"
Nov 02 13:51:01.978109 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:51:01.978091    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81cb4194-b6ef-43f0-aed2-58146101fa75-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-2jwl6\" (UID: \"81cb4194-b6ef-43f0-aed2-58146101fa75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2jwl6"
Nov 02 13:51:02.079143 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:51:02.079066    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhqx4\" (UniqueName: \"kubernetes.io/projected/81cb4194-b6ef-43f0-aed2-58146101fa75-kube-api-access-jhqx4\") pod \"maas-api-revoke-sub-3d4156f1-2jwl6\" (UID: \"81cb4194-b6ef-43f0-aed2-58146101fa75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2jwl6"
Nov 02 13:51:02.079143 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:51:02.079128    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81cb4194-b6ef-43f0-aed2-58146101fa75-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-2jwl6\" (UID: \"81cb4194-b6ef-43f0-aed2-58146101fa75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2jwl6"
Nov 02 13:51:02.079353 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:51:02.079193    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81cb4194-b6ef-43f0-aed2-58146101fa75-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-2jwl6\" (UID: \"81cb4194-b6ef-43f0-aed2-58146101fa75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2jwl6"
Nov 02 13:51:02.079849 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:51:02.079829    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81cb4194-b6ef-43f0-aed2-58146101fa75-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-2jwl6\" (UID: \"81cb4194-b6ef-43f0-aed2-58146101fa75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2jwl6"
Nov 02 13:51:02.081511 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:51:02.081487    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81cb4194-b6ef-43f0-aed2-58146101fa75-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-2jwl6\" (UID: \"81cb4194-b6ef-43f0-aed2-58146101fa75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2jwl6"
Nov 02 13:51:02.103332 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:51:02.103308    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhqx4\" (UniqueName: \"kubernetes.io/projected/81cb4194-b6ef-43f0-aed2-58146101fa75-kube-api-access-jhqx4\") pod \"maas-api-revoke-sub-3d4156f1-2jwl6\" (UID: \"81cb4194-b6ef-43f0-aed2-58146101fa75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2jwl6"
Nov 02 13:51:02.192142 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:51:02.192119    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2jwl6"
Nov 02 13:51:02.488495 ip-10-0-128-132 kubenswrapper[4559]: W1102 13:51:02.488466    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81cb4194_b6ef_43f0_aed2_58146101fa75.slice/crio-7b469e2d8fdc9331922686131ef80599e29eb1d5563a1673b4f609b12fe8d60c WatchSource:0}: Error finding container 7b469e2d8fdc9331922686131ef80599e29eb1d5563a1673b4f609b12fe8d60c: Status 404 returned error can't find the container with id 7b469e2d8fdc9331922686131ef80599e29eb1d5563a1673b4f609b12fe8d60c
Nov 02 13:51:02.489048 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:51:02.489024    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2jwl6"]
Nov 02 13:51:02.689102 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:51:02.689072    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2jwl6" event={"ID":"81cb4194-b6ef-43f0-aed2-58146101fa75","Type":"ContainerStarted","Data":"7b469e2d8fdc9331922686131ef80599e29eb1d5563a1673b4f609b12fe8d60c"}
Nov 02 13:51:03.696130 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:51:03.696094    4559 generic.go:356] "Generic (PLEG): container finished" podID="81cb4194-b6ef-43f0-aed2-58146101fa75" containerID="42533ec5c78f54720a8298b7e2dbe49c1bf5dce2adf6c022de3889be1c6ac494" exitCode=0
Nov 02 13:51:03.696534 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:51:03.696185    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2jwl6" event={"ID":"81cb4194-b6ef-43f0-aed2-58146101fa75","Type":"ContainerDied","Data":"42533ec5c78f54720a8298b7e2dbe49c1bf5dce2adf6c022de3889be1c6ac494"}
Nov 02 13:51:04.859579 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:51:04.859554    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2jwl6"
Nov 02 13:51:04.904708 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:51:04.904684    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81cb4194-b6ef-43f0-aed2-58146101fa75-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81cb4194-b6ef-43f0-aed2-58146101fa75-maas-api-openshift-ca\") pod \"81cb4194-b6ef-43f0-aed2-58146101fa75\" (UID: \"81cb4194-b6ef-43f0-aed2-58146101fa75\") "
Nov 02 13:51:04.904878 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:51:04.904732    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/81cb4194-b6ef-43f0-aed2-58146101fa75-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81cb4194-b6ef-43f0-aed2-58146101fa75-maas-api-xks-ca\") pod \"81cb4194-b6ef-43f0-aed2-58146101fa75\" (UID: \"81cb4194-b6ef-43f0-aed2-58146101fa75\") "
Nov 02 13:51:04.904878 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:51:04.904854    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81cb4194-b6ef-43f0-aed2-58146101fa75-kube-api-access-jhqx4\" (UniqueName: \"kubernetes.io/projected/81cb4194-b6ef-43f0-aed2-58146101fa75-kube-api-access-jhqx4\") pod \"81cb4194-b6ef-43f0-aed2-58146101fa75\" (UID: \"81cb4194-b6ef-43f0-aed2-58146101fa75\") "
Nov 02 13:51:04.905094 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:51:04.905065    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81cb4194-b6ef-43f0-aed2-58146101fa75-maas-api-openshift-ca" pod "81cb4194-b6ef-43f0-aed2-58146101fa75" (UID: "81cb4194-b6ef-43f0-aed2-58146101fa75"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:51:04.907143 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:51:04.907115    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81cb4194-b6ef-43f0-aed2-58146101fa75-kube-api-access-jhqx4" pod "81cb4194-b6ef-43f0-aed2-58146101fa75" (UID: "81cb4194-b6ef-43f0-aed2-58146101fa75"). InnerVolumeSpecName "kube-api-access-jhqx4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:51:04.907253 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:51:04.907116    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/81cb4194-b6ef-43f0-aed2-58146101fa75-maas-api-xks-ca" pod "81cb4194-b6ef-43f0-aed2-58146101fa75" (UID: "81cb4194-b6ef-43f0-aed2-58146101fa75"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:51:05.005840 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:51:05.005786    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81cb4194-b6ef-43f0-aed2-58146101fa75-maas-api-openshift-ca\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:51:05.005840 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:51:05.005809    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81cb4194-b6ef-43f0-aed2-58146101fa75-maas-api-xks-ca\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:51:05.005840 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:51:05.005818    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jhqx4\" (UniqueName: \"kubernetes.io/projected/81cb4194-b6ef-43f0-aed2-58146101fa75-kube-api-access-jhqx4\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:51:05.707660 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:51:05.707605    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2jwl6"
Nov 02 13:51:05.707841 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:51:05.707666    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2jwl6" event={"ID":"81cb4194-b6ef-43f0-aed2-58146101fa75","Type":"ContainerDied","Data":"7b469e2d8fdc9331922686131ef80599e29eb1d5563a1673b4f609b12fe8d60c"}
Nov 02 13:51:05.707841 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:51:05.707712    4559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7b469e2d8fdc9331922686131ef80599e29eb1d5563a1673b4f609b12fe8d60c"
Nov 02 13:51:14.009194 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:51:14.009153    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"]
Nov 02 13:51:15.084926 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:51:15.084881    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqcmj" containerName="extractor"
Nov 02 13:51:19.196491 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:51:19.196457    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"]
Nov 02 13:51:25.996933 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:51:25.996861    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"]
Nov 02 13:51:28.763782 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:51:28.763738    4559 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-qmrsp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.15:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 13:51:28.764433 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:51:28.764391    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qmrsp" podUID="a534baf3-8ff8-4dca-bc27-9022feaaca75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.15:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:51:29.011792 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:51:29.011752    4559 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-jr7g7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.16:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 13:51:29.012116 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:51:29.012074    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jr7g7" podUID="a15c4957-178b-41d5-ae44-e79c37fb55d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.16:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:51:32.618034 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:51:32.617994    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-jk7jx"]
Nov 02 13:51:32.618434 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:51:32.618383    4559 cpu_manager.go:414] "containerMap: removing container" podUID="81cb4194-b6ef-43f0-aed2-58146101fa75" containerName="revoke-keys"
Nov 02 13:51:32.618434 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:51:32.618396    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81cb4194-b6ef-43f0-aed2-58146101fa75" containerName="revoke-keys"
Nov 02 13:51:32.618511 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:51:32.618488    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="81cb4194-b6ef-43f0-aed2-58146101fa75" containerName="revoke-keys"
Nov 02 13:51:32.621195 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:51:32.621173    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-jk7jx"
Nov 02 13:51:32.645995 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:51:32.645961    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-jk7jx"]
Nov 02 13:51:32.655932 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:51:32.655905    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 13:51:32.656092 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:51:32.655948    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-45fzn\""
Nov 02 13:51:32.715143 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:51:32.715111    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 13:51:32.715143 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:51:32.715111    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 13:51:32.793842 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:51:32.793808    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7060ccc8-9297-45df-8de9-2ea1955a0692-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-jk7jx\" (UID: \"7060ccc8-9297-45df-8de9-2ea1955a0692\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-jk7jx"
Nov 02 13:51:32.793842 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:51:32.793847    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7060ccc8-9297-45df-8de9-2ea1955a0692-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-jk7jx\" (UID: \"7060ccc8-9297-45df-8de9-2ea1955a0692\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-jk7jx"
Nov 02 13:51:32.794078 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:51:32.793970    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjmwj\" (UniqueName: \"kubernetes.io/projected/7060ccc8-9297-45df-8de9-2ea1955a0692-kube-api-access-zjmwj\") pod \"maas-api-revoke-sub-e2fc9c5b-jk7jx\" (UID: \"7060ccc8-9297-45df-8de9-2ea1955a0692\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-jk7jx"
Nov 02 13:51:32.895205 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:51:32.895119    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7060ccc8-9297-45df-8de9-2ea1955a0692-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-jk7jx\" (UID: \"7060ccc8-9297-45df-8de9-2ea1955a0692\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-jk7jx"
Nov 02 13:51:32.895205 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:51:32.895165    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7060ccc8-9297-45df-8de9-2ea1955a0692-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-jk7jx\" (UID: \"7060ccc8-9297-45df-8de9-2ea1955a0692\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-jk7jx"
Nov 02 13:51:32.895415 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:51:32.895218    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjmwj\" (UniqueName: \"kubernetes.io/projected/7060ccc8-9297-45df-8de9-2ea1955a0692-kube-api-access-zjmwj\") pod \"maas-api-revoke-sub-e2fc9c5b-jk7jx\" (UID: \"7060ccc8-9297-45df-8de9-2ea1955a0692\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-jk7jx"
Nov 02 13:51:32.895759 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:51:32.895735    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7060ccc8-9297-45df-8de9-2ea1955a0692-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-jk7jx\" (UID: \"7060ccc8-9297-45df-8de9-2ea1955a0692\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-jk7jx"
Nov 02 13:51:32.897593 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:51:32.897564    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7060ccc8-9297-45df-8de9-2ea1955a0692-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-jk7jx\" (UID: \"7060ccc8-9297-45df-8de9-2ea1955a0692\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-jk7jx"
Nov 02 13:51:32.919295 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:51:32.919274    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjmwj\" (UniqueName: \"kubernetes.io/projected/7060ccc8-9297-45df-8de9-2ea1955a0692-kube-api-access-zjmwj\") pod \"maas-api-revoke-sub-e2fc9c5b-jk7jx\" (UID: \"7060ccc8-9297-45df-8de9-2ea1955a0692\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-jk7jx"
Nov 02 13:51:32.934042 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:51:32.934021    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-jk7jx"
Nov 02 13:51:33.233630 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:51:33.233600    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-jk7jx"]
Nov 02 13:51:33.234660 ip-10-0-128-132 kubenswrapper[4559]: W1102 13:51:33.234613    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7060ccc8_9297_45df_8de9_2ea1955a0692.slice/crio-c8e6b22de6a344dded3f42b1dad69482b1f7619efaff2ac3186a8daac34e0c63 WatchSource:0}: Error finding container c8e6b22de6a344dded3f42b1dad69482b1f7619efaff2ac3186a8daac34e0c63: Status 404 returned error can't find the container with id c8e6b22de6a344dded3f42b1dad69482b1f7619efaff2ac3186a8daac34e0c63
Nov 02 13:51:33.873982 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:51:33.873896    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-jk7jx" event={"ID":"7060ccc8-9297-45df-8de9-2ea1955a0692","Type":"ContainerStarted","Data":"eb1c7c5d3f180b4d158267b9b63c1993efb0d5152fba5c0fef225be45afc676d"}
Nov 02 13:51:33.873982 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:51:33.873947    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-jk7jx" event={"ID":"7060ccc8-9297-45df-8de9-2ea1955a0692","Type":"ContainerStarted","Data":"c8e6b22de6a344dded3f42b1dad69482b1f7619efaff2ac3186a8daac34e0c63"}
Nov 02 13:51:33.938775 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:51:33.938717    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-jk7jx" podStartSLOduration=1.938699755 podStartE2EDuration="1.938699755s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:51:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:51:33.938394629 +0000 UTC m=+5090.193793724" watchObservedRunningTime="2026-11-02 13:51:33.938699755 +0000 UTC m=+5090.194098851"
Nov 02 13:51:34.880688 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:51:34.880630    4559 generic.go:356] "Generic (PLEG): container finished" podID="7060ccc8-9297-45df-8de9-2ea1955a0692" containerID="eb1c7c5d3f180b4d158267b9b63c1993efb0d5152fba5c0fef225be45afc676d" exitCode=0
Nov 02 13:51:34.880688 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:51:34.880669    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-jk7jx" event={"ID":"7060ccc8-9297-45df-8de9-2ea1955a0692","Type":"ContainerDied","Data":"eb1c7c5d3f180b4d158267b9b63c1993efb0d5152fba5c0fef225be45afc676d"}
Nov 02 13:51:36.018363 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:51:36.018336    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-jk7jx"
Nov 02 13:51:36.124582 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:51:36.124552    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7060ccc8-9297-45df-8de9-2ea1955a0692-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7060ccc8-9297-45df-8de9-2ea1955a0692-maas-api-xks-ca\") pod \"7060ccc8-9297-45df-8de9-2ea1955a0692\" (UID: \"7060ccc8-9297-45df-8de9-2ea1955a0692\") "
Nov 02 13:51:36.124764 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:51:36.124592    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7060ccc8-9297-45df-8de9-2ea1955a0692-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7060ccc8-9297-45df-8de9-2ea1955a0692-maas-api-openshift-ca\") pod \"7060ccc8-9297-45df-8de9-2ea1955a0692\" (UID: \"7060ccc8-9297-45df-8de9-2ea1955a0692\") "
Nov 02 13:51:36.124764 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:51:36.124724    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7060ccc8-9297-45df-8de9-2ea1955a0692-kube-api-access-zjmwj\" (UniqueName: \"kubernetes.io/projected/7060ccc8-9297-45df-8de9-2ea1955a0692-kube-api-access-zjmwj\") pod \"7060ccc8-9297-45df-8de9-2ea1955a0692\" (UID: \"7060ccc8-9297-45df-8de9-2ea1955a0692\") "
Nov 02 13:51:36.124974 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:51:36.124953    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7060ccc8-9297-45df-8de9-2ea1955a0692-maas-api-openshift-ca" pod "7060ccc8-9297-45df-8de9-2ea1955a0692" (UID: "7060ccc8-9297-45df-8de9-2ea1955a0692"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:51:36.126723 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:51:36.126691    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7060ccc8-9297-45df-8de9-2ea1955a0692-maas-api-xks-ca" pod "7060ccc8-9297-45df-8de9-2ea1955a0692" (UID: "7060ccc8-9297-45df-8de9-2ea1955a0692"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:51:36.126836 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:51:36.126815    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7060ccc8-9297-45df-8de9-2ea1955a0692-kube-api-access-zjmwj" pod "7060ccc8-9297-45df-8de9-2ea1955a0692" (UID: "7060ccc8-9297-45df-8de9-2ea1955a0692"). InnerVolumeSpecName "kube-api-access-zjmwj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:51:36.225383 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:51:36.225344    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7060ccc8-9297-45df-8de9-2ea1955a0692-maas-api-openshift-ca\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:51:36.225383 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:51:36.225382    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zjmwj\" (UniqueName: \"kubernetes.io/projected/7060ccc8-9297-45df-8de9-2ea1955a0692-kube-api-access-zjmwj\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:51:36.225383 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:51:36.225394    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7060ccc8-9297-45df-8de9-2ea1955a0692-maas-api-xks-ca\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:51:36.891832 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:51:36.891800    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-jk7jx"
Nov 02 13:51:36.892016 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:51:36.891799    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-jk7jx" event={"ID":"7060ccc8-9297-45df-8de9-2ea1955a0692","Type":"ContainerDied","Data":"c8e6b22de6a344dded3f42b1dad69482b1f7619efaff2ac3186a8daac34e0c63"}
Nov 02 13:51:36.892016 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:51:36.891922    4559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c8e6b22de6a344dded3f42b1dad69482b1f7619efaff2ac3186a8daac34e0c63"
Nov 02 13:51:43.052407 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:51:43.052371    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-9286p"]
Nov 02 13:51:43.053016 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:51:43.052993    4559 cpu_manager.go:414] "containerMap: removing container" podUID="7060ccc8-9297-45df-8de9-2ea1955a0692" containerName="revoke-keys"
Nov 02 13:51:43.053116 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:51:43.053017    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7060ccc8-9297-45df-8de9-2ea1955a0692" containerName="revoke-keys"
Nov 02 13:51:43.053175 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:51:43.053141    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="7060ccc8-9297-45df-8de9-2ea1955a0692" containerName="revoke-keys"
Nov 02 13:51:43.056331 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:51:43.056303    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-9286p"
Nov 02 13:51:43.076130 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:51:43.076100    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 13:51:43.076130 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:51:43.076101    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-45fzn\""
Nov 02 13:51:43.082328 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:51:43.082295    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwr9f\" (UniqueName: \"kubernetes.io/projected/a56aac52-8bf3-4a5b-98ca-91afba77505f-kube-api-access-xwr9f\") pod \"maas-api-revoke-sub-5a29efb3-9286p\" (UID: \"a56aac52-8bf3-4a5b-98ca-91afba77505f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-9286p"
Nov 02 13:51:43.082524 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:51:43.082375    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a56aac52-8bf3-4a5b-98ca-91afba77505f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-9286p\" (UID: \"a56aac52-8bf3-4a5b-98ca-91afba77505f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-9286p"
Nov 02 13:51:43.082612 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:51:43.082533    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a56aac52-8bf3-4a5b-98ca-91afba77505f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-9286p\" (UID: \"a56aac52-8bf3-4a5b-98ca-91afba77505f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-9286p"
Nov 02 13:51:43.089608 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:51:43.089142    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 13:51:43.089608 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:51:43.089535    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 13:51:43.092330 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:51:43.092305    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-9286p"]
Nov 02 13:51:43.183344 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:51:43.183307    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a56aac52-8bf3-4a5b-98ca-91afba77505f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-9286p\" (UID: \"a56aac52-8bf3-4a5b-98ca-91afba77505f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-9286p"
Nov 02 13:51:43.183562 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:51:43.183355    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwr9f\" (UniqueName: \"kubernetes.io/projected/a56aac52-8bf3-4a5b-98ca-91afba77505f-kube-api-access-xwr9f\") pod \"maas-api-revoke-sub-5a29efb3-9286p\" (UID: \"a56aac52-8bf3-4a5b-98ca-91afba77505f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-9286p"
Nov 02 13:51:43.183562 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:51:43.183397    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a56aac52-8bf3-4a5b-98ca-91afba77505f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-9286p\" (UID: \"a56aac52-8bf3-4a5b-98ca-91afba77505f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-9286p"
Nov 02 13:51:43.183990 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:51:43.183967    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a56aac52-8bf3-4a5b-98ca-91afba77505f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-9286p\" (UID: \"a56aac52-8bf3-4a5b-98ca-91afba77505f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-9286p"
Nov 02 13:51:43.185736 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:51:43.185712    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a56aac52-8bf3-4a5b-98ca-91afba77505f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-9286p\" (UID: \"a56aac52-8bf3-4a5b-98ca-91afba77505f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-9286p"
Nov 02 13:51:43.207744 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:51:43.207720    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwr9f\" (UniqueName: \"kubernetes.io/projected/a56aac52-8bf3-4a5b-98ca-91afba77505f-kube-api-access-xwr9f\") pod \"maas-api-revoke-sub-5a29efb3-9286p\" (UID: \"a56aac52-8bf3-4a5b-98ca-91afba77505f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-9286p"
Nov 02 13:51:43.369234 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:51:43.369147    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-9286p"
Nov 02 13:51:43.669462 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:51:43.669424    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-9286p"]
Nov 02 13:51:43.670182 ip-10-0-128-132 kubenswrapper[4559]: W1102 13:51:43.670156    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda56aac52_8bf3_4a5b_98ca_91afba77505f.slice/crio-643ca6cefedd683fb2848897785a73c25c01b90b9b2646aece4d1fa328db9de3 WatchSource:0}: Error finding container 643ca6cefedd683fb2848897785a73c25c01b90b9b2646aece4d1fa328db9de3: Status 404 returned error can't find the container with id 643ca6cefedd683fb2848897785a73c25c01b90b9b2646aece4d1fa328db9de3
Nov 02 13:51:43.926010 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:51:43.925932    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-9286p" event={"ID":"a56aac52-8bf3-4a5b-98ca-91afba77505f","Type":"ContainerStarted","Data":"643ca6cefedd683fb2848897785a73c25c01b90b9b2646aece4d1fa328db9de3"}
Nov 02 13:51:43.988686 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:51:43.988656    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"]
Nov 02 13:51:44.936230 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:51:44.936193    4559 generic.go:356] "Generic (PLEG): container finished" podID="a56aac52-8bf3-4a5b-98ca-91afba77505f" containerID="e73c7eebf0ac29a38d2d1d7d27efd35be212551b5fea495c7df99fc904fa8c59" exitCode=0
Nov 02 13:51:44.936920 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:51:44.936265    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-9286p" event={"ID":"a56aac52-8bf3-4a5b-98ca-91afba77505f","Type":"ContainerDied","Data":"e73c7eebf0ac29a38d2d1d7d27efd35be212551b5fea495c7df99fc904fa8c59"}
Nov 02 13:51:45.099658 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:51:45.099605    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqcmj" containerName="extractor"
Nov 02 13:51:46.693915 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:51:46.693876    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-9286p"
Nov 02 13:51:46.806962 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:51:46.806915    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a56aac52-8bf3-4a5b-98ca-91afba77505f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a56aac52-8bf3-4a5b-98ca-91afba77505f-maas-api-openshift-ca\") pod \"a56aac52-8bf3-4a5b-98ca-91afba77505f\" (UID: \"a56aac52-8bf3-4a5b-98ca-91afba77505f\") "
Nov 02 13:51:46.831782 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:51:46.831742    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a56aac52-8bf3-4a5b-98ca-91afba77505f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a56aac52-8bf3-4a5b-98ca-91afba77505f-maas-api-xks-ca\") pod \"a56aac52-8bf3-4a5b-98ca-91afba77505f\" (UID: \"a56aac52-8bf3-4a5b-98ca-91afba77505f\") "
Nov 02 13:51:46.832141 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:51:46.832116    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a56aac52-8bf3-4a5b-98ca-91afba77505f-kube-api-access-xwr9f\" (UniqueName: \"kubernetes.io/projected/a56aac52-8bf3-4a5b-98ca-91afba77505f-kube-api-access-xwr9f\") pod \"a56aac52-8bf3-4a5b-98ca-91afba77505f\" (UID: \"a56aac52-8bf3-4a5b-98ca-91afba77505f\") "
Nov 02 13:51:46.858724 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:51:46.827869    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a56aac52-8bf3-4a5b-98ca-91afba77505f-maas-api-openshift-ca" pod "a56aac52-8bf3-4a5b-98ca-91afba77505f" (UID: "a56aac52-8bf3-4a5b-98ca-91afba77505f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:51:46.884665 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:51:46.883892    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a56aac52-8bf3-4a5b-98ca-91afba77505f-kube-api-access-xwr9f" pod "a56aac52-8bf3-4a5b-98ca-91afba77505f" (UID: "a56aac52-8bf3-4a5b-98ca-91afba77505f"). InnerVolumeSpecName "kube-api-access-xwr9f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:51:46.904395 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:51:46.904331    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a56aac52-8bf3-4a5b-98ca-91afba77505f-maas-api-xks-ca" pod "a56aac52-8bf3-4a5b-98ca-91afba77505f" (UID: "a56aac52-8bf3-4a5b-98ca-91afba77505f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:51:46.936321 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:51:46.936261    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a56aac52-8bf3-4a5b-98ca-91afba77505f-maas-api-openshift-ca\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:51:46.936605 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:51:46.936581    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a56aac52-8bf3-4a5b-98ca-91afba77505f-maas-api-xks-ca\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:51:46.944903 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:51:46.944871    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xwr9f\" (UniqueName: \"kubernetes.io/projected/a56aac52-8bf3-4a5b-98ca-91afba77505f-kube-api-access-xwr9f\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:51:46.986279 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:51:46.986235    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-9286p" event={"ID":"a56aac52-8bf3-4a5b-98ca-91afba77505f","Type":"ContainerDied","Data":"643ca6cefedd683fb2848897785a73c25c01b90b9b2646aece4d1fa328db9de3"}
Nov 02 13:51:46.986605 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:51:46.986584    4559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="643ca6cefedd683fb2848897785a73c25c01b90b9b2646aece4d1fa328db9de3"
Nov 02 13:51:46.986873 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:51:46.986854    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-9286p"
Nov 02 13:51:47.981700 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:51:47.981666    4559 scope.go:122] "RemoveContainer" containerID="8e91d4488d114218e9666f89ae316055530e5644d351d02a2d174a4dbbdfb4df"
Nov 02 13:51:54.407723 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:51:54.407690    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"]
Nov 02 13:51:59.141259 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:51:59.141225    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2jwl6"]
Nov 02 13:51:59.157530 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:51:59.157504    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2jwl6"]
Nov 02 13:51:59.193008 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:51:59.192978    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dkvnp"]
Nov 02 13:51:59.193384 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:51:59.193372    4559 cpu_manager.go:414] "containerMap: removing container" podUID="a56aac52-8bf3-4a5b-98ca-91afba77505f" containerName="revoke-keys"
Nov 02 13:51:59.193430 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:51:59.193385    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a56aac52-8bf3-4a5b-98ca-91afba77505f" containerName="revoke-keys"
Nov 02 13:51:59.193464 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:51:59.193449    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="a56aac52-8bf3-4a5b-98ca-91afba77505f" containerName="revoke-keys"
Nov 02 13:51:59.196486 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:51:59.196469    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dkvnp"
Nov 02 13:51:59.213790 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:51:59.213761    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 13:51:59.213949 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:51:59.213906    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-45fzn\""
Nov 02 13:51:59.226396 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:51:59.226368    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 13:51:59.226396 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:51:59.226381    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 13:51:59.234127 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:51:59.234090    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dkvnp"]
Nov 02 13:51:59.354691 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:51:59.354654    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba787db2-d345-419e-b4c5-200abdf92503-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-dkvnp\" (UID: \"ba787db2-d345-419e-b4c5-200abdf92503\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dkvnp"
Nov 02 13:51:59.354899 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:51:59.354707    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba787db2-d345-419e-b4c5-200abdf92503-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-dkvnp\" (UID: \"ba787db2-d345-419e-b4c5-200abdf92503\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dkvnp"
Nov 02 13:51:59.354899 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:51:59.354816    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdbc4\" (UniqueName: \"kubernetes.io/projected/ba787db2-d345-419e-b4c5-200abdf92503-kube-api-access-rdbc4\") pod \"maas-api-revoke-sub-3d4156f1-dkvnp\" (UID: \"ba787db2-d345-419e-b4c5-200abdf92503\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dkvnp"
Nov 02 13:51:59.455974 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:51:59.455940    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba787db2-d345-419e-b4c5-200abdf92503-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-dkvnp\" (UID: \"ba787db2-d345-419e-b4c5-200abdf92503\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dkvnp"
Nov 02 13:51:59.456165 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:51:59.455984    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdbc4\" (UniqueName: \"kubernetes.io/projected/ba787db2-d345-419e-b4c5-200abdf92503-kube-api-access-rdbc4\") pod \"maas-api-revoke-sub-3d4156f1-dkvnp\" (UID: \"ba787db2-d345-419e-b4c5-200abdf92503\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dkvnp"
Nov 02 13:51:59.456165 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:51:59.456055    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba787db2-d345-419e-b4c5-200abdf92503-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-dkvnp\" (UID: \"ba787db2-d345-419e-b4c5-200abdf92503\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dkvnp"
Nov 02 13:51:59.456675 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:51:59.456654    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba787db2-d345-419e-b4c5-200abdf92503-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-dkvnp\" (UID: \"ba787db2-d345-419e-b4c5-200abdf92503\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dkvnp"
Nov 02 13:51:59.458232 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:51:59.458212    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba787db2-d345-419e-b4c5-200abdf92503-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-dkvnp\" (UID: \"ba787db2-d345-419e-b4c5-200abdf92503\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dkvnp"
Nov 02 13:51:59.479947 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:51:59.479920    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdbc4\" (UniqueName: \"kubernetes.io/projected/ba787db2-d345-419e-b4c5-200abdf92503-kube-api-access-rdbc4\") pod \"maas-api-revoke-sub-3d4156f1-dkvnp\" (UID: \"ba787db2-d345-419e-b4c5-200abdf92503\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dkvnp"
Nov 02 13:51:59.512221 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:51:59.512196    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dkvnp"
Nov 02 13:51:59.817770 ip-10-0-128-132 kubenswrapper[4559]: W1102 13:51:59.817740    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba787db2_d345_419e_b4c5_200abdf92503.slice/crio-5d308fd72addbe8f0ea7dafef4a0b28914c6209ed35a065d598829f304cb1efa WatchSource:0}: Error finding container 5d308fd72addbe8f0ea7dafef4a0b28914c6209ed35a065d598829f304cb1efa: Status 404 returned error can't find the container with id 5d308fd72addbe8f0ea7dafef4a0b28914c6209ed35a065d598829f304cb1efa
Nov 02 13:51:59.818827 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:51:59.818795    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dkvnp"]
Nov 02 13:51:59.891192 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:51:59.891163    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81cb4194-b6ef-43f0-aed2-58146101fa75" path="/var/lib/kubelet/pods/81cb4194-b6ef-43f0-aed2-58146101fa75/volumes"
Nov 02 13:52:00.050701 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:52:00.050667    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dkvnp" event={"ID":"ba787db2-d345-419e-b4c5-200abdf92503","Type":"ContainerStarted","Data":"5d308fd72addbe8f0ea7dafef4a0b28914c6209ed35a065d598829f304cb1efa"}
Nov 02 13:52:01.056613 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:52:01.056574    4559 generic.go:356] "Generic (PLEG): container finished" podID="ba787db2-d345-419e-b4c5-200abdf92503" containerID="86e22255d58277b3cd24a08499bed83486a7063a37240ec3c83b5968d677f5ac" exitCode=0
Nov 02 13:52:01.057028 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:52:01.056665    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dkvnp" event={"ID":"ba787db2-d345-419e-b4c5-200abdf92503","Type":"ContainerDied","Data":"86e22255d58277b3cd24a08499bed83486a7063a37240ec3c83b5968d677f5ac"}
Nov 02 13:52:02.196857 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:52:02.196827    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dkvnp"
Nov 02 13:52:02.385736 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:52:02.385660    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ba787db2-d345-419e-b4c5-200abdf92503-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba787db2-d345-419e-b4c5-200abdf92503-maas-api-xks-ca\") pod \"ba787db2-d345-419e-b4c5-200abdf92503\" (UID: \"ba787db2-d345-419e-b4c5-200abdf92503\") "
Nov 02 13:52:02.385736 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:52:02.385704    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba787db2-d345-419e-b4c5-200abdf92503-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba787db2-d345-419e-b4c5-200abdf92503-maas-api-openshift-ca\") pod \"ba787db2-d345-419e-b4c5-200abdf92503\" (UID: \"ba787db2-d345-419e-b4c5-200abdf92503\") "
Nov 02 13:52:02.385959 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:52:02.385749    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba787db2-d345-419e-b4c5-200abdf92503-kube-api-access-rdbc4\" (UniqueName: \"kubernetes.io/projected/ba787db2-d345-419e-b4c5-200abdf92503-kube-api-access-rdbc4\") pod \"ba787db2-d345-419e-b4c5-200abdf92503\" (UID: \"ba787db2-d345-419e-b4c5-200abdf92503\") "
Nov 02 13:52:02.386143 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:52:02.386106    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba787db2-d345-419e-b4c5-200abdf92503-maas-api-openshift-ca" pod "ba787db2-d345-419e-b4c5-200abdf92503" (UID: "ba787db2-d345-419e-b4c5-200abdf92503"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:52:02.387901 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:52:02.387880    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba787db2-d345-419e-b4c5-200abdf92503-kube-api-access-rdbc4" pod "ba787db2-d345-419e-b4c5-200abdf92503" (UID: "ba787db2-d345-419e-b4c5-200abdf92503"). InnerVolumeSpecName "kube-api-access-rdbc4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:52:02.387981 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:52:02.387899    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba787db2-d345-419e-b4c5-200abdf92503-maas-api-xks-ca" pod "ba787db2-d345-419e-b4c5-200abdf92503" (UID: "ba787db2-d345-419e-b4c5-200abdf92503"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:52:02.486789 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:52:02.486763    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rdbc4\" (UniqueName: \"kubernetes.io/projected/ba787db2-d345-419e-b4c5-200abdf92503-kube-api-access-rdbc4\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:52:02.486789 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:52:02.486787    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba787db2-d345-419e-b4c5-200abdf92503-maas-api-xks-ca\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:52:02.486948 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:52:02.486797    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba787db2-d345-419e-b4c5-200abdf92503-maas-api-openshift-ca\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:52:03.068918 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:52:03.068876    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dkvnp" event={"ID":"ba787db2-d345-419e-b4c5-200abdf92503","Type":"ContainerDied","Data":"5d308fd72addbe8f0ea7dafef4a0b28914c6209ed35a065d598829f304cb1efa"}
Nov 02 13:52:03.069096 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:52:03.068934    4559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5d308fd72addbe8f0ea7dafef4a0b28914c6209ed35a065d598829f304cb1efa"
Nov 02 13:52:03.069096 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:52:03.068897    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dkvnp"
Nov 02 13:52:11.326092 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:52:11.326054    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"]
Nov 02 13:52:14.812799 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:52:14.812763    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-593fe3-openshift-default-56579c548fcn26x"]
Nov 02 13:52:14.813210 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:52:14.813161    4559 cpu_manager.go:414] "containerMap: removing container" podUID="ba787db2-d345-419e-b4c5-200abdf92503" containerName="revoke-keys"
Nov 02 13:52:14.813210 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:52:14.813172    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba787db2-d345-419e-b4c5-200abdf92503" containerName="revoke-keys"
Nov 02 13:52:14.813308 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:52:14.813246    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba787db2-d345-419e-b4c5-200abdf92503" containerName="revoke-keys"
Nov 02 13:52:14.817628 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:52:14.817606    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-593fe3-openshift-default-56579c548fcn26x"
Nov 02 13:52:14.840611 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:52:14.840582    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-593fe3-openshift-default-dockercfg-cwbc2\""
Nov 02 13:52:14.841968 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:52:14.841936    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-593fe3-openshift-default-56579c548fcn26x"]
Nov 02 13:52:14.924607 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:52:14.924568    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d98abfdb-03ee-463d-b059-be87d7f9a774-istiod-ca-cert\") pod \"e2e-worker-single-main-593fe3-openshift-default-56579c548fcn26x\" (UID: \"d98abfdb-03ee-463d-b059-be87d7f9a774\") " pod="openshift-ingress/e2e-worker-single-main-593fe3-openshift-default-56579c548fcn26x"
Nov 02 13:52:14.924607 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:52:14.924617    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d98abfdb-03ee-463d-b059-be87d7f9a774-workload-socket\") pod \"e2e-worker-single-main-593fe3-openshift-default-56579c548fcn26x\" (UID: \"d98abfdb-03ee-463d-b059-be87d7f9a774\") " pod="openshift-ingress/e2e-worker-single-main-593fe3-openshift-default-56579c548fcn26x"
Nov 02 13:52:14.924918 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:52:14.924699    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d98abfdb-03ee-463d-b059-be87d7f9a774-istio-data\") pod \"e2e-worker-single-main-593fe3-openshift-default-56579c548fcn26x\" (UID: \"d98abfdb-03ee-463d-b059-be87d7f9a774\") " pod="openshift-ingress/e2e-worker-single-main-593fe3-openshift-default-56579c548fcn26x"
Nov 02 13:52:14.924918 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:52:14.924768    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6m6fx\" (UniqueName: \"kubernetes.io/projected/d98abfdb-03ee-463d-b059-be87d7f9a774-kube-api-access-6m6fx\") pod \"e2e-worker-single-main-593fe3-openshift-default-56579c548fcn26x\" (UID: \"d98abfdb-03ee-463d-b059-be87d7f9a774\") " pod="openshift-ingress/e2e-worker-single-main-593fe3-openshift-default-56579c548fcn26x"
Nov 02 13:52:14.924918 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:52:14.924803    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d98abfdb-03ee-463d-b059-be87d7f9a774-istio-podinfo\") pod \"e2e-worker-single-main-593fe3-openshift-default-56579c548fcn26x\" (UID: \"d98abfdb-03ee-463d-b059-be87d7f9a774\") " pod="openshift-ingress/e2e-worker-single-main-593fe3-openshift-default-56579c548fcn26x"
Nov 02 13:52:14.924918 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:52:14.924843    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d98abfdb-03ee-463d-b059-be87d7f9a774-workload-certs\") pod \"e2e-worker-single-main-593fe3-openshift-default-56579c548fcn26x\" (UID: \"d98abfdb-03ee-463d-b059-be87d7f9a774\") " pod="openshift-ingress/e2e-worker-single-main-593fe3-openshift-default-56579c548fcn26x"
Nov 02 13:52:14.924918 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:52:14.924871    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d98abfdb-03ee-463d-b059-be87d7f9a774-istio-envoy\") pod \"e2e-worker-single-main-593fe3-openshift-default-56579c548fcn26x\" (UID: \"d98abfdb-03ee-463d-b059-be87d7f9a774\") " pod="openshift-ingress/e2e-worker-single-main-593fe3-openshift-default-56579c548fcn26x"
Nov 02 13:52:14.924918 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:52:14.924907    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d98abfdb-03ee-463d-b059-be87d7f9a774-istio-token\") pod \"e2e-worker-single-main-593fe3-openshift-default-56579c548fcn26x\" (UID: \"d98abfdb-03ee-463d-b059-be87d7f9a774\") " pod="openshift-ingress/e2e-worker-single-main-593fe3-openshift-default-56579c548fcn26x"
Nov 02 13:52:14.925271 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:52:14.924945    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d98abfdb-03ee-463d-b059-be87d7f9a774-credential-socket\") pod \"e2e-worker-single-main-593fe3-openshift-default-56579c548fcn26x\" (UID: \"d98abfdb-03ee-463d-b059-be87d7f9a774\") " pod="openshift-ingress/e2e-worker-single-main-593fe3-openshift-default-56579c548fcn26x"
Nov 02 13:52:15.025516 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:52:15.025479    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6m6fx\" (UniqueName: \"kubernetes.io/projected/d98abfdb-03ee-463d-b059-be87d7f9a774-kube-api-access-6m6fx\") pod \"e2e-worker-single-main-593fe3-openshift-default-56579c548fcn26x\" (UID: \"d98abfdb-03ee-463d-b059-be87d7f9a774\") " pod="openshift-ingress/e2e-worker-single-main-593fe3-openshift-default-56579c548fcn26x"
Nov 02 13:52:15.025736 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:52:15.025532    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d98abfdb-03ee-463d-b059-be87d7f9a774-istio-podinfo\") pod \"e2e-worker-single-main-593fe3-openshift-default-56579c548fcn26x\" (UID: \"d98abfdb-03ee-463d-b059-be87d7f9a774\") " pod="openshift-ingress/e2e-worker-single-main-593fe3-openshift-default-56579c548fcn26x"
Nov 02 13:52:15.025736 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:52:15.025573    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d98abfdb-03ee-463d-b059-be87d7f9a774-workload-certs\") pod \"e2e-worker-single-main-593fe3-openshift-default-56579c548fcn26x\" (UID: \"d98abfdb-03ee-463d-b059-be87d7f9a774\") " pod="openshift-ingress/e2e-worker-single-main-593fe3-openshift-default-56579c548fcn26x"
Nov 02 13:52:15.025736 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:52:15.025604    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d98abfdb-03ee-463d-b059-be87d7f9a774-istio-envoy\") pod \"e2e-worker-single-main-593fe3-openshift-default-56579c548fcn26x\" (UID: \"d98abfdb-03ee-463d-b059-be87d7f9a774\") " pod="openshift-ingress/e2e-worker-single-main-593fe3-openshift-default-56579c548fcn26x"
Nov 02 13:52:15.025736 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:52:15.025690    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d98abfdb-03ee-463d-b059-be87d7f9a774-istio-token\") pod \"e2e-worker-single-main-593fe3-openshift-default-56579c548fcn26x\" (UID: \"d98abfdb-03ee-463d-b059-be87d7f9a774\") " pod="openshift-ingress/e2e-worker-single-main-593fe3-openshift-default-56579c548fcn26x"
Nov 02 13:52:15.025995 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:52:15.025749    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d98abfdb-03ee-463d-b059-be87d7f9a774-credential-socket\") pod \"e2e-worker-single-main-593fe3-openshift-default-56579c548fcn26x\" (UID: \"d98abfdb-03ee-463d-b059-be87d7f9a774\") " pod="openshift-ingress/e2e-worker-single-main-593fe3-openshift-default-56579c548fcn26x"
Nov 02 13:52:15.025995 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:52:15.025796    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d98abfdb-03ee-463d-b059-be87d7f9a774-istiod-ca-cert\") pod \"e2e-worker-single-main-593fe3-openshift-default-56579c548fcn26x\" (UID: \"d98abfdb-03ee-463d-b059-be87d7f9a774\") " pod="openshift-ingress/e2e-worker-single-main-593fe3-openshift-default-56579c548fcn26x"
Nov 02 13:52:15.025995 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:52:15.025820    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d98abfdb-03ee-463d-b059-be87d7f9a774-workload-socket\") pod \"e2e-worker-single-main-593fe3-openshift-default-56579c548fcn26x\" (UID: \"d98abfdb-03ee-463d-b059-be87d7f9a774\") " pod="openshift-ingress/e2e-worker-single-main-593fe3-openshift-default-56579c548fcn26x"
Nov 02 13:52:15.025995 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:52:15.025883    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d98abfdb-03ee-463d-b059-be87d7f9a774-istio-data\") pod \"e2e-worker-single-main-593fe3-openshift-default-56579c548fcn26x\" (UID: \"d98abfdb-03ee-463d-b059-be87d7f9a774\") " pod="openshift-ingress/e2e-worker-single-main-593fe3-openshift-default-56579c548fcn26x"
Nov 02 13:52:15.026200 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:52:15.026151    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d98abfdb-03ee-463d-b059-be87d7f9a774-workload-certs\") pod \"e2e-worker-single-main-593fe3-openshift-default-56579c548fcn26x\" (UID: \"d98abfdb-03ee-463d-b059-be87d7f9a774\") " pod="openshift-ingress/e2e-worker-single-main-593fe3-openshift-default-56579c548fcn26x"
Nov 02 13:52:15.026295 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:52:15.026266    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d98abfdb-03ee-463d-b059-be87d7f9a774-credential-socket\") pod \"e2e-worker-single-main-593fe3-openshift-default-56579c548fcn26x\" (UID: \"d98abfdb-03ee-463d-b059-be87d7f9a774\") " pod="openshift-ingress/e2e-worker-single-main-593fe3-openshift-default-56579c548fcn26x"
Nov 02 13:52:15.026439 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:52:15.026343    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d98abfdb-03ee-463d-b059-be87d7f9a774-workload-socket\") pod \"e2e-worker-single-main-593fe3-openshift-default-56579c548fcn26x\" (UID: \"d98abfdb-03ee-463d-b059-be87d7f9a774\") " pod="openshift-ingress/e2e-worker-single-main-593fe3-openshift-default-56579c548fcn26x"
Nov 02 13:52:15.026439 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:52:15.026371    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d98abfdb-03ee-463d-b059-be87d7f9a774-istio-data\") pod \"e2e-worker-single-main-593fe3-openshift-default-56579c548fcn26x\" (UID: \"d98abfdb-03ee-463d-b059-be87d7f9a774\") " pod="openshift-ingress/e2e-worker-single-main-593fe3-openshift-default-56579c548fcn26x"
Nov 02 13:52:15.026615 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:52:15.026593    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d98abfdb-03ee-463d-b059-be87d7f9a774-istiod-ca-cert\") pod \"e2e-worker-single-main-593fe3-openshift-default-56579c548fcn26x\" (UID: \"d98abfdb-03ee-463d-b059-be87d7f9a774\") " pod="openshift-ingress/e2e-worker-single-main-593fe3-openshift-default-56579c548fcn26x"
Nov 02 13:52:15.028039 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:52:15.028019    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d98abfdb-03ee-463d-b059-be87d7f9a774-istio-envoy\") pod \"e2e-worker-single-main-593fe3-openshift-default-56579c548fcn26x\" (UID: \"d98abfdb-03ee-463d-b059-be87d7f9a774\") " pod="openshift-ingress/e2e-worker-single-main-593fe3-openshift-default-56579c548fcn26x"
Nov 02 13:52:15.028190 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:52:15.028172    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d98abfdb-03ee-463d-b059-be87d7f9a774-istio-podinfo\") pod \"e2e-worker-single-main-593fe3-openshift-default-56579c548fcn26x\" (UID: \"d98abfdb-03ee-463d-b059-be87d7f9a774\") " pod="openshift-ingress/e2e-worker-single-main-593fe3-openshift-default-56579c548fcn26x"
Nov 02 13:52:15.084428 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:52:15.084346    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqcmj" containerName="extractor"
Nov 02 13:52:15.101947 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:52:15.101913    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d98abfdb-03ee-463d-b059-be87d7f9a774-istio-token\") pod \"e2e-worker-single-main-593fe3-openshift-default-56579c548fcn26x\" (UID: \"d98abfdb-03ee-463d-b059-be87d7f9a774\") " pod="openshift-ingress/e2e-worker-single-main-593fe3-openshift-default-56579c548fcn26x"
Nov 02 13:52:15.102185 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:52:15.102160    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6m6fx\" (UniqueName: \"kubernetes.io/projected/d98abfdb-03ee-463d-b059-be87d7f9a774-kube-api-access-6m6fx\") pod \"e2e-worker-single-main-593fe3-openshift-default-56579c548fcn26x\" (UID: \"d98abfdb-03ee-463d-b059-be87d7f9a774\") " pod="openshift-ingress/e2e-worker-single-main-593fe3-openshift-default-56579c548fcn26x"
Nov 02 13:52:15.130841 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:52:15.130803    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-593fe3-openshift-default-56579c548fcn26x"
Nov 02 13:52:15.463081 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:52:15.463054    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-593fe3-openshift-default-56579c548fcn26x"]
Nov 02 13:52:15.463749 ip-10-0-128-132 kubenswrapper[4559]: W1102 13:52:15.463720    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd98abfdb_03ee_463d_b059_be87d7f9a774.slice/crio-57a1b61b7981d026b7ee9f364daf8b6c0f58df108d58f1276c84e58de7a0ebf2 WatchSource:0}: Error finding container 57a1b61b7981d026b7ee9f364daf8b6c0f58df108d58f1276c84e58de7a0ebf2: Status 404 returned error can't find the container with id 57a1b61b7981d026b7ee9f364daf8b6c0f58df108d58f1276c84e58de7a0ebf2
Nov 02 13:52:15.465856 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:52:15.465819    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 13:52:15.465941 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:52:15.465883    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 13:52:15.465941 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:52:15.465918    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 13:52:16.163980 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:52:16.163940    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-593fe3-openshift-default-56579c548fcn26x" event={"ID":"d98abfdb-03ee-463d-b059-be87d7f9a774","Type":"ContainerStarted","Data":"2ea6151341e410f1eb3bbf92e7acc9fee40ee799d9960b029348b7da28dee1df"}
Nov 02 13:52:16.164478 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:52:16.164006    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-593fe3-openshift-default-56579c548fcn26x" event={"ID":"d98abfdb-03ee-463d-b059-be87d7f9a774","Type":"ContainerStarted","Data":"57a1b61b7981d026b7ee9f364daf8b6c0f58df108d58f1276c84e58de7a0ebf2"}
Nov 02 13:52:16.225317 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:52:16.225255    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-593fe3-openshift-default-56579c548fcn26x" podStartSLOduration=2.2252396660000002 podStartE2EDuration="2.225239666s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:52:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:52:16.22311661 +0000 UTC m=+5132.478515707" watchObservedRunningTime="2026-11-02 13:52:16.225239666 +0000 UTC m=+5132.480638761"
Nov 02 13:52:17.131369 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:52:17.131332    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-593fe3-openshift-default-56579c548fcn26x"
Nov 02 13:52:17.136373 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:52:17.136348    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-593fe3-openshift-default-56579c548fcn26x"
Nov 02 13:52:17.169530 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:52:17.169502    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-593fe3-openshift-default-56579c548fcn26x"
Nov 02 13:52:17.170498 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:52:17.170481    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-593fe3-openshift-default-56579c548fcn26x"
Nov 02 13:52:18.108138 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:52:18.108102    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"]
Nov 02 13:52:20.298038 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:52:20.297993    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-593fe3-6ff88d8bbf-hndjt"]
Nov 02 13:52:20.302097 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:52:20.302075    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-593fe3-6ff88d8bbf-hndjt"
Nov 02 13:52:20.333927 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:52:20.333897    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 13:52:20.334236 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:52:20.334216    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-593fe3-serving-cert\""
Nov 02 13:52:20.334537 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:52:20.334513    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 13:52:20.334676 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:52:20.334606    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Nov 02 13:52:20.334778 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:52:20.334697    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-sr5s9\""
Nov 02 13:52:20.337552 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:52:20.337525    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-593fe3-6ff88d8bbf-hndjt"]
Nov 02 13:52:20.379895 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:52:20.379867    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b800fd1c-eda5-4828-be75-1996c79f6340-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-593fe3-6ff88d8bbf-hndjt\" (UID: \"b800fd1c-eda5-4828-be75-1996c79f6340\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-593fe3-6ff88d8bbf-hndjt"
Nov 02 13:52:20.380082 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:52:20.379903    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b800fd1c-eda5-4828-be75-1996c79f6340-metrics-certs\") pod \"maas-api-e2e-worker-single-main-593fe3-6ff88d8bbf-hndjt\" (UID: \"b800fd1c-eda5-4828-be75-1996c79f6340\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-593fe3-6ff88d8bbf-hndjt"
Nov 02 13:52:20.380142 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:52:20.380097    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b800fd1c-eda5-4828-be75-1996c79f6340-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-593fe3-6ff88d8bbf-hndjt\" (UID: \"b800fd1c-eda5-4828-be75-1996c79f6340\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-593fe3-6ff88d8bbf-hndjt"
Nov 02 13:52:20.380200 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:52:20.380166    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzjwr\" (UniqueName: \"kubernetes.io/projected/b800fd1c-eda5-4828-be75-1996c79f6340-kube-api-access-fzjwr\") pod \"maas-api-e2e-worker-single-main-593fe3-6ff88d8bbf-hndjt\" (UID: \"b800fd1c-eda5-4828-be75-1996c79f6340\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-593fe3-6ff88d8bbf-hndjt"
Nov 02 13:52:20.480974 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:52:20.480938    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b800fd1c-eda5-4828-be75-1996c79f6340-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-593fe3-6ff88d8bbf-hndjt\" (UID: \"b800fd1c-eda5-4828-be75-1996c79f6340\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-593fe3-6ff88d8bbf-hndjt"
Nov 02 13:52:20.481152 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:52:20.480982    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b800fd1c-eda5-4828-be75-1996c79f6340-metrics-certs\") pod \"maas-api-e2e-worker-single-main-593fe3-6ff88d8bbf-hndjt\" (UID: \"b800fd1c-eda5-4828-be75-1996c79f6340\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-593fe3-6ff88d8bbf-hndjt"
Nov 02 13:52:20.481152 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:52:20.481040    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b800fd1c-eda5-4828-be75-1996c79f6340-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-593fe3-6ff88d8bbf-hndjt\" (UID: \"b800fd1c-eda5-4828-be75-1996c79f6340\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-593fe3-6ff88d8bbf-hndjt"
Nov 02 13:52:20.481152 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:52:20.481074    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzjwr\" (UniqueName: \"kubernetes.io/projected/b800fd1c-eda5-4828-be75-1996c79f6340-kube-api-access-fzjwr\") pod \"maas-api-e2e-worker-single-main-593fe3-6ff88d8bbf-hndjt\" (UID: \"b800fd1c-eda5-4828-be75-1996c79f6340\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-593fe3-6ff88d8bbf-hndjt"
Nov 02 13:52:20.481619 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:52:20.481582    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b800fd1c-eda5-4828-be75-1996c79f6340-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-593fe3-6ff88d8bbf-hndjt\" (UID: \"b800fd1c-eda5-4828-be75-1996c79f6340\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-593fe3-6ff88d8bbf-hndjt"
Nov 02 13:52:20.483511 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:52:20.483488    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b800fd1c-eda5-4828-be75-1996c79f6340-metrics-certs\") pod \"maas-api-e2e-worker-single-main-593fe3-6ff88d8bbf-hndjt\" (UID: \"b800fd1c-eda5-4828-be75-1996c79f6340\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-593fe3-6ff88d8bbf-hndjt"
Nov 02 13:52:20.483630 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:52:20.483534    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b800fd1c-eda5-4828-be75-1996c79f6340-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-593fe3-6ff88d8bbf-hndjt\" (UID: \"b800fd1c-eda5-4828-be75-1996c79f6340\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-593fe3-6ff88d8bbf-hndjt"
Nov 02 13:52:20.509276 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:52:20.509248    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzjwr\" (UniqueName: \"kubernetes.io/projected/b800fd1c-eda5-4828-be75-1996c79f6340-kube-api-access-fzjwr\") pod \"maas-api-e2e-worker-single-main-593fe3-6ff88d8bbf-hndjt\" (UID: \"b800fd1c-eda5-4828-be75-1996c79f6340\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-593fe3-6ff88d8bbf-hndjt"
Nov 02 13:52:20.619437 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:52:20.619350    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-593fe3-6ff88d8bbf-hndjt"
Nov 02 13:52:20.656420 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:52:20.656390    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-593fe3-664c4sq4d9"]
Nov 02 13:52:20.661496 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:52:20.661467    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-593fe3-664c4sq4d9"
Nov 02 13:52:20.682513 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:52:20.682481    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-593fe3-664c4sq4d9"]
Nov 02 13:52:20.684489 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:52:20.684455    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-f2c1c28a-dockercfg-gvg6l\""
Nov 02 13:52:20.684611 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:52:20.684557    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-593fe3\""
Nov 02 13:52:20.770324 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:52:20.770289    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-593fe3-b5847f68fvhh5t"]
Nov 02 13:52:20.774309 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:52:20.774286    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-593fe3-b5847f68fvhh5t"
Nov 02 13:52:20.786199 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:52:20.785930    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvzb7\" (UniqueName: \"kubernetes.io/projected/a953b628-392e-48e9-a256-f63bc2ae654a-kube-api-access-kvzb7\") pod \"payload-pre-processing-e2e-worker-single-main-593fe3-664c4sq4d9\" (UID: \"a953b628-392e-48e9-a256-f63bc2ae654a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-593fe3-664c4sq4d9"
Nov 02 13:52:20.786199 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:52:20.786032    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a953b628-392e-48e9-a256-f63bc2ae654a-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-593fe3-664c4sq4d9\" (UID: \"a953b628-392e-48e9-a256-f63bc2ae654a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-593fe3-664c4sq4d9"
Nov 02 13:52:20.796583 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:52:20.796553    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-593fe3-b5847f68fvhh5t"]
Nov 02 13:52:20.887456 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:52:20.887387    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvzb7\" (UniqueName: \"kubernetes.io/projected/a953b628-392e-48e9-a256-f63bc2ae654a-kube-api-access-kvzb7\") pod \"payload-pre-processing-e2e-worker-single-main-593fe3-664c4sq4d9\" (UID: \"a953b628-392e-48e9-a256-f63bc2ae654a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-593fe3-664c4sq4d9"
Nov 02 13:52:20.887456 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:52:20.887448    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a953b628-392e-48e9-a256-f63bc2ae654a-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-593fe3-664c4sq4d9\" (UID: \"a953b628-392e-48e9-a256-f63bc2ae654a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-593fe3-664c4sq4d9"
Nov 02 13:52:20.887674 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:52:20.887477    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k79k4\" (UniqueName: \"kubernetes.io/projected/b6b47f8f-149a-4511-8d45-dee306e485d3-kube-api-access-k79k4\") pod \"payload-processing-e2e-worker-single-main-593fe3-b5847f68fvhh5t\" (UID: \"b6b47f8f-149a-4511-8d45-dee306e485d3\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-593fe3-b5847f68fvhh5t"
Nov 02 13:52:20.887674 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:52:20.887501    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6b47f8f-149a-4511-8d45-dee306e485d3-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-593fe3-b5847f68fvhh5t\" (UID: \"b6b47f8f-149a-4511-8d45-dee306e485d3\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-593fe3-b5847f68fvhh5t"
Nov 02 13:52:20.888172 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:52:20.888151    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a953b628-392e-48e9-a256-f63bc2ae654a-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-593fe3-664c4sq4d9\" (UID: \"a953b628-392e-48e9-a256-f63bc2ae654a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-593fe3-664c4sq4d9"
Nov 02 13:52:20.911747 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:52:20.911724    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvzb7\" (UniqueName: \"kubernetes.io/projected/a953b628-392e-48e9-a256-f63bc2ae654a-kube-api-access-kvzb7\") pod \"payload-pre-processing-e2e-worker-single-main-593fe3-664c4sq4d9\" (UID: \"a953b628-392e-48e9-a256-f63bc2ae654a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-593fe3-664c4sq4d9"
Nov 02 13:52:20.932924 ip-10-0-128-132 kubenswrapper[4559]: W1102 13:52:20.932898    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb800fd1c_eda5_4828_be75_1996c79f6340.slice/crio-08c226b7df2796aed735862b912a2b5c3f44ba0f24733096c179f3541e8eb1cb WatchSource:0}: Error finding container 08c226b7df2796aed735862b912a2b5c3f44ba0f24733096c179f3541e8eb1cb: Status 404 returned error can't find the container with id 08c226b7df2796aed735862b912a2b5c3f44ba0f24733096c179f3541e8eb1cb
Nov 02 13:52:20.933415 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:52:20.933385    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-593fe3-6ff88d8bbf-hndjt"]
Nov 02 13:52:20.934285 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:52:20.934269    4559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 13:52:20.987962 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:52:20.987937    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-593fe3-664c4sq4d9"
Nov 02 13:52:20.988217 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:52:20.988190    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k79k4\" (UniqueName: \"kubernetes.io/projected/b6b47f8f-149a-4511-8d45-dee306e485d3-kube-api-access-k79k4\") pod \"payload-processing-e2e-worker-single-main-593fe3-b5847f68fvhh5t\" (UID: \"b6b47f8f-149a-4511-8d45-dee306e485d3\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-593fe3-b5847f68fvhh5t"
Nov 02 13:52:20.988350 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:52:20.988239    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6b47f8f-149a-4511-8d45-dee306e485d3-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-593fe3-b5847f68fvhh5t\" (UID: \"b6b47f8f-149a-4511-8d45-dee306e485d3\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-593fe3-b5847f68fvhh5t"
Nov 02 13:52:20.988981 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:52:20.988960    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6b47f8f-149a-4511-8d45-dee306e485d3-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-593fe3-b5847f68fvhh5t\" (UID: \"b6b47f8f-149a-4511-8d45-dee306e485d3\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-593fe3-b5847f68fvhh5t"
Nov 02 13:52:21.015250 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:52:21.015226    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k79k4\" (UniqueName: \"kubernetes.io/projected/b6b47f8f-149a-4511-8d45-dee306e485d3-kube-api-access-k79k4\") pod \"payload-processing-e2e-worker-single-main-593fe3-b5847f68fvhh5t\" (UID: \"b6b47f8f-149a-4511-8d45-dee306e485d3\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-593fe3-b5847f68fvhh5t"
Nov 02 13:52:21.101810 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:52:21.101778    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-593fe3-b5847f68fvhh5t"
Nov 02 13:52:21.195231 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:52:21.195184    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-593fe3-6ff88d8bbf-hndjt" event={"ID":"b800fd1c-eda5-4828-be75-1996c79f6340","Type":"ContainerStarted","Data":"08c226b7df2796aed735862b912a2b5c3f44ba0f24733096c179f3541e8eb1cb"}
Nov 02 13:52:21.286700 ip-10-0-128-132 kubenswrapper[4559]: W1102 13:52:21.286670    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda953b628_392e_48e9_a256_f63bc2ae654a.slice/crio-7519726978d6e8bd6d94024711a9d78254fc9b2c1819d3e8ad791867988a7cae WatchSource:0}: Error finding container 7519726978d6e8bd6d94024711a9d78254fc9b2c1819d3e8ad791867988a7cae: Status 404 returned error can't find the container with id 7519726978d6e8bd6d94024711a9d78254fc9b2c1819d3e8ad791867988a7cae
Nov 02 13:52:21.287634 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:52:21.287608    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-593fe3-664c4sq4d9"]
Nov 02 13:52:21.405226 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:52:21.405201    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-593fe3-b5847f68fvhh5t"]
Nov 02 13:52:21.405684 ip-10-0-128-132 kubenswrapper[4559]: W1102 13:52:21.405610    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6b47f8f_149a_4511_8d45_dee306e485d3.slice/crio-c733dd656d61469e2f78e7d8500786c3cc2b940feef079163c4c9b2d933d9173 WatchSource:0}: Error finding container c733dd656d61469e2f78e7d8500786c3cc2b940feef079163c4c9b2d933d9173: Status 404 returned error can't find the container with id c733dd656d61469e2f78e7d8500786c3cc2b940feef079163c4c9b2d933d9173
Nov 02 13:52:22.202383 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:52:22.202342    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-593fe3-664c4sq4d9" event={"ID":"a953b628-392e-48e9-a256-f63bc2ae654a","Type":"ContainerStarted","Data":"3d43de6fd8f31499a503441838b93983ffc44798564bfe75420829206bc88117"}
Nov 02 13:52:22.202580 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:52:22.202409    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-593fe3-664c4sq4d9" event={"ID":"a953b628-392e-48e9-a256-f63bc2ae654a","Type":"ContainerStarted","Data":"7519726978d6e8bd6d94024711a9d78254fc9b2c1819d3e8ad791867988a7cae"}
Nov 02 13:52:22.202580 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:52:22.202439    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-593fe3-664c4sq4d9"
Nov 02 13:52:22.203973 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:52:22.203950    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-593fe3-b5847f68fvhh5t" event={"ID":"b6b47f8f-149a-4511-8d45-dee306e485d3","Type":"ContainerStarted","Data":"8e3c89fb67e5c21f3af1d2142677e9bf564cd72d9ac413d34811eaa8e1336c29"}
Nov 02 13:52:22.204095 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:52:22.203986    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-593fe3-b5847f68fvhh5t" event={"ID":"b6b47f8f-149a-4511-8d45-dee306e485d3","Type":"ContainerStarted","Data":"c733dd656d61469e2f78e7d8500786c3cc2b940feef079163c4c9b2d933d9173"}
Nov 02 13:52:22.204095 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:52:22.204012    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-593fe3-b5847f68fvhh5t"
Nov 02 13:52:22.205362 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:52:22.205340    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-593fe3-6ff88d8bbf-hndjt" event={"ID":"b800fd1c-eda5-4828-be75-1996c79f6340","Type":"ContainerStarted","Data":"6e364da172522b3e27ef1403cf39d939ddfb01b42c7156c1c6d6b2302d695531"}
Nov 02 13:52:22.205488 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:52:22.205473    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-593fe3-6ff88d8bbf-hndjt"
Nov 02 13:52:22.247821 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:52:22.247770    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-593fe3-664c4sq4d9" podStartSLOduration=2.247753938 podStartE2EDuration="2.247753938s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:52:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:52:22.247501988 +0000 UTC m=+5138.502901083" watchObservedRunningTime="2026-11-02 13:52:22.247753938 +0000 UTC m=+5138.503153033"
Nov 02 13:52:22.298952 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:52:22.298907    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-593fe3-6ff88d8bbf-hndjt" podStartSLOduration=1.8544137969999999 podStartE2EDuration="2.298895204s" totalImagesPullingTime="444.481407ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:52:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:52:22.298311539 +0000 UTC m=+5138.553710633" watchObservedRunningTime="2026-11-02 13:52:22.298895204 +0000 UTC m=+5138.554294298"
Nov 02 13:52:22.346876 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:52:22.346815    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-593fe3-b5847f68fvhh5t" podStartSLOduration=2.346801002 podStartE2EDuration="2.346801002s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:52:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:52:22.346462244 +0000 UTC m=+5138.601861343" watchObservedRunningTime="2026-11-02 13:52:22.346801002 +0000 UTC m=+5138.602200100"
Nov 02 13:52:23.942661 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:52:23.942604    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"]
Nov 02 13:52:28.216501 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:52:28.216472    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-593fe3-6ff88d8bbf-hndjt"
Nov 02 13:52:33.210834 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:52:33.210787    4559 patch_prober.go:29] interesting pod/e2e-worker-single-main-593fe3-openshift-default-56579c548fcn26x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 13:52:33.211361 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:52:33.210863    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-593fe3-openshift-default-56579c548fcn26x" podUID="d98abfdb-03ee-463d-b059-be87d7f9a774" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:52:33.213519 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:52:33.213495    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-593fe3-b5847f68fvhh5t"
Nov 02 13:52:33.213682 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:52:33.213560    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-593fe3-664c4sq4d9"
Nov 02 13:52:34.273873 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:52:34.273838    4559 generic.go:356] "Generic (PLEG): container finished" podID="d98abfdb-03ee-463d-b059-be87d7f9a774" containerID="2ea6151341e410f1eb3bbf92e7acc9fee40ee799d9960b029348b7da28dee1df" exitCode=0
Nov 02 13:52:34.274348 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:52:34.273911    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-593fe3-openshift-default-56579c548fcn26x" event={"ID":"d98abfdb-03ee-463d-b059-be87d7f9a774","Type":"ContainerDied","Data":"2ea6151341e410f1eb3bbf92e7acc9fee40ee799d9960b029348b7da28dee1df"}
Nov 02 13:52:34.274348 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:52:34.274336    4559 scope.go:122] "RemoveContainer" containerID="2ea6151341e410f1eb3bbf92e7acc9fee40ee799d9960b029348b7da28dee1df"
Nov 02 13:52:34.275613 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:52:34.275583    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 13:52:34.275688 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:52:34.275675    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 13:52:34.275728 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:52:34.275709    4559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Nov 02 13:52:35.286664 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:52:35.282813    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-593fe3-openshift-default-56579c548fcn26x" event={"ID":"d98abfdb-03ee-463d-b059-be87d7f9a774","Type":"ContainerStarted","Data":"e37795122e54d58b30fd2484f926cb091ded123000e57760e3742d91661122f8"}
Nov 02 13:52:36.132006 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:52:36.131968    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-593fe3-openshift-default-56579c548fcn26x"
Nov 02 13:52:36.132272 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:52:36.132245    4559 patch_prober.go:29] interesting pod/e2e-worker-single-main-593fe3-openshift-default-56579c548fcn26x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.56:15021/healthz/ready\": dial tcp 10.133.1.56:15021: connect: connection refused" start-of-body=
Nov 02 13:52:36.132322 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:52:36.132300    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-593fe3-openshift-default-56579c548fcn26x" podUID="d98abfdb-03ee-463d-b059-be87d7f9a774" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.56:15021/healthz/ready\": dial tcp 10.133.1.56:15021: connect: connection refused"
Nov 02 13:52:36.245016 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:52:36.244977    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-593fe3/simulator-single-main-593fe3-kserve-7d5755f66f-dcztx"]
Nov 02 13:52:36.249533 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:52:36.249507    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-593fe3/simulator-single-main-593fe3-kserve-7d5755f66f-dcztx"
Nov 02 13:52:36.267919 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:52:36.267892    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-593fe3/simulator-single-main-593fe3-kserve-7d5755f66f-dcztx"]
Nov 02 13:52:36.277349 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:52:36.277326    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-593fe3\"/\"simulator-single-main-593fe3-kserve-self-signed-certs\""
Nov 02 13:52:36.277880 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:52:36.277855    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-593fe3\"/\"kube-root-ca.crt\""
Nov 02 13:52:36.278006 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:52:36.277945    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-593fe3\"/\"openshift-service-ca.crt\""
Nov 02 13:52:36.294885 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:52:36.294853    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-593fe3\"/\"default-dockercfg-jlk47\""
Nov 02 13:52:36.338813 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:52:36.338775    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c0d5da9-b867-44cf-a3a3-facb9793683b-dshm\") pod \"simulator-single-main-593fe3-kserve-7d5755f66f-dcztx\" (UID: \"1c0d5da9-b867-44cf-a3a3-facb9793683b\") " pod="e2e-models-e2e-worker-single-main-593fe3/simulator-single-main-593fe3-kserve-7d5755f66f-dcztx"
Nov 02 13:52:36.338990 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:52:36.338822    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2mhd\" (UniqueName: \"kubernetes.io/projected/1c0d5da9-b867-44cf-a3a3-facb9793683b-kube-api-access-s2mhd\") pod \"simulator-single-main-593fe3-kserve-7d5755f66f-dcztx\" (UID: \"1c0d5da9-b867-44cf-a3a3-facb9793683b\") " pod="e2e-models-e2e-worker-single-main-593fe3/simulator-single-main-593fe3-kserve-7d5755f66f-dcztx"
Nov 02 13:52:36.338990 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:52:36.338947    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c0d5da9-b867-44cf-a3a3-facb9793683b-model-cache\") pod \"simulator-single-main-593fe3-kserve-7d5755f66f-dcztx\" (UID: \"1c0d5da9-b867-44cf-a3a3-facb9793683b\") " pod="e2e-models-e2e-worker-single-main-593fe3/simulator-single-main-593fe3-kserve-7d5755f66f-dcztx"
Nov 02 13:52:36.339067 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:52:36.339005    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c0d5da9-b867-44cf-a3a3-facb9793683b-tls-certs\") pod \"simulator-single-main-593fe3-kserve-7d5755f66f-dcztx\" (UID: \"1c0d5da9-b867-44cf-a3a3-facb9793683b\") " pod="e2e-models-e2e-worker-single-main-593fe3/simulator-single-main-593fe3-kserve-7d5755f66f-dcztx"
Nov 02 13:52:36.339109 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:52:36.339095    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c0d5da9-b867-44cf-a3a3-facb9793683b-tmp-dir\") pod \"simulator-single-main-593fe3-kserve-7d5755f66f-dcztx\" (UID: \"1c0d5da9-b867-44cf-a3a3-facb9793683b\") " pod="e2e-models-e2e-worker-single-main-593fe3/simulator-single-main-593fe3-kserve-7d5755f66f-dcztx"
Nov 02 13:52:36.339147 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:52:36.339114    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c0d5da9-b867-44cf-a3a3-facb9793683b-home\") pod \"simulator-single-main-593fe3-kserve-7d5755f66f-dcztx\" (UID: \"1c0d5da9-b867-44cf-a3a3-facb9793683b\") " pod="e2e-models-e2e-worker-single-main-593fe3/simulator-single-main-593fe3-kserve-7d5755f66f-dcztx"
Nov 02 13:52:36.440616 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:52:36.440580    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c0d5da9-b867-44cf-a3a3-facb9793683b-dshm\") pod \"simulator-single-main-593fe3-kserve-7d5755f66f-dcztx\" (UID: \"1c0d5da9-b867-44cf-a3a3-facb9793683b\") " pod="e2e-models-e2e-worker-single-main-593fe3/simulator-single-main-593fe3-kserve-7d5755f66f-dcztx"
Nov 02 13:52:36.440838 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:52:36.440750    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2mhd\" (UniqueName: \"kubernetes.io/projected/1c0d5da9-b867-44cf-a3a3-facb9793683b-kube-api-access-s2mhd\") pod \"simulator-single-main-593fe3-kserve-7d5755f66f-dcztx\" (UID: \"1c0d5da9-b867-44cf-a3a3-facb9793683b\") " pod="e2e-models-e2e-worker-single-main-593fe3/simulator-single-main-593fe3-kserve-7d5755f66f-dcztx"
Nov 02 13:52:36.440923 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:52:36.440896    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c0d5da9-b867-44cf-a3a3-facb9793683b-model-cache\") pod \"simulator-single-main-593fe3-kserve-7d5755f66f-dcztx\" (UID: \"1c0d5da9-b867-44cf-a3a3-facb9793683b\") " pod="e2e-models-e2e-worker-single-main-593fe3/simulator-single-main-593fe3-kserve-7d5755f66f-dcztx"
Nov 02 13:52:36.441107 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:52:36.441085    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c0d5da9-b867-44cf-a3a3-facb9793683b-tls-certs\") pod \"simulator-single-main-593fe3-kserve-7d5755f66f-dcztx\" (UID: \"1c0d5da9-b867-44cf-a3a3-facb9793683b\") " pod="e2e-models-e2e-worker-single-main-593fe3/simulator-single-main-593fe3-kserve-7d5755f66f-dcztx"
Nov 02 13:52:36.441181 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:52:36.441147    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c0d5da9-b867-44cf-a3a3-facb9793683b-tmp-dir\") pod \"simulator-single-main-593fe3-kserve-7d5755f66f-dcztx\" (UID: \"1c0d5da9-b867-44cf-a3a3-facb9793683b\") " pod="e2e-models-e2e-worker-single-main-593fe3/simulator-single-main-593fe3-kserve-7d5755f66f-dcztx"
Nov 02 13:52:36.441181 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:52:36.441172    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c0d5da9-b867-44cf-a3a3-facb9793683b-home\") pod \"simulator-single-main-593fe3-kserve-7d5755f66f-dcztx\" (UID: \"1c0d5da9-b867-44cf-a3a3-facb9793683b\") " pod="e2e-models-e2e-worker-single-main-593fe3/simulator-single-main-593fe3-kserve-7d5755f66f-dcztx"
Nov 02 13:52:36.441300 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:52:36.441280    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c0d5da9-b867-44cf-a3a3-facb9793683b-model-cache\") pod \"simulator-single-main-593fe3-kserve-7d5755f66f-dcztx\" (UID: \"1c0d5da9-b867-44cf-a3a3-facb9793683b\") " pod="e2e-models-e2e-worker-single-main-593fe3/simulator-single-main-593fe3-kserve-7d5755f66f-dcztx"
Nov 02 13:52:36.441444 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:52:36.441429    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c0d5da9-b867-44cf-a3a3-facb9793683b-tmp-dir\") pod \"simulator-single-main-593fe3-kserve-7d5755f66f-dcztx\" (UID: \"1c0d5da9-b867-44cf-a3a3-facb9793683b\") " pod="e2e-models-e2e-worker-single-main-593fe3/simulator-single-main-593fe3-kserve-7d5755f66f-dcztx"
Nov 02 13:52:36.441484 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:52:36.441468    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c0d5da9-b867-44cf-a3a3-facb9793683b-home\") pod \"simulator-single-main-593fe3-kserve-7d5755f66f-dcztx\" (UID: \"1c0d5da9-b867-44cf-a3a3-facb9793683b\") " pod="e2e-models-e2e-worker-single-main-593fe3/simulator-single-main-593fe3-kserve-7d5755f66f-dcztx"
Nov 02 13:52:36.443011 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:52:36.442987    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c0d5da9-b867-44cf-a3a3-facb9793683b-dshm\") pod \"simulator-single-main-593fe3-kserve-7d5755f66f-dcztx\" (UID: \"1c0d5da9-b867-44cf-a3a3-facb9793683b\") " pod="e2e-models-e2e-worker-single-main-593fe3/simulator-single-main-593fe3-kserve-7d5755f66f-dcztx"
Nov 02 13:52:36.443572 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:52:36.443554    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c0d5da9-b867-44cf-a3a3-facb9793683b-tls-certs\") pod \"simulator-single-main-593fe3-kserve-7d5755f66f-dcztx\" (UID: \"1c0d5da9-b867-44cf-a3a3-facb9793683b\") " pod="e2e-models-e2e-worker-single-main-593fe3/simulator-single-main-593fe3-kserve-7d5755f66f-dcztx"
Nov 02 13:52:36.519217 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:52:36.519179    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2mhd\" (UniqueName: \"kubernetes.io/projected/1c0d5da9-b867-44cf-a3a3-facb9793683b-kube-api-access-s2mhd\") pod \"simulator-single-main-593fe3-kserve-7d5755f66f-dcztx\" (UID: \"1c0d5da9-b867-44cf-a3a3-facb9793683b\") " pod="e2e-models-e2e-worker-single-main-593fe3/simulator-single-main-593fe3-kserve-7d5755f66f-dcztx"
Nov 02 13:52:36.562758 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:52:36.562717    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-593fe3/simulator-single-main-593fe3-kserve-7d5755f66f-dcztx"
Nov 02 13:52:36.909212 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:52:36.909132    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-593fe3/simulator-single-main-593fe3-kserve-7d5755f66f-dcztx"]
Nov 02 13:52:36.909481 ip-10-0-128-132 kubenswrapper[4559]: W1102 13:52:36.909454    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1c0d5da9_b867_44cf_a3a3_facb9793683b.slice/crio-6c96c66e43690133fc8e25ee1ac99d5978c8fa278713014a42616e432117f80e WatchSource:0}: Error finding container 6c96c66e43690133fc8e25ee1ac99d5978c8fa278713014a42616e432117f80e: Status 404 returned error can't find the container with id 6c96c66e43690133fc8e25ee1ac99d5978c8fa278713014a42616e432117f80e
Nov 02 13:52:37.131340 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:52:37.131300    4559 patch_prober.go:29] interesting pod/e2e-worker-single-main-593fe3-openshift-default-56579c548fcn26x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.56:15021/healthz/ready\": dial tcp 10.133.1.56:15021: connect: connection refused" start-of-body=
Nov 02 13:52:37.131534 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:52:37.131358    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-593fe3-openshift-default-56579c548fcn26x" podUID="d98abfdb-03ee-463d-b059-be87d7f9a774" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.56:15021/healthz/ready\": dial tcp 10.133.1.56:15021: connect: connection refused"
Nov 02 13:52:37.320113 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:52:37.320078    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-593fe3/simulator-single-main-593fe3-kserve-7d5755f66f-dcztx" event={"ID":"1c0d5da9-b867-44cf-a3a3-facb9793683b","Type":"ContainerStarted","Data":"af59d5a7d985896dc096d61e458d76ab7c186728cc41daf2466f9775502fe680"}
Nov 02 13:52:37.320575 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:52:37.320121    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-593fe3/simulator-single-main-593fe3-kserve-7d5755f66f-dcztx" event={"ID":"1c0d5da9-b867-44cf-a3a3-facb9793683b","Type":"ContainerStarted","Data":"6c96c66e43690133fc8e25ee1ac99d5978c8fa278713014a42616e432117f80e"}
Nov 02 13:52:37.373310 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:52:37.373257    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-593fe3/simulator-single-main-593fe3-kserve-7d5755f66f-dcztx" podStartSLOduration=1.3732420379999999 podStartE2EDuration="1.373242038s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:52:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:52:37.370301199 +0000 UTC m=+5153.625700293" watchObservedRunningTime="2026-11-02 13:52:37.373242038 +0000 UTC m=+5153.628641132"
Nov 02 13:52:37.563401 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:52:37.563363    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-593fe3/simulator-single-main-593fe3-kserve-7d5755f66f-dcztx"
Nov 02 13:52:38.131340 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:52:38.131299    4559 patch_prober.go:29] interesting pod/e2e-worker-single-main-593fe3-openshift-default-56579c548fcn26x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.56:15021/healthz/ready\": dial tcp 10.133.1.56:15021: connect: connection refused" start-of-body=
Nov 02 13:52:38.131528 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:52:38.131357    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-593fe3-openshift-default-56579c548fcn26x" podUID="d98abfdb-03ee-463d-b059-be87d7f9a774" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.56:15021/healthz/ready\": dial tcp 10.133.1.56:15021: connect: connection refused"
Nov 02 13:52:38.513522 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:52:38.513481    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"]
Nov 02 13:52:39.131516 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:52:39.131478    4559 patch_prober.go:29] interesting pod/e2e-worker-single-main-593fe3-openshift-default-56579c548fcn26x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.56:15021/healthz/ready\": dial tcp 10.133.1.56:15021: connect: connection refused" start-of-body=
Nov 02 13:52:39.131722 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:52:39.131531    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-593fe3-openshift-default-56579c548fcn26x" podUID="d98abfdb-03ee-463d-b059-be87d7f9a774" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.56:15021/healthz/ready\": dial tcp 10.133.1.56:15021: connect: connection refused"
Nov 02 13:52:40.132074 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:52:40.132034    4559 patch_prober.go:29] interesting pod/e2e-worker-single-main-593fe3-openshift-default-56579c548fcn26x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.56:15021/healthz/ready\": dial tcp 10.133.1.56:15021: connect: connection refused" start-of-body=
Nov 02 13:52:40.132581 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:52:40.132099    4559 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-593fe3-openshift-default-56579c548fcn26x" podUID="d98abfdb-03ee-463d-b059-be87d7f9a774" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.56:15021/healthz/ready\": dial tcp 10.133.1.56:15021: connect: connection refused"
Nov 02 13:52:41.135525 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:52:41.135488    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-593fe3-openshift-default-56579c548fcn26x"
Nov 02 13:52:41.136005 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:52:41.135836    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-593fe3-openshift-default-56579c548fcn26x"
Nov 02 13:52:41.136669 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:52:41.136648    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-593fe3-openshift-default-56579c548fcn26x"
Nov 02 13:52:45.085182 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:52:45.085147    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqcmj" containerName="extractor"
Nov 02 13:52:46.563346 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:52:46.563310    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-593fe3/simulator-single-main-593fe3-kserve-7d5755f66f-dcztx"
Nov 02 13:52:46.576413 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:52:46.576390    4559 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-593fe3/simulator-single-main-593fe3-kserve-7d5755f66f-dcztx"
Nov 02 13:52:47.388793 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:52:47.388761    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-593fe3/simulator-single-main-593fe3-kserve-7d5755f66f-dcztx"
Nov 02 13:52:54.138099 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:52:54.138054    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"]
Nov 02 13:52:58.041294 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:52:58.041213    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-hmknb"]
Nov 02 13:52:58.054233 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:52:58.054204    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-hmknb"]
Nov 02 13:52:59.891284 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:52:59.891246    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6974e987-7d18-48d9-88df-a35017f430e6" path="/var/lib/kubelet/pods/6974e987-7d18-48d9-88df-a35017f430e6/volumes"
Nov 02 13:53:15.085167 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:53:15.085131    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqcmj" containerName="extractor"
Nov 02 13:53:20.211358 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:53:20.211318    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"]
Nov 02 13:53:22.037078 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:53:22.037043    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-9wzwg"]
Nov 02 13:53:22.048486 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:53:22.048460    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-9wzwg"]
Nov 02 13:53:23.891655 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:53:23.891609    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b18195c-ef6c-41ba-83fb-bc46f1cb0a75" path="/var/lib/kubelet/pods/3b18195c-ef6c-41ba-83fb-bc46f1cb0a75/volumes"
Nov 02 13:53:23.939251 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:53:23.939220    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-593fe3/simulator-single-main-593fe3-kserve-7d5755f66f-dcztx"]
Nov 02 13:53:25.273082 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:53:25.273042    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-593fe3-b5847f68fvhh5t"]
Nov 02 13:53:25.322060 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:53:25.322024    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-593fe3-664c4sq4d9"]
Nov 02 13:53:26.139848 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:53:26.139813    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-593fe3-6ff88d8bbf-hndjt"]
Nov 02 13:53:26.140081 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:53:26.140054    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-593fe3-6ff88d8bbf-hndjt" podUID="b800fd1c-eda5-4828-be75-1996c79f6340" containerName="maas-api" containerID="cri-o://6e364da172522b3e27ef1403cf39d939ddfb01b42c7156c1c6d6b2302d695531" gracePeriod=30
Nov 02 13:53:26.625951 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:53:26.625924    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-593fe3-6ff88d8bbf-hndjt"
Nov 02 13:53:26.630373 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:53:26.630343    4559 generic.go:356] "Generic (PLEG): container finished" podID="b800fd1c-eda5-4828-be75-1996c79f6340" containerID="6e364da172522b3e27ef1403cf39d939ddfb01b42c7156c1c6d6b2302d695531" exitCode=0
Nov 02 13:53:26.630531 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:53:26.630378    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-593fe3-6ff88d8bbf-hndjt"
Nov 02 13:53:26.630531 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:53:26.630417    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-593fe3-6ff88d8bbf-hndjt" event={"ID":"b800fd1c-eda5-4828-be75-1996c79f6340","Type":"ContainerDied","Data":"6e364da172522b3e27ef1403cf39d939ddfb01b42c7156c1c6d6b2302d695531"}
Nov 02 13:53:26.630531 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:53:26.630469    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-593fe3-6ff88d8bbf-hndjt" event={"ID":"b800fd1c-eda5-4828-be75-1996c79f6340","Type":"ContainerDied","Data":"08c226b7df2796aed735862b912a2b5c3f44ba0f24733096c179f3541e8eb1cb"}
Nov 02 13:53:26.630531 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:53:26.630499    4559 scope.go:122] "RemoveContainer" containerID="6e364da172522b3e27ef1403cf39d939ddfb01b42c7156c1c6d6b2302d695531"
Nov 02 13:53:26.639323 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:53:26.639295    4559 scope.go:122] "RemoveContainer" containerID="6e364da172522b3e27ef1403cf39d939ddfb01b42c7156c1c6d6b2302d695531"
Nov 02 13:53:26.639594 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:53:26.639575    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e364da172522b3e27ef1403cf39d939ddfb01b42c7156c1c6d6b2302d695531\": container with ID starting with 6e364da172522b3e27ef1403cf39d939ddfb01b42c7156c1c6d6b2302d695531 not found: ID does not exist" containerID="6e364da172522b3e27ef1403cf39d939ddfb01b42c7156c1c6d6b2302d695531"
Nov 02 13:53:26.639664 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:53:26.639602    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e364da172522b3e27ef1403cf39d939ddfb01b42c7156c1c6d6b2302d695531"} err="failed to get container status \"6e364da172522b3e27ef1403cf39d939ddfb01b42c7156c1c6d6b2302d695531\": rpc error: code = NotFound desc = could not find container \"6e364da172522b3e27ef1403cf39d939ddfb01b42c7156c1c6d6b2302d695531\": container with ID starting with 6e364da172522b3e27ef1403cf39d939ddfb01b42c7156c1c6d6b2302d695531 not found: ID does not exist"
Nov 02 13:53:26.759049 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:53:26.758969    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b800fd1c-eda5-4828-be75-1996c79f6340-kube-api-access-fzjwr\" (UniqueName: \"kubernetes.io/projected/b800fd1c-eda5-4828-be75-1996c79f6340-kube-api-access-fzjwr\") pod \"b800fd1c-eda5-4828-be75-1996c79f6340\" (UID: \"b800fd1c-eda5-4828-be75-1996c79f6340\") "
Nov 02 13:53:26.759049 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:53:26.759045    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b800fd1c-eda5-4828-be75-1996c79f6340-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b800fd1c-eda5-4828-be75-1996c79f6340-maas-api-tls\") pod \"b800fd1c-eda5-4828-be75-1996c79f6340\" (UID: \"b800fd1c-eda5-4828-be75-1996c79f6340\") "
Nov 02 13:53:26.759376 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:53:26.759074    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b800fd1c-eda5-4828-be75-1996c79f6340-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b800fd1c-eda5-4828-be75-1996c79f6340-openshift-service-ca\") pod \"b800fd1c-eda5-4828-be75-1996c79f6340\" (UID: \"b800fd1c-eda5-4828-be75-1996c79f6340\") "
Nov 02 13:53:26.759376 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:53:26.759148    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b800fd1c-eda5-4828-be75-1996c79f6340-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b800fd1c-eda5-4828-be75-1996c79f6340-metrics-certs\") pod \"b800fd1c-eda5-4828-be75-1996c79f6340\" (UID: \"b800fd1c-eda5-4828-be75-1996c79f6340\") "
Nov 02 13:53:26.759527 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:53:26.759499    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b800fd1c-eda5-4828-be75-1996c79f6340-openshift-service-ca" pod "b800fd1c-eda5-4828-be75-1996c79f6340" (UID: "b800fd1c-eda5-4828-be75-1996c79f6340"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:53:26.761230 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:53:26.761204    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b800fd1c-eda5-4828-be75-1996c79f6340-metrics-certs" pod "b800fd1c-eda5-4828-be75-1996c79f6340" (UID: "b800fd1c-eda5-4828-be75-1996c79f6340"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:53:26.761230 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:53:26.761214    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b800fd1c-eda5-4828-be75-1996c79f6340-kube-api-access-fzjwr" pod "b800fd1c-eda5-4828-be75-1996c79f6340" (UID: "b800fd1c-eda5-4828-be75-1996c79f6340"). InnerVolumeSpecName "kube-api-access-fzjwr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:53:26.761402 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:53:26.761222    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b800fd1c-eda5-4828-be75-1996c79f6340-maas-api-tls" pod "b800fd1c-eda5-4828-be75-1996c79f6340" (UID: "b800fd1c-eda5-4828-be75-1996c79f6340"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:53:26.860399 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:53:26.860370    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b800fd1c-eda5-4828-be75-1996c79f6340-maas-api-tls\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:53:26.860399 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:53:26.860394    4559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b800fd1c-eda5-4828-be75-1996c79f6340-openshift-service-ca\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:53:26.860562 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:53:26.860407    4559 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b800fd1c-eda5-4828-be75-1996c79f6340-metrics-certs\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:53:26.860562 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:53:26.860419    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fzjwr\" (UniqueName: \"kubernetes.io/projected/b800fd1c-eda5-4828-be75-1996c79f6340-kube-api-access-fzjwr\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:53:26.980661 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:53:26.980623    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-593fe3-6ff88d8bbf-hndjt"]
Nov 02 13:53:26.993284 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:53:26.993260    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-593fe3-6ff88d8bbf-hndjt"]
Nov 02 13:53:27.891892 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:53:27.891850    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b800fd1c-eda5-4828-be75-1996c79f6340" path="/var/lib/kubelet/pods/b800fd1c-eda5-4828-be75-1996c79f6340/volumes"
Nov 02 13:53:28.202366 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:53:28.202327    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"]
Nov 02 13:53:30.274418 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:53:30.274335    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-593fe3-b5847f68fvhh5t" podUID="b6b47f8f-149a-4511-8d45-dee306e485d3" containerName="payload-processing" containerID="cri-o://8e3c89fb67e5c21f3af1d2142677e9bf564cd72d9ac413d34811eaa8e1336c29" gracePeriod=25
Nov 02 13:53:30.323287 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:53:30.323209    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-593fe3-664c4sq4d9" podUID="a953b628-392e-48e9-a256-f63bc2ae654a" containerName="payload-pre-processing" containerID="cri-o://3d43de6fd8f31499a503441838b93983ffc44798564bfe75420829206bc88117" gracePeriod=25
Nov 02 13:53:33.212170 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:53:33.212129    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-593fe3-b5847f68fvhh5t" podUID="b6b47f8f-149a-4511-8d45-dee306e485d3" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 13:53:33.212560 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:53:33.212197    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-593fe3-664c4sq4d9" podUID="a953b628-392e-48e9-a256-f63bc2ae654a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 13:53:38.959225 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:53:38.959188    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-593fe3/simulator-single-main-593fe3-kserve-7d5755f66f-dcztx" podUID="1c0d5da9-b867-44cf-a3a3-facb9793683b" containerName="main" containerID="cri-o://af59d5a7d985896dc096d61e458d76ab7c186728cc41daf2466f9775502fe680" gracePeriod=45
Nov 02 13:53:39.208324 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:53:39.208293    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-593fe3/simulator-single-main-593fe3-kserve-7d5755f66f-dcztx"
Nov 02 13:53:39.374857 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:53:39.374778    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c0d5da9-b867-44cf-a3a3-facb9793683b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c0d5da9-b867-44cf-a3a3-facb9793683b-dshm\") pod \"1c0d5da9-b867-44cf-a3a3-facb9793683b\" (UID: \"1c0d5da9-b867-44cf-a3a3-facb9793683b\") "
Nov 02 13:53:39.374857 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:53:39.374825    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c0d5da9-b867-44cf-a3a3-facb9793683b-home\" (UniqueName: \"kubernetes.io/empty-dir/1c0d5da9-b867-44cf-a3a3-facb9793683b-home\") pod \"1c0d5da9-b867-44cf-a3a3-facb9793683b\" (UID: \"1c0d5da9-b867-44cf-a3a3-facb9793683b\") "
Nov 02 13:53:39.374857 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:53:39.374844    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c0d5da9-b867-44cf-a3a3-facb9793683b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c0d5da9-b867-44cf-a3a3-facb9793683b-model-cache\") pod \"1c0d5da9-b867-44cf-a3a3-facb9793683b\" (UID: \"1c0d5da9-b867-44cf-a3a3-facb9793683b\") "
Nov 02 13:53:39.375159 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:53:39.374891    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c0d5da9-b867-44cf-a3a3-facb9793683b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c0d5da9-b867-44cf-a3a3-facb9793683b-tmp-dir\") pod \"1c0d5da9-b867-44cf-a3a3-facb9793683b\" (UID: \"1c0d5da9-b867-44cf-a3a3-facb9793683b\") "
Nov 02 13:53:39.375159 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:53:39.374935    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c0d5da9-b867-44cf-a3a3-facb9793683b-kube-api-access-s2mhd\" (UniqueName: \"kubernetes.io/projected/1c0d5da9-b867-44cf-a3a3-facb9793683b-kube-api-access-s2mhd\") pod \"1c0d5da9-b867-44cf-a3a3-facb9793683b\" (UID: \"1c0d5da9-b867-44cf-a3a3-facb9793683b\") "
Nov 02 13:53:39.375159 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:53:39.375004    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1c0d5da9-b867-44cf-a3a3-facb9793683b-tls-certs\" (UniqueName: \"kubernetes.io/secret/1c0d5da9-b867-44cf-a3a3-facb9793683b-tls-certs\") pod \"1c0d5da9-b867-44cf-a3a3-facb9793683b\" (UID: \"1c0d5da9-b867-44cf-a3a3-facb9793683b\") "
Nov 02 13:53:39.375159 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:53:39.375104    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c0d5da9-b867-44cf-a3a3-facb9793683b-home" pod "1c0d5da9-b867-44cf-a3a3-facb9793683b" (UID: "1c0d5da9-b867-44cf-a3a3-facb9793683b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:53:39.375349 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:53:39.375200    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c0d5da9-b867-44cf-a3a3-facb9793683b-model-cache" pod "1c0d5da9-b867-44cf-a3a3-facb9793683b" (UID: "1c0d5da9-b867-44cf-a3a3-facb9793683b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:53:39.375349 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:53:39.375216    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c0d5da9-b867-44cf-a3a3-facb9793683b-tmp-dir" pod "1c0d5da9-b867-44cf-a3a3-facb9793683b" (UID: "1c0d5da9-b867-44cf-a3a3-facb9793683b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:53:39.375472 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:53:39.375408    4559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c0d5da9-b867-44cf-a3a3-facb9793683b-home\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:53:39.375472 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:53:39.375422    4559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c0d5da9-b867-44cf-a3a3-facb9793683b-model-cache\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:53:39.375472 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:53:39.375431    4559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c0d5da9-b867-44cf-a3a3-facb9793683b-tmp-dir\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:53:39.377084 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:53:39.377056    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c0d5da9-b867-44cf-a3a3-facb9793683b-dshm" pod "1c0d5da9-b867-44cf-a3a3-facb9793683b" (UID: "1c0d5da9-b867-44cf-a3a3-facb9793683b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:53:39.377207 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:53:39.377098    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c0d5da9-b867-44cf-a3a3-facb9793683b-kube-api-access-s2mhd" pod "1c0d5da9-b867-44cf-a3a3-facb9793683b" (UID: "1c0d5da9-b867-44cf-a3a3-facb9793683b"). InnerVolumeSpecName "kube-api-access-s2mhd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:53:39.377207 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:53:39.377166    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c0d5da9-b867-44cf-a3a3-facb9793683b-tls-certs" pod "1c0d5da9-b867-44cf-a3a3-facb9793683b" (UID: "1c0d5da9-b867-44cf-a3a3-facb9793683b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:53:39.476888 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:53:39.476848    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s2mhd\" (UniqueName: \"kubernetes.io/projected/1c0d5da9-b867-44cf-a3a3-facb9793683b-kube-api-access-s2mhd\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:53:39.476888 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:53:39.476886    4559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c0d5da9-b867-44cf-a3a3-facb9793683b-tls-certs\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:53:39.476888 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:53:39.476896    4559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c0d5da9-b867-44cf-a3a3-facb9793683b-dshm\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:53:39.697633 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:53:39.697599    4559 generic.go:356] "Generic (PLEG): container finished" podID="1c0d5da9-b867-44cf-a3a3-facb9793683b" containerID="af59d5a7d985896dc096d61e458d76ab7c186728cc41daf2466f9775502fe680" exitCode=0
Nov 02 13:53:39.697807 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:53:39.697678    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-593fe3/simulator-single-main-593fe3-kserve-7d5755f66f-dcztx"
Nov 02 13:53:39.697807 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:53:39.697682    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-593fe3/simulator-single-main-593fe3-kserve-7d5755f66f-dcztx" event={"ID":"1c0d5da9-b867-44cf-a3a3-facb9793683b","Type":"ContainerDied","Data":"af59d5a7d985896dc096d61e458d76ab7c186728cc41daf2466f9775502fe680"}
Nov 02 13:53:39.697807 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:53:39.697794    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-593fe3/simulator-single-main-593fe3-kserve-7d5755f66f-dcztx" event={"ID":"1c0d5da9-b867-44cf-a3a3-facb9793683b","Type":"ContainerDied","Data":"6c96c66e43690133fc8e25ee1ac99d5978c8fa278713014a42616e432117f80e"}
Nov 02 13:53:39.698005 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:53:39.697812    4559 scope.go:122] "RemoveContainer" containerID="af59d5a7d985896dc096d61e458d76ab7c186728cc41daf2466f9775502fe680"
Nov 02 13:53:39.707084 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:53:39.707066    4559 scope.go:122] "RemoveContainer" containerID="af59d5a7d985896dc096d61e458d76ab7c186728cc41daf2466f9775502fe680"
Nov 02 13:53:39.707377 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:53:39.707354    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af59d5a7d985896dc096d61e458d76ab7c186728cc41daf2466f9775502fe680\": container with ID starting with af59d5a7d985896dc096d61e458d76ab7c186728cc41daf2466f9775502fe680 not found: ID does not exist" containerID="af59d5a7d985896dc096d61e458d76ab7c186728cc41daf2466f9775502fe680"
Nov 02 13:53:39.707496 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:53:39.707389    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af59d5a7d985896dc096d61e458d76ab7c186728cc41daf2466f9775502fe680"} err="failed to get container status \"af59d5a7d985896dc096d61e458d76ab7c186728cc41daf2466f9775502fe680\": rpc error: code = NotFound desc = could not find container \"af59d5a7d985896dc096d61e458d76ab7c186728cc41daf2466f9775502fe680\": container with ID starting with af59d5a7d985896dc096d61e458d76ab7c186728cc41daf2466f9775502fe680 not found: ID does not exist"
Nov 02 13:53:39.751147 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:53:39.751124    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-593fe3/simulator-single-main-593fe3-kserve-7d5755f66f-dcztx"]
Nov 02 13:53:39.765234 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:53:39.765213    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-593fe3/simulator-single-main-593fe3-kserve-7d5755f66f-dcztx"]
Nov 02 13:53:39.891144 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:53:39.891107    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c0d5da9-b867-44cf-a3a3-facb9793683b" path="/var/lib/kubelet/pods/1c0d5da9-b867-44cf-a3a3-facb9793683b/volumes"
Nov 02 13:53:41.031713 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:53:41.031670    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-dpt95"]
Nov 02 13:53:41.044096 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:53:41.044069    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-dpt95"]
Nov 02 13:53:41.891768 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:53:41.891739    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f8f5849-a389-40e9-86e3-1697da21291d" path="/var/lib/kubelet/pods/6f8f5849-a389-40e9-86e3-1697da21291d/volumes"
Nov 02 13:53:43.212115 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:53:43.212073    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-593fe3-b5847f68fvhh5t" podUID="b6b47f8f-149a-4511-8d45-dee306e485d3" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 13:53:43.212115 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:53:43.212109    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-593fe3-664c4sq4d9" podUID="a953b628-392e-48e9-a256-f63bc2ae654a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 13:53:45.084893 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:53:45.084859    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqcmj" containerName="extractor"
Nov 02 13:53:50.559431 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:53:50.559402    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-593fe3-b5847f68fvhh5t"
Nov 02 13:53:50.589650 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:53:50.589618    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-593fe3-664c4sq4d9"
Nov 02 13:53:50.667531 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:53:50.667502    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6b47f8f-149a-4511-8d45-dee306e485d3-kube-api-access-k79k4\" (UniqueName: \"kubernetes.io/projected/b6b47f8f-149a-4511-8d45-dee306e485d3-kube-api-access-k79k4\") pod \"b6b47f8f-149a-4511-8d45-dee306e485d3\" (UID: \"b6b47f8f-149a-4511-8d45-dee306e485d3\") "
Nov 02 13:53:50.667731 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:53:50.667695    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a953b628-392e-48e9-a256-f63bc2ae654a-kube-api-access-kvzb7\" (UniqueName: \"kubernetes.io/projected/a953b628-392e-48e9-a256-f63bc2ae654a-kube-api-access-kvzb7\") pod \"a953b628-392e-48e9-a256-f63bc2ae654a\" (UID: \"a953b628-392e-48e9-a256-f63bc2ae654a\") "
Nov 02 13:53:50.667826 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:53:50.667743    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6b47f8f-149a-4511-8d45-dee306e485d3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6b47f8f-149a-4511-8d45-dee306e485d3-plugins-config-volume\") pod \"b6b47f8f-149a-4511-8d45-dee306e485d3\" (UID: \"b6b47f8f-149a-4511-8d45-dee306e485d3\") "
Nov 02 13:53:50.667826 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:53:50.667776    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a953b628-392e-48e9-a256-f63bc2ae654a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a953b628-392e-48e9-a256-f63bc2ae654a-plugins-config-volume\") pod \"a953b628-392e-48e9-a256-f63bc2ae654a\" (UID: \"a953b628-392e-48e9-a256-f63bc2ae654a\") "
Nov 02 13:53:50.668163 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:53:50.668132    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6b47f8f-149a-4511-8d45-dee306e485d3-plugins-config-volume" pod "b6b47f8f-149a-4511-8d45-dee306e485d3" (UID: "b6b47f8f-149a-4511-8d45-dee306e485d3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:53:50.668286 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:53:50.668178    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a953b628-392e-48e9-a256-f63bc2ae654a-plugins-config-volume" pod "a953b628-392e-48e9-a256-f63bc2ae654a" (UID: "a953b628-392e-48e9-a256-f63bc2ae654a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:53:50.669701 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:53:50.669678    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6b47f8f-149a-4511-8d45-dee306e485d3-kube-api-access-k79k4" pod "b6b47f8f-149a-4511-8d45-dee306e485d3" (UID: "b6b47f8f-149a-4511-8d45-dee306e485d3"). InnerVolumeSpecName "kube-api-access-k79k4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:53:50.669913 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:53:50.669890    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a953b628-392e-48e9-a256-f63bc2ae654a-kube-api-access-kvzb7" pod "a953b628-392e-48e9-a256-f63bc2ae654a" (UID: "a953b628-392e-48e9-a256-f63bc2ae654a"). InnerVolumeSpecName "kube-api-access-kvzb7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:53:50.755929 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:53:50.755906    4559 generic.go:356] "Generic (PLEG): container finished" podID="b6b47f8f-149a-4511-8d45-dee306e485d3" containerID="8e3c89fb67e5c21f3af1d2142677e9bf564cd72d9ac413d34811eaa8e1336c29" exitCode=0
Nov 02 13:53:50.756056 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:53:50.755944    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-593fe3-b5847f68fvhh5t" event={"ID":"b6b47f8f-149a-4511-8d45-dee306e485d3","Type":"ContainerDied","Data":"8e3c89fb67e5c21f3af1d2142677e9bf564cd72d9ac413d34811eaa8e1336c29"}
Nov 02 13:53:50.756056 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:53:50.755963    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-593fe3-b5847f68fvhh5t"
Nov 02 13:53:50.756056 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:53:50.756001    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-593fe3-b5847f68fvhh5t" event={"ID":"b6b47f8f-149a-4511-8d45-dee306e485d3","Type":"ContainerDied","Data":"c733dd656d61469e2f78e7d8500786c3cc2b940feef079163c4c9b2d933d9173"}
Nov 02 13:53:50.756056 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:53:50.756029    4559 scope.go:122] "RemoveContainer" containerID="8e3c89fb67e5c21f3af1d2142677e9bf564cd72d9ac413d34811eaa8e1336c29"
Nov 02 13:53:50.757576 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:53:50.757549    4559 generic.go:356] "Generic (PLEG): container finished" podID="a953b628-392e-48e9-a256-f63bc2ae654a" containerID="3d43de6fd8f31499a503441838b93983ffc44798564bfe75420829206bc88117" exitCode=0
Nov 02 13:53:50.757740 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:53:50.757593    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-593fe3-664c4sq4d9"
Nov 02 13:53:50.757740 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:53:50.757672    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-593fe3-664c4sq4d9" event={"ID":"a953b628-392e-48e9-a256-f63bc2ae654a","Type":"ContainerDied","Data":"3d43de6fd8f31499a503441838b93983ffc44798564bfe75420829206bc88117"}
Nov 02 13:53:50.757740 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:53:50.757721    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-593fe3-664c4sq4d9" event={"ID":"a953b628-392e-48e9-a256-f63bc2ae654a","Type":"ContainerDied","Data":"7519726978d6e8bd6d94024711a9d78254fc9b2c1819d3e8ad791867988a7cae"}
Nov 02 13:53:50.765844 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:53:50.765824    4559 scope.go:122] "RemoveContainer" containerID="8e3c89fb67e5c21f3af1d2142677e9bf564cd72d9ac413d34811eaa8e1336c29"
Nov 02 13:53:50.766581 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:53:50.766445    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e3c89fb67e5c21f3af1d2142677e9bf564cd72d9ac413d34811eaa8e1336c29\": container with ID starting with 8e3c89fb67e5c21f3af1d2142677e9bf564cd72d9ac413d34811eaa8e1336c29 not found: ID does not exist" containerID="8e3c89fb67e5c21f3af1d2142677e9bf564cd72d9ac413d34811eaa8e1336c29"
Nov 02 13:53:50.766581 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:53:50.766480    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e3c89fb67e5c21f3af1d2142677e9bf564cd72d9ac413d34811eaa8e1336c29"} err="failed to get container status \"8e3c89fb67e5c21f3af1d2142677e9bf564cd72d9ac413d34811eaa8e1336c29\": rpc error: code = NotFound desc = could not find container \"8e3c89fb67e5c21f3af1d2142677e9bf564cd72d9ac413d34811eaa8e1336c29\": container with ID starting with 8e3c89fb67e5c21f3af1d2142677e9bf564cd72d9ac413d34811eaa8e1336c29 not found: ID does not exist"
Nov 02 13:53:50.766581 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:53:50.766500    4559 scope.go:122] "RemoveContainer" containerID="3d43de6fd8f31499a503441838b93983ffc44798564bfe75420829206bc88117"
Nov 02 13:53:50.770905 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:53:50.769002    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k79k4\" (UniqueName: \"kubernetes.io/projected/b6b47f8f-149a-4511-8d45-dee306e485d3-kube-api-access-k79k4\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:53:50.770905 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:53:50.769026    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kvzb7\" (UniqueName: \"kubernetes.io/projected/a953b628-392e-48e9-a256-f63bc2ae654a-kube-api-access-kvzb7\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:53:50.770905 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:53:50.769040    4559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6b47f8f-149a-4511-8d45-dee306e485d3-plugins-config-volume\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:53:50.770905 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:53:50.769055    4559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a953b628-392e-48e9-a256-f63bc2ae654a-plugins-config-volume\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:53:50.775422 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:53:50.775407    4559 scope.go:122] "RemoveContainer" containerID="3d43de6fd8f31499a503441838b93983ffc44798564bfe75420829206bc88117"
Nov 02 13:53:50.775764 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:53:50.775745    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d43de6fd8f31499a503441838b93983ffc44798564bfe75420829206bc88117\": container with ID starting with 3d43de6fd8f31499a503441838b93983ffc44798564bfe75420829206bc88117 not found: ID does not exist" containerID="3d43de6fd8f31499a503441838b93983ffc44798564bfe75420829206bc88117"
Nov 02 13:53:50.775824 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:53:50.775771    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d43de6fd8f31499a503441838b93983ffc44798564bfe75420829206bc88117"} err="failed to get container status \"3d43de6fd8f31499a503441838b93983ffc44798564bfe75420829206bc88117\": rpc error: code = NotFound desc = could not find container \"3d43de6fd8f31499a503441838b93983ffc44798564bfe75420829206bc88117\": container with ID starting with 3d43de6fd8f31499a503441838b93983ffc44798564bfe75420829206bc88117 not found: ID does not exist"
Nov 02 13:53:50.810289 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:53:50.810238    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-593fe3-b5847f68fvhh5t"]
Nov 02 13:53:50.816138 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:53:50.816113    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-593fe3-b5847f68fvhh5t"]
Nov 02 13:53:50.853134 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:53:50.853114    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-593fe3-664c4sq4d9"]
Nov 02 13:53:50.860558 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:53:50.860539    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-593fe3-664c4sq4d9"]
Nov 02 13:53:51.891109 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:53:51.891075    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a953b628-392e-48e9-a256-f63bc2ae654a" path="/var/lib/kubelet/pods/a953b628-392e-48e9-a256-f63bc2ae654a/volumes"
Nov 02 13:53:51.891573 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:53:51.891429    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6b47f8f-149a-4511-8d45-dee306e485d3" path="/var/lib/kubelet/pods/b6b47f8f-149a-4511-8d45-dee306e485d3/volumes"
Nov 02 13:53:52.032033 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:53:52.031994    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-2hnrw"]
Nov 02 13:53:52.044496 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:53:52.044466    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-2hnrw"]
Nov 02 13:53:53.892257 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:53:53.892226    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce8b223b-090f-4bb7-ad38-deddcc9f6dd6" path="/var/lib/kubelet/pods/ce8b223b-090f-4bb7-ad38-deddcc9f6dd6/volumes"
Nov 02 13:53:58.279369 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:53:58.279329    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-593fe3-6ff88d8bbf-std2b"]
Nov 02 13:53:58.279779 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:53:58.279714    4559 cpu_manager.go:414] "containerMap: removing container" podUID="b6b47f8f-149a-4511-8d45-dee306e485d3" containerName="payload-processing"
Nov 02 13:53:58.279779 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:53:58.279726    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6b47f8f-149a-4511-8d45-dee306e485d3" containerName="payload-processing"
Nov 02 13:53:58.279779 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:53:58.279741    4559 cpu_manager.go:414] "containerMap: removing container" podUID="a953b628-392e-48e9-a256-f63bc2ae654a" containerName="payload-pre-processing"
Nov 02 13:53:58.279779 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:53:58.279748    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a953b628-392e-48e9-a256-f63bc2ae654a" containerName="payload-pre-processing"
Nov 02 13:53:58.279779 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:53:58.279761    4559 cpu_manager.go:414] "containerMap: removing container" podUID="b800fd1c-eda5-4828-be75-1996c79f6340" containerName="maas-api"
Nov 02 13:53:58.279779 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:53:58.279765    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b800fd1c-eda5-4828-be75-1996c79f6340" containerName="maas-api"
Nov 02 13:53:58.279779 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:53:58.279778    4559 cpu_manager.go:414] "containerMap: removing container" podUID="1c0d5da9-b867-44cf-a3a3-facb9793683b" containerName="main"
Nov 02 13:53:58.279779 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:53:58.279782    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c0d5da9-b867-44cf-a3a3-facb9793683b" containerName="main"
Nov 02 13:53:58.280051 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:53:58.279839    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6b47f8f-149a-4511-8d45-dee306e485d3" containerName="payload-processing"
Nov 02 13:53:58.280051 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:53:58.279848    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="a953b628-392e-48e9-a256-f63bc2ae654a" containerName="payload-pre-processing"
Nov 02 13:53:58.280051 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:53:58.279855    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="b800fd1c-eda5-4828-be75-1996c79f6340" containerName="maas-api"
Nov 02 13:53:58.280051 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:53:58.279863    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c0d5da9-b867-44cf-a3a3-facb9793683b" containerName="main"
Nov 02 13:53:58.284306 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:53:58.284285    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-593fe3-6ff88d8bbf-std2b"
Nov 02 13:53:58.310142 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:53:58.310106    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 13:53:58.310142 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:53:58.310114    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 13:53:58.320338 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:53:58.320284    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Nov 02 13:53:58.320338 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:53:58.320305    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-593fe3-serving-cert\""
Nov 02 13:53:58.320338 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:53:58.320285    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-sr5s9\""
Nov 02 13:53:58.323869 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:53:58.323846    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-593fe3-6ff88d8bbf-std2b"]
Nov 02 13:53:58.334010 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:53:58.333985    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a3e6dd6-12bb-41c5-b53d-79bd4394b269-metrics-certs\") pod \"maas-api-e2e-worker-single-main-593fe3-6ff88d8bbf-std2b\" (UID: \"3a3e6dd6-12bb-41c5-b53d-79bd4394b269\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-593fe3-6ff88d8bbf-std2b"
Nov 02 13:53:58.334155 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:53:58.334123    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9w858\" (UniqueName: \"kubernetes.io/projected/3a3e6dd6-12bb-41c5-b53d-79bd4394b269-kube-api-access-9w858\") pod \"maas-api-e2e-worker-single-main-593fe3-6ff88d8bbf-std2b\" (UID: \"3a3e6dd6-12bb-41c5-b53d-79bd4394b269\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-593fe3-6ff88d8bbf-std2b"
Nov 02 13:53:58.334356 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:53:58.334330    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a3e6dd6-12bb-41c5-b53d-79bd4394b269-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-593fe3-6ff88d8bbf-std2b\" (UID: \"3a3e6dd6-12bb-41c5-b53d-79bd4394b269\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-593fe3-6ff88d8bbf-std2b"
Nov 02 13:53:58.334802 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:53:58.334403    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a3e6dd6-12bb-41c5-b53d-79bd4394b269-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-593fe3-6ff88d8bbf-std2b\" (UID: \"3a3e6dd6-12bb-41c5-b53d-79bd4394b269\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-593fe3-6ff88d8bbf-std2b"
Nov 02 13:53:58.436113 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:53:58.436084    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a3e6dd6-12bb-41c5-b53d-79bd4394b269-metrics-certs\") pod \"maas-api-e2e-worker-single-main-593fe3-6ff88d8bbf-std2b\" (UID: \"3a3e6dd6-12bb-41c5-b53d-79bd4394b269\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-593fe3-6ff88d8bbf-std2b"
Nov 02 13:53:58.436262 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:53:58.436125    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9w858\" (UniqueName: \"kubernetes.io/projected/3a3e6dd6-12bb-41c5-b53d-79bd4394b269-kube-api-access-9w858\") pod \"maas-api-e2e-worker-single-main-593fe3-6ff88d8bbf-std2b\" (UID: \"3a3e6dd6-12bb-41c5-b53d-79bd4394b269\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-593fe3-6ff88d8bbf-std2b"
Nov 02 13:53:58.436262 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:53:58.436147    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a3e6dd6-12bb-41c5-b53d-79bd4394b269-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-593fe3-6ff88d8bbf-std2b\" (UID: \"3a3e6dd6-12bb-41c5-b53d-79bd4394b269\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-593fe3-6ff88d8bbf-std2b"
Nov 02 13:53:58.436262 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:53:58.436178    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a3e6dd6-12bb-41c5-b53d-79bd4394b269-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-593fe3-6ff88d8bbf-std2b\" (UID: \"3a3e6dd6-12bb-41c5-b53d-79bd4394b269\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-593fe3-6ff88d8bbf-std2b"
Nov 02 13:53:58.436830 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:53:58.436804    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a3e6dd6-12bb-41c5-b53d-79bd4394b269-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-593fe3-6ff88d8bbf-std2b\" (UID: \"3a3e6dd6-12bb-41c5-b53d-79bd4394b269\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-593fe3-6ff88d8bbf-std2b"
Nov 02 13:53:58.438541 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:53:58.438521    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a3e6dd6-12bb-41c5-b53d-79bd4394b269-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-593fe3-6ff88d8bbf-std2b\" (UID: \"3a3e6dd6-12bb-41c5-b53d-79bd4394b269\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-593fe3-6ff88d8bbf-std2b"
Nov 02 13:53:58.438664 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:53:58.438592    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a3e6dd6-12bb-41c5-b53d-79bd4394b269-metrics-certs\") pod \"maas-api-e2e-worker-single-main-593fe3-6ff88d8bbf-std2b\" (UID: \"3a3e6dd6-12bb-41c5-b53d-79bd4394b269\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-593fe3-6ff88d8bbf-std2b"
Nov 02 13:53:58.467740 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:53:58.467718    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9w858\" (UniqueName: \"kubernetes.io/projected/3a3e6dd6-12bb-41c5-b53d-79bd4394b269-kube-api-access-9w858\") pod \"maas-api-e2e-worker-single-main-593fe3-6ff88d8bbf-std2b\" (UID: \"3a3e6dd6-12bb-41c5-b53d-79bd4394b269\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-593fe3-6ff88d8bbf-std2b"
Nov 02 13:53:58.597823 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:53:58.597747    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-593fe3-6ff88d8bbf-std2b"
Nov 02 13:53:58.837432 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:53:58.837404    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-593fe3-664c458jbc"]
Nov 02 13:53:58.841266 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:53:58.841246    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-593fe3-664c458jbc"
Nov 02 13:53:58.876986 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:53:58.876965    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-593fe3\""
Nov 02 13:53:58.878839 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:53:58.878814    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-f2c1c28a-dockercfg-pw7rt\""
Nov 02 13:53:58.879130 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:53:58.879105    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-593fe3-664c458jbc"]
Nov 02 13:53:58.907971 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:53:58.907944    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-593fe3-b5847f68fd8795"]
Nov 02 13:53:58.911817 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:53:58.911802    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-593fe3-b5847f68fd8795"
Nov 02 13:53:58.919359 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:53:58.918910    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-593fe3-6ff88d8bbf-std2b"]
Nov 02 13:53:58.920222 ip-10-0-128-132 kubenswrapper[4559]: W1102 13:53:58.920201    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a3e6dd6_12bb_41c5_b53d_79bd4394b269.slice/crio-788b606841d6338d39739fd1680390224afba5d95fbe73b52d866c1b7e9db071 WatchSource:0}: Error finding container 788b606841d6338d39739fd1680390224afba5d95fbe73b52d866c1b7e9db071: Status 404 returned error can't find the container with id 788b606841d6338d39739fd1680390224afba5d95fbe73b52d866c1b7e9db071
Nov 02 13:53:58.942527 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:53:58.942422    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8bfcc94b-a517-483d-a6d1-ee5fe682b119-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-593fe3-664c458jbc\" (UID: \"8bfcc94b-a517-483d-a6d1-ee5fe682b119\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-593fe3-664c458jbc"
Nov 02 13:53:58.942527 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:53:58.942471    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qstnt\" (UniqueName: \"kubernetes.io/projected/8bfcc94b-a517-483d-a6d1-ee5fe682b119-kube-api-access-qstnt\") pod \"payload-pre-processing-e2e-worker-single-main-593fe3-664c458jbc\" (UID: \"8bfcc94b-a517-483d-a6d1-ee5fe682b119\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-593fe3-664c458jbc"
Nov 02 13:53:58.943175 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:53:58.943100    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9zss\" (UniqueName: \"kubernetes.io/projected/0cd959d9-52db-4e4e-a874-0811c8178ca8-kube-api-access-g9zss\") pod \"payload-processing-e2e-worker-single-main-593fe3-b5847f68fd8795\" (UID: \"0cd959d9-52db-4e4e-a874-0811c8178ca8\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-593fe3-b5847f68fd8795"
Nov 02 13:53:58.943332 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:53:58.943157    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0cd959d9-52db-4e4e-a874-0811c8178ca8-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-593fe3-b5847f68fd8795\" (UID: \"0cd959d9-52db-4e4e-a874-0811c8178ca8\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-593fe3-b5847f68fd8795"
Nov 02 13:53:58.948501 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:53:58.948474    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-593fe3-b5847f68fd8795"]
Nov 02 13:53:59.044512 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:53:59.044484    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8bfcc94b-a517-483d-a6d1-ee5fe682b119-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-593fe3-664c458jbc\" (UID: \"8bfcc94b-a517-483d-a6d1-ee5fe682b119\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-593fe3-664c458jbc"
Nov 02 13:53:59.044512 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:53:59.044517    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qstnt\" (UniqueName: \"kubernetes.io/projected/8bfcc94b-a517-483d-a6d1-ee5fe682b119-kube-api-access-qstnt\") pod \"payload-pre-processing-e2e-worker-single-main-593fe3-664c458jbc\" (UID: \"8bfcc94b-a517-483d-a6d1-ee5fe682b119\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-593fe3-664c458jbc"
Nov 02 13:53:59.044782 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:53:59.044548    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9zss\" (UniqueName: \"kubernetes.io/projected/0cd959d9-52db-4e4e-a874-0811c8178ca8-kube-api-access-g9zss\") pod \"payload-processing-e2e-worker-single-main-593fe3-b5847f68fd8795\" (UID: \"0cd959d9-52db-4e4e-a874-0811c8178ca8\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-593fe3-b5847f68fd8795"
Nov 02 13:53:59.044782 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:53:59.044762    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0cd959d9-52db-4e4e-a874-0811c8178ca8-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-593fe3-b5847f68fd8795\" (UID: \"0cd959d9-52db-4e4e-a874-0811c8178ca8\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-593fe3-b5847f68fd8795"
Nov 02 13:53:59.045291 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:53:59.045273    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8bfcc94b-a517-483d-a6d1-ee5fe682b119-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-593fe3-664c458jbc\" (UID: \"8bfcc94b-a517-483d-a6d1-ee5fe682b119\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-593fe3-664c458jbc"
Nov 02 13:53:59.045407 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:53:59.045389    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0cd959d9-52db-4e4e-a874-0811c8178ca8-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-593fe3-b5847f68fd8795\" (UID: \"0cd959d9-52db-4e4e-a874-0811c8178ca8\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-593fe3-b5847f68fd8795"
Nov 02 13:53:59.068733 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:53:59.068704    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qstnt\" (UniqueName: \"kubernetes.io/projected/8bfcc94b-a517-483d-a6d1-ee5fe682b119-kube-api-access-qstnt\") pod \"payload-pre-processing-e2e-worker-single-main-593fe3-664c458jbc\" (UID: \"8bfcc94b-a517-483d-a6d1-ee5fe682b119\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-593fe3-664c458jbc"
Nov 02 13:53:59.071329 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:53:59.071303    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9zss\" (UniqueName: \"kubernetes.io/projected/0cd959d9-52db-4e4e-a874-0811c8178ca8-kube-api-access-g9zss\") pod \"payload-processing-e2e-worker-single-main-593fe3-b5847f68fd8795\" (UID: \"0cd959d9-52db-4e4e-a874-0811c8178ca8\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-593fe3-b5847f68fd8795"
Nov 02 13:53:59.155115 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:53:59.155047    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-593fe3-664c458jbc"
Nov 02 13:53:59.226714 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:53:59.226518    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-593fe3-b5847f68fd8795"
Nov 02 13:53:59.454398 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:53:59.454372    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-593fe3-664c458jbc"]
Nov 02 13:53:59.455297 ip-10-0-128-132 kubenswrapper[4559]: W1102 13:53:59.455273    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bfcc94b_a517_483d_a6d1_ee5fe682b119.slice/crio-e38e2c7799fbab72d0fda525b0fef91e8c357648ab1ab25b78aba9ffaea5d07c WatchSource:0}: Error finding container e38e2c7799fbab72d0fda525b0fef91e8c357648ab1ab25b78aba9ffaea5d07c: Status 404 returned error can't find the container with id e38e2c7799fbab72d0fda525b0fef91e8c357648ab1ab25b78aba9ffaea5d07c
Nov 02 13:53:59.530398 ip-10-0-128-132 kubenswrapper[4559]: W1102 13:53:59.530368    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cd959d9_52db_4e4e_a874_0811c8178ca8.slice/crio-abdb7bdad6ca2190173f82a62a02affecb9a332129b27128f6981c5473ef1b0e WatchSource:0}: Error finding container abdb7bdad6ca2190173f82a62a02affecb9a332129b27128f6981c5473ef1b0e: Status 404 returned error can't find the container with id abdb7bdad6ca2190173f82a62a02affecb9a332129b27128f6981c5473ef1b0e
Nov 02 13:53:59.530765 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:53:59.530743    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-593fe3-b5847f68fd8795"]
Nov 02 13:53:59.807209 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:53:59.807126    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-593fe3-6ff88d8bbf-std2b" event={"ID":"3a3e6dd6-12bb-41c5-b53d-79bd4394b269","Type":"ContainerStarted","Data":"84297aa42a04326227b901991d881559c6e3b5d053418c5739994f9b8bb693ce"}
Nov 02 13:53:59.807387 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:53:59.807214    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-593fe3-6ff88d8bbf-std2b" event={"ID":"3a3e6dd6-12bb-41c5-b53d-79bd4394b269","Type":"ContainerStarted","Data":"788b606841d6338d39739fd1680390224afba5d95fbe73b52d866c1b7e9db071"}
Nov 02 13:53:59.807387 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:53:59.807364    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-593fe3-6ff88d8bbf-std2b"
Nov 02 13:53:59.808935 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:53:59.808905    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-593fe3-b5847f68fd8795" event={"ID":"0cd959d9-52db-4e4e-a874-0811c8178ca8","Type":"ContainerStarted","Data":"f6c8505c601c1f7b7740ca1b13b4ef3511cd0e8a303b762d546b229fdba1d1e6"}
Nov 02 13:53:59.809072 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:53:59.808956    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-593fe3-b5847f68fd8795"
Nov 02 13:53:59.809072 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:53:59.808975    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-593fe3-b5847f68fd8795" event={"ID":"0cd959d9-52db-4e4e-a874-0811c8178ca8","Type":"ContainerStarted","Data":"abdb7bdad6ca2190173f82a62a02affecb9a332129b27128f6981c5473ef1b0e"}
Nov 02 13:53:59.810507 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:53:59.810485    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-593fe3-664c458jbc" event={"ID":"8bfcc94b-a517-483d-a6d1-ee5fe682b119","Type":"ContainerStarted","Data":"64bada63d9bb6c45e27a14e0ee307ec108442337c8088c7230ff1c7f6acc55e2"}
Nov 02 13:53:59.810600 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:53:59.810532    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-593fe3-664c458jbc"
Nov 02 13:53:59.810600 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:53:59.810552    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-593fe3-664c458jbc" event={"ID":"8bfcc94b-a517-483d-a6d1-ee5fe682b119","Type":"ContainerStarted","Data":"e38e2c7799fbab72d0fda525b0fef91e8c357648ab1ab25b78aba9ffaea5d07c"}
Nov 02 13:53:59.854246 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:53:59.854207    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-593fe3-6ff88d8bbf-std2b" podStartSLOduration=1.505734494 podStartE2EDuration="1.854194711s" totalImagesPullingTime="348.460217ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:53:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:53:59.854096268 +0000 UTC m=+5236.109495362" watchObservedRunningTime="2026-11-02 13:53:59.854194711 +0000 UTC m=+5236.109593846"
Nov 02 13:53:59.897160 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:53:59.897116    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-593fe3-664c458jbc" podStartSLOduration=1.8971028479999998 podStartE2EDuration="1.897102848s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:53:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:53:59.894707816 +0000 UTC m=+5236.150106911" watchObservedRunningTime="2026-11-02 13:53:59.897102848 +0000 UTC m=+5236.152502002"
Nov 02 13:53:59.939534 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:53:59.939488    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-593fe3-b5847f68fd8795" podStartSLOduration=1.939475772 podStartE2EDuration="1.939475772s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:53:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:53:59.938307394 +0000 UTC m=+5236.193706489" watchObservedRunningTime="2026-11-02 13:53:59.939475772 +0000 UTC m=+5236.194874868"
Nov 02 13:54:02.425687 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:54:02.424283    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"]
Nov 02 13:54:05.820731 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:54:05.820700    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-593fe3-6ff88d8bbf-std2b"
Nov 02 13:54:10.818597 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:54:10.818566    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-593fe3-664c458jbc"
Nov 02 13:54:10.819041 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:54:10.818683    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-593fe3-b5847f68fd8795"
Nov 02 13:54:15.085773 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:54:15.085732    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqcmj" containerName="extractor"
Nov 02 13:54:18.115777 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:54:18.115742    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-593fe3-b5847f68fd8795"]
Nov 02 13:54:18.213131 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:54:18.213098    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-593fe3-664c458jbc"]
Nov 02 13:54:19.030330 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:54:19.030287    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-qzrs5"]
Nov 02 13:54:19.032860 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:54:19.032839    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-qzrs5"]
Nov 02 13:54:19.891354 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:54:19.891320    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb8ac514-d0c1-4520-a68c-d1559d32d439" path="/var/lib/kubelet/pods/bb8ac514-d0c1-4520-a68c-d1559d32d439/volumes"
Nov 02 13:54:20.151114 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:54:20.151036    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-593fe3-6ff88d8bbf-std2b"]
Nov 02 13:54:20.151280 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:54:20.151241    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-593fe3-6ff88d8bbf-std2b" podUID="3a3e6dd6-12bb-41c5-b53d-79bd4394b269" containerName="maas-api" containerID="cri-o://84297aa42a04326227b901991d881559c6e3b5d053418c5739994f9b8bb693ce" gracePeriod=30
Nov 02 13:54:20.405250 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:54:20.405196    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-593fe3-6ff88d8bbf-std2b"
Nov 02 13:54:20.562393 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:54:20.562358    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3a3e6dd6-12bb-41c5-b53d-79bd4394b269-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a3e6dd6-12bb-41c5-b53d-79bd4394b269-metrics-certs\") pod \"3a3e6dd6-12bb-41c5-b53d-79bd4394b269\" (UID: \"3a3e6dd6-12bb-41c5-b53d-79bd4394b269\") "
Nov 02 13:54:20.562598 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:54:20.562428    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3a3e6dd6-12bb-41c5-b53d-79bd4394b269-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a3e6dd6-12bb-41c5-b53d-79bd4394b269-maas-api-tls\") pod \"3a3e6dd6-12bb-41c5-b53d-79bd4394b269\" (UID: \"3a3e6dd6-12bb-41c5-b53d-79bd4394b269\") "
Nov 02 13:54:20.562598 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:54:20.562497    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a3e6dd6-12bb-41c5-b53d-79bd4394b269-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a3e6dd6-12bb-41c5-b53d-79bd4394b269-openshift-service-ca\") pod \"3a3e6dd6-12bb-41c5-b53d-79bd4394b269\" (UID: \"3a3e6dd6-12bb-41c5-b53d-79bd4394b269\") "
Nov 02 13:54:20.562598 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:54:20.562527    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a3e6dd6-12bb-41c5-b53d-79bd4394b269-kube-api-access-9w858\" (UniqueName: \"kubernetes.io/projected/3a3e6dd6-12bb-41c5-b53d-79bd4394b269-kube-api-access-9w858\") pod \"3a3e6dd6-12bb-41c5-b53d-79bd4394b269\" (UID: \"3a3e6dd6-12bb-41c5-b53d-79bd4394b269\") "
Nov 02 13:54:20.563327 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:54:20.563191    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a3e6dd6-12bb-41c5-b53d-79bd4394b269-openshift-service-ca" pod "3a3e6dd6-12bb-41c5-b53d-79bd4394b269" (UID: "3a3e6dd6-12bb-41c5-b53d-79bd4394b269"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:54:20.565094 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:54:20.565038    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a3e6dd6-12bb-41c5-b53d-79bd4394b269-kube-api-access-9w858" pod "3a3e6dd6-12bb-41c5-b53d-79bd4394b269" (UID: "3a3e6dd6-12bb-41c5-b53d-79bd4394b269"). InnerVolumeSpecName "kube-api-access-9w858". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:54:20.565233 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:54:20.565119    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a3e6dd6-12bb-41c5-b53d-79bd4394b269-metrics-certs" pod "3a3e6dd6-12bb-41c5-b53d-79bd4394b269" (UID: "3a3e6dd6-12bb-41c5-b53d-79bd4394b269"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:54:20.565353 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:54:20.565253    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a3e6dd6-12bb-41c5-b53d-79bd4394b269-maas-api-tls" pod "3a3e6dd6-12bb-41c5-b53d-79bd4394b269" (UID: "3a3e6dd6-12bb-41c5-b53d-79bd4394b269"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:54:20.664067 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:54:20.663991    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a3e6dd6-12bb-41c5-b53d-79bd4394b269-maas-api-tls\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:54:20.664067 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:54:20.664022    4559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a3e6dd6-12bb-41c5-b53d-79bd4394b269-openshift-service-ca\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:54:20.664067 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:54:20.664032    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9w858\" (UniqueName: \"kubernetes.io/projected/3a3e6dd6-12bb-41c5-b53d-79bd4394b269-kube-api-access-9w858\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:54:20.664067 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:54:20.664041    4559 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a3e6dd6-12bb-41c5-b53d-79bd4394b269-metrics-certs\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:54:20.917316 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:54:20.917284    4559 generic.go:356] "Generic (PLEG): container finished" podID="3a3e6dd6-12bb-41c5-b53d-79bd4394b269" containerID="84297aa42a04326227b901991d881559c6e3b5d053418c5739994f9b8bb693ce" exitCode=0
Nov 02 13:54:20.917802 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:54:20.917326    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-593fe3-6ff88d8bbf-std2b"
Nov 02 13:54:20.917802 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:54:20.917373    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-593fe3-6ff88d8bbf-std2b" event={"ID":"3a3e6dd6-12bb-41c5-b53d-79bd4394b269","Type":"ContainerDied","Data":"84297aa42a04326227b901991d881559c6e3b5d053418c5739994f9b8bb693ce"}
Nov 02 13:54:20.917802 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:54:20.917425    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-593fe3-6ff88d8bbf-std2b" event={"ID":"3a3e6dd6-12bb-41c5-b53d-79bd4394b269","Type":"ContainerDied","Data":"788b606841d6338d39739fd1680390224afba5d95fbe73b52d866c1b7e9db071"}
Nov 02 13:54:20.917802 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:54:20.917449    4559 scope.go:122] "RemoveContainer" containerID="84297aa42a04326227b901991d881559c6e3b5d053418c5739994f9b8bb693ce"
Nov 02 13:54:20.927322 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:54:20.927303    4559 scope.go:122] "RemoveContainer" containerID="84297aa42a04326227b901991d881559c6e3b5d053418c5739994f9b8bb693ce"
Nov 02 13:54:20.927605 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:54:20.927586    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84297aa42a04326227b901991d881559c6e3b5d053418c5739994f9b8bb693ce\": container with ID starting with 84297aa42a04326227b901991d881559c6e3b5d053418c5739994f9b8bb693ce not found: ID does not exist" containerID="84297aa42a04326227b901991d881559c6e3b5d053418c5739994f9b8bb693ce"
Nov 02 13:54:20.927665 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:54:20.927612    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84297aa42a04326227b901991d881559c6e3b5d053418c5739994f9b8bb693ce"} err="failed to get container status \"84297aa42a04326227b901991d881559c6e3b5d053418c5739994f9b8bb693ce\": rpc error: code = NotFound desc = could not find container \"84297aa42a04326227b901991d881559c6e3b5d053418c5739994f9b8bb693ce\": container with ID starting with 84297aa42a04326227b901991d881559c6e3b5d053418c5739994f9b8bb693ce not found: ID does not exist"
Nov 02 13:54:20.978178 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:54:20.978144    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-593fe3-6ff88d8bbf-std2b"]
Nov 02 13:54:20.981450 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:54:20.981423    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-593fe3-6ff88d8bbf-std2b"]
Nov 02 13:54:21.374273 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:54:21.374198    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-593fe3-openshift-default-56579c548fcn26x"]
Nov 02 13:54:21.374420 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:54:21.374390    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-593fe3-openshift-default-56579c548fcn26x" podUID="d98abfdb-03ee-463d-b059-be87d7f9a774" containerName="istio-proxy" containerID="cri-o://e37795122e54d58b30fd2484f926cb091ded123000e57760e3742d91661122f8" gracePeriod=30
Nov 02 13:54:21.891784 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:54:21.891753    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a3e6dd6-12bb-41c5-b53d-79bd4394b269" path="/var/lib/kubelet/pods/3a3e6dd6-12bb-41c5-b53d-79bd4394b269/volumes"
Nov 02 13:54:22.650561 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:54:22.650526    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dkvnp"]
Nov 02 13:54:22.663033 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:54:22.663003    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dkvnp"]
Nov 02 13:54:22.706423 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:54:22.706391    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8sq95"]
Nov 02 13:54:22.706964 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:54:22.706949    4559 cpu_manager.go:414] "containerMap: removing container" podUID="3a3e6dd6-12bb-41c5-b53d-79bd4394b269" containerName="maas-api"
Nov 02 13:54:22.707021 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:54:22.706966    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a3e6dd6-12bb-41c5-b53d-79bd4394b269" containerName="maas-api"
Nov 02 13:54:22.707094 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:54:22.707083    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a3e6dd6-12bb-41c5-b53d-79bd4394b269" containerName="maas-api"
Nov 02 13:54:22.711227 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:54:22.711206    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8sq95"
Nov 02 13:54:22.729203 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:54:22.729161    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 13:54:22.729347 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:54:22.729237    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-45fzn\""
Nov 02 13:54:22.730409 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:54:22.729717    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 13:54:22.730409 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:54:22.729803    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 13:54:22.731768 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:54:22.731747    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8sq95"]
Nov 02 13:54:22.886814 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:54:22.886782    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97175416-27c8-4665-8376-07a963a467c2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-8sq95\" (UID: \"97175416-27c8-4665-8376-07a963a467c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8sq95"
Nov 02 13:54:22.887030 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:54:22.886831    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-str2f\" (UniqueName: \"kubernetes.io/projected/97175416-27c8-4665-8376-07a963a467c2-kube-api-access-str2f\") pod \"maas-api-revoke-sub-3d4156f1-8sq95\" (UID: \"97175416-27c8-4665-8376-07a963a467c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8sq95"
Nov 02 13:54:22.887030 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:54:22.886932    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97175416-27c8-4665-8376-07a963a467c2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-8sq95\" (UID: \"97175416-27c8-4665-8376-07a963a467c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8sq95"
Nov 02 13:54:22.988147 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:54:22.988112    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97175416-27c8-4665-8376-07a963a467c2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-8sq95\" (UID: \"97175416-27c8-4665-8376-07a963a467c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8sq95"
Nov 02 13:54:22.988358 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:54:22.988206    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-str2f\" (UniqueName: \"kubernetes.io/projected/97175416-27c8-4665-8376-07a963a467c2-kube-api-access-str2f\") pod \"maas-api-revoke-sub-3d4156f1-8sq95\" (UID: \"97175416-27c8-4665-8376-07a963a467c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8sq95"
Nov 02 13:54:22.988358 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:54:22.988294    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97175416-27c8-4665-8376-07a963a467c2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-8sq95\" (UID: \"97175416-27c8-4665-8376-07a963a467c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8sq95"
Nov 02 13:54:22.988983 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:54:22.988959    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97175416-27c8-4665-8376-07a963a467c2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-8sq95\" (UID: \"97175416-27c8-4665-8376-07a963a467c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8sq95"
Nov 02 13:54:22.990716 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:54:22.990694    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97175416-27c8-4665-8376-07a963a467c2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-8sq95\" (UID: \"97175416-27c8-4665-8376-07a963a467c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8sq95"
Nov 02 13:54:23.017090 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:54:23.017065    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-str2f\" (UniqueName: \"kubernetes.io/projected/97175416-27c8-4665-8376-07a963a467c2-kube-api-access-str2f\") pod \"maas-api-revoke-sub-3d4156f1-8sq95\" (UID: \"97175416-27c8-4665-8376-07a963a467c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8sq95"
Nov 02 13:54:23.026898 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:54:23.026877    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8sq95"
Nov 02 13:54:23.116872 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:54:23.116828    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-593fe3-b5847f68fd8795" podUID="0cd959d9-52db-4e4e-a874-0811c8178ca8" containerName="payload-processing" containerID="cri-o://f6c8505c601c1f7b7740ca1b13b4ef3511cd0e8a303b762d546b229fdba1d1e6" gracePeriod=25
Nov 02 13:54:23.199170 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:54:23.199138    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"]
Nov 02 13:54:23.213583 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:54:23.213546    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-593fe3-664c458jbc" podUID="8bfcc94b-a517-483d-a6d1-ee5fe682b119" containerName="payload-pre-processing" containerID="cri-o://64bada63d9bb6c45e27a14e0ee307ec108442337c8088c7230ff1c7f6acc55e2" gracePeriod=25
Nov 02 13:54:23.326818 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:54:23.326789    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8sq95"]
Nov 02 13:54:23.328235 ip-10-0-128-132 kubenswrapper[4559]: W1102 13:54:23.328201    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97175416_27c8_4665_8376_07a963a467c2.slice/crio-cd2a6196bbc3d473e5e3897da78c19acd701259e26a8688df988e1a7c938d235 WatchSource:0}: Error finding container cd2a6196bbc3d473e5e3897da78c19acd701259e26a8688df988e1a7c938d235: Status 404 returned error can't find the container with id cd2a6196bbc3d473e5e3897da78c19acd701259e26a8688df988e1a7c938d235
Nov 02 13:54:23.462356 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:54:23.462332    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-593fe3-b5847f68fd8795"
Nov 02 13:54:23.492417 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:54:23.492390    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-593fe3-664c458jbc"
Nov 02 13:54:23.597910 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:54:23.597842    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8bfcc94b-a517-483d-a6d1-ee5fe682b119-kube-api-access-qstnt\" (UniqueName: \"kubernetes.io/projected/8bfcc94b-a517-483d-a6d1-ee5fe682b119-kube-api-access-qstnt\") pod \"8bfcc94b-a517-483d-a6d1-ee5fe682b119\" (UID: \"8bfcc94b-a517-483d-a6d1-ee5fe682b119\") "
Nov 02 13:54:23.598079 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:54:23.597929    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0cd959d9-52db-4e4e-a874-0811c8178ca8-kube-api-access-g9zss\" (UniqueName: \"kubernetes.io/projected/0cd959d9-52db-4e4e-a874-0811c8178ca8-kube-api-access-g9zss\") pod \"0cd959d9-52db-4e4e-a874-0811c8178ca8\" (UID: \"0cd959d9-52db-4e4e-a874-0811c8178ca8\") "
Nov 02 13:54:23.598079 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:54:23.597959    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8bfcc94b-a517-483d-a6d1-ee5fe682b119-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8bfcc94b-a517-483d-a6d1-ee5fe682b119-plugins-config-volume\") pod \"8bfcc94b-a517-483d-a6d1-ee5fe682b119\" (UID: \"8bfcc94b-a517-483d-a6d1-ee5fe682b119\") "
Nov 02 13:54:23.598079 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:54:23.597981    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0cd959d9-52db-4e4e-a874-0811c8178ca8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0cd959d9-52db-4e4e-a874-0811c8178ca8-plugins-config-volume\") pod \"0cd959d9-52db-4e4e-a874-0811c8178ca8\" (UID: \"0cd959d9-52db-4e4e-a874-0811c8178ca8\") "
Nov 02 13:54:23.598356 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:54:23.598334    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bfcc94b-a517-483d-a6d1-ee5fe682b119-plugins-config-volume" pod "8bfcc94b-a517-483d-a6d1-ee5fe682b119" (UID: "8bfcc94b-a517-483d-a6d1-ee5fe682b119"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:54:23.598426 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:54:23.598357    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cd959d9-52db-4e4e-a874-0811c8178ca8-plugins-config-volume" pod "0cd959d9-52db-4e4e-a874-0811c8178ca8" (UID: "0cd959d9-52db-4e4e-a874-0811c8178ca8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:54:23.600119 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:54:23.600097    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cd959d9-52db-4e4e-a874-0811c8178ca8-kube-api-access-g9zss" pod "0cd959d9-52db-4e4e-a874-0811c8178ca8" (UID: "0cd959d9-52db-4e4e-a874-0811c8178ca8"). InnerVolumeSpecName "kube-api-access-g9zss". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:54:23.600210 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:54:23.600134    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bfcc94b-a517-483d-a6d1-ee5fe682b119-kube-api-access-qstnt" pod "8bfcc94b-a517-483d-a6d1-ee5fe682b119" (UID: "8bfcc94b-a517-483d-a6d1-ee5fe682b119"). InnerVolumeSpecName "kube-api-access-qstnt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:54:23.699304 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:54:23.699274    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qstnt\" (UniqueName: \"kubernetes.io/projected/8bfcc94b-a517-483d-a6d1-ee5fe682b119-kube-api-access-qstnt\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:54:23.699304 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:54:23.699298    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g9zss\" (UniqueName: \"kubernetes.io/projected/0cd959d9-52db-4e4e-a874-0811c8178ca8-kube-api-access-g9zss\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:54:23.699304 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:54:23.699309    4559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8bfcc94b-a517-483d-a6d1-ee5fe682b119-plugins-config-volume\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:54:23.699862 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:54:23.699318    4559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0cd959d9-52db-4e4e-a874-0811c8178ca8-plugins-config-volume\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:54:23.891074 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:54:23.890991    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba787db2-d345-419e-b4c5-200abdf92503" path="/var/lib/kubelet/pods/ba787db2-d345-419e-b4c5-200abdf92503/volumes"
Nov 02 13:54:23.935267 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:54:23.935215    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8sq95" event={"ID":"97175416-27c8-4665-8376-07a963a467c2","Type":"ContainerStarted","Data":"3ca94fa1914f41ad1288f1baf4507f66ef47ea887187d332c375497ef9ec301c"}
Nov 02 13:54:23.935439 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:54:23.935364    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8sq95" event={"ID":"97175416-27c8-4665-8376-07a963a467c2","Type":"ContainerStarted","Data":"cd2a6196bbc3d473e5e3897da78c19acd701259e26a8688df988e1a7c938d235"}
Nov 02 13:54:23.936712 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:54:23.936677    4559 generic.go:356] "Generic (PLEG): container finished" podID="0cd959d9-52db-4e4e-a874-0811c8178ca8" containerID="f6c8505c601c1f7b7740ca1b13b4ef3511cd0e8a303b762d546b229fdba1d1e6" exitCode=0
Nov 02 13:54:23.936827 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:54:23.936740    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-593fe3-b5847f68fd8795" event={"ID":"0cd959d9-52db-4e4e-a874-0811c8178ca8","Type":"ContainerDied","Data":"f6c8505c601c1f7b7740ca1b13b4ef3511cd0e8a303b762d546b229fdba1d1e6"}
Nov 02 13:54:23.936827 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:54:23.936815    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-593fe3-b5847f68fd8795" event={"ID":"0cd959d9-52db-4e4e-a874-0811c8178ca8","Type":"ContainerDied","Data":"abdb7bdad6ca2190173f82a62a02affecb9a332129b27128f6981c5473ef1b0e"}
Nov 02 13:54:23.936908 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:54:23.936819    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-593fe3-b5847f68fd8795"
Nov 02 13:54:23.936908 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:54:23.936861    4559 scope.go:122] "RemoveContainer" containerID="f6c8505c601c1f7b7740ca1b13b4ef3511cd0e8a303b762d546b229fdba1d1e6"
Nov 02 13:54:23.938193 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:54:23.938167    4559 generic.go:356] "Generic (PLEG): container finished" podID="8bfcc94b-a517-483d-a6d1-ee5fe682b119" containerID="64bada63d9bb6c45e27a14e0ee307ec108442337c8088c7230ff1c7f6acc55e2" exitCode=0
Nov 02 13:54:23.938328 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:54:23.938198    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-593fe3-664c458jbc" event={"ID":"8bfcc94b-a517-483d-a6d1-ee5fe682b119","Type":"ContainerDied","Data":"64bada63d9bb6c45e27a14e0ee307ec108442337c8088c7230ff1c7f6acc55e2"}
Nov 02 13:54:23.938328 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:54:23.938221    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-593fe3-664c458jbc"
Nov 02 13:54:23.938328 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:54:23.938248    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-593fe3-664c458jbc" event={"ID":"8bfcc94b-a517-483d-a6d1-ee5fe682b119","Type":"ContainerDied","Data":"e38e2c7799fbab72d0fda525b0fef91e8c357648ab1ab25b78aba9ffaea5d07c"}
Nov 02 13:54:23.948082 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:54:23.947973    4559 scope.go:122] "RemoveContainer" containerID="f6c8505c601c1f7b7740ca1b13b4ef3511cd0e8a303b762d546b229fdba1d1e6"
Nov 02 13:54:23.948391 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:54:23.948361    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6c8505c601c1f7b7740ca1b13b4ef3511cd0e8a303b762d546b229fdba1d1e6\": container with ID starting with f6c8505c601c1f7b7740ca1b13b4ef3511cd0e8a303b762d546b229fdba1d1e6 not found: ID does not exist" containerID="f6c8505c601c1f7b7740ca1b13b4ef3511cd0e8a303b762d546b229fdba1d1e6"
Nov 02 13:54:23.948495 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:54:23.948397    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6c8505c601c1f7b7740ca1b13b4ef3511cd0e8a303b762d546b229fdba1d1e6"} err="failed to get container status \"f6c8505c601c1f7b7740ca1b13b4ef3511cd0e8a303b762d546b229fdba1d1e6\": rpc error: code = NotFound desc = could not find container \"f6c8505c601c1f7b7740ca1b13b4ef3511cd0e8a303b762d546b229fdba1d1e6\": container with ID starting with f6c8505c601c1f7b7740ca1b13b4ef3511cd0e8a303b762d546b229fdba1d1e6 not found: ID does not exist"
Nov 02 13:54:23.948495 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:54:23.948414    4559 scope.go:122] "RemoveContainer" containerID="64bada63d9bb6c45e27a14e0ee307ec108442337c8088c7230ff1c7f6acc55e2"
Nov 02 13:54:23.956918 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:54:23.956898    4559 scope.go:122] "RemoveContainer" containerID="64bada63d9bb6c45e27a14e0ee307ec108442337c8088c7230ff1c7f6acc55e2"
Nov 02 13:54:23.957175 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:54:23.957154    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64bada63d9bb6c45e27a14e0ee307ec108442337c8088c7230ff1c7f6acc55e2\": container with ID starting with 64bada63d9bb6c45e27a14e0ee307ec108442337c8088c7230ff1c7f6acc55e2 not found: ID does not exist" containerID="64bada63d9bb6c45e27a14e0ee307ec108442337c8088c7230ff1c7f6acc55e2"
Nov 02 13:54:23.957245 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:54:23.957187    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64bada63d9bb6c45e27a14e0ee307ec108442337c8088c7230ff1c7f6acc55e2"} err="failed to get container status \"64bada63d9bb6c45e27a14e0ee307ec108442337c8088c7230ff1c7f6acc55e2\": rpc error: code = NotFound desc = could not find container \"64bada63d9bb6c45e27a14e0ee307ec108442337c8088c7230ff1c7f6acc55e2\": container with ID starting with 64bada63d9bb6c45e27a14e0ee307ec108442337c8088c7230ff1c7f6acc55e2 not found: ID does not exist"
Nov 02 13:54:23.984758 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:54:23.984717    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8sq95" podStartSLOduration=1.9847041760000002 podStartE2EDuration="1.984704176s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:54:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:54:23.983618054 +0000 UTC m=+5260.239017167" watchObservedRunningTime="2026-11-02 13:54:23.984704176 +0000 UTC m=+5260.240103270"
Nov 02 13:54:24.022684 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:54:24.022659    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-593fe3-664c458jbc"]
Nov 02 13:54:24.030999 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:54:24.030980    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-593fe3-664c458jbc"]
Nov 02 13:54:24.068525 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:54:24.068503    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-593fe3-b5847f68fd8795"]
Nov 02 13:54:24.075507 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:54:24.075488    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-593fe3-b5847f68fd8795"]
Nov 02 13:54:24.946671 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:54:24.946558    4559 generic.go:356] "Generic (PLEG): container finished" podID="97175416-27c8-4665-8376-07a963a467c2" containerID="3ca94fa1914f41ad1288f1baf4507f66ef47ea887187d332c375497ef9ec301c" exitCode=0
Nov 02 13:54:24.947039 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:54:24.946661    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8sq95" event={"ID":"97175416-27c8-4665-8376-07a963a467c2","Type":"ContainerDied","Data":"3ca94fa1914f41ad1288f1baf4507f66ef47ea887187d332c375497ef9ec301c"}
Nov 02 13:54:25.891254 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:54:25.891222    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0cd959d9-52db-4e4e-a874-0811c8178ca8" path="/var/lib/kubelet/pods/0cd959d9-52db-4e4e-a874-0811c8178ca8/volumes"
Nov 02 13:54:25.891604 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:54:25.891591    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8bfcc94b-a517-483d-a6d1-ee5fe682b119" path="/var/lib/kubelet/pods/8bfcc94b-a517-483d-a6d1-ee5fe682b119/volumes"
Nov 02 13:54:26.091450 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:54:26.091425    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8sq95"
Nov 02 13:54:26.117150 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:54:26.117124    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/97175416-27c8-4665-8376-07a963a467c2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97175416-27c8-4665-8376-07a963a467c2-maas-api-xks-ca\") pod \"97175416-27c8-4665-8376-07a963a467c2\" (UID: \"97175416-27c8-4665-8376-07a963a467c2\") "
Nov 02 13:54:26.117271 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:54:26.117169    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97175416-27c8-4665-8376-07a963a467c2-kube-api-access-str2f\" (UniqueName: \"kubernetes.io/projected/97175416-27c8-4665-8376-07a963a467c2-kube-api-access-str2f\") pod \"97175416-27c8-4665-8376-07a963a467c2\" (UID: \"97175416-27c8-4665-8376-07a963a467c2\") "
Nov 02 13:54:26.117370 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:54:26.117352    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97175416-27c8-4665-8376-07a963a467c2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97175416-27c8-4665-8376-07a963a467c2-maas-api-openshift-ca\") pod \"97175416-27c8-4665-8376-07a963a467c2\" (UID: \"97175416-27c8-4665-8376-07a963a467c2\") "
Nov 02 13:54:26.117777 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:54:26.117749    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97175416-27c8-4665-8376-07a963a467c2-maas-api-openshift-ca" pod "97175416-27c8-4665-8376-07a963a467c2" (UID: "97175416-27c8-4665-8376-07a963a467c2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:54:26.119126 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:54:26.119095    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97175416-27c8-4665-8376-07a963a467c2-maas-api-xks-ca" pod "97175416-27c8-4665-8376-07a963a467c2" (UID: "97175416-27c8-4665-8376-07a963a467c2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:54:26.119259 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:54:26.119240    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97175416-27c8-4665-8376-07a963a467c2-kube-api-access-str2f" pod "97175416-27c8-4665-8376-07a963a467c2" (UID: "97175416-27c8-4665-8376-07a963a467c2"). InnerVolumeSpecName "kube-api-access-str2f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:54:26.138600 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:54:26.138568    4559 patch_prober.go:29] interesting pod/e2e-worker-single-main-593fe3-openshift-default-56579c548fcn26x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 13:54:26.138770 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:54:26.138619    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-593fe3-openshift-default-56579c548fcn26x" podUID="d98abfdb-03ee-463d-b059-be87d7f9a774" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 13:54:26.218180 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:54:26.218149    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97175416-27c8-4665-8376-07a963a467c2-maas-api-openshift-ca\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:54:26.218180 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:54:26.218179    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97175416-27c8-4665-8376-07a963a467c2-maas-api-xks-ca\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:54:26.218387 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:54:26.218189    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-str2f\" (UniqueName: \"kubernetes.io/projected/97175416-27c8-4665-8376-07a963a467c2-kube-api-access-str2f\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:54:26.608718 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:54:26.608694    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-593fe3-openshift-default-56579c548fcn26x"
Nov 02 13:54:26.620776 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:54:26.620757    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d98abfdb-03ee-463d-b059-be87d7f9a774-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d98abfdb-03ee-463d-b059-be87d7f9a774-workload-socket\") pod \"d98abfdb-03ee-463d-b059-be87d7f9a774\" (UID: \"d98abfdb-03ee-463d-b059-be87d7f9a774\") "
Nov 02 13:54:26.620850 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:54:26.620795    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d98abfdb-03ee-463d-b059-be87d7f9a774-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d98abfdb-03ee-463d-b059-be87d7f9a774-istio-data\") pod \"d98abfdb-03ee-463d-b059-be87d7f9a774\" (UID: \"d98abfdb-03ee-463d-b059-be87d7f9a774\") "
Nov 02 13:54:26.620850 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:54:26.620829    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d98abfdb-03ee-463d-b059-be87d7f9a774-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d98abfdb-03ee-463d-b059-be87d7f9a774-istio-envoy\") pod \"d98abfdb-03ee-463d-b059-be87d7f9a774\" (UID: \"d98abfdb-03ee-463d-b059-be87d7f9a774\") "
Nov 02 13:54:26.620969 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:54:26.620851    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d98abfdb-03ee-463d-b059-be87d7f9a774-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d98abfdb-03ee-463d-b059-be87d7f9a774-istiod-ca-cert\") pod \"d98abfdb-03ee-463d-b059-be87d7f9a774\" (UID: \"d98abfdb-03ee-463d-b059-be87d7f9a774\") "
Nov 02 13:54:26.620969 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:54:26.620871    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d98abfdb-03ee-463d-b059-be87d7f9a774-kube-api-access-6m6fx\" (UniqueName: \"kubernetes.io/projected/d98abfdb-03ee-463d-b059-be87d7f9a774-kube-api-access-6m6fx\") pod \"d98abfdb-03ee-463d-b059-be87d7f9a774\" (UID: \"d98abfdb-03ee-463d-b059-be87d7f9a774\") "
Nov 02 13:54:26.620969 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:54:26.620896    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d98abfdb-03ee-463d-b059-be87d7f9a774-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d98abfdb-03ee-463d-b059-be87d7f9a774-istio-podinfo\") pod \"d98abfdb-03ee-463d-b059-be87d7f9a774\" (UID: \"d98abfdb-03ee-463d-b059-be87d7f9a774\") "
Nov 02 13:54:26.621143 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:54:26.620986    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d98abfdb-03ee-463d-b059-be87d7f9a774-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d98abfdb-03ee-463d-b059-be87d7f9a774-credential-socket\") pod \"d98abfdb-03ee-463d-b059-be87d7f9a774\" (UID: \"d98abfdb-03ee-463d-b059-be87d7f9a774\") "
Nov 02 13:54:26.621143 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:54:26.621026    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d98abfdb-03ee-463d-b059-be87d7f9a774-workload-socket" pod "d98abfdb-03ee-463d-b059-be87d7f9a774" (UID: "d98abfdb-03ee-463d-b059-be87d7f9a774"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:54:26.621143 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:54:26.621071    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d98abfdb-03ee-463d-b059-be87d7f9a774-istio-token\" (UniqueName: \"kubernetes.io/projected/d98abfdb-03ee-463d-b059-be87d7f9a774-istio-token\") pod \"d98abfdb-03ee-463d-b059-be87d7f9a774\" (UID: \"d98abfdb-03ee-463d-b059-be87d7f9a774\") "
Nov 02 13:54:26.621143 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:54:26.621101    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d98abfdb-03ee-463d-b059-be87d7f9a774-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d98abfdb-03ee-463d-b059-be87d7f9a774-workload-certs\") pod \"d98abfdb-03ee-463d-b059-be87d7f9a774\" (UID: \"d98abfdb-03ee-463d-b059-be87d7f9a774\") "
Nov 02 13:54:26.621372 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:54:26.621172    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d98abfdb-03ee-463d-b059-be87d7f9a774-istio-data" pod "d98abfdb-03ee-463d-b059-be87d7f9a774" (UID: "d98abfdb-03ee-463d-b059-be87d7f9a774"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:54:26.621372 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:54:26.621320    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d98abfdb-03ee-463d-b059-be87d7f9a774-credential-socket" pod "d98abfdb-03ee-463d-b059-be87d7f9a774" (UID: "d98abfdb-03ee-463d-b059-be87d7f9a774"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:54:26.621475 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:54:26.621438    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d98abfdb-03ee-463d-b059-be87d7f9a774-workload-certs" pod "d98abfdb-03ee-463d-b059-be87d7f9a774" (UID: "d98abfdb-03ee-463d-b059-be87d7f9a774"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:54:26.621543 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:54:26.621478    4559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d98abfdb-03ee-463d-b059-be87d7f9a774-workload-socket\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:54:26.621543 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:54:26.621500    4559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d98abfdb-03ee-463d-b059-be87d7f9a774-istio-data\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:54:26.621543 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:54:26.621513    4559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d98abfdb-03ee-463d-b059-be87d7f9a774-credential-socket\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:54:26.621736 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:54:26.621701    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d98abfdb-03ee-463d-b059-be87d7f9a774-istiod-ca-cert" pod "d98abfdb-03ee-463d-b059-be87d7f9a774" (UID: "d98abfdb-03ee-463d-b059-be87d7f9a774"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:54:26.623029 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:54:26.623005    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d98abfdb-03ee-463d-b059-be87d7f9a774-istio-token" pod "d98abfdb-03ee-463d-b059-be87d7f9a774" (UID: "d98abfdb-03ee-463d-b059-be87d7f9a774"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:54:26.623116 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:54:26.623044    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d98abfdb-03ee-463d-b059-be87d7f9a774-istio-podinfo" pod "d98abfdb-03ee-463d-b059-be87d7f9a774" (UID: "d98abfdb-03ee-463d-b059-be87d7f9a774"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 13:54:26.623116 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:54:26.623061    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d98abfdb-03ee-463d-b059-be87d7f9a774-istio-envoy" pod "d98abfdb-03ee-463d-b059-be87d7f9a774" (UID: "d98abfdb-03ee-463d-b059-be87d7f9a774"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:54:26.623116 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:54:26.623065    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d98abfdb-03ee-463d-b059-be87d7f9a774-kube-api-access-6m6fx" pod "d98abfdb-03ee-463d-b059-be87d7f9a774" (UID: "d98abfdb-03ee-463d-b059-be87d7f9a774"). InnerVolumeSpecName "kube-api-access-6m6fx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:54:26.722118 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:54:26.722086    4559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d98abfdb-03ee-463d-b059-be87d7f9a774-istio-envoy\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:54:26.722258 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:54:26.722123    4559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d98abfdb-03ee-463d-b059-be87d7f9a774-istiod-ca-cert\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:54:26.722258 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:54:26.722140    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6m6fx\" (UniqueName: \"kubernetes.io/projected/d98abfdb-03ee-463d-b059-be87d7f9a774-kube-api-access-6m6fx\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:54:26.722258 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:54:26.722155    4559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d98abfdb-03ee-463d-b059-be87d7f9a774-istio-podinfo\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:54:26.722258 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:54:26.722169    4559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d98abfdb-03ee-463d-b059-be87d7f9a774-istio-token\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:54:26.722258 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:54:26.722184    4559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d98abfdb-03ee-463d-b059-be87d7f9a774-workload-certs\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:54:26.959172 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:54:26.959143    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8sq95"
Nov 02 13:54:26.959352 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:54:26.959183    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8sq95" event={"ID":"97175416-27c8-4665-8376-07a963a467c2","Type":"ContainerDied","Data":"cd2a6196bbc3d473e5e3897da78c19acd701259e26a8688df988e1a7c938d235"}
Nov 02 13:54:26.959352 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:54:26.959236    4559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cd2a6196bbc3d473e5e3897da78c19acd701259e26a8688df988e1a7c938d235"
Nov 02 13:54:26.960989 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:54:26.960967    4559 generic.go:356] "Generic (PLEG): container finished" podID="d98abfdb-03ee-463d-b059-be87d7f9a774" containerID="e37795122e54d58b30fd2484f926cb091ded123000e57760e3742d91661122f8" exitCode=0
Nov 02 13:54:26.961113 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:54:26.961027    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-593fe3-openshift-default-56579c548fcn26x"
Nov 02 13:54:26.961113 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:54:26.961039    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-593fe3-openshift-default-56579c548fcn26x" event={"ID":"d98abfdb-03ee-463d-b059-be87d7f9a774","Type":"ContainerDied","Data":"e37795122e54d58b30fd2484f926cb091ded123000e57760e3742d91661122f8"}
Nov 02 13:54:26.961113 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:54:26.961073    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-593fe3-openshift-default-56579c548fcn26x" event={"ID":"d98abfdb-03ee-463d-b059-be87d7f9a774","Type":"ContainerDied","Data":"57a1b61b7981d026b7ee9f364daf8b6c0f58df108d58f1276c84e58de7a0ebf2"}
Nov 02 13:54:26.961113 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:54:26.961094    4559 scope.go:122] "RemoveContainer" containerID="e37795122e54d58b30fd2484f926cb091ded123000e57760e3742d91661122f8"
Nov 02 13:54:26.970075 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:54:26.970034    4559 scope.go:122] "RemoveContainer" containerID="2ea6151341e410f1eb3bbf92e7acc9fee40ee799d9960b029348b7da28dee1df"
Nov 02 13:54:26.978165 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:54:26.978151    4559 scope.go:122] "RemoveContainer" containerID="e37795122e54d58b30fd2484f926cb091ded123000e57760e3742d91661122f8"
Nov 02 13:54:26.978403 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:54:26.978386    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e37795122e54d58b30fd2484f926cb091ded123000e57760e3742d91661122f8\": container with ID starting with e37795122e54d58b30fd2484f926cb091ded123000e57760e3742d91661122f8 not found: ID does not exist" containerID="e37795122e54d58b30fd2484f926cb091ded123000e57760e3742d91661122f8"
Nov 02 13:54:26.978450 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:54:26.978411    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e37795122e54d58b30fd2484f926cb091ded123000e57760e3742d91661122f8"} err="failed to get container status \"e37795122e54d58b30fd2484f926cb091ded123000e57760e3742d91661122f8\": rpc error: code = NotFound desc = could not find container \"e37795122e54d58b30fd2484f926cb091ded123000e57760e3742d91661122f8\": container with ID starting with e37795122e54d58b30fd2484f926cb091ded123000e57760e3742d91661122f8 not found: ID does not exist"
Nov 02 13:54:26.978450 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:54:26.978425    4559 scope.go:122] "RemoveContainer" containerID="2ea6151341e410f1eb3bbf92e7acc9fee40ee799d9960b029348b7da28dee1df"
Nov 02 13:54:26.978672 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:54:26.978630    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ea6151341e410f1eb3bbf92e7acc9fee40ee799d9960b029348b7da28dee1df\": container with ID starting with 2ea6151341e410f1eb3bbf92e7acc9fee40ee799d9960b029348b7da28dee1df not found: ID does not exist" containerID="2ea6151341e410f1eb3bbf92e7acc9fee40ee799d9960b029348b7da28dee1df"
Nov 02 13:54:26.978738 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:54:26.978676    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ea6151341e410f1eb3bbf92e7acc9fee40ee799d9960b029348b7da28dee1df"} err="failed to get container status \"2ea6151341e410f1eb3bbf92e7acc9fee40ee799d9960b029348b7da28dee1df\": rpc error: code = NotFound desc = could not find container \"2ea6151341e410f1eb3bbf92e7acc9fee40ee799d9960b029348b7da28dee1df\": container with ID starting with 2ea6151341e410f1eb3bbf92e7acc9fee40ee799d9960b029348b7da28dee1df not found: ID does not exist"
Nov 02 13:54:27.058085 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:54:27.058060    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-593fe3-openshift-default-56579c548fcn26x"]
Nov 02 13:54:27.077463 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:54:27.077444    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-593fe3-openshift-default-56579c548fcn26x"]
Nov 02 13:54:27.895801 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:54:27.895766    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d98abfdb-03ee-463d-b059-be87d7f9a774" path="/var/lib/kubelet/pods/d98abfdb-03ee-463d-b059-be87d7f9a774/volumes"
Nov 02 13:54:30.030099 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:54:30.030065    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-mj4v4"]
Nov 02 13:54:30.042702 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:54:30.042673    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-mj4v4"]
Nov 02 13:54:31.892131 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:54:31.892099    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ead5881-7f3d-401a-95c9-e3466dd4cfc2" path="/var/lib/kubelet/pods/7ead5881-7f3d-401a-95c9-e3466dd4cfc2/volumes"
Nov 02 13:54:34.702089 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:54:34.702051    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"]
Nov 02 13:54:40.528664 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:54:40.528619    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"]
Nov 02 13:54:45.084617 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:54:45.084580    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqcmj" containerName="extractor"
Nov 02 13:54:48.024926 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:54:48.024894    4559 scope.go:122] "RemoveContainer" containerID="464df2712fa14316ea20741c1247e65907196e8fde6e4c6b7f0d6048ebf9c774"
Nov 02 13:54:48.033597 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:54:48.033578    4559 scope.go:122] "RemoveContainer" containerID="2a32d24005943611c69297614643660dbb12fcf2393e4c2aeda277d553a06130"
Nov 02 13:54:48.041053 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:54:48.041032    4559 scope.go:122] "RemoveContainer" containerID="4075b58eeda95097fcabadd34fb6456670b3b3d385a354900ac086dac006a748"
Nov 02 13:54:52.031050 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:54:52.031018    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-7xwkv"]
Nov 02 13:54:52.043696 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:54:52.043665    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-7xwkv"]
Nov 02 13:54:53.891202 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:54:53.891170    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ffc78c45-ff69-42fd-ab06-8c8cf4623bf3" path="/var/lib/kubelet/pods/ffc78c45-ff69-42fd-ab06-8c8cf4623bf3/volumes"
Nov 02 13:55:07.030701 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:55:07.030659    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-9nglq"]
Nov 02 13:55:07.043137 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:55:07.043109    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-9nglq"]
Nov 02 13:55:07.891245 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:55:07.891213    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c3598e7-1cb1-42b0-a291-656fc5e14505" path="/var/lib/kubelet/pods/4c3598e7-1cb1-42b0-a291-656fc5e14505/volumes"
Nov 02 13:55:15.085211 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:55:15.085175    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqcmj" containerName="extractor"
Nov 02 13:55:24.031376 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:55:24.031341    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-qnvts"]
Nov 02 13:55:24.043516 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:55:24.043490    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-qnvts"]
Nov 02 13:55:25.891622 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:55:25.891585    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5ff9c16-f825-418d-ba32-ce9faaa39351" path="/var/lib/kubelet/pods/c5ff9c16-f825-418d-ba32-ce9faaa39351/volumes"
Nov 02 13:55:38.031976 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:55:38.031935    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-qz8b5"]
Nov 02 13:55:38.042451 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:55:38.042426    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-qz8b5"]
Nov 02 13:55:39.891732 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:55:39.891702    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="804809f5-f9da-436a-9800-e9866dd8d586" path="/var/lib/kubelet/pods/804809f5-f9da-436a-9800-e9866dd8d586/volumes"
Nov 02 13:55:45.085821 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:55:45.085781    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqcmj" containerName="extractor"
Nov 02 13:55:48.091914 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:55:48.091881    4559 scope.go:122] "RemoveContainer" containerID="56ada7fdcb518c82330803dc59d83103843e0901647c5d97444e9ba51573d8f1"
Nov 02 13:55:48.100186 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:55:48.100165    4559 scope.go:122] "RemoveContainer" containerID="44e3bae57c20d4380447b1ec09653bd48e2530f4139e366629870f4c6ebcee60"
Nov 02 13:55:48.107680 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:55:48.107632    4559 scope.go:122] "RemoveContainer" containerID="a6fab3a2b765ee10ffa2711fac46ddf53736a54049d117e170bcb13cdcad4ed5"
Nov 02 13:56:08.059665 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:56:08.048021    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"]
Nov 02 13:56:09.203829 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:56:09.203792    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-tcrqn"]
Nov 02 13:56:09.204400 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:56:09.204350    4559 cpu_manager.go:414] "containerMap: removing container" podUID="d98abfdb-03ee-463d-b059-be87d7f9a774" containerName="istio-proxy"
Nov 02 13:56:09.204400 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:56:09.204371    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d98abfdb-03ee-463d-b059-be87d7f9a774" containerName="istio-proxy"
Nov 02 13:56:09.204535 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:56:09.204406    4559 cpu_manager.go:414] "containerMap: removing container" podUID="0cd959d9-52db-4e4e-a874-0811c8178ca8" containerName="payload-processing"
Nov 02 13:56:09.204535 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:56:09.204416    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0cd959d9-52db-4e4e-a874-0811c8178ca8" containerName="payload-processing"
Nov 02 13:56:09.204535 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:56:09.204431    4559 cpu_manager.go:414] "containerMap: removing container" podUID="97175416-27c8-4665-8376-07a963a467c2" containerName="revoke-keys"
Nov 02 13:56:09.204535 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:56:09.204438    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97175416-27c8-4665-8376-07a963a467c2" containerName="revoke-keys"
Nov 02 13:56:09.204535 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:56:09.204455    4559 cpu_manager.go:414] "containerMap: removing container" podUID="8bfcc94b-a517-483d-a6d1-ee5fe682b119" containerName="payload-pre-processing"
Nov 02 13:56:09.204535 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:56:09.204462    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8bfcc94b-a517-483d-a6d1-ee5fe682b119" containerName="payload-pre-processing"
Nov 02 13:56:09.204878 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:56:09.204558    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="d98abfdb-03ee-463d-b059-be87d7f9a774" containerName="istio-proxy"
Nov 02 13:56:09.204878 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:56:09.204575    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="0cd959d9-52db-4e4e-a874-0811c8178ca8" containerName="payload-processing"
Nov 02 13:56:09.204878 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:56:09.204588    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="97175416-27c8-4665-8376-07a963a467c2" containerName="revoke-keys"
Nov 02 13:56:09.204878 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:56:09.204598    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="8bfcc94b-a517-483d-a6d1-ee5fe682b119" containerName="payload-pre-processing"
Nov 02 13:56:09.204878 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:56:09.204610    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="d98abfdb-03ee-463d-b059-be87d7f9a774" containerName="istio-proxy"
Nov 02 13:56:09.207960 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:56:09.207940    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-tcrqn"
Nov 02 13:56:09.223767 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:56:09.223741    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 13:56:09.236413 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:56:09.236356    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 13:56:09.236583 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:56:09.236391    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 13:56:09.236583 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:56:09.236413    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-45fzn\""
Nov 02 13:56:09.238808 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:56:09.238786    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-tcrqn"]
Nov 02 13:56:09.250658 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:56:09.250613    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8afdd12c-bbde-47d5-9ac6-7ed7ff7ae147-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-tcrqn\" (UID: \"8afdd12c-bbde-47d5-9ac6-7ed7ff7ae147\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-tcrqn"
Nov 02 13:56:09.250796 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:56:09.250734    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2q5gr\" (UniqueName: \"kubernetes.io/projected/8afdd12c-bbde-47d5-9ac6-7ed7ff7ae147-kube-api-access-2q5gr\") pod \"maas-api-revoke-sub-d576e68b-tcrqn\" (UID: \"8afdd12c-bbde-47d5-9ac6-7ed7ff7ae147\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-tcrqn"
Nov 02 13:56:09.250796 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:56:09.250779    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8afdd12c-bbde-47d5-9ac6-7ed7ff7ae147-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-tcrqn\" (UID: \"8afdd12c-bbde-47d5-9ac6-7ed7ff7ae147\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-tcrqn"
Nov 02 13:56:09.352172 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:56:09.352139    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8afdd12c-bbde-47d5-9ac6-7ed7ff7ae147-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-tcrqn\" (UID: \"8afdd12c-bbde-47d5-9ac6-7ed7ff7ae147\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-tcrqn"
Nov 02 13:56:09.352366 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:56:09.352231    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8afdd12c-bbde-47d5-9ac6-7ed7ff7ae147-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-tcrqn\" (UID: \"8afdd12c-bbde-47d5-9ac6-7ed7ff7ae147\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-tcrqn"
Nov 02 13:56:09.352435 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:56:09.352405    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2q5gr\" (UniqueName: \"kubernetes.io/projected/8afdd12c-bbde-47d5-9ac6-7ed7ff7ae147-kube-api-access-2q5gr\") pod \"maas-api-revoke-sub-d576e68b-tcrqn\" (UID: \"8afdd12c-bbde-47d5-9ac6-7ed7ff7ae147\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-tcrqn"
Nov 02 13:56:09.352953 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:56:09.352928    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8afdd12c-bbde-47d5-9ac6-7ed7ff7ae147-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-tcrqn\" (UID: \"8afdd12c-bbde-47d5-9ac6-7ed7ff7ae147\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-tcrqn"
Nov 02 13:56:09.354780 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:56:09.354756    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8afdd12c-bbde-47d5-9ac6-7ed7ff7ae147-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-tcrqn\" (UID: \"8afdd12c-bbde-47d5-9ac6-7ed7ff7ae147\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-tcrqn"
Nov 02 13:56:09.376341 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:56:09.376314    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2q5gr\" (UniqueName: \"kubernetes.io/projected/8afdd12c-bbde-47d5-9ac6-7ed7ff7ae147-kube-api-access-2q5gr\") pod \"maas-api-revoke-sub-d576e68b-tcrqn\" (UID: \"8afdd12c-bbde-47d5-9ac6-7ed7ff7ae147\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-tcrqn"
Nov 02 13:56:09.520659 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:56:09.520559    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-tcrqn"
Nov 02 13:56:09.823082 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:56:09.823049    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-tcrqn"]
Nov 02 13:56:09.824017 ip-10-0-128-132 kubenswrapper[4559]: W1102 13:56:09.823986    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8afdd12c_bbde_47d5_9ac6_7ed7ff7ae147.slice/crio-8bfa6fbe1c742fd9e94e690e9b4c0392e0ce883ef8500f563045d2503fd3cade WatchSource:0}: Error finding container 8bfa6fbe1c742fd9e94e690e9b4c0392e0ce883ef8500f563045d2503fd3cade: Status 404 returned error can't find the container with id 8bfa6fbe1c742fd9e94e690e9b4c0392e0ce883ef8500f563045d2503fd3cade
Nov 02 13:56:10.499445 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:56:10.499403    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-tcrqn" event={"ID":"8afdd12c-bbde-47d5-9ac6-7ed7ff7ae147","Type":"ContainerStarted","Data":"bab48d37cd401c45fb7dc953500e20a6eddb6b618702b83eee2c81eae33915ee"}
Nov 02 13:56:10.499918 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:56:10.499461    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-tcrqn" event={"ID":"8afdd12c-bbde-47d5-9ac6-7ed7ff7ae147","Type":"ContainerStarted","Data":"8bfa6fbe1c742fd9e94e690e9b4c0392e0ce883ef8500f563045d2503fd3cade"}
Nov 02 13:56:10.541757 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:56:10.541700    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-tcrqn" podStartSLOduration=1.541686041 podStartE2EDuration="1.541686041s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:56:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:56:10.54047378 +0000 UTC m=+5366.795872875" watchObservedRunningTime="2026-11-02 13:56:10.541686041 +0000 UTC m=+5366.797085136"
Nov 02 13:56:11.505233 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:56:11.505195    4559 generic.go:356] "Generic (PLEG): container finished" podID="8afdd12c-bbde-47d5-9ac6-7ed7ff7ae147" containerID="bab48d37cd401c45fb7dc953500e20a6eddb6b618702b83eee2c81eae33915ee" exitCode=0
Nov 02 13:56:11.505685 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:56:11.505262    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-tcrqn" event={"ID":"8afdd12c-bbde-47d5-9ac6-7ed7ff7ae147","Type":"ContainerDied","Data":"bab48d37cd401c45fb7dc953500e20a6eddb6b618702b83eee2c81eae33915ee"}
Nov 02 13:56:12.643747 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:56:12.643719    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-tcrqn"
Nov 02 13:56:12.783088 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:56:12.782996    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8afdd12c-bbde-47d5-9ac6-7ed7ff7ae147-kube-api-access-2q5gr\" (UniqueName: \"kubernetes.io/projected/8afdd12c-bbde-47d5-9ac6-7ed7ff7ae147-kube-api-access-2q5gr\") pod \"8afdd12c-bbde-47d5-9ac6-7ed7ff7ae147\" (UID: \"8afdd12c-bbde-47d5-9ac6-7ed7ff7ae147\") "
Nov 02 13:56:12.783088 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:56:12.783034    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8afdd12c-bbde-47d5-9ac6-7ed7ff7ae147-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8afdd12c-bbde-47d5-9ac6-7ed7ff7ae147-maas-api-xks-ca\") pod \"8afdd12c-bbde-47d5-9ac6-7ed7ff7ae147\" (UID: \"8afdd12c-bbde-47d5-9ac6-7ed7ff7ae147\") "
Nov 02 13:56:12.783088 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:56:12.783082    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8afdd12c-bbde-47d5-9ac6-7ed7ff7ae147-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8afdd12c-bbde-47d5-9ac6-7ed7ff7ae147-maas-api-openshift-ca\") pod \"8afdd12c-bbde-47d5-9ac6-7ed7ff7ae147\" (UID: \"8afdd12c-bbde-47d5-9ac6-7ed7ff7ae147\") "
Nov 02 13:56:12.783577 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:56:12.783550    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8afdd12c-bbde-47d5-9ac6-7ed7ff7ae147-maas-api-openshift-ca" pod "8afdd12c-bbde-47d5-9ac6-7ed7ff7ae147" (UID: "8afdd12c-bbde-47d5-9ac6-7ed7ff7ae147"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:56:12.785169 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:56:12.785136    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8afdd12c-bbde-47d5-9ac6-7ed7ff7ae147-kube-api-access-2q5gr" pod "8afdd12c-bbde-47d5-9ac6-7ed7ff7ae147" (UID: "8afdd12c-bbde-47d5-9ac6-7ed7ff7ae147"). InnerVolumeSpecName "kube-api-access-2q5gr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:56:12.785277 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:56:12.785173    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8afdd12c-bbde-47d5-9ac6-7ed7ff7ae147-maas-api-xks-ca" pod "8afdd12c-bbde-47d5-9ac6-7ed7ff7ae147" (UID: "8afdd12c-bbde-47d5-9ac6-7ed7ff7ae147"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:56:12.884464 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:56:12.884434    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2q5gr\" (UniqueName: \"kubernetes.io/projected/8afdd12c-bbde-47d5-9ac6-7ed7ff7ae147-kube-api-access-2q5gr\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:56:12.884464 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:56:12.884462    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8afdd12c-bbde-47d5-9ac6-7ed7ff7ae147-maas-api-xks-ca\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:56:12.884464 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:56:12.884472    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8afdd12c-bbde-47d5-9ac6-7ed7ff7ae147-maas-api-openshift-ca\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:56:13.515423 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:56:13.515392    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-tcrqn"
Nov 02 13:56:13.515423 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:56:13.515413    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-tcrqn" event={"ID":"8afdd12c-bbde-47d5-9ac6-7ed7ff7ae147","Type":"ContainerDied","Data":"8bfa6fbe1c742fd9e94e690e9b4c0392e0ce883ef8500f563045d2503fd3cade"}
Nov 02 13:56:13.515692 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:56:13.515462    4559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8bfa6fbe1c742fd9e94e690e9b4c0392e0ce883ef8500f563045d2503fd3cade"
Nov 02 13:56:15.085226 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:56:15.085187    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqcmj" containerName="extractor"
Nov 02 13:56:20.607752 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:56:20.607721    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"]
Nov 02 13:56:24.334796 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:56:24.334758    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-zxrzc"]
Nov 02 13:56:24.335947 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:56:24.335918    4559 cpu_manager.go:414] "containerMap: removing container" podUID="d98abfdb-03ee-463d-b059-be87d7f9a774" containerName="istio-proxy"
Nov 02 13:56:24.336109 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:56:24.336092    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d98abfdb-03ee-463d-b059-be87d7f9a774" containerName="istio-proxy"
Nov 02 13:56:24.336221 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:56:24.336208    4559 cpu_manager.go:414] "containerMap: removing container" podUID="8afdd12c-bbde-47d5-9ac6-7ed7ff7ae147" containerName="revoke-keys"
Nov 02 13:56:24.336315 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:56:24.336303    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8afdd12c-bbde-47d5-9ac6-7ed7ff7ae147" containerName="revoke-keys"
Nov 02 13:56:24.336550 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:56:24.336532    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="8afdd12c-bbde-47d5-9ac6-7ed7ff7ae147" containerName="revoke-keys"
Nov 02 13:56:24.390898 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:56:24.390861    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-zxrzc"]
Nov 02 13:56:24.391244 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:56:24.391226    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-zxrzc"
Nov 02 13:56:24.423049 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:56:24.421323    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 13:56:24.423049 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:56:24.421450    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-45fzn\""
Nov 02 13:56:24.445038 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:56:24.444913    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 13:56:24.445038 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:56:24.445000    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 13:56:24.494153 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:56:24.494110    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/296049f9-5f5e-4ec2-b170-534105e1b937-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-zxrzc\" (UID: \"296049f9-5f5e-4ec2-b170-534105e1b937\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-zxrzc"
Nov 02 13:56:24.519063 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:56:24.519027    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/296049f9-5f5e-4ec2-b170-534105e1b937-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-zxrzc\" (UID: \"296049f9-5f5e-4ec2-b170-534105e1b937\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-zxrzc"
Nov 02 13:56:24.519303 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:56:24.519283    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhfgk\" (UniqueName: \"kubernetes.io/projected/296049f9-5f5e-4ec2-b170-534105e1b937-kube-api-access-qhfgk\") pod \"maas-api-revoke-sub-d2bc93bc-zxrzc\" (UID: \"296049f9-5f5e-4ec2-b170-534105e1b937\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-zxrzc"
Nov 02 13:56:24.627248 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:56:24.627137    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/296049f9-5f5e-4ec2-b170-534105e1b937-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-zxrzc\" (UID: \"296049f9-5f5e-4ec2-b170-534105e1b937\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-zxrzc"
Nov 02 13:56:24.638662 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:56:24.628033    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/296049f9-5f5e-4ec2-b170-534105e1b937-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-zxrzc\" (UID: \"296049f9-5f5e-4ec2-b170-534105e1b937\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-zxrzc"
Nov 02 13:56:24.648413 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:56:24.648382    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhfgk\" (UniqueName: \"kubernetes.io/projected/296049f9-5f5e-4ec2-b170-534105e1b937-kube-api-access-qhfgk\") pod \"maas-api-revoke-sub-d2bc93bc-zxrzc\" (UID: \"296049f9-5f5e-4ec2-b170-534105e1b937\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-zxrzc"
Nov 02 13:56:24.655816 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:56:24.630417    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/296049f9-5f5e-4ec2-b170-534105e1b937-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-zxrzc\" (UID: \"296049f9-5f5e-4ec2-b170-534105e1b937\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-zxrzc"
Nov 02 13:56:24.674665 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:56:24.664120    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/296049f9-5f5e-4ec2-b170-534105e1b937-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-zxrzc\" (UID: \"296049f9-5f5e-4ec2-b170-534105e1b937\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-zxrzc"
Nov 02 13:56:24.709664 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:56:24.702941    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhfgk\" (UniqueName: \"kubernetes.io/projected/296049f9-5f5e-4ec2-b170-534105e1b937-kube-api-access-qhfgk\") pod \"maas-api-revoke-sub-d2bc93bc-zxrzc\" (UID: \"296049f9-5f5e-4ec2-b170-534105e1b937\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-zxrzc"
Nov 02 13:56:24.777827 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:56:24.777790    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-zxrzc"
Nov 02 13:56:25.429710 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:56:25.404069    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-zxrzc"]
Nov 02 13:56:25.588925 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:56:25.588895    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-zxrzc" event={"ID":"296049f9-5f5e-4ec2-b170-534105e1b937","Type":"ContainerStarted","Data":"013763ab765c18b26fad4a91d406200bbf8493ebe9bdd578876aae5c0ef7b6ea"}
Nov 02 13:56:26.037685 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:56:26.037621    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"]
Nov 02 13:56:26.300938 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:56:26.300817    4559 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-gbd89 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.13:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 13:56:26.301265 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:56:26.301230    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-gbd89" podUID="f55beaae-7351-41b7-89ca-3142739266cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.13:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:56:26.594956 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:56:26.594873    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-zxrzc" event={"ID":"296049f9-5f5e-4ec2-b170-534105e1b937","Type":"ContainerStarted","Data":"c93444359d071ac4b6e66ec08030801fd4102f740a694a4b61c805e7bbadd068"}
Nov 02 13:56:26.638856 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:56:26.638813    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-zxrzc" podStartSLOduration=2.638801044 podStartE2EDuration="2.638801044s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:56:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:56:26.63661856 +0000 UTC m=+5382.892017655" watchObservedRunningTime="2026-11-02 13:56:26.638801044 +0000 UTC m=+5382.894200138"
Nov 02 13:56:27.601917 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:56:27.601880    4559 generic.go:356] "Generic (PLEG): container finished" podID="296049f9-5f5e-4ec2-b170-534105e1b937" containerID="c93444359d071ac4b6e66ec08030801fd4102f740a694a4b61c805e7bbadd068" exitCode=0
Nov 02 13:56:27.602368 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:56:27.601967    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-zxrzc" event={"ID":"296049f9-5f5e-4ec2-b170-534105e1b937","Type":"ContainerDied","Data":"c93444359d071ac4b6e66ec08030801fd4102f740a694a4b61c805e7bbadd068"}
Nov 02 13:56:28.733407 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:56:28.733379    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-zxrzc"
Nov 02 13:56:28.867924 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:56:28.867856    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/296049f9-5f5e-4ec2-b170-534105e1b937-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/296049f9-5f5e-4ec2-b170-534105e1b937-maas-api-openshift-ca\") pod \"296049f9-5f5e-4ec2-b170-534105e1b937\" (UID: \"296049f9-5f5e-4ec2-b170-534105e1b937\") "
Nov 02 13:56:28.867924 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:56:28.867902    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/296049f9-5f5e-4ec2-b170-534105e1b937-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/296049f9-5f5e-4ec2-b170-534105e1b937-maas-api-xks-ca\") pod \"296049f9-5f5e-4ec2-b170-534105e1b937\" (UID: \"296049f9-5f5e-4ec2-b170-534105e1b937\") "
Nov 02 13:56:28.868156 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:56:28.867932    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/296049f9-5f5e-4ec2-b170-534105e1b937-kube-api-access-qhfgk\" (UniqueName: \"kubernetes.io/projected/296049f9-5f5e-4ec2-b170-534105e1b937-kube-api-access-qhfgk\") pod \"296049f9-5f5e-4ec2-b170-534105e1b937\" (UID: \"296049f9-5f5e-4ec2-b170-534105e1b937\") "
Nov 02 13:56:28.868226 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:56:28.868203    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/296049f9-5f5e-4ec2-b170-534105e1b937-maas-api-openshift-ca" pod "296049f9-5f5e-4ec2-b170-534105e1b937" (UID: "296049f9-5f5e-4ec2-b170-534105e1b937"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:56:28.870029 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:56:28.870001    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/296049f9-5f5e-4ec2-b170-534105e1b937-kube-api-access-qhfgk" pod "296049f9-5f5e-4ec2-b170-534105e1b937" (UID: "296049f9-5f5e-4ec2-b170-534105e1b937"). InnerVolumeSpecName "kube-api-access-qhfgk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:56:28.870137 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:56:28.870049    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/296049f9-5f5e-4ec2-b170-534105e1b937-maas-api-xks-ca" pod "296049f9-5f5e-4ec2-b170-534105e1b937" (UID: "296049f9-5f5e-4ec2-b170-534105e1b937"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:56:28.968675 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:56:28.968634    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/296049f9-5f5e-4ec2-b170-534105e1b937-maas-api-xks-ca\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:56:28.968675 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:56:28.968671    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qhfgk\" (UniqueName: \"kubernetes.io/projected/296049f9-5f5e-4ec2-b170-534105e1b937-kube-api-access-qhfgk\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:56:28.968675 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:56:28.968681    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/296049f9-5f5e-4ec2-b170-534105e1b937-maas-api-openshift-ca\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:56:29.612009 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:56:29.611978    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-zxrzc"
Nov 02 13:56:29.612181 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:56:29.612015    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-zxrzc" event={"ID":"296049f9-5f5e-4ec2-b170-534105e1b937","Type":"ContainerDied","Data":"013763ab765c18b26fad4a91d406200bbf8493ebe9bdd578876aae5c0ef7b6ea"}
Nov 02 13:56:29.612181 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:56:29.612065    4559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="013763ab765c18b26fad4a91d406200bbf8493ebe9bdd578876aae5c0ef7b6ea"
Nov 02 13:56:30.511903 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:56:30.511865    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"]
Nov 02 13:56:36.401126 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:56:36.401091    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-65f6775b5-7t49d"]
Nov 02 13:56:36.402208 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:56:36.401668    4559 cpu_manager.go:414] "containerMap: removing container" podUID="296049f9-5f5e-4ec2-b170-534105e1b937" containerName="revoke-keys"
Nov 02 13:56:36.402208 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:56:36.401690    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="296049f9-5f5e-4ec2-b170-534105e1b937" containerName="revoke-keys"
Nov 02 13:56:36.402208 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:56:36.401812    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="296049f9-5f5e-4ec2-b170-534105e1b937" containerName="revoke-keys"
Nov 02 13:56:36.405168 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:56:36.405138    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-65f6775b5-7t49d"
Nov 02 13:56:36.424900 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:56:36.424871    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-756c48cb77-5k2qx"]
Nov 02 13:56:36.428450 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:56:36.428424    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-756c48cb77-5k2qx"
Nov 02 13:56:36.436072 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:56:36.436029    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-sr5s9\""
Nov 02 13:56:36.440357 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:56:36.439669    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Nov 02 13:56:36.440357 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:56:36.439750    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 13:56:36.440357 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:56:36.439799    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 13:56:36.442140 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:56:36.442111    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-65f6775b5-7t49d"]
Nov 02 13:56:36.448903 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:56:36.448882    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Nov 02 13:56:36.451874 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:56:36.451850    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-756c48cb77-5k2qx"]
Nov 02 13:56:36.538047 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:56:36.538013    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/45ebae56-61c5-4e2b-85c1-36f500719234-metrics-certs\") pod \"maas-api-65f6775b5-7t49d\" (UID: \"45ebae56-61c5-4e2b-85c1-36f500719234\") " pod="odh-ai-gateway-infra/maas-api-65f6775b5-7t49d"
Nov 02 13:56:36.538218 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:56:36.538062    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hsm74\" (UniqueName: \"kubernetes.io/projected/ac5dc138-69ca-41b7-8b92-b2b08109dd52-kube-api-access-hsm74\") pod \"maas-api-756c48cb77-5k2qx\" (UID: \"ac5dc138-69ca-41b7-8b92-b2b08109dd52\") " pod="odh-ai-gateway-infra/maas-api-756c48cb77-5k2qx"
Nov 02 13:56:36.538218 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:56:36.538155    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac5dc138-69ca-41b7-8b92-b2b08109dd52-openshift-service-ca\") pod \"maas-api-756c48cb77-5k2qx\" (UID: \"ac5dc138-69ca-41b7-8b92-b2b08109dd52\") " pod="odh-ai-gateway-infra/maas-api-756c48cb77-5k2qx"
Nov 02 13:56:36.538315 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:56:36.538233    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/45ebae56-61c5-4e2b-85c1-36f500719234-maas-api-tls\") pod \"maas-api-65f6775b5-7t49d\" (UID: \"45ebae56-61c5-4e2b-85c1-36f500719234\") " pod="odh-ai-gateway-infra/maas-api-65f6775b5-7t49d"
Nov 02 13:56:36.538315 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:56:36.538263    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac5dc138-69ca-41b7-8b92-b2b08109dd52-maas-api-tls\") pod \"maas-api-756c48cb77-5k2qx\" (UID: \"ac5dc138-69ca-41b7-8b92-b2b08109dd52\") " pod="odh-ai-gateway-infra/maas-api-756c48cb77-5k2qx"
Nov 02 13:56:36.538394 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:56:36.538333    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac5dc138-69ca-41b7-8b92-b2b08109dd52-metrics-certs\") pod \"maas-api-756c48cb77-5k2qx\" (UID: \"ac5dc138-69ca-41b7-8b92-b2b08109dd52\") " pod="odh-ai-gateway-infra/maas-api-756c48cb77-5k2qx"
Nov 02 13:56:36.538394 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:56:36.538383    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lrfx\" (UniqueName: \"kubernetes.io/projected/45ebae56-61c5-4e2b-85c1-36f500719234-kube-api-access-8lrfx\") pod \"maas-api-65f6775b5-7t49d\" (UID: \"45ebae56-61c5-4e2b-85c1-36f500719234\") " pod="odh-ai-gateway-infra/maas-api-65f6775b5-7t49d"
Nov 02 13:56:36.538472 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:56:36.538408    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/45ebae56-61c5-4e2b-85c1-36f500719234-openshift-service-ca\") pod \"maas-api-65f6775b5-7t49d\" (UID: \"45ebae56-61c5-4e2b-85c1-36f500719234\") " pod="odh-ai-gateway-infra/maas-api-65f6775b5-7t49d"
Nov 02 13:56:36.639265 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:56:36.639231    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lrfx\" (UniqueName: \"kubernetes.io/projected/45ebae56-61c5-4e2b-85c1-36f500719234-kube-api-access-8lrfx\") pod \"maas-api-65f6775b5-7t49d\" (UID: \"45ebae56-61c5-4e2b-85c1-36f500719234\") " pod="odh-ai-gateway-infra/maas-api-65f6775b5-7t49d"
Nov 02 13:56:36.639265 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:56:36.639270    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/45ebae56-61c5-4e2b-85c1-36f500719234-openshift-service-ca\") pod \"maas-api-65f6775b5-7t49d\" (UID: \"45ebae56-61c5-4e2b-85c1-36f500719234\") " pod="odh-ai-gateway-infra/maas-api-65f6775b5-7t49d"
Nov 02 13:56:36.639541 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:56:36.639305    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/45ebae56-61c5-4e2b-85c1-36f500719234-metrics-certs\") pod \"maas-api-65f6775b5-7t49d\" (UID: \"45ebae56-61c5-4e2b-85c1-36f500719234\") " pod="odh-ai-gateway-infra/maas-api-65f6775b5-7t49d"
Nov 02 13:56:36.639541 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:56:36.639343    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsm74\" (UniqueName: \"kubernetes.io/projected/ac5dc138-69ca-41b7-8b92-b2b08109dd52-kube-api-access-hsm74\") pod \"maas-api-756c48cb77-5k2qx\" (UID: \"ac5dc138-69ca-41b7-8b92-b2b08109dd52\") " pod="odh-ai-gateway-infra/maas-api-756c48cb77-5k2qx"
Nov 02 13:56:36.639541 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:56:36.639371    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac5dc138-69ca-41b7-8b92-b2b08109dd52-openshift-service-ca\") pod \"maas-api-756c48cb77-5k2qx\" (UID: \"ac5dc138-69ca-41b7-8b92-b2b08109dd52\") " pod="odh-ai-gateway-infra/maas-api-756c48cb77-5k2qx"
Nov 02 13:56:36.639541 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:56:36.639413    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/45ebae56-61c5-4e2b-85c1-36f500719234-maas-api-tls\") pod \"maas-api-65f6775b5-7t49d\" (UID: \"45ebae56-61c5-4e2b-85c1-36f500719234\") " pod="odh-ai-gateway-infra/maas-api-65f6775b5-7t49d"
Nov 02 13:56:36.639541 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:56:36.639441    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac5dc138-69ca-41b7-8b92-b2b08109dd52-maas-api-tls\") pod \"maas-api-756c48cb77-5k2qx\" (UID: \"ac5dc138-69ca-41b7-8b92-b2b08109dd52\") " pod="odh-ai-gateway-infra/maas-api-756c48cb77-5k2qx"
Nov 02 13:56:36.639541 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:56:36.639467    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac5dc138-69ca-41b7-8b92-b2b08109dd52-metrics-certs\") pod \"maas-api-756c48cb77-5k2qx\" (UID: \"ac5dc138-69ca-41b7-8b92-b2b08109dd52\") " pod="odh-ai-gateway-infra/maas-api-756c48cb77-5k2qx"
Nov 02 13:56:36.640199 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:56:36.640143    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/45ebae56-61c5-4e2b-85c1-36f500719234-openshift-service-ca\") pod \"maas-api-65f6775b5-7t49d\" (UID: \"45ebae56-61c5-4e2b-85c1-36f500719234\") " pod="odh-ai-gateway-infra/maas-api-65f6775b5-7t49d"
Nov 02 13:56:36.640550 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:56:36.640517    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac5dc138-69ca-41b7-8b92-b2b08109dd52-openshift-service-ca\") pod \"maas-api-756c48cb77-5k2qx\" (UID: \"ac5dc138-69ca-41b7-8b92-b2b08109dd52\") " pod="odh-ai-gateway-infra/maas-api-756c48cb77-5k2qx"
Nov 02 13:56:36.642239 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:56:36.642203    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/45ebae56-61c5-4e2b-85c1-36f500719234-metrics-certs\") pod \"maas-api-65f6775b5-7t49d\" (UID: \"45ebae56-61c5-4e2b-85c1-36f500719234\") " pod="odh-ai-gateway-infra/maas-api-65f6775b5-7t49d"
Nov 02 13:56:36.642538 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:56:36.642509    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac5dc138-69ca-41b7-8b92-b2b08109dd52-metrics-certs\") pod \"maas-api-756c48cb77-5k2qx\" (UID: \"ac5dc138-69ca-41b7-8b92-b2b08109dd52\") " pod="odh-ai-gateway-infra/maas-api-756c48cb77-5k2qx"
Nov 02 13:56:36.642699 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:56:36.642680    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/45ebae56-61c5-4e2b-85c1-36f500719234-maas-api-tls\") pod \"maas-api-65f6775b5-7t49d\" (UID: \"45ebae56-61c5-4e2b-85c1-36f500719234\") " pod="odh-ai-gateway-infra/maas-api-65f6775b5-7t49d"
Nov 02 13:56:36.642754 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:56:36.642680    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac5dc138-69ca-41b7-8b92-b2b08109dd52-maas-api-tls\") pod \"maas-api-756c48cb77-5k2qx\" (UID: \"ac5dc138-69ca-41b7-8b92-b2b08109dd52\") " pod="odh-ai-gateway-infra/maas-api-756c48cb77-5k2qx"
Nov 02 13:56:36.663295 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:56:36.663237    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lrfx\" (UniqueName: \"kubernetes.io/projected/45ebae56-61c5-4e2b-85c1-36f500719234-kube-api-access-8lrfx\") pod \"maas-api-65f6775b5-7t49d\" (UID: \"45ebae56-61c5-4e2b-85c1-36f500719234\") " pod="odh-ai-gateway-infra/maas-api-65f6775b5-7t49d"
Nov 02 13:56:36.663393 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:56:36.663321    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hsm74\" (UniqueName: \"kubernetes.io/projected/ac5dc138-69ca-41b7-8b92-b2b08109dd52-kube-api-access-hsm74\") pod \"maas-api-756c48cb77-5k2qx\" (UID: \"ac5dc138-69ca-41b7-8b92-b2b08109dd52\") " pod="odh-ai-gateway-infra/maas-api-756c48cb77-5k2qx"
Nov 02 13:56:36.718847 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:56:36.718817    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-65f6775b5-7t49d"
Nov 02 13:56:36.746112 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:56:36.746081    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-756c48cb77-5k2qx"
Nov 02 13:56:37.022068 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:56:37.022038    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-65f6775b5-7t49d"]
Nov 02 13:56:37.022213 ip-10-0-128-132 kubenswrapper[4559]: W1102 13:56:37.022188    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45ebae56_61c5_4e2b_85c1_36f500719234.slice/crio-dc6d3c199893fc3375811e24e4e010b1d39cf9b132dbcd5376ba7449ac1025a1 WatchSource:0}: Error finding container dc6d3c199893fc3375811e24e4e010b1d39cf9b132dbcd5376ba7449ac1025a1: Status 404 returned error can't find the container with id dc6d3c199893fc3375811e24e4e010b1d39cf9b132dbcd5376ba7449ac1025a1
Nov 02 13:56:37.030212 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:56:37.030185    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-jk7jx"]
Nov 02 13:56:37.033110 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:56:37.033084    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-jk7jx"]
Nov 02 13:56:37.059114 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:56:37.059084    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-756c48cb77-5k2qx"]
Nov 02 13:56:37.059509 ip-10-0-128-132 kubenswrapper[4559]: W1102 13:56:37.059489    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac5dc138_69ca_41b7_8b92_b2b08109dd52.slice/crio-21cf4c844005cb015c42fecaf753cfc1be7b8f2a62f6b3a6732b8916ae189dc0 WatchSource:0}: Error finding container 21cf4c844005cb015c42fecaf753cfc1be7b8f2a62f6b3a6732b8916ae189dc0: Status 404 returned error can't find the container with id 21cf4c844005cb015c42fecaf753cfc1be7b8f2a62f6b3a6732b8916ae189dc0
Nov 02 13:56:37.652477 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:56:37.652441    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-756c48cb77-5k2qx" event={"ID":"ac5dc138-69ca-41b7-8b92-b2b08109dd52","Type":"ContainerStarted","Data":"9fa9024d942303d2cbfd5be24e682dda83bbd202d1f191757db86ddfde8d5f7b"}
Nov 02 13:56:37.653004 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:56:37.652525    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-756c48cb77-5k2qx" event={"ID":"ac5dc138-69ca-41b7-8b92-b2b08109dd52","Type":"ContainerStarted","Data":"21cf4c844005cb015c42fecaf753cfc1be7b8f2a62f6b3a6732b8916ae189dc0"}
Nov 02 13:56:37.653004 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:56:37.652556    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-756c48cb77-5k2qx"
Nov 02 13:56:37.654538 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:56:37.654496    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-65f6775b5-7t49d" event={"ID":"45ebae56-61c5-4e2b-85c1-36f500719234","Type":"ContainerStarted","Data":"634fe511ffa9cbc53efd81330d9fd60d6be67df1e071e8e5184770075da793b3"}
Nov 02 13:56:37.654745 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:56:37.654554    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-65f6775b5-7t49d" event={"ID":"45ebae56-61c5-4e2b-85c1-36f500719234","Type":"ContainerStarted","Data":"dc6d3c199893fc3375811e24e4e010b1d39cf9b132dbcd5376ba7449ac1025a1"}
Nov 02 13:56:37.654745 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:56:37.654617    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-65f6775b5-7t49d"
Nov 02 13:56:37.696537 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:56:37.696436    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-756c48cb77-5k2qx" podStartSLOduration=1.29267826 podStartE2EDuration="1.696421376s" totalImagesPullingTime="403.743116ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:56:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:56:37.694971521 +0000 UTC m=+5393.950370628" watchObservedRunningTime="2026-11-02 13:56:37.696421376 +0000 UTC m=+5393.951820502"
Nov 02 13:56:37.739763 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:56:37.739704    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-65f6775b5-7t49d" podStartSLOduration=1.3183205 podStartE2EDuration="1.739690182s" totalImagesPullingTime="421.369682ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:56:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:56:37.737854225 +0000 UTC m=+5393.993253320" watchObservedRunningTime="2026-11-02 13:56:37.739690182 +0000 UTC m=+5393.995089277"
Nov 02 13:56:37.892086 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:56:37.892053    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7060ccc8-9297-45df-8de9-2ea1955a0692" path="/var/lib/kubelet/pods/7060ccc8-9297-45df-8de9-2ea1955a0692/volumes"
Nov 02 13:56:43.664929 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:56:43.664894    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-756c48cb77-5k2qx"
Nov 02 13:56:43.665393 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:56:43.665235    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-65f6775b5-7t49d"
Nov 02 13:56:43.744504 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:56:43.744469    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-65f6775b5-7t49d"]
Nov 02 13:56:43.744959 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:56:43.744926    4559 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-65f6775b5-7t49d" podUID="45ebae56-61c5-4e2b-85c1-36f500719234" containerName="maas-api" containerID="cri-o://634fe511ffa9cbc53efd81330d9fd60d6be67df1e071e8e5184770075da793b3" gracePeriod=30
Nov 02 13:56:44.003585 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:56:44.003557    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-65f6775b5-7t49d"
Nov 02 13:56:44.108653 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:56:44.108610    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/45ebae56-61c5-4e2b-85c1-36f500719234-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/45ebae56-61c5-4e2b-85c1-36f500719234-maas-api-tls\") pod \"45ebae56-61c5-4e2b-85c1-36f500719234\" (UID: \"45ebae56-61c5-4e2b-85c1-36f500719234\") "
Nov 02 13:56:44.108835 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:56:44.108694    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45ebae56-61c5-4e2b-85c1-36f500719234-kube-api-access-8lrfx\" (UniqueName: \"kubernetes.io/projected/45ebae56-61c5-4e2b-85c1-36f500719234-kube-api-access-8lrfx\") pod \"45ebae56-61c5-4e2b-85c1-36f500719234\" (UID: \"45ebae56-61c5-4e2b-85c1-36f500719234\") "
Nov 02 13:56:44.108835 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:56:44.108747    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45ebae56-61c5-4e2b-85c1-36f500719234-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/45ebae56-61c5-4e2b-85c1-36f500719234-openshift-service-ca\") pod \"45ebae56-61c5-4e2b-85c1-36f500719234\" (UID: \"45ebae56-61c5-4e2b-85c1-36f500719234\") "
Nov 02 13:56:44.108986 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:56:44.108880    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/45ebae56-61c5-4e2b-85c1-36f500719234-metrics-certs\" (UniqueName: \"kubernetes.io/secret/45ebae56-61c5-4e2b-85c1-36f500719234-metrics-certs\") pod \"45ebae56-61c5-4e2b-85c1-36f500719234\" (UID: \"45ebae56-61c5-4e2b-85c1-36f500719234\") "
Nov 02 13:56:44.109149 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:56:44.109121    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45ebae56-61c5-4e2b-85c1-36f500719234-openshift-service-ca" pod "45ebae56-61c5-4e2b-85c1-36f500719234" (UID: "45ebae56-61c5-4e2b-85c1-36f500719234"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:56:44.110848 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:56:44.110817    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45ebae56-61c5-4e2b-85c1-36f500719234-maas-api-tls" pod "45ebae56-61c5-4e2b-85c1-36f500719234" (UID: "45ebae56-61c5-4e2b-85c1-36f500719234"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:56:44.110989 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:56:44.110915    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45ebae56-61c5-4e2b-85c1-36f500719234-kube-api-access-8lrfx" pod "45ebae56-61c5-4e2b-85c1-36f500719234" (UID: "45ebae56-61c5-4e2b-85c1-36f500719234"). InnerVolumeSpecName "kube-api-access-8lrfx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:56:44.110989 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:56:44.110922    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45ebae56-61c5-4e2b-85c1-36f500719234-metrics-certs" pod "45ebae56-61c5-4e2b-85c1-36f500719234" (UID: "45ebae56-61c5-4e2b-85c1-36f500719234"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:56:44.210515 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:56:44.210433    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/45ebae56-61c5-4e2b-85c1-36f500719234-maas-api-tls\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:56:44.210515 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:56:44.210460    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8lrfx\" (UniqueName: \"kubernetes.io/projected/45ebae56-61c5-4e2b-85c1-36f500719234-kube-api-access-8lrfx\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:56:44.210515 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:56:44.210472    4559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/45ebae56-61c5-4e2b-85c1-36f500719234-openshift-service-ca\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:56:44.210515 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:56:44.210481    4559 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/45ebae56-61c5-4e2b-85c1-36f500719234-metrics-certs\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 13:56:44.690021 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:56:44.689983    4559 generic.go:356] "Generic (PLEG): container finished" podID="45ebae56-61c5-4e2b-85c1-36f500719234" containerID="634fe511ffa9cbc53efd81330d9fd60d6be67df1e071e8e5184770075da793b3" exitCode=0
Nov 02 13:56:44.690478 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:56:44.690034    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-65f6775b5-7t49d" event={"ID":"45ebae56-61c5-4e2b-85c1-36f500719234","Type":"ContainerDied","Data":"634fe511ffa9cbc53efd81330d9fd60d6be67df1e071e8e5184770075da793b3"}
Nov 02 13:56:44.690478 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:56:44.690042    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-65f6775b5-7t49d"
Nov 02 13:56:44.690478 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:56:44.690075    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-65f6775b5-7t49d" event={"ID":"45ebae56-61c5-4e2b-85c1-36f500719234","Type":"ContainerDied","Data":"dc6d3c199893fc3375811e24e4e010b1d39cf9b132dbcd5376ba7449ac1025a1"}
Nov 02 13:56:44.690478 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:56:44.690096    4559 scope.go:122] "RemoveContainer" containerID="634fe511ffa9cbc53efd81330d9fd60d6be67df1e071e8e5184770075da793b3"
Nov 02 13:56:44.699712 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:56:44.699685    4559 scope.go:122] "RemoveContainer" containerID="634fe511ffa9cbc53efd81330d9fd60d6be67df1e071e8e5184770075da793b3"
Nov 02 13:56:44.700039 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:56:44.700017    4559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"634fe511ffa9cbc53efd81330d9fd60d6be67df1e071e8e5184770075da793b3\": container with ID starting with 634fe511ffa9cbc53efd81330d9fd60d6be67df1e071e8e5184770075da793b3 not found: ID does not exist" containerID="634fe511ffa9cbc53efd81330d9fd60d6be67df1e071e8e5184770075da793b3"
Nov 02 13:56:44.700128 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:56:44.700052    4559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"634fe511ffa9cbc53efd81330d9fd60d6be67df1e071e8e5184770075da793b3"} err="failed to get container status \"634fe511ffa9cbc53efd81330d9fd60d6be67df1e071e8e5184770075da793b3\": rpc error: code = NotFound desc = could not find container \"634fe511ffa9cbc53efd81330d9fd60d6be67df1e071e8e5184770075da793b3\": container with ID starting with 634fe511ffa9cbc53efd81330d9fd60d6be67df1e071e8e5184770075da793b3 not found: ID does not exist"
Nov 02 13:56:44.744004 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:56:44.743961    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-65f6775b5-7t49d"]
Nov 02 13:56:44.757083 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:56:44.757045    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-65f6775b5-7t49d"]
Nov 02 13:56:45.085608 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:56:45.085527    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqcmj" containerName="extractor"
Nov 02 13:56:45.891878 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:56:45.891842    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45ebae56-61c5-4e2b-85c1-36f500719234" path="/var/lib/kubelet/pods/45ebae56-61c5-4e2b-85c1-36f500719234/volumes"
Nov 02 13:56:47.032307 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:56:47.032271    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-9286p"]
Nov 02 13:56:47.044255 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:56:47.044223    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-9286p"]
Nov 02 13:56:47.627151 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:56:47.627116    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"]
Nov 02 13:56:47.891597 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:56:47.891520    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a56aac52-8bf3-4a5b-98ca-91afba77505f" path="/var/lib/kubelet/pods/a56aac52-8bf3-4a5b-98ca-91afba77505f/volumes"
Nov 02 13:56:48.136270 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:56:48.136224    4559 scope.go:122] "RemoveContainer" containerID="e560ea6bf1cc319621214e3a56c604fb17847c8910919741840d1d5aa2e2eddc"
Nov 02 13:56:48.148381 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:56:48.148352    4559 scope.go:122] "RemoveContainer" containerID="32dc46760cd32df531b5f9b8fee582eab3b32522e6c2880e9723499d9e124f6d"
Nov 02 13:56:48.159463 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:56:48.159437    4559 scope.go:122] "RemoveContainer" containerID="2fd96168dc310ca0bf1cb443d3017ad839de0881c7f6742c95c60f6c8be18a28"
Nov 02 13:56:48.169914 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:56:48.169891    4559 scope.go:122] "RemoveContainer" containerID="ccab5b95f29bdc19dcf86ad102d16e5a8ed709afe347d32fe162214310396021"
Nov 02 13:56:48.180356 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:56:48.180331    4559 scope.go:122] "RemoveContainer" containerID="fd9c2b45dce73cad68b17ac3ab5c789d5a400e606ec1ae1b5490e796e52b32dd"
Nov 02 13:57:15.084739 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:57:15.084703    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqcmj" containerName="extractor"
Nov 02 13:57:16.785530 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:57:16.785498    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"]
Nov 02 13:57:21.523425 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:57:21.523388    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"]
Nov 02 13:57:38.425453 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:57:38.425413    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"]
Nov 02 13:57:43.122216 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:57:43.122176    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"]
Nov 02 13:57:45.187405 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:57:45.187355    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqcmj" containerName="extractor"
Nov 02 13:57:48.232111 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:57:48.232082    4559 scope.go:122] "RemoveContainer" containerID="e73c7eebf0ac29a38d2d1d7d27efd35be212551b5fea495c7df99fc904fa8c59"
Nov 02 13:57:48.240721 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:57:48.240700    4559 scope.go:122] "RemoveContainer" containerID="eb1c7c5d3f180b4d158267b9b63c1993efb0d5152fba5c0fef225be45afc676d"
Nov 02 13:57:48.248024 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:57:48.248007    4559 scope.go:122] "RemoveContainer" containerID="42533ec5c78f54720a8298b7e2dbe49c1bf5dce2adf6c022de3889be1c6ac494"
Nov 02 13:57:48.630822 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:57:48.630743    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"]
Nov 02 13:58:11.118749 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:58:11.118714    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"]
Nov 02 13:58:15.085115 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:58:15.085077    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqcmj" containerName="extractor"
Nov 02 13:58:21.421791 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:58:21.421754    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"]
Nov 02 13:58:26.128515 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:58:26.128481    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"]
Nov 02 13:58:28.764667 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:58:28.762726    4559 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-qmrsp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.15:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 13:58:28.764667 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:58:28.762790    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qmrsp" podUID="a534baf3-8ff8-4dca-bc27-9022feaaca75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.15:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:58:29.010795 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:58:29.010738    4559 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-jr7g7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.16:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 13:58:29.011126 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:58:29.011079    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jr7g7" podUID="a15c4957-178b-41d5-ae44-e79c37fb55d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.16:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:58:34.182239 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:58:34.182206    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"]
Nov 02 13:58:45.084771 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:58:45.084736    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqcmj" containerName="extractor"
Nov 02 13:58:48.271564 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:58:48.271530    4559 scope.go:122] "RemoveContainer" containerID="86e22255d58277b3cd24a08499bed83486a7063a37240ec3c83b5968d677f5ac"
Nov 02 13:58:53.240131 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:58:53.240097    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"]
Nov 02 13:58:56.229711 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:58:56.228893    4559 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-gbd89 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.13:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 13:58:56.229711 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:58:56.228974    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-gbd89" podUID="f55beaae-7351-41b7-89ca-3142739266cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.13:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:59:02.729569 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:59:02.729536    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"]
Nov 02 13:59:08.196187 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:59:08.196145    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"]
Nov 02 13:59:15.085072 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:59:15.085030    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqcmj" containerName="extractor"
Nov 02 13:59:19.593318 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:59:19.593286    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"]
Nov 02 13:59:27.029436 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:59:27.029401    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8sq95"]
Nov 02 13:59:27.041878 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:59:27.041852    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8sq95"]
Nov 02 13:59:27.891289 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:59:27.891255    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97175416-27c8-4665-8376-07a963a467c2" path="/var/lib/kubelet/pods/97175416-27c8-4665-8376-07a963a467c2/volumes"
Nov 02 13:59:35.401843 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:59:35.401805    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"]
Nov 02 13:59:45.084793 ip-10-0-128-132 kubenswrapper[4559]: E1102 13:59:45.084759    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqcmj" containerName="extractor"
Nov 02 13:59:58.774665 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:59:58.771094    4559 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-qmrsp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.15:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 13:59:58.774665 ip-10-0-128-132 kubenswrapper[4559]: I1102 13:59:58.771163    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qmrsp" podUID="a534baf3-8ff8-4dca-bc27-9022feaaca75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.15:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:00:05.487868 ip-10-0-128-132 kubenswrapper[4559]: I1102 14:00:05.487827    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-84c23cd0-m5bck"]
Nov 02 14:00:05.488273 ip-10-0-128-132 kubenswrapper[4559]: I1102 14:00:05.488202    4559 cpu_manager.go:414] "containerMap: removing container" podUID="45ebae56-61c5-4e2b-85c1-36f500719234" containerName="maas-api"
Nov 02 14:00:05.488273 ip-10-0-128-132 kubenswrapper[4559]: I1102 14:00:05.488213    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45ebae56-61c5-4e2b-85c1-36f500719234" containerName="maas-api"
Nov 02 14:00:05.488354 ip-10-0-128-132 kubenswrapper[4559]: I1102 14:00:05.488295    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="45ebae56-61c5-4e2b-85c1-36f500719234" containerName="maas-api"
Nov 02 14:00:05.491336 ip-10-0-128-132 kubenswrapper[4559]: I1102 14:00:05.491318    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84c23cd0-m5bck"
Nov 02 14:00:05.511046 ip-10-0-128-132 kubenswrapper[4559]: I1102 14:00:05.511004    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-84c23cd0-m5bck"]
Nov 02 14:00:05.521538 ip-10-0-128-132 kubenswrapper[4559]: I1102 14:00:05.521513    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 14:00:05.521683 ip-10-0-128-132 kubenswrapper[4559]: I1102 14:00:05.521572    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-45fzn\""
Nov 02 14:00:05.566995 ip-10-0-128-132 kubenswrapper[4559]: I1102 14:00:05.566965    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7ee87ed-29f7-4fd2-bdc8-3a22513daf12-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-84c23cd0-m5bck\" (UID: \"b7ee87ed-29f7-4fd2-bdc8-3a22513daf12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84c23cd0-m5bck"
Nov 02 14:00:05.567118 ip-10-0-128-132 kubenswrapper[4559]: I1102 14:00:05.567008    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzn76\" (UniqueName: \"kubernetes.io/projected/b7ee87ed-29f7-4fd2-bdc8-3a22513daf12-kube-api-access-rzn76\") pod \"maas-api-revoke-sub-84c23cd0-m5bck\" (UID: \"b7ee87ed-29f7-4fd2-bdc8-3a22513daf12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84c23cd0-m5bck"
Nov 02 14:00:05.567165 ip-10-0-128-132 kubenswrapper[4559]: I1102 14:00:05.567147    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7ee87ed-29f7-4fd2-bdc8-3a22513daf12-maas-api-xks-ca\") pod \"maas-api-revoke-sub-84c23cd0-m5bck\" (UID: \"b7ee87ed-29f7-4fd2-bdc8-3a22513daf12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84c23cd0-m5bck"
Nov 02 14:00:05.668175 ip-10-0-128-132 kubenswrapper[4559]: I1102 14:00:05.668151    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7ee87ed-29f7-4fd2-bdc8-3a22513daf12-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-84c23cd0-m5bck\" (UID: \"b7ee87ed-29f7-4fd2-bdc8-3a22513daf12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84c23cd0-m5bck"
Nov 02 14:00:05.668326 ip-10-0-128-132 kubenswrapper[4559]: I1102 14:00:05.668189    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzn76\" (UniqueName: \"kubernetes.io/projected/b7ee87ed-29f7-4fd2-bdc8-3a22513daf12-kube-api-access-rzn76\") pod \"maas-api-revoke-sub-84c23cd0-m5bck\" (UID: \"b7ee87ed-29f7-4fd2-bdc8-3a22513daf12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84c23cd0-m5bck"
Nov 02 14:00:05.668326 ip-10-0-128-132 kubenswrapper[4559]: I1102 14:00:05.668235    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7ee87ed-29f7-4fd2-bdc8-3a22513daf12-maas-api-xks-ca\") pod \"maas-api-revoke-sub-84c23cd0-m5bck\" (UID: \"b7ee87ed-29f7-4fd2-bdc8-3a22513daf12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84c23cd0-m5bck"
Nov 02 14:00:05.668827 ip-10-0-128-132 kubenswrapper[4559]: I1102 14:00:05.668804    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7ee87ed-29f7-4fd2-bdc8-3a22513daf12-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-84c23cd0-m5bck\" (UID: \"b7ee87ed-29f7-4fd2-bdc8-3a22513daf12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84c23cd0-m5bck"
Nov 02 14:00:05.670381 ip-10-0-128-132 kubenswrapper[4559]: I1102 14:00:05.670361    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7ee87ed-29f7-4fd2-bdc8-3a22513daf12-maas-api-xks-ca\") pod \"maas-api-revoke-sub-84c23cd0-m5bck\" (UID: \"b7ee87ed-29f7-4fd2-bdc8-3a22513daf12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84c23cd0-m5bck"
Nov 02 14:00:05.692221 ip-10-0-128-132 kubenswrapper[4559]: I1102 14:00:05.692194    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzn76\" (UniqueName: \"kubernetes.io/projected/b7ee87ed-29f7-4fd2-bdc8-3a22513daf12-kube-api-access-rzn76\") pod \"maas-api-revoke-sub-84c23cd0-m5bck\" (UID: \"b7ee87ed-29f7-4fd2-bdc8-3a22513daf12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84c23cd0-m5bck"
Nov 02 14:00:05.803405 ip-10-0-128-132 kubenswrapper[4559]: I1102 14:00:05.803344    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84c23cd0-m5bck"
Nov 02 14:00:06.103245 ip-10-0-128-132 kubenswrapper[4559]: I1102 14:00:06.103200    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-84c23cd0-m5bck"]
Nov 02 14:00:06.104128 ip-10-0-128-132 kubenswrapper[4559]: W1102 14:00:06.104085    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7ee87ed_29f7_4fd2_bdc8_3a22513daf12.slice/crio-72034af4989db4a5be84e2ac360286f2d65039f31f630e531d8f0b0ed0502d39 WatchSource:0}: Error finding container 72034af4989db4a5be84e2ac360286f2d65039f31f630e531d8f0b0ed0502d39: Status 404 returned error can't find the container with id 72034af4989db4a5be84e2ac360286f2d65039f31f630e531d8f0b0ed0502d39
Nov 02 14:00:07.012500 ip-10-0-128-132 kubenswrapper[4559]: I1102 14:00:07.012457    4559 generic.go:356] "Generic (PLEG): container finished" podID="b7ee87ed-29f7-4fd2-bdc8-3a22513daf12" containerID="386ca1bfd13349ae24873f4c137b73ae39e44a2f8e6f26d7f0477de4477578d7" exitCode=0
Nov 02 14:00:07.013073 ip-10-0-128-132 kubenswrapper[4559]: I1102 14:00:07.012559    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84c23cd0-m5bck" event={"ID":"b7ee87ed-29f7-4fd2-bdc8-3a22513daf12","Type":"ContainerDied","Data":"386ca1bfd13349ae24873f4c137b73ae39e44a2f8e6f26d7f0477de4477578d7"}
Nov 02 14:00:07.013073 ip-10-0-128-132 kubenswrapper[4559]: I1102 14:00:07.012608    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84c23cd0-m5bck" event={"ID":"b7ee87ed-29f7-4fd2-bdc8-3a22513daf12","Type":"ContainerStarted","Data":"72034af4989db4a5be84e2ac360286f2d65039f31f630e531d8f0b0ed0502d39"}
Nov 02 14:00:08.692248 ip-10-0-128-132 kubenswrapper[4559]: I1102 14:00:08.692216    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84c23cd0-m5bck"
Nov 02 14:00:08.860149 ip-10-0-128-132 kubenswrapper[4559]: I1102 14:00:08.860039    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7ee87ed-29f7-4fd2-bdc8-3a22513daf12-kube-api-access-rzn76\" (UniqueName: \"kubernetes.io/projected/b7ee87ed-29f7-4fd2-bdc8-3a22513daf12-kube-api-access-rzn76\") pod \"b7ee87ed-29f7-4fd2-bdc8-3a22513daf12\" (UID: \"b7ee87ed-29f7-4fd2-bdc8-3a22513daf12\") "
Nov 02 14:00:08.870666 ip-10-0-128-132 kubenswrapper[4559]: I1102 14:00:08.861766    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b7ee87ed-29f7-4fd2-bdc8-3a22513daf12-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7ee87ed-29f7-4fd2-bdc8-3a22513daf12-maas-api-xks-ca\") pod \"b7ee87ed-29f7-4fd2-bdc8-3a22513daf12\" (UID: \"b7ee87ed-29f7-4fd2-bdc8-3a22513daf12\") "
Nov 02 14:00:08.870666 ip-10-0-128-132 kubenswrapper[4559]: I1102 14:00:08.861890    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7ee87ed-29f7-4fd2-bdc8-3a22513daf12-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7ee87ed-29f7-4fd2-bdc8-3a22513daf12-maas-api-openshift-ca\") pod \"b7ee87ed-29f7-4fd2-bdc8-3a22513daf12\" (UID: \"b7ee87ed-29f7-4fd2-bdc8-3a22513daf12\") "
Nov 02 14:00:08.870666 ip-10-0-128-132 kubenswrapper[4559]: I1102 14:00:08.862522    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7ee87ed-29f7-4fd2-bdc8-3a22513daf12-maas-api-openshift-ca" pod "b7ee87ed-29f7-4fd2-bdc8-3a22513daf12" (UID: "b7ee87ed-29f7-4fd2-bdc8-3a22513daf12"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:00:08.946832 ip-10-0-128-132 kubenswrapper[4559]: I1102 14:00:08.946783    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7ee87ed-29f7-4fd2-bdc8-3a22513daf12-maas-api-xks-ca" pod "b7ee87ed-29f7-4fd2-bdc8-3a22513daf12" (UID: "b7ee87ed-29f7-4fd2-bdc8-3a22513daf12"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:00:08.955810 ip-10-0-128-132 kubenswrapper[4559]: I1102 14:00:08.955771    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7ee87ed-29f7-4fd2-bdc8-3a22513daf12-kube-api-access-rzn76" pod "b7ee87ed-29f7-4fd2-bdc8-3a22513daf12" (UID: "b7ee87ed-29f7-4fd2-bdc8-3a22513daf12"). InnerVolumeSpecName "kube-api-access-rzn76". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:00:08.970664 ip-10-0-128-132 kubenswrapper[4559]: I1102 14:00:08.969674    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7ee87ed-29f7-4fd2-bdc8-3a22513daf12-maas-api-openshift-ca\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 14:00:08.970664 ip-10-0-128-132 kubenswrapper[4559]: I1102 14:00:08.969714    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rzn76\" (UniqueName: \"kubernetes.io/projected/b7ee87ed-29f7-4fd2-bdc8-3a22513daf12-kube-api-access-rzn76\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 14:00:08.970664 ip-10-0-128-132 kubenswrapper[4559]: I1102 14:00:08.969748    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7ee87ed-29f7-4fd2-bdc8-3a22513daf12-maas-api-xks-ca\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 14:00:09.057668 ip-10-0-128-132 kubenswrapper[4559]: I1102 14:00:09.048801    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84c23cd0-m5bck" event={"ID":"b7ee87ed-29f7-4fd2-bdc8-3a22513daf12","Type":"ContainerDied","Data":"72034af4989db4a5be84e2ac360286f2d65039f31f630e531d8f0b0ed0502d39"}
Nov 02 14:00:09.057668 ip-10-0-128-132 kubenswrapper[4559]: I1102 14:00:09.048861    4559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="72034af4989db4a5be84e2ac360286f2d65039f31f630e531d8f0b0ed0502d39"
Nov 02 14:00:09.057668 ip-10-0-128-132 kubenswrapper[4559]: I1102 14:00:09.048932    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84c23cd0-m5bck"
Nov 02 14:00:15.084432 ip-10-0-128-132 kubenswrapper[4559]: E1102 14:00:15.084397    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqcmj" containerName="extractor"
Nov 02 14:00:15.847950 ip-10-0-128-132 kubenswrapper[4559]: I1102 14:00:15.845024    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5219fe2-lz8cr"]
Nov 02 14:00:15.847950 ip-10-0-128-132 kubenswrapper[4559]: I1102 14:00:15.846045    4559 cpu_manager.go:414] "containerMap: removing container" podUID="b7ee87ed-29f7-4fd2-bdc8-3a22513daf12" containerName="revoke-keys"
Nov 02 14:00:15.847950 ip-10-0-128-132 kubenswrapper[4559]: I1102 14:00:15.846066    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7ee87ed-29f7-4fd2-bdc8-3a22513daf12" containerName="revoke-keys"
Nov 02 14:00:15.847950 ip-10-0-128-132 kubenswrapper[4559]: I1102 14:00:15.846221    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7ee87ed-29f7-4fd2-bdc8-3a22513daf12" containerName="revoke-keys"
Nov 02 14:00:15.850445 ip-10-0-128-132 kubenswrapper[4559]: I1102 14:00:15.850413    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5219fe2-lz8cr"
Nov 02 14:00:15.852506 ip-10-0-128-132 kubenswrapper[4559]: I1102 14:00:15.852473    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2n5lq\" (UniqueName: \"kubernetes.io/projected/9dd5edb8-96ca-46ac-ae54-5b78d4911c22-kube-api-access-2n5lq\") pod \"maas-api-revoke-sub-e5219fe2-lz8cr\" (UID: \"9dd5edb8-96ca-46ac-ae54-5b78d4911c22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5219fe2-lz8cr"
Nov 02 14:00:15.852694 ip-10-0-128-132 kubenswrapper[4559]: I1102 14:00:15.852674    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9dd5edb8-96ca-46ac-ae54-5b78d4911c22-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e5219fe2-lz8cr\" (UID: \"9dd5edb8-96ca-46ac-ae54-5b78d4911c22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5219fe2-lz8cr"
Nov 02 14:00:15.852842 ip-10-0-128-132 kubenswrapper[4559]: I1102 14:00:15.852719    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9dd5edb8-96ca-46ac-ae54-5b78d4911c22-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e5219fe2-lz8cr\" (UID: \"9dd5edb8-96ca-46ac-ae54-5b78d4911c22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5219fe2-lz8cr"
Nov 02 14:00:15.868908 ip-10-0-128-132 kubenswrapper[4559]: I1102 14:00:15.868882    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5219fe2-lz8cr"]
Nov 02 14:00:15.879039 ip-10-0-128-132 kubenswrapper[4559]: I1102 14:00:15.879017    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 14:00:15.879139 ip-10-0-128-132 kubenswrapper[4559]: I1102 14:00:15.879039    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-45fzn\""
Nov 02 14:00:15.953790 ip-10-0-128-132 kubenswrapper[4559]: I1102 14:00:15.953762    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2n5lq\" (UniqueName: \"kubernetes.io/projected/9dd5edb8-96ca-46ac-ae54-5b78d4911c22-kube-api-access-2n5lq\") pod \"maas-api-revoke-sub-e5219fe2-lz8cr\" (UID: \"9dd5edb8-96ca-46ac-ae54-5b78d4911c22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5219fe2-lz8cr"
Nov 02 14:00:15.953977 ip-10-0-128-132 kubenswrapper[4559]: I1102 14:00:15.953808    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9dd5edb8-96ca-46ac-ae54-5b78d4911c22-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e5219fe2-lz8cr\" (UID: \"9dd5edb8-96ca-46ac-ae54-5b78d4911c22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5219fe2-lz8cr"
Nov 02 14:00:15.953977 ip-10-0-128-132 kubenswrapper[4559]: I1102 14:00:15.953920    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9dd5edb8-96ca-46ac-ae54-5b78d4911c22-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e5219fe2-lz8cr\" (UID: \"9dd5edb8-96ca-46ac-ae54-5b78d4911c22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5219fe2-lz8cr"
Nov 02 14:00:15.954451 ip-10-0-128-132 kubenswrapper[4559]: I1102 14:00:15.954427    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9dd5edb8-96ca-46ac-ae54-5b78d4911c22-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e5219fe2-lz8cr\" (UID: \"9dd5edb8-96ca-46ac-ae54-5b78d4911c22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5219fe2-lz8cr"
Nov 02 14:00:15.956187 ip-10-0-128-132 kubenswrapper[4559]: I1102 14:00:15.956169    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9dd5edb8-96ca-46ac-ae54-5b78d4911c22-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e5219fe2-lz8cr\" (UID: \"9dd5edb8-96ca-46ac-ae54-5b78d4911c22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5219fe2-lz8cr"
Nov 02 14:00:15.977735 ip-10-0-128-132 kubenswrapper[4559]: I1102 14:00:15.977712    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2n5lq\" (UniqueName: \"kubernetes.io/projected/9dd5edb8-96ca-46ac-ae54-5b78d4911c22-kube-api-access-2n5lq\") pod \"maas-api-revoke-sub-e5219fe2-lz8cr\" (UID: \"9dd5edb8-96ca-46ac-ae54-5b78d4911c22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5219fe2-lz8cr"
Nov 02 14:00:16.167129 ip-10-0-128-132 kubenswrapper[4559]: I1102 14:00:16.167078    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5219fe2-lz8cr"
Nov 02 14:00:16.465515 ip-10-0-128-132 kubenswrapper[4559]: I1102 14:00:16.465482    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5219fe2-lz8cr"]
Nov 02 14:00:16.466800 ip-10-0-128-132 kubenswrapper[4559]: W1102 14:00:16.466776    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9dd5edb8_96ca_46ac_ae54_5b78d4911c22.slice/crio-cddf67116c75ef40bda730305bf52f064304548da9c3a5d5fdece946ef90c114 WatchSource:0}: Error finding container cddf67116c75ef40bda730305bf52f064304548da9c3a5d5fdece946ef90c114: Status 404 returned error can't find the container with id cddf67116c75ef40bda730305bf52f064304548da9c3a5d5fdece946ef90c114
Nov 02 14:00:17.088190 ip-10-0-128-132 kubenswrapper[4559]: I1102 14:00:17.088117    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5219fe2-lz8cr" event={"ID":"9dd5edb8-96ca-46ac-ae54-5b78d4911c22","Type":"ContainerStarted","Data":"585f982e95c0e3bf3cf21b4936d92c6d035a3a9a90e27f16bebbaa93cc961cec"}
Nov 02 14:00:17.088190 ip-10-0-128-132 kubenswrapper[4559]: I1102 14:00:17.088164    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5219fe2-lz8cr" event={"ID":"9dd5edb8-96ca-46ac-ae54-5b78d4911c22","Type":"ContainerStarted","Data":"cddf67116c75ef40bda730305bf52f064304548da9c3a5d5fdece946ef90c114"}
Nov 02 14:00:17.130022 ip-10-0-128-132 kubenswrapper[4559]: I1102 14:00:17.129964    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5219fe2-lz8cr" podStartSLOduration=2.129946822 podStartE2EDuration="2.129946822s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:00:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:00:17.129012611 +0000 UTC m=+5613.384411705" watchObservedRunningTime="2026-11-02 14:00:17.129946822 +0000 UTC m=+5613.385345918"
Nov 02 14:00:18.119326 ip-10-0-128-132 kubenswrapper[4559]: I1102 14:00:18.119272    4559 generic.go:356] "Generic (PLEG): container finished" podID="9dd5edb8-96ca-46ac-ae54-5b78d4911c22" containerID="585f982e95c0e3bf3cf21b4936d92c6d035a3a9a90e27f16bebbaa93cc961cec" exitCode=0
Nov 02 14:00:18.120000 ip-10-0-128-132 kubenswrapper[4559]: I1102 14:00:18.119948    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5219fe2-lz8cr" event={"ID":"9dd5edb8-96ca-46ac-ae54-5b78d4911c22","Type":"ContainerDied","Data":"585f982e95c0e3bf3cf21b4936d92c6d035a3a9a90e27f16bebbaa93cc961cec"}
Nov 02 14:00:19.989923 ip-10-0-128-132 kubenswrapper[4559]: I1102 14:00:19.989897    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5219fe2-lz8cr"
Nov 02 14:00:20.139521 ip-10-0-128-132 kubenswrapper[4559]: I1102 14:00:20.139484    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9dd5edb8-96ca-46ac-ae54-5b78d4911c22-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9dd5edb8-96ca-46ac-ae54-5b78d4911c22-maas-api-openshift-ca\") pod \"9dd5edb8-96ca-46ac-ae54-5b78d4911c22\" (UID: \"9dd5edb8-96ca-46ac-ae54-5b78d4911c22\") "
Nov 02 14:00:20.139736 ip-10-0-128-132 kubenswrapper[4559]: I1102 14:00:20.139620    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9dd5edb8-96ca-46ac-ae54-5b78d4911c22-kube-api-access-2n5lq\" (UniqueName: \"kubernetes.io/projected/9dd5edb8-96ca-46ac-ae54-5b78d4911c22-kube-api-access-2n5lq\") pod \"9dd5edb8-96ca-46ac-ae54-5b78d4911c22\" (UID: \"9dd5edb8-96ca-46ac-ae54-5b78d4911c22\") "
Nov 02 14:00:20.139736 ip-10-0-128-132 kubenswrapper[4559]: I1102 14:00:20.139705    4559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9dd5edb8-96ca-46ac-ae54-5b78d4911c22-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9dd5edb8-96ca-46ac-ae54-5b78d4911c22-maas-api-xks-ca\") pod \"9dd5edb8-96ca-46ac-ae54-5b78d4911c22\" (UID: \"9dd5edb8-96ca-46ac-ae54-5b78d4911c22\") "
Nov 02 14:00:20.139848 ip-10-0-128-132 kubenswrapper[4559]: I1102 14:00:20.139827    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9dd5edb8-96ca-46ac-ae54-5b78d4911c22-maas-api-openshift-ca" pod "9dd5edb8-96ca-46ac-ae54-5b78d4911c22" (UID: "9dd5edb8-96ca-46ac-ae54-5b78d4911c22"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:00:20.140073 ip-10-0-128-132 kubenswrapper[4559]: I1102 14:00:20.140052    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9dd5edb8-96ca-46ac-ae54-5b78d4911c22-maas-api-openshift-ca\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 14:00:20.141813 ip-10-0-128-132 kubenswrapper[4559]: I1102 14:00:20.141785    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9dd5edb8-96ca-46ac-ae54-5b78d4911c22-kube-api-access-2n5lq" pod "9dd5edb8-96ca-46ac-ae54-5b78d4911c22" (UID: "9dd5edb8-96ca-46ac-ae54-5b78d4911c22"). InnerVolumeSpecName "kube-api-access-2n5lq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:00:20.141813 ip-10-0-128-132 kubenswrapper[4559]: I1102 14:00:20.141786    4559 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9dd5edb8-96ca-46ac-ae54-5b78d4911c22-maas-api-xks-ca" pod "9dd5edb8-96ca-46ac-ae54-5b78d4911c22" (UID: "9dd5edb8-96ca-46ac-ae54-5b78d4911c22"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:00:20.240856 ip-10-0-128-132 kubenswrapper[4559]: I1102 14:00:20.240824    4559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2n5lq\" (UniqueName: \"kubernetes.io/projected/9dd5edb8-96ca-46ac-ae54-5b78d4911c22-kube-api-access-2n5lq\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 14:00:20.240856 ip-10-0-128-132 kubenswrapper[4559]: I1102 14:00:20.240850    4559 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9dd5edb8-96ca-46ac-ae54-5b78d4911c22-maas-api-xks-ca\") on node \"ip-10-0-128-132.ec2.internal\" DevicePath \"\""
Nov 02 14:00:20.288731 ip-10-0-128-132 kubenswrapper[4559]: I1102 14:00:20.288710    4559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5219fe2-lz8cr"
Nov 02 14:00:20.288731 ip-10-0-128-132 kubenswrapper[4559]: I1102 14:00:20.288714    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5219fe2-lz8cr" event={"ID":"9dd5edb8-96ca-46ac-ae54-5b78d4911c22","Type":"ContainerDied","Data":"cddf67116c75ef40bda730305bf52f064304548da9c3a5d5fdece946ef90c114"}
Nov 02 14:00:20.288906 ip-10-0-128-132 kubenswrapper[4559]: I1102 14:00:20.288758    4559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cddf67116c75ef40bda730305bf52f064304548da9c3a5d5fdece946ef90c114"
Nov 02 14:00:45.085852 ip-10-0-128-132 kubenswrapper[4559]: E1102 14:00:45.085762    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqcmj" containerName="extractor"
Nov 02 14:00:48.298288 ip-10-0-128-132 kubenswrapper[4559]: I1102 14:00:48.298256    4559 scope.go:122] "RemoveContainer" containerID="3ca94fa1914f41ad1288f1baf4507f66ef47ea887187d332c375497ef9ec301c"
Nov 02 14:01:11.242805 ip-10-0-128-132 kubenswrapper[4559]: I1102 14:01:11.242761    4559 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-gbd89 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.13:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 14:01:11.243537 ip-10-0-128-132 kubenswrapper[4559]: I1102 14:01:11.243489    4559 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-gbd89" podUID="f55beaae-7351-41b7-89ca-3142739266cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.13:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:01:13.030152 ip-10-0-128-132 kubenswrapper[4559]: I1102 14:01:13.030114    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-tcrqn"]
Nov 02 14:01:13.043570 ip-10-0-128-132 kubenswrapper[4559]: I1102 14:01:13.043544    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-tcrqn"]
Nov 02 14:01:13.890988 ip-10-0-128-132 kubenswrapper[4559]: I1102 14:01:13.890953    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8afdd12c-bbde-47d5-9ac6-7ed7ff7ae147" path="/var/lib/kubelet/pods/8afdd12c-bbde-47d5-9ac6-7ed7ff7ae147/volumes"
Nov 02 14:01:15.084942 ip-10-0-128-132 kubenswrapper[4559]: E1102 14:01:15.084907    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqcmj" containerName="extractor"
Nov 02 14:01:29.036359 ip-10-0-128-132 kubenswrapper[4559]: I1102 14:01:29.036322    4559 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-zxrzc"]
Nov 02 14:01:29.049547 ip-10-0-128-132 kubenswrapper[4559]: I1102 14:01:29.049519    4559 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-zxrzc"]
Nov 02 14:01:29.416886 ip-10-0-128-132 kubenswrapper[4559]: I1102 14:01:29.416809    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l9pbf"]
Nov 02 14:01:29.904242 ip-10-0-128-132 kubenswrapper[4559]: I1102 14:01:29.904193    4559 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="296049f9-5f5e-4ec2-b170-534105e1b937" path="/var/lib/kubelet/pods/296049f9-5f5e-4ec2-b170-534105e1b937/volumes"
Nov 02 14:01:38.646531 ip-10-0-128-132 kubenswrapper[4559]: I1102 14:01:38.646500    4559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6768bb47c6-q87t7_a00817af-1ee2-4bad-b007-b058928a12b7/authorino/0.log"
Nov 02 14:01:45.086307 ip-10-0-128-132 kubenswrapper[4559]: E1102 14:01:45.086256    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqcmj" containerName="extractor"
Nov 02 14:01:50.627732 ip-10-0-128-132 kubenswrapper[4559]: I1102 14:01:50.627698    4559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-64577f84c6-5nmh8_4af87e02-6dc0-42ef-93df-26e174ec85df/manager/0.log"
Nov 02 14:01:51.551361 ip-10-0-128-132 kubenswrapper[4559]: I1102 14:01:51.551328    4559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5597f6b9f6-cbcqg_b01f338f-5aac-4cbb-9c18-178defd659c1/manager/0.log"
Nov 02 14:01:53.139364 ip-10-0-128-132 kubenswrapper[4559]: I1102 14:01:53.139329    4559 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-756c48cb77-5k2qx_ac5dc138-69ca-41b7-8b92-b2b08109dd52/maas-api/0.log"
Nov 02 14:01:54.578874 ip-10-0-128-132 kubenswrapper[4559]: I1102 14:01:54.578847    4559 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-84c23cd0-m5bck_b7ee87ed-29f7-4fd2-bdc8-3a22513daf12/revoke-keys/0.log"
Nov 02 14:01:56.331114 ip-10-0-128-132 kubenswrapper[4559]: I1102 14:01:56.331086    4559 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-e5219fe2-lz8cr_9dd5edb8-96ca-46ac-ae54-5b78d4911c22/revoke-keys/0.log"
Nov 02 14:01:59.221338 ip-10-0-128-132 kubenswrapper[4559]: I1102 14:01:59.221306    4559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6768bb47c6-q87t7_a00817af-1ee2-4bad-b007-b058928a12b7/authorino/0.log"
Nov 02 14:02:00.337410 ip-10-0-128-132 kubenswrapper[4559]: I1102 14:02:00.337384    4559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-l9pbf_ae914cec-8158-4c77-9870-e0fa6573c7b6/limitador/0.log"
Nov 02 14:02:01.912457 ip-10-0-128-132 kubenswrapper[4559]: I1102 14:02:01.912429    4559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-9c9fcfcf5-9pxlh_4aea6613-d504-4ab3-80cd-a1c4010709a7/kube-auth-proxy/0.log"
Nov 02 14:02:02.717450 ip-10-0-128-132 kubenswrapper[4559]: I1102 14:02:02.717417    4559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-5wtjm_b9661a3b-8a07-4a30-9893-20ab1d95a2a7/istio-proxy/0.log"
Nov 02 14:02:03.536083 ip-10-0-128-132 kubenswrapper[4559]: I1102 14:02:03.536047    4559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-gbd89_f55beaae-7351-41b7-89ca-3142739266cd/istio-proxy/0.log"
Nov 02 14:02:03.730675 ip-10-0-128-132 kubenswrapper[4559]: I1102 14:02:03.730635    4559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-jr7g7_a15c4957-178b-41d5-ae44-e79c37fb55d1/istio-proxy/0.log"
Nov 02 14:02:04.164365 ip-10-0-128-132 kubenswrapper[4559]: I1102 14:02:04.164331    4559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-qmrsp_a534baf3-8ff8-4dca-bc27-9022feaaca75/istio-proxy/0.log"
Nov 02 14:02:04.533674 ip-10-0-128-132 kubenswrapper[4559]: I1102 14:02:04.533623    4559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-86f4845867-n579j_d6760973-a200-4c89-9304-103bcb3d846e/payload-pre-processing/0.log"
Nov 02 14:02:04.708002 ip-10-0-128-132 kubenswrapper[4559]: I1102 14:02:04.707976    4559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-7dbf4f4996-66pcj_d6b257bf-cee7-4330-91f7-74b8ba1f1a1f/payload-processing/0.log"
Nov 02 14:02:05.897613 ip-10-0-128-132 kubenswrapper[4559]: I1102 14:02:05.897585    4559 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-hsqc5_b3be8164-4f0a-490f-bd56-bc2270f12bbc/main/0.log"
Nov 02 14:02:06.070570 ip-10-0-128-132 kubenswrapper[4559]: I1102 14:02:06.070535    4559 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-tz6wl_1a48eaf5-31e6-410e-b93f-24b21ab5b871/main/0.log"
Nov 02 14:02:06.441135 ip-10-0-128-132 kubenswrapper[4559]: I1102 14:02:06.441103    4559 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-9frnx_e6bda1dc-2b3a-477c-a148-1d5be4611146/main/0.log"
Nov 02 14:02:06.842936 ip-10-0-128-132 kubenswrapper[4559]: I1102 14:02:06.842868    4559 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-b82wj_538a0b23-da86-458d-baf6-bb864c837a8d/main/0.log"
Nov 02 14:02:15.084473 ip-10-0-128-132 kubenswrapper[4559]: E1102 14:02:15.084432    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqcmj" containerName="extractor"
Nov 02 14:02:17.726667 ip-10-0-128-132 kubenswrapper[4559]: I1102 14:02:17.726619    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-n7wh7/must-gather-xdgmr"]
Nov 02 14:02:17.727481 ip-10-0-128-132 kubenswrapper[4559]: I1102 14:02:17.727053    4559 cpu_manager.go:414] "containerMap: removing container" podUID="9dd5edb8-96ca-46ac-ae54-5b78d4911c22" containerName="revoke-keys"
Nov 02 14:02:17.727481 ip-10-0-128-132 kubenswrapper[4559]: I1102 14:02:17.727077    4559 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9dd5edb8-96ca-46ac-ae54-5b78d4911c22" containerName="revoke-keys"
Nov 02 14:02:17.727481 ip-10-0-128-132 kubenswrapper[4559]: I1102 14:02:17.727200    4559 memory_manager.go:365] "RemoveStaleState removing state" podUID="9dd5edb8-96ca-46ac-ae54-5b78d4911c22" containerName="revoke-keys"
Nov 02 14:02:17.730594 ip-10-0-128-132 kubenswrapper[4559]: I1102 14:02:17.730579    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-n7wh7/must-gather-xdgmr"
Nov 02 14:02:17.746286 ip-10-0-128-132 kubenswrapper[4559]: I1102 14:02:17.746264    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/2baa2153-0f12-41cb-aae2-87f03594192a-must-gather-output\") pod \"must-gather-xdgmr\" (UID: \"2baa2153-0f12-41cb-aae2-87f03594192a\") " pod="openshift-must-gather-n7wh7/must-gather-xdgmr"
Nov 02 14:02:17.746402 ip-10-0-128-132 kubenswrapper[4559]: I1102 14:02:17.746337    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzxqd\" (UniqueName: \"kubernetes.io/projected/2baa2153-0f12-41cb-aae2-87f03594192a-kube-api-access-qzxqd\") pod \"must-gather-xdgmr\" (UID: \"2baa2153-0f12-41cb-aae2-87f03594192a\") " pod="openshift-must-gather-n7wh7/must-gather-xdgmr"
Nov 02 14:02:17.768678 ip-10-0-128-132 kubenswrapper[4559]: I1102 14:02:17.768654    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-n7wh7/must-gather-xdgmr"]
Nov 02 14:02:17.778521 ip-10-0-128-132 kubenswrapper[4559]: I1102 14:02:17.778503    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-n7wh7\"/\"openshift-service-ca.crt\""
Nov 02 14:02:17.778657 ip-10-0-128-132 kubenswrapper[4559]: I1102 14:02:17.778542    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-n7wh7\"/\"kube-root-ca.crt\""
Nov 02 14:02:17.778657 ip-10-0-128-132 kubenswrapper[4559]: I1102 14:02:17.778571    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-n7wh7\"/\"default-dockercfg-mhwb5\""
Nov 02 14:02:17.831049 ip-10-0-128-132 kubenswrapper[4559]: I1102 14:02:17.831021    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-trmxf/must-gather-4pzzj"]
Nov 02 14:02:17.834566 ip-10-0-128-132 kubenswrapper[4559]: I1102 14:02:17.834551    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-trmxf/must-gather-4pzzj"
Nov 02 14:02:17.847649 ip-10-0-128-132 kubenswrapper[4559]: I1102 14:02:17.847625    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/2baa2153-0f12-41cb-aae2-87f03594192a-must-gather-output\") pod \"must-gather-xdgmr\" (UID: \"2baa2153-0f12-41cb-aae2-87f03594192a\") " pod="openshift-must-gather-n7wh7/must-gather-xdgmr"
Nov 02 14:02:17.847743 ip-10-0-128-132 kubenswrapper[4559]: I1102 14:02:17.847706    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzxqd\" (UniqueName: \"kubernetes.io/projected/2baa2153-0f12-41cb-aae2-87f03594192a-kube-api-access-qzxqd\") pod \"must-gather-xdgmr\" (UID: \"2baa2153-0f12-41cb-aae2-87f03594192a\") " pod="openshift-must-gather-n7wh7/must-gather-xdgmr"
Nov 02 14:02:17.847927 ip-10-0-128-132 kubenswrapper[4559]: I1102 14:02:17.847908    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/2baa2153-0f12-41cb-aae2-87f03594192a-must-gather-output\") pod \"must-gather-xdgmr\" (UID: \"2baa2153-0f12-41cb-aae2-87f03594192a\") " pod="openshift-must-gather-n7wh7/must-gather-xdgmr"
Nov 02 14:02:17.858429 ip-10-0-128-132 kubenswrapper[4559]: I1102 14:02:17.858406    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-trmxf/must-gather-4pzzj"]
Nov 02 14:02:17.873942 ip-10-0-128-132 kubenswrapper[4559]: I1102 14:02:17.873912    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzxqd\" (UniqueName: \"kubernetes.io/projected/2baa2153-0f12-41cb-aae2-87f03594192a-kube-api-access-qzxqd\") pod \"must-gather-xdgmr\" (UID: \"2baa2153-0f12-41cb-aae2-87f03594192a\") " pod="openshift-must-gather-n7wh7/must-gather-xdgmr"
Nov 02 14:02:17.874497 ip-10-0-128-132 kubenswrapper[4559]: I1102 14:02:17.874480    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-trmxf\"/\"kube-root-ca.crt\""
Nov 02 14:02:17.874580 ip-10-0-128-132 kubenswrapper[4559]: I1102 14:02:17.874508    4559 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-trmxf\"/\"openshift-service-ca.crt\""
Nov 02 14:02:17.877525 ip-10-0-128-132 kubenswrapper[4559]: I1102 14:02:17.877507    4559 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-trmxf\"/\"default-dockercfg-p5wzj\""
Nov 02 14:02:17.948419 ip-10-0-128-132 kubenswrapper[4559]: I1102 14:02:17.948391    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/fc51f5b7-0bff-4d79-bd95-fe5d95457ae8-must-gather-output\") pod \"must-gather-4pzzj\" (UID: \"fc51f5b7-0bff-4d79-bd95-fe5d95457ae8\") " pod="openshift-must-gather-trmxf/must-gather-4pzzj"
Nov 02 14:02:17.948419 ip-10-0-128-132 kubenswrapper[4559]: I1102 14:02:17.948417    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mns6\" (UniqueName: \"kubernetes.io/projected/fc51f5b7-0bff-4d79-bd95-fe5d95457ae8-kube-api-access-4mns6\") pod \"must-gather-4pzzj\" (UID: \"fc51f5b7-0bff-4d79-bd95-fe5d95457ae8\") " pod="openshift-must-gather-trmxf/must-gather-4pzzj"
Nov 02 14:02:18.044758 ip-10-0-128-132 kubenswrapper[4559]: I1102 14:02:18.044702    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-n7wh7/must-gather-xdgmr"
Nov 02 14:02:18.049942 ip-10-0-128-132 kubenswrapper[4559]: I1102 14:02:18.049922    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/fc51f5b7-0bff-4d79-bd95-fe5d95457ae8-must-gather-output\") pod \"must-gather-4pzzj\" (UID: \"fc51f5b7-0bff-4d79-bd95-fe5d95457ae8\") " pod="openshift-must-gather-trmxf/must-gather-4pzzj"
Nov 02 14:02:18.050015 ip-10-0-128-132 kubenswrapper[4559]: I1102 14:02:18.049952    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mns6\" (UniqueName: \"kubernetes.io/projected/fc51f5b7-0bff-4d79-bd95-fe5d95457ae8-kube-api-access-4mns6\") pod \"must-gather-4pzzj\" (UID: \"fc51f5b7-0bff-4d79-bd95-fe5d95457ae8\") " pod="openshift-must-gather-trmxf/must-gather-4pzzj"
Nov 02 14:02:18.050288 ip-10-0-128-132 kubenswrapper[4559]: I1102 14:02:18.050270    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/fc51f5b7-0bff-4d79-bd95-fe5d95457ae8-must-gather-output\") pod \"must-gather-4pzzj\" (UID: \"fc51f5b7-0bff-4d79-bd95-fe5d95457ae8\") " pod="openshift-must-gather-trmxf/must-gather-4pzzj"
Nov 02 14:02:18.076954 ip-10-0-128-132 kubenswrapper[4559]: I1102 14:02:18.076926    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mns6\" (UniqueName: \"kubernetes.io/projected/fc51f5b7-0bff-4d79-bd95-fe5d95457ae8-kube-api-access-4mns6\") pod \"must-gather-4pzzj\" (UID: \"fc51f5b7-0bff-4d79-bd95-fe5d95457ae8\") " pod="openshift-must-gather-trmxf/must-gather-4pzzj"
Nov 02 14:02:18.147389 ip-10-0-128-132 kubenswrapper[4559]: I1102 14:02:18.147358    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-trmxf/must-gather-4pzzj"
Nov 02 14:02:18.349884 ip-10-0-128-132 kubenswrapper[4559]: I1102 14:02:18.349852    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-n7wh7/must-gather-xdgmr"]
Nov 02 14:02:18.350741 ip-10-0-128-132 kubenswrapper[4559]: W1102 14:02:18.350713    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2baa2153_0f12_41cb_aae2_87f03594192a.slice/crio-5e4a9d3363b4de7c2b820b36cf273d20fbc3b4f6b80029bd9e10d88f5daa588a WatchSource:0}: Error finding container 5e4a9d3363b4de7c2b820b36cf273d20fbc3b4f6b80029bd9e10d88f5daa588a: Status 404 returned error can't find the container with id 5e4a9d3363b4de7c2b820b36cf273d20fbc3b4f6b80029bd9e10d88f5daa588a
Nov 02 14:02:18.352540 ip-10-0-128-132 kubenswrapper[4559]: I1102 14:02:18.352520    4559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 14:02:18.450407 ip-10-0-128-132 kubenswrapper[4559]: I1102 14:02:18.448770    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-trmxf/must-gather-4pzzj"]
Nov 02 14:02:18.452936 ip-10-0-128-132 kubenswrapper[4559]: W1102 14:02:18.452913    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfc51f5b7_0bff_4d79_bd95_fe5d95457ae8.slice/crio-230d8f5df992442d71876ca4a5e12fd65eae410628c16669631e08e8f2cdae20 WatchSource:0}: Error finding container 230d8f5df992442d71876ca4a5e12fd65eae410628c16669631e08e8f2cdae20: Status 404 returned error can't find the container with id 230d8f5df992442d71876ca4a5e12fd65eae410628c16669631e08e8f2cdae20
Nov 02 14:02:19.070926 ip-10-0-128-132 kubenswrapper[4559]: I1102 14:02:19.070888    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-trmxf/must-gather-4pzzj" event={"ID":"fc51f5b7-0bff-4d79-bd95-fe5d95457ae8","Type":"ContainerStarted","Data":"230d8f5df992442d71876ca4a5e12fd65eae410628c16669631e08e8f2cdae20"}
Nov 02 14:02:19.072163 ip-10-0-128-132 kubenswrapper[4559]: I1102 14:02:19.072136    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-n7wh7/must-gather-xdgmr" event={"ID":"2baa2153-0f12-41cb-aae2-87f03594192a","Type":"ContainerStarted","Data":"5e4a9d3363b4de7c2b820b36cf273d20fbc3b4f6b80029bd9e10d88f5daa588a"}
Nov 02 14:02:20.078605 ip-10-0-128-132 kubenswrapper[4559]: I1102 14:02:20.078565    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-trmxf/must-gather-4pzzj" event={"ID":"fc51f5b7-0bff-4d79-bd95-fe5d95457ae8","Type":"ContainerStarted","Data":"450be1c7f43eb034f9052abe0830c40212f45096259affcf43bdbb3da5ad195c"}
Nov 02 14:02:20.078605 ip-10-0-128-132 kubenswrapper[4559]: I1102 14:02:20.078612    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-trmxf/must-gather-4pzzj" event={"ID":"fc51f5b7-0bff-4d79-bd95-fe5d95457ae8","Type":"ContainerStarted","Data":"a0d96808527aa403738a810bb3a48a05b9d032f9752b4e10c0c196c038b5a701"}
Nov 02 14:02:20.127954 ip-10-0-128-132 kubenswrapper[4559]: I1102 14:02:20.127862    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-trmxf/must-gather-4pzzj" podStartSLOduration=2.383865679 podStartE2EDuration="3.127846478s" totalImagesPullingTime="743.980799ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:02:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:02:20.126018102 +0000 UTC m=+5736.381417197" watchObservedRunningTime="2026-11-02 14:02:20.127846478 +0000 UTC m=+5736.383245573"
Nov 02 14:02:22.399414 ip-10-0-128-132 kubenswrapper[4559]: I1102 14:02:22.399384    4559 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-6grht_83964713-5989-4b35-91e8-3d3993688ccd/global-pull-secret-syncer/0.log"
Nov 02 14:02:23.718743 ip-10-0-128-132 kubenswrapper[4559]: I1102 14:02:23.718709    4559 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-dnm8d_f7c6ba1b-8808-4b6b-8965-2e4e7a067030/konnectivity-agent/0.log"
Nov 02 14:02:23.986532 ip-10-0-128-132 kubenswrapper[4559]: I1102 14:02:23.986429    4559 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-128-132.ec2.internal_9f29c1002c42911f93b255b0f3495fd0/haproxy/0.log"
Nov 02 14:02:29.977929 ip-10-0-128-132 kubenswrapper[4559]: I1102 14:02:29.977888    4559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6768bb47c6-q87t7_a00817af-1ee2-4bad-b007-b058928a12b7/authorino/0.log"
Nov 02 14:02:31.484132 ip-10-0-128-132 kubenswrapper[4559]: I1102 14:02:31.484103    4559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-l9pbf_ae914cec-8158-4c77-9870-e0fa6573c7b6/limitador/0.log"
Nov 02 14:02:32.320055 ip-10-0-128-132 kubenswrapper[4559]: I1102 14:02:32.320015    4559 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-trmxf/perf-node-gather-daemonset-49srk"]
Nov 02 14:02:32.325788 ip-10-0-128-132 kubenswrapper[4559]: I1102 14:02:32.325759    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-trmxf/perf-node-gather-daemonset-49srk"
Nov 02 14:02:32.341387 ip-10-0-128-132 kubenswrapper[4559]: I1102 14:02:32.341357    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-trmxf/perf-node-gather-daemonset-49srk"]
Nov 02 14:02:32.517268 ip-10-0-128-132 kubenswrapper[4559]: I1102 14:02:32.517231    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/26aa6519-a1cc-4993-af4f-ed30930a096b-sys\") pod \"perf-node-gather-daemonset-49srk\" (UID: \"26aa6519-a1cc-4993-af4f-ed30930a096b\") " pod="openshift-must-gather-trmxf/perf-node-gather-daemonset-49srk"
Nov 02 14:02:32.517741 ip-10-0-128-132 kubenswrapper[4559]: I1102 14:02:32.517307    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/26aa6519-a1cc-4993-af4f-ed30930a096b-podres\") pod \"perf-node-gather-daemonset-49srk\" (UID: \"26aa6519-a1cc-4993-af4f-ed30930a096b\") " pod="openshift-must-gather-trmxf/perf-node-gather-daemonset-49srk"
Nov 02 14:02:32.517741 ip-10-0-128-132 kubenswrapper[4559]: I1102 14:02:32.517492    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n62hw\" (UniqueName: \"kubernetes.io/projected/26aa6519-a1cc-4993-af4f-ed30930a096b-kube-api-access-n62hw\") pod \"perf-node-gather-daemonset-49srk\" (UID: \"26aa6519-a1cc-4993-af4f-ed30930a096b\") " pod="openshift-must-gather-trmxf/perf-node-gather-daemonset-49srk"
Nov 02 14:02:32.517741 ip-10-0-128-132 kubenswrapper[4559]: I1102 14:02:32.517535    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/26aa6519-a1cc-4993-af4f-ed30930a096b-lib-modules\") pod \"perf-node-gather-daemonset-49srk\" (UID: \"26aa6519-a1cc-4993-af4f-ed30930a096b\") " pod="openshift-must-gather-trmxf/perf-node-gather-daemonset-49srk"
Nov 02 14:02:32.517741 ip-10-0-128-132 kubenswrapper[4559]: I1102 14:02:32.517564    4559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/26aa6519-a1cc-4993-af4f-ed30930a096b-proc\") pod \"perf-node-gather-daemonset-49srk\" (UID: \"26aa6519-a1cc-4993-af4f-ed30930a096b\") " pod="openshift-must-gather-trmxf/perf-node-gather-daemonset-49srk"
Nov 02 14:02:32.618494 ip-10-0-128-132 kubenswrapper[4559]: I1102 14:02:32.618403    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/26aa6519-a1cc-4993-af4f-ed30930a096b-sys\") pod \"perf-node-gather-daemonset-49srk\" (UID: \"26aa6519-a1cc-4993-af4f-ed30930a096b\") " pod="openshift-must-gather-trmxf/perf-node-gather-daemonset-49srk"
Nov 02 14:02:32.618494 ip-10-0-128-132 kubenswrapper[4559]: I1102 14:02:32.618469    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/26aa6519-a1cc-4993-af4f-ed30930a096b-podres\") pod \"perf-node-gather-daemonset-49srk\" (UID: \"26aa6519-a1cc-4993-af4f-ed30930a096b\") " pod="openshift-must-gather-trmxf/perf-node-gather-daemonset-49srk"
Nov 02 14:02:32.618775 ip-10-0-128-132 kubenswrapper[4559]: I1102 14:02:32.618517    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n62hw\" (UniqueName: \"kubernetes.io/projected/26aa6519-a1cc-4993-af4f-ed30930a096b-kube-api-access-n62hw\") pod \"perf-node-gather-daemonset-49srk\" (UID: \"26aa6519-a1cc-4993-af4f-ed30930a096b\") " pod="openshift-must-gather-trmxf/perf-node-gather-daemonset-49srk"
Nov 02 14:02:32.618775 ip-10-0-128-132 kubenswrapper[4559]: I1102 14:02:32.618536    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/26aa6519-a1cc-4993-af4f-ed30930a096b-lib-modules\") pod \"perf-node-gather-daemonset-49srk\" (UID: \"26aa6519-a1cc-4993-af4f-ed30930a096b\") " pod="openshift-must-gather-trmxf/perf-node-gather-daemonset-49srk"
Nov 02 14:02:32.618775 ip-10-0-128-132 kubenswrapper[4559]: I1102 14:02:32.618549    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/26aa6519-a1cc-4993-af4f-ed30930a096b-sys\") pod \"perf-node-gather-daemonset-49srk\" (UID: \"26aa6519-a1cc-4993-af4f-ed30930a096b\") " pod="openshift-must-gather-trmxf/perf-node-gather-daemonset-49srk"
Nov 02 14:02:32.618775 ip-10-0-128-132 kubenswrapper[4559]: I1102 14:02:32.618584    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/26aa6519-a1cc-4993-af4f-ed30930a096b-podres\") pod \"perf-node-gather-daemonset-49srk\" (UID: \"26aa6519-a1cc-4993-af4f-ed30930a096b\") " pod="openshift-must-gather-trmxf/perf-node-gather-daemonset-49srk"
Nov 02 14:02:32.618775 ip-10-0-128-132 kubenswrapper[4559]: I1102 14:02:32.618558    4559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/26aa6519-a1cc-4993-af4f-ed30930a096b-proc\") pod \"perf-node-gather-daemonset-49srk\" (UID: \"26aa6519-a1cc-4993-af4f-ed30930a096b\") " pod="openshift-must-gather-trmxf/perf-node-gather-daemonset-49srk"
Nov 02 14:02:32.618775 ip-10-0-128-132 kubenswrapper[4559]: I1102 14:02:32.618616    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/26aa6519-a1cc-4993-af4f-ed30930a096b-proc\") pod \"perf-node-gather-daemonset-49srk\" (UID: \"26aa6519-a1cc-4993-af4f-ed30930a096b\") " pod="openshift-must-gather-trmxf/perf-node-gather-daemonset-49srk"
Nov 02 14:02:32.618775 ip-10-0-128-132 kubenswrapper[4559]: I1102 14:02:32.618627    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/26aa6519-a1cc-4993-af4f-ed30930a096b-lib-modules\") pod \"perf-node-gather-daemonset-49srk\" (UID: \"26aa6519-a1cc-4993-af4f-ed30930a096b\") " pod="openshift-must-gather-trmxf/perf-node-gather-daemonset-49srk"
Nov 02 14:02:32.648680 ip-10-0-128-132 kubenswrapper[4559]: I1102 14:02:32.644893    4559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n62hw\" (UniqueName: \"kubernetes.io/projected/26aa6519-a1cc-4993-af4f-ed30930a096b-kube-api-access-n62hw\") pod \"perf-node-gather-daemonset-49srk\" (UID: \"26aa6519-a1cc-4993-af4f-ed30930a096b\") " pod="openshift-must-gather-trmxf/perf-node-gather-daemonset-49srk"
Nov 02 14:02:32.942398 ip-10-0-128-132 kubenswrapper[4559]: I1102 14:02:32.942360    4559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-trmxf/perf-node-gather-daemonset-49srk"
Nov 02 14:02:33.255995 ip-10-0-128-132 kubenswrapper[4559]: I1102 14:02:33.255942    4559 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-trmxf/perf-node-gather-daemonset-49srk"]
Nov 02 14:02:33.255995 ip-10-0-128-132 kubenswrapper[4559]: W1102 14:02:33.255979    4559 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod26aa6519_a1cc_4993_af4f_ed30930a096b.slice/crio-58f202801e48ee1153ec273e65a7e059728ac7c7d052040f30fd028b073f4063 WatchSource:0}: Error finding container 58f202801e48ee1153ec273e65a7e059728ac7c7d052040f30fd028b073f4063: Status 404 returned error can't find the container with id 58f202801e48ee1153ec273e65a7e059728ac7c7d052040f30fd028b073f4063
Nov 02 14:02:34.214467 ip-10-0-128-132 kubenswrapper[4559]: I1102 14:02:34.214426    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-trmxf/perf-node-gather-daemonset-49srk" event={"ID":"26aa6519-a1cc-4993-af4f-ed30930a096b","Type":"ContainerStarted","Data":"2c42c139ea4d25f03399c7ada5f29d407e8820c41ef7db8ae6b920f5a3c17c54"}
Nov 02 14:02:34.214904 ip-10-0-128-132 kubenswrapper[4559]: I1102 14:02:34.214496    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-trmxf/perf-node-gather-daemonset-49srk"
Nov 02 14:02:34.214904 ip-10-0-128-132 kubenswrapper[4559]: I1102 14:02:34.214522    4559 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-trmxf/perf-node-gather-daemonset-49srk" event={"ID":"26aa6519-a1cc-4993-af4f-ed30930a096b","Type":"ContainerStarted","Data":"58f202801e48ee1153ec273e65a7e059728ac7c7d052040f30fd028b073f4063"}
Nov 02 14:02:34.257884 ip-10-0-128-132 kubenswrapper[4559]: I1102 14:02:34.257830    4559 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-trmxf/perf-node-gather-daemonset-49srk" podStartSLOduration=2.257817342 podStartE2EDuration="2.257817342s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 14:02:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:02:34.255753167 +0000 UTC m=+5750.511152262" watchObservedRunningTime="2026-11-02 14:02:34.257817342 +0000 UTC m=+5750.513216478"
Nov 02 14:02:35.378561 ip-10-0-128-132 kubenswrapper[4559]: I1102 14:02:35.378531    4559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-wt562_614c03cd-6d2c-4fee-8718-27a60c6a2d0d/node-exporter/0.log"
Nov 02 14:02:35.502298 ip-10-0-128-132 kubenswrapper[4559]: I1102 14:02:35.502271    4559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-wt562_614c03cd-6d2c-4fee-8718-27a60c6a2d0d/kube-rbac-proxy/0.log"
Nov 02 14:02:35.634162 ip-10-0-128-132 kubenswrapper[4559]: I1102 14:02:35.634098    4559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-wt562_614c03cd-6d2c-4fee-8718-27a60c6a2d0d/init-textfile/0.log"
Nov 02 14:02:37.016472 ip-10-0-128-132 kubenswrapper[4559]: I1102 14:02:37.016435    4559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-565jh_b7da0f69-d13d-42db-96ee-32949d5bb600/prometheus-operator/0.log"
Nov 02 14:02:37.138310 ip-10-0-128-132 kubenswrapper[4559]: I1102 14:02:37.138269    4559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-565jh_b7da0f69-d13d-42db-96ee-32949d5bb600/kube-rbac-proxy/0.log"
Nov 02 14:02:37.262587 ip-10-0-128-132 kubenswrapper[4559]: I1102 14:02:37.262549    4559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-4vfzb_7134dbc9-ca71-47f2-886f-d80d2a6ae912/prometheus-operator-admission-webhook/0.log"
Nov 02 14:02:40.229994 ip-10-0-128-132 kubenswrapper[4559]: I1102 14:02:40.229964    4559 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-trmxf/perf-node-gather-daemonset-49srk"
Nov 02 14:02:41.964100 ip-10-0-128-132 kubenswrapper[4559]: I1102 14:02:41.964071    4559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-bbb695c57-n2z5q_c89c6465-eeec-408e-b2e3-f9c131be5730/console/0.log"
Nov 02 14:02:42.110568 ip-10-0-128-132 kubenswrapper[4559]: I1102 14:02:42.110533    4559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-6dd789c8b7-9lc22_9f83980d-62d1-448f-bdbc-95114e3cd8f3/download-server/0.log"
Nov 02 14:02:45.098964 ip-10-0-128-132 kubenswrapper[4559]: E1102 14:02:45.098923    4559 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wqcmj" containerName="extractor"
Nov 02 14:02:45.826428 ip-10-0-128-132 kubenswrapper[4559]: I1102 14:02:45.826391    4559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-bg5xh_7b688512-ffc4-4b6a-b9ab-8fcf8ea0afe2/dns/0.log"
Nov 02 14:02:45.967369 ip-10-0-128-132 kubenswrapper[4559]: I1102 14:02:45.967340    4559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-bg5xh_7b688512-ffc4-4b6a-b9ab-8fcf8ea0afe2/kube-rbac-proxy/0.log"
Nov 02 14:02:46.502276 ip-10-0-128-132 kubenswrapper[4559]: I1102 14:02:46.502241    4559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-7trhz_5507fd15-57b9-422a-af1d-5daf266bd9ca/dns-node-resolver/0.log"
Nov 02 14:02:47.792135 ip-10-0-128-132 kubenswrapper[4559]: I1102 14:02:47.792105    4559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-75fd5f49d8-7t6q5_96cbbd04-22df-4987-8fd5-5ee673985f5e/registry/0.log"
Nov 02 14:02:47.933536 ip-10-0-128-132 kubenswrapper[4559]: I1102 14:02:47.933490    4559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-9c9fcfcf5-9pxlh_4aea6613-d504-4ab3-80cd-a1c4010709a7/kube-auth-proxy/0.log"
Nov 02 14:02:47.935902 ip-10-0-128-132 kubenswrapper[4559]: I1102 14:02:47.935883    4559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-9t4f5_20cc95cd-9718-49ee-8082-e9ac00fcc147/node-ca/0.log"
Nov 02 14:02:48.219293 ip-10-0-128-132 kubenswrapper[4559]: I1102 14:02:48.219250    4559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-5wtjm_b9661a3b-8a07-4a30-9893-20ab1d95a2a7/istio-proxy/0.log"
Nov 02 14:02:48.325451 ip-10-0-128-132 kubenswrapper[4559]: I1102 14:02:48.325421    4559 scope.go:122] "RemoveContainer" containerID="c93444359d071ac4b6e66ec08030801fd4102f740a694a4b61c805e7bbadd068"
Nov 02 14:02:48.335482 ip-10-0-128-132 kubenswrapper[4559]: I1102 14:02:48.335456    4559 scope.go:122] "RemoveContainer" containerID="bab48d37cd401c45fb7dc953500e20a6eddb6b618702b83eee2c81eae33915ee"
Nov 02 14:02:48.506432 ip-10-0-128-132 kubenswrapper[4559]: I1102 14:02:48.506352    4559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-gbd89_f55beaae-7351-41b7-89ca-3142739266cd/istio-proxy/0.log"
Nov 02 14:02:48.571373 ip-10-0-128-132 kubenswrapper[4559]: I1102 14:02:48.571350    4559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-jr7g7_a15c4957-178b-41d5-ae44-e79c37fb55d1/istio-proxy/0.log"
Nov 02 14:02:48.708231 ip-10-0-128-132 kubenswrapper[4559]: I1102 14:02:48.708204    4559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-qmrsp_a534baf3-8ff8-4dca-bc27-9022feaaca75/istio-proxy/0.log"
Nov 02 14:02:48.849372 ip-10-0-128-132 kubenswrapper[4559]: I1102 14:02:48.849305    4559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-86f4845867-n579j_d6760973-a200-4c89-9304-103bcb3d846e/payload-pre-processing/0.log"
Nov 02 14:02:48.916986 ip-10-0-128-132 kubenswrapper[4559]: I1102 14:02:48.916956    4559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-7dbf4f4996-66pcj_d6b257bf-cee7-4330-91f7-74b8ba1f1a1f/payload-processing/0.log"